Can Language Models Be Specific? How? (2023)

Abstract: “He is a person”, “Paris is located on the earth”. Both statements are correct but meaningless - due to lack of specificity. In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is. To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. For instance, given “Toronto is located in [MASK].”, we want to test whether a more specific answer will be better filled in by PLMs, e.g., Ontario instead of Canada. From our evaluations, we show that existing PLMs have only a slight preference for more specific answers. We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity. Results show that the specificity of the models can be improved by the proposed methods without additional training. We hope this work can bring to awareness the notion of specificity of language models and encourage the research community to further explore this important but understudied problem.

## introduction pre-trained language models (plms) such as bert @xcite and gpt-2/3 @xcite have achieved quite impressive results in various natural language processing tasks. recent works show that the parameters of these models contain significant amounts of knowledge @xcite @xcite , and knowledge stored in plms can be extracted by predicting the mask token(s) using prompts. for instance, given prompt "j. k. rowling was born in @xcite .", plms can predict the birthplace of rowling based on its knowledge. toronto is located on the earth. dante is a person. cat is a subclass of animal. however, there may exist multiple answers for a query, while not all answers are equally specific. in many situations, we desire a specific answer. for the example above, the masked token can be replaced by yate (a town), gloucestershire (a county), or england (a country). to acquire the maximum knowledge (in this example, the town, the county, and the country where rowling was born), we may prefer the model to fill in yate since gloucestershire and england can be further predicted using prompts, e.g., "yate is located in @xcite ." this means, if the prediction is more specific, we can retrieve more fine-grained information from language models, and further acquire more information. besides, sometimes, the less specific answer is not useful. for instance, it is well known that chicago is located in the usa, users will not get additional information if the model only predicts chicago is located in the usa instead of illinois. more examples are shown in figure 1 . to make an analogy: a good speaker not only needs to be correct, but also has the ability to be specific when desired. the same is true for language models. although there are works on measuring how much knowledge is stored in plms or improving the correctness of the predictions @xcite @xcite , few attempted to measure or improve the specificity of predictions made by plms. noteworthy exceptions include the work by @xcite ; @xcite , who evaluated the specificity of conversational language models. in their research, specificity was defined and measured within a conversational context -for instance, the response "me too. i love eurovision songs" is deemed more specific than simply "me too" to the statement "i love eurovision". understanding how specific the language of plms is can help us better understand the behavior of language models and facilitate downstream applications such as question answering, text generation, and information extraction @xcite @xcite , e.g., making the generated answers/sentences or extracted information more specific or fine-grained. therefore, we propose to build a benchmark to measure the specificity of the language of plms. for reducing human effort and easier to further expand the dataset (e.g., to specific domains), we introduce a novel way to construct test data automatically based on transitive relations in wikidata @xcite . specifically, we extract reasoning paths from wikidata, e.g., (j. k. rowling, birthplace, yate, location, gloucestershire, location, england). based on the average distance of each object to the subject and the property of transitive relations, we form masked-token-prediction based probing tasks to measure the specificity, e.g., whether the masked token in "j. k. rowling was born in @xcite ." is better filled by yate than england by plms. the resulting benchmark dataset contains more than 20,000 probes covering queries from 5 different categories. the quality of the benchmark is high, where the judgment on which answer is more specific is ∼ 97% consistent with humans. we provide in-depth analyses on model specificity and study two factors that affect the specificity with our benchmark. as shown by our evaluations in section 4, existing plms, e.g., bert and gpt-2, similarly have only a slight preference for more specific answers (in only about 60% of cases where a more specific answer is preferred). we also show that, in general, plms prefer less specific answers without subjects given, and they only have a weak ability to differentiate coarse-grained/finegrained objects by measuring their similarities to subjects. the results indicate that specificity was neglected by existing research on language models. how to improve and control it is undoubtedly an interesting and valuable problem. based on our observations and analyses, we propose two techniques to improve the specificity of the predictions by modifying the prompts without additional training: few-shot prompting, where demonstrations with more specific answers are provided to guide the models to produce more specific answers; and cascade prompting, where which clauses are added as suffixes to bias the predictions to be more specific. results show that few-shot prompting can improve the specificity for unidirectional language models like gpt-2 well, while cascade prompting works well for bidirectional language models such as bert. the main contributions of our work are summarized as follows: knowledge retrieval from lms and prompting: previous works have worked on extracting factual knowledge from plms without incorporating external knowledge, which is usually achieved by creating prompts and letting plms predict the masked token(s) @xcite @xcite . they demonstrated that plms contain a significant amount of knowledge. by creating appropriate prompts with some additional training, such methods can even achieve performance comparable to sota for some specific tasks @xcite . our work is inspired by these works; but different from these works, where the focus is to measure or improve the correctness of the predictions, our work focuses on measuring and improving the specificity of the predictions. ## s-test: specificity testing in this section, we introduce our specificity testing (s-test) task, describe the creation process of the dataset, and design the metric to measure the specificity of predictions made by plms. ## analysis in this section, we first analyze the results of s-test and then identify and study two underlying factors that affect the specificity of predictions produced by pre-trained language models. ## experiments in this section, we conduct experiments with the prompt-based methods proposed in section 5. ## discussion specificity testing in more general scenarios: in this work, we test the specificity of plms on several relations with manually crafted prompts, with test data created automatically based on the property of transitive relations. for future work, we may test the specificity in more general scenarios. for instance, for numerical knowledge (lin et al., 2020), we can test how specifically plms describe the numbers, e.g., obama was born in 1961 vs obama was born in 1960s, a car has four wheels vs a car has several wheels. in addition, we may test on multi-token answers @xcite , and measure the specificity of sentences generated by plms @xcite @xcite , e.g., this is a very good paper. i really like it. vs this paper conducts a very novel and interesting study, which provides a new insight for future work on language models. further improvement of specificity: in this paper, we propose few-shot prompting and cascade prompting to improve the specificity of plms without any additional training. future work may improve the specificity by including prompt-based fine-tuning @xcite . the observation also encourages future work to take into account the specificity, e.g., adding constraints regarding specificity, in the pre-training process. it is also interesting to design methods to control the degree of specificity for different usage scenarios @xcite . ## conclusion in this paper, we build a benchmark to measure the specificity of predictions produced by pre-trained language models. to achieve this, we propose a novel approach to construct test data for specificity testing automatically. from our evaluations, we show that existing plms have only a slight preference for more specific answers. we also propose two prompt-based methods, i.e., few-shot prompting and cascade prompting, to improve the specificity of the predictions. extensive experiments and in-depth analyses demonstrate the effectiveness of the proposed methods. we hope this work can encourage future research in this direction and give some insights to improve downstream tasks such as question answering, information extraction, and text generation: 1) to make the answers, the extracted information, or the generated sentences more specific; 2) to control the degree of specificity for different usage scenarios.