Comparing Template-based and Template-free Language Model Probing (2024)

Abstract: The differences between cloze-task language model (LM) probing with 1) expert-made templates and 2) naturally-occurring text have often been overlooked. Here, we evaluate 16 different LMs on 10 probing English datasets – 4 template-based and 6 template-free – in general and biomedical domains to answer the following research questions: (RQ1) Do model rankings differ between the two approaches? (RQ2) Do models’ absolute scores differ between the two approaches? (RQ3) Do the answers to RQ1 and RQ2 differ between general and domain-specific models? Our findings are: 1) Template-free and template-based approaches often rank models differently, except for the top domain- specific models. 2) Scores decrease by up to 42% Acc@1 when comparing parallel template-free and template-based prompts. 3) Perplexity is negatively correlated with accuracy in the template-free approach, but, counter-intuitively, they are positively correlated for template-based probing. 4) Models tend to predict the same answers frequently across prompts for template-based probing, which is less common when employing template-free techniques.

## introduction in the past few years there has been a growing interest in understanding what parametric knowledge language models (lms) contain @xcite . one standard approach of probing lms for knowledge consists of using "fill-in-the-blank" cloze statements @xcite @xcite @xcite , where models are tasked with predicting a masked entity given a prompt, e.g., "dante was born in [mask]". template-based probing template-free probing template: "[x] (born [mask])" n/a peter f. martin (born [mask]) peter f martin (born [mask]) is an american politician [...] dennis b. sullivan (born [mask]) sullivan was born in chippewa falls wisconsin in [mask] tan jiexi (born [mask]) tan jiexi (born december 2, [mask] in shenzhen,china), is a chinese singer-songwriter tasos neroutsos (born [mask]) while this has been studied in various settings, including multilingual @xcite , single token predictions @xcite @xcite , multi-token predictions @xcite , and prompt optimization @xcite , the differences between the two types of prompts -template-based and template-free; see although in both methods whether a lm knows a fact is defined by its ability to successfully predict the masked object in a prompt @xcite , the template-based approach uses templates (which are often manually created) to create the prompts, where each template is used to create many prompts which are identical except for the subject entity. in comparison, the template-free approach does not use templates and prompts lms with an often unique prompt per entity. each of the two methods has its pros and cons. for example, while the template-based approach generally guarantees that the prompt is evaluating the required knowledge, it requires expensive domain experts. and while the prompts in the template-free approach are more similar to the training data lms are trained on, as they come from real-world text and not from artificial templates, they may contain additional irrelevant information. we hypothesize that this may result in different rankings for the same models when being prompted via the two approaches. here, we 1) evaluate 16 different lms on 10 probing datasets (4 template-based and 6 templatefree) in multiple domains; 2) we propose a method to create template-free domain-specific datasets and use it to develop the first template-free biomedical probing dataset, which allows us to compare the effect of the two probing approaches in two different domains; 3) ask the following research questions: (rq1) do model rankings differ between template-based and template-free probing? (rq2) do models' absolute scores differ between the two approaches? (rq3) do the answers to the two previous questions differ between general and domain-specific models? our study's results can be summarized as follows: 1) there is a discrepancy in ranking models between template-free and template-based methods, except for the top domain-specific models. 2) scores decrease by up to 42% acc@1 when comparing parallel template-free and template-based prompts (i.e., similar subject entities and masked objects). 3) perplexity is negatively correlated with accuracy in the template-free approach, but, counter-intuitively, they are positively correlated for template-based probing. 4) models have a tendency to predict similar objects to various prompts, even when the subjects change, when utilizing template-based probing, which is less common when employing template-free techniques. ## experiments the lms' input for both template-free and template-based probing is a prompt with one masked entity; see while most previous work focuses solely on single-token mask prediction, many entities are composed of more than one token. hence, we follow @xcite that expand the probing technique to multi-token prediction and show that it is a better method to investigate knowledge captured by lms. in particular, we use the same alternative to the "fill-in-the-blank" querying by framing the task as entity ranking. however, while @xcite limit the prediction to entities of the type required by the prompt, we relax this limitation because 1) few existing datasets have entity type information, and using external resource to classify entity types may be inaccurate and skew results; 2) this simplifies the problem for the models which, again, may skew results. instead, we allow our models to predict any entity from the dataset's entity list. lastly, we experiment in both general and biomedical domains to analyze whether our experiments generalize. hence, a key portion of the experiments pertains to biomedical models, as in addition to generic english models, we use both biomedical fine-tuned models and biomedical models. evaluation metric following prior work by @xcite , we use top-k accuracy (acc@k), wherein a score of 1 is given if the correct entity appears among the top k predicted entities, and 0 otherwise. since entities are often related to numerous other entities (n-to-m connections), we use acc@1, acc@5, and acc@10. ## results tables 3 and 4 show our main results on the template-based and template-free datasets. model rankings in both the template-based and template-free biomedical datasets -ctd, biomed-wikidata, and our novel lipid datasets -pubmed-bert performs best, followed by bioformer and biom-electra: both techniques clearly separate models that were trained solely on biomedical data (in blue) as opposed to those who were fine-tuned on biomedicine (in green) or general domain (in red). in comparison, on general-domain datasets, bert-large performs best, followed by either bert-base or distilbert. furthermore, the top-5 general-domain models are roughly the same across all general-domain datasets. however, as the rank increases, the pattern is less obvious and there is no clear separation between models in blue and those in green, e.g., while pubmedbert is the 6th best, followed by bioformer as 7th or 8th, its rank changes to 15 (i.e., second to last) on google-re template-based. this is especially surprising as on the same google-re dataset, but in the templatefree setting, bioformer ranks 7th. similarly, large changes in ranking can be seen for roberta-base, which moves from rank 12 for template-free to 7 in the template-based google-re. this is also visible for the t-rex dataset, where roberta-large moves from rank 7 in the template-free to 12 in the template-based setting. similar ranking differences between datasets also appear in general models that are fine-tuned on biomedical data (in green). notably, model rankings change between two general-domain datasets (e.g., google-re and t-rex), between domain-specific datasets (e.g., ctd and biomed-wikidata), and between both the template-free and template-based approaches (e.g., both google-re and t-rex settings). model scores since both google-re and t-rex are composed of parallel template-free and template-based datasets in which each template has a corresponding template-free text, see we see the largest performance difference in bert-large, which obtains a score of 0.72 acc@1 on template-free t-rex, but a score of 0.3 acc@1 on the corresponding template-based data. while t-rex and google-re are the only parallel datasets we have, allowing us to directly compare between the datasets, we can also see that the scores are different in general between templatebased and template-free datasets: e.g., the average acc@1 on the ctd and biomed-wikidata are 0.002 and 0.011, while on our lipid datasets of the same biomedical domain, the average acc@1 are 0.12 on genes and 0.18 on chemicals. another strange model behavior we see for the template-based datasets is the effect of model size: larger models generally perform better than their smaller counterparts. this can be seen across all base and large models on all template-free datasets. however, this is not the case in the templatebased datasets: e.g., on ctd and biomed-wikidata, albert-base outperforms its larger counterpart, and, on google-re, roberta-base outperforms roberta-large. we can further see this pattern with bert on the biomed-wikidata dataset, where on t-rex and google-re the bert-base version performs as well as bert-large. ## discussion and analysis we now discuss and investigate why the aforementioned differences in scores and rankings occur between datasets and the two probing approaches. ## conclusion our study demonstrates that the choice of probing technique -template-based vs. template-freesignificantly affects the assessment of lms knowledge. using "fill-in-the-blank" cloze statements, we compare 16 lms across 10 probing datasets, and find substantial disparities between the two approaches. we further propose a method to create template-free domain-specific datasets and use it to develop lipid: the first template-free biomedical probing dataset, making it possible to compare the effect of the two probing techniques in different domains. our findings emphasize the necessity of employing multiple evaluation methods to obtain a comprehensive understanding of lms' knowledge. limitations while we try to be extensive in our analysis of the potential reasons for the differences between template-base and template-free probing, other reasons may exist. that being said, we find many patterns that highlight that existing approaches are significantly different in the amount of knowledge they are capable of extracting which opens avenues for future work. additionally, a lot of research these days is focusing on extremely large models with billions of parameters. while we expect our results to generalize as we evaluate a significant number of systems, more work can be done on evaluating template-free and templatebased probing of such extremely large models. ethics statement the motivation for this paper is to develop a better evaluation strategy of the amount of lms' parametric knowledge. we believe that it is crucial that future work continues to evaluate and improve models' factual knowledge in biomedicine as well as other domains.