How to Make the Most of LLM s’ Grammatical Knowledge for Acceptability Judgments (2025)

Abstract: The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates. We therefore investigate how to derive the most accurate acceptability judgments from LLMs to comprehensively evaluate their grammatical knowledge. Through extensive experiments in both English and Chinese, we compare nine judgment methods and demonstrate that two of them, in-template LP (a probability readout method) and Yes/No probability computing (a prompting-based method), achieve higher accuracy than the conventional approach. Our analysis reveals that the top two methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge. We find that ensembling the two methods achieves even higher accuracy. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs.

## introduction the grammatical knowledge of language models (lms) is often measured using acceptability judgments @xcite . there are two main categories of acceptability judgment benchmarks, the single-sentence one and the minimal-pair (mp) one, as detailed in section 2. we focus on the latter because it allows us to directly measure lms' grammatical knowledge with1 our codes and templates are published at https:// github.com/yusuke196/llm-acceptability . meanwhile, the recent scaling up of model sizes and training data for lms has made it possible to solve a wide range of tasks using few-shot or zeroshot prompting, without the need for task-specific fine-tuning @xcite , popularizing the term large language model (llm). incorporating learning techniques such as instruction tuning @xcite and direct preference optimization @xcite further improved the alignment of llm outputs with human preferences. the llms trained by these techniques achieve good performance by prompting, i.e., guidance on what knowledge to elicit. in this light, various methods can be developed to obtain more accurate acceptability judgments by providing llms with targeted guidance. however, as discussed in section 2, most previous studies simply input the sentences into the (l)lm, calculate their probabilities, and consider the sentence with the higher probability as the acceptable one. although @xcite compared multiple methods of obtaining acceptability judgments from llms (see their experiment 3b), their probability readout method and prompting method were limited to basic ones. as a result, they broadly claim that prompting is ineffective, whereas our experiments demonstrate that it can be highly effective with the proper technique. we contribute to this area by comparing (1) the conventional sentence probability readout 2 methods, (2) our novel probability readout methods in figure 1 : conceptual illustration of our top methods. differences between paired sentences are underlined. both methods judge the sentence that results in a higher (log) probability acceptable. see section 3 for the details. to rigorously compare these methods, we conduct experiments using eight llms and two mp benchmarks (one for english and one for chinese). the results show the effectiveness of the two methods. in-template lp consistently outperforms conventional methods, achieving the highest accuracies on the chinese benchmark. yes/no prob comp achieves the highest accuracies on the english benchmark in all but one setting. moreover, our analysis demonstrates the following key findings. (1) in-template lp and yes/no prob comp have different strengths; for example, yes/no prob comp is robust against token-length bias. this indicates that they access different as-pects of llms' grammatical knowledge, contributing to a more comprehensive evaluation. (2) ensembling the two methods further improves the accuracy, revealing their complementary capabilities. the highest score, achieved with qwen2, is 1.6 percentage points higher than humans on the english benchmark. based on these findings, we recommend the following: if possible, ensemble the two methods; otherwise, use in-template lp. (3) we identify a common weakness across all our combinations of the llms and methods: they struggle to make correct judgments on linguistic phenomena where the unacceptable sentence can be obtained by shuffling the words in the acceptable one, which presents a challenge for future work. ## related work benchmarks of acceptability judgments can be divided into two categories: single-sentence benchmarks and mp benchmarks. the single-sentence benchmarks pose a binary classification of single sentences as seen in cola @xcite , a dataset composed of sentences each labeled acceptable or unacceptable. cola was incorporated into the natural language understanding benchmark glue @xcite and has been used to evaluate a wide range of models, including lms. however, single-sentence benchmarks cannot measure lms' grammatical knowledge directly because they require training a supervised classifier before the evaluation. this makes it difficult to distinguish between the knowledge of the model itself and what is learned by training the classifier @xcite . in contrast, mp benchmarks present minimally different pairs and the task is to determine which is the most acceptable sentence, eliminating the need for a classifier. as another advantage of the mp benchmark, minimal pairs can be automatically generated in a controlled manner, providing a sufficient amount of quality data for model evaluation @xcite . in conventional experiments using an mp benchmark, judgments are made based on sentence probabilities. models are evaluated by whether they assign a higher probability to the acceptable sentence in each minimal pair. this method, which we call sentence probability readout, has been dominantly employed for mp acceptability judgments across languages @xcite @xcite @xcite . experiments with prompting llms have been conducted on both single-sentence benchmarks and mp benchmarks. @xcite compared various models, including llms, on a single-sentence benchmark. on mp benchmarks, hu and levy (2023) compared the sentence probability readout and prompting. however, their probability readout and prompt-based methods relied on basic implementations, without systematically exploring alternative ways to calculate sentence probabilities or optimize prompting strategies. another line of work has studied biases that affect the performance of sentence probability readout. for example, it is known that the sentence probabilities given by lms tend to decline as the token length of the sentence grows; normalized measures such as penlp @xcite have been shown to mitigate some of this bias @xcite ), but they do not eliminate it @xcite . ## methods we compare three different groups of methods to extract acceptability judgments from the llms. ## experimental setup models we use eight state-of-the-art llms, among which llama-3-70b (meta, 2024), mixtral-8x7b-v0.1 @xcite , qwen2-57b-a14b (qwen team, 2024), and yi-1.5-34b @xcite are base models, while llama-3-70b-instruct, mixtral-8x7b-instruct-v0.1, qwen2-57b-a14b-instruct, and yi-1.5-34b-chat are instruct models based on the pre-trained counterparts. these models ranked relatively high in the leaderboard of english language understanding foot_6 or chinese llms foot_7 at the time of model selection. for chinese experiments, we substituted yi-1.5 for mixtral because mixtral is not explicitly trained for chinese tasks, while yi-1.5 ranked high in the chinese llm leaderboard. we hereafter abbreviate these models, e.g., to llama-3, omitting the model sizes and minor versions. post-training for the three instruct models includes supervised fine-tuning on an instruction dataset, i.e., instruction-tuning and dpo. they are publicly available on hugging face hub. on inference, we perform 4-bit quantization using bitsandbytes foot_8 to compress the models. the computational budgets are described in appendix c. ## results ## analysis given the remarkable performance of yes/no prob comp and in-template lp, this section further investigates their strengths and weaknesses. yes/no prob comp is robust against token-length bias. figure 2 illustrates the correlations between the token-length difference and the accuracy. the in-template readout and yes/no prob comp excel in different phenomena. figure 3 illustrates the accuracy of in-template lp, yes/no prob comp, and the humans by linguistic phenomenon; the scores of humans are from @xcite and @xcite . for in-template lp and yes/no prob comp, the result of the bestperforming template is shown. here we find that the two methods have different strengths. on blimp, yes/no prob comp excels at phenomena such as subject-verb agreement (s-v agr.) and binding for most (at least five out of six) models (see appendix b.1 for examples of these phenomena). in contrast, in-template lp is superior in ellipsis and quantifiers for most models. on climp, yes/no prob comp is good at coverb and in-template lp at np head finality (np head). this indicates that each method harnesses different aspects of the models' grammatical knowledge. given the aforementioned token-length bias, one hypothesis to explain this difference would be that yes/no prob comp is more accurate in phenomena with large token-length differences. our analysis on blimp, however, does not support this, as detailed in appendix d.3. meanwhile, some phenomena are challenging for both methods. as figure 3 shows, on blimp, the two methods underperform humans for all models in island effects and quantifiers, which were shown to be challenging also by @xcite . on climp, our methods struggle with phenomena such as binding and passive. voting ensembles of the top two methods further improve the performance. given the different strengths of in-template lp and yes/no prob comp, we ensemble these methods to see if they can complement each other to achieve higher accuracy. now we have 10 sets of predictions by the two methods, as each has five templates. to compare ensembling single-method predictions and ensembling multi-method predictions on equal terms, we sample five without replacement from the 10 and perform majority voting by the five. we prepare the following four settings, which differ in the balance between the two methods: p-only, mix-p3, mix-l3, and l-only. p-only and l-only are ensembles of predictions by yes/no prob comp only and in-template lp only, respectively. mix-p3 and mix-l3 use three predictions from yes/no prob comp and in-template lp, respectively, with two predictions from the other method. we report the mean accuracy of 10 trials for these settings as the result is non-deterministic due to sampling. word-shuffling paradigms are challenging. blimp's 67 paradigms can be divided into two categories based on whether paired sentences of the paradigm have the same bag of words when the cases are ignored. we call the paradigms where this is true word-shuffling paradigms. in other words, the unacceptable sentence in wordshuffling paradigms can be obtained by shuffling the words in the acceptable counterpart. here, we focus on blimp because climp does not have word-shuffling paradigms. following is an example pair from a word-shuffling paradigm, coordi-nate_structure_constraint_complex_left_branch. figure 4 shows the accuracy by paradigm, paradigm type-word-shuffling or not, demonstrating that the word-shuffling paradigms have much lower accuracy than other phenomena across methods and models. the accuracy of word-shuffling paradigms averaged over models and methods is 71.6% compared to 87.9% of other paradigms. the best accuracy on word-shuffling paradigms, achieved by yes/no prob comp with qwen2, was i. lp y/n p.c. 0.0 0.5 1.0 accuracy llama-3 i. lp y/n p.c. mixtral i. lp y/n p.c. qwen2 i. lp y/n p.c. llama-3-instruct i. lp y/n p.c. mixtral-instruct i. lp y/n p.c. ## conclusion we investigate how to derive the most accurate acceptability judgments from llms by comparing nine methods, using eight llms and two benchmarks. our experiments reveal that in-template lp consistently outperforms conventional sentence probability readout methods while yes/no prob comp achieves the highest accuracies on the english benchmark. our analysis demonstrates that the two methods excel in different phenomena, suggesting they harness different aspects of llms' grammatical knowledge. we also find that ensembling the two methods achieves even higher accuracy. consequently, we recommend ensembling the two methods or employing in-template lp as more effective alternatives to conventional approaches. meanwhile, we show that word-shuffling paradigms remain difficult for all our methods, posing a challenge for future work.