Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments (2025)

Abstract: Large language models (LLMs) can explain grammatical rules, yet they often fail to apply those rules when judging sentence acceptability. We present grammar prompting , an explain-then-process paradigm: a large LLM first produces a concise explanation of the relevant syntactic phenomenon, then that explanation is fed back as additional context to the target model – either an LLM or a smaller language model (SLM) – before deciding which sentence of a minimal pair is grammatical. On the English BLiMP, Chinese SLING, and Russian RuBLiMP benchmarks, this simple prompt design yields substantial improvements over strong baselines across a wide range of syntactic phenomena. Feeding an LLM’s metalinguistic explanation back to the target model bridges the gap between knowing a rule and using it. On SLMs, grammar prompting alone trims the average LLM-SLM accuracy gap by 20%, and when paired with chain-of-thought, by 56% (13.0 pp \to 5.8 pp), all at negligible cost. The lightweight, language-agnostic cue lets low-cost SLMs approach frontier-LLM performance in multilingual settings.

## introduction large language models (llms) occupy an uncanny valley: they appear almost human in their language use, excelling at tasks from text generation to translation to complex reasoning. yet these same models show surprising gaps, particularly when asked to make explicit grammatical judgments -a limitation that becomes especially apparent in multilingual contexts @xcite . while modern language models generate fluent text, they often struggle to reflect on grammatical structure, suggesting a disconnect between their ability to use language and their capacity to reason about it, a gap that is particularly pronounced in smaller models. consider this example. we asked claude 3.5 sonnet to reason out which minimal pair sentence @xcite ) is more grammatically acceptable: we observed this pattern across different llms and linguistic phenomena: when making acceptability judgments, models typically begin by paraphrasing (or translating) the sentences, and in this process frequently obscure the grammatical features crucial for accurate judgments. how can we help models focus more directly on linguistic structure? to explore this question, we adopt a classic psycholinguistics task: grammatical acceptability judgments @xcite . specifically, we evaluate whether models can reliably distinguish grammatically acceptable forms from grammatically unacceptable forms by using minimal pairs that differ by only one syntactic feature (e.g., a missing question particle). we focus ## related work recent theoretical work by @xcite suggests that llms possess considerable latent linguistic knowledge that can be activated through in-context learning (icl) and introducing new linguistic information. for example, @xcite and @xcite demonstrate the effectiveness of targeted explanations and validation-tuned prompts to improve ability in specific tasks. notably, the machine translation from one book (mtob) paradigm found that llms could improve translation between english and a zero-resource papuan language, kalamang, by consulting one grammar book @xcite . follow-up work showed that incorporating explicit rules can help in some cases @xcite , but might be insufficient without additional strategies such as validation-tuned prompts @xcite . grammatical acceptability judgment tasks have long been a fundamental method in psycholinguistics for assessing linguistic competence, typically presenting single sentences for evaluation @xcite . in computational linguistics, this task has been systematized using minimal pairssentence pairs that differ by only one syntactic feature-allowing for more controlled comparisons. this approach led to the development of benchmark datasets such as the english-language cola and blimp @xcite , which have inspired similar resources for other languages, including chinese @xcite , russian @xcite , arabic @xcite , dutch @xcite , and indonesian @xcite . while some computational approaches to minimal pair evaluation still rely on intrinsic metrics like perplexity or model-assigned probabilities, there is growing interest in prompt-based methods that explicitly ask models to make grammatical acceptability judgments @xcite , aligning more closely with how llms are used in interactive settings. ## explain-then-process grammar prompting we hypothesize that while llms can often describe grammatical rules, they do not consistently apply these rules when making grammatical acceptability judgments. to address this, we propose an explain-then-process framework for grammar prompting allowing us to test whether llms can integrate explicit metalinguistic knowledge into their grammatical reasoning. by explicitly linking grammatical descriptions to acceptability judgments, this approach tests whether llms can internalize syntactic rules. task: explain, to a [novice learner/expert linguist], the 'left branch island echo question' in english. context: help a [novice learner/expert linguist] understand and identify correct usage of this concept to pass a test. requirements: • clear, non-technical title • explain key rules and patterns • include specific grammatical details • ... examples (for reference, do not include): good: many boys will embarrass what patient? bad: what will many boys embarrass patient? good: sara was insulting what student? bad: what was sara insulting student? target length: 250 words this is particularly important in multilingual settings, where access to explicit grammatical explanations is limited for many languages. unlike english, which benefits from extensive linguistic resources, many languages lack formalized grammatical descriptions. explain-then-process prompting allows llms to self-generate grammatical explanations, reducing reliance on external resources. to elicit principled and consistent grammatical explanations from llms, we first design an instruction template (figure 2 shows a sample template) so as to ensures systematic coverage across languages and structures. each template includes the grammatical paradigm name (e.g., npi licensing), sample minimal pairs, and an instruction to avoid full example sentences, preventing reasoning based on examples @xcite . the instruction template also specifies a target audience: we request either a description for beginners with clear, straightforward explanations, or for experts with technical descriptions and formal terminology, following mtob @xcite . next, we use these instructions to ask llms to generate targeted grammatical explanations, or grammar prompts. figure 3 shows a grammatical explanation generated by sonnet. given that the models we examine have english as their dominant training language @xcite , we used english prompts throughout, while the mini- ## results and analysis we present the results (macro-average accuracy across paradigm categories) of four main experiments: (1) comparing beginner vs. expert-oriented grammar prompts (section 5.1), (2) evaluating control and textbook conditions (section 5.2), (3) assessing performance on blimp, sling, and rublimp (sections 5.3-5.5), and (4) comparison with three-shot results (section 5.6). in section 5.7 we examine how grammar prompting narrows the gap between llm and slm performance. ## discussion our results across english, chinese and russian reveal clear patterns how grammar prompting works for different linguistic phenomena. it is most effective for phenomena governed by simple distributional constraints and categorical distinctions, such as negative polarity items in english, aspect marking in russian and chinese, or alternative questions in chinese. these often reach near-perfect accuracy under grammar prompting, likely because they involve specific functional elements (e.g., any, haishi, sebya) that operate under specific licensing environments or paradigmatic in contrast, phenomena that demand finegrained constituent recognition (e.g. english island constraints) or language-specific lexical knowledge (e.g. chinese classifier-noun agreement, wh-fronting) show smaller but still consistent accuracy gains. appendix c documents instances of successes and characteristic failure modes: when the model correctly identifies the relevant constituent, grammar prompting enables rule-based judgments even on island violations; when it misparses or loses track of its own reasoning (as gpt-3.5 occasionally does on chinese wh-fronting), the benefit is muted. these errors appear to stem less from the prompting paradigm itself than from (i) occasional constituent identification failures, (ii) the added cognitive load of explicit chain-of-thought reasoning, and (iii) our deliberate decision to omit illustrative examples. addressing any of these factors -better parsing, better reasoning models, or simply adding in-context examples -could therefore push performance on the "hard" paradigms substantially higher. we opened this paper with an incorrect conclusion drawn by sonnet using cot. with grammar prompting, sonnet now reasons: "the key difference is the placement of "only". in sentence a, it's correctly placed at the beginning of the entire noun phrase, scoping over 'a popsicle that danielle admires'. in sentence b, it's incorrectly placed inside the relative clause, which changes the meaning and doesn't follow the grammar rule explained. therefore, sentence a is more grammatically acceptable according to the provided explanation." in the introduction, we observed how models can err by abstracting away surface but essential linguistic form in the attempt to decipher communicative intent; now, we see how grammar prompting reduces problematic 'semantically-driven reasoning' by 1) harnessing the model's metalinguistic ability to generate a grammar prompt and 2) feeding the grammar prompt back to the model, thereby directing its reasoning process to operate over the relevant linguistic structure. ## conclusion llms are trained to distill away surface forms to capture meaning. this 'meaning-first' strategy powers functional tasks such as translation but can sabotage syntactic reasoning. grammar prompting recalibrates that bias and does so as a lowcost equalizer between model sizes. with a single rule explanation generated by a large llm and fed back with chain-of-thought, three slms finish on average within 5.8 pp of gpt-4-class models across english, chinese, and russian, cutting the capacity gap by 56%. from general to specific reasoning. grammar prompting anchors the model's attention on relevant grammatical categories and constraints. from implicit to explicit knowledge. it turns latent metalinguistic knowledge into an explicit rule the model can apply, boosting accuracy even when data-driven cues are sparse. the explain-then-process paradigm is attractive for (i) low-resource languages, where training data are scarce, and (ii) deployments that must rely on smaller, cheaper models when frontier llms are impractical.