Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models (2024)

Abstract: We introduce Holmes, a new benchmark designed to assess language models’ (LMs’) linguistic competence—their unconscious understanding of linguistic phenomena. Specifically, we use classifier-based probing to examine LMs’ internal representations regarding distinct linguistic phenomena (e.g., part-of-speech tagging). As a result, we meet recent calls to disentangle LMs’ linguistic competence from other cognitive abilities, such as following instructions in prompting-based evaluations. Composing Holmes, we review over 270 probing studies and include more than 200 datasets to assess syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing over 50 LMs reveals that, aligned with known trends, their linguistic competence correlates with model size. However, surprisingly, model architecture and instruction tuning also significantly influence performance, particularly in morphology and syntax. Finally, we propose FlashHolmes, a streamlined version that reduces the computation load while maintaining high-ranking precision.

## introduction linguistic competence is the unconscious understanding of language @xcite , like the syntactic structure of a sentence. as language models (lms) are trained on simple tasks such as next word prediction @xcite , one might naturally wonder: what is the linguistic competence of lms, and how do they differ? to answer such questions, contemporary benchmarks estimate cognitive abilities, as done for mathematical reasoning @xcite or factual knowledge @xcite . however, such benchmarks rely on lms' use of language (textual responses), known as linguistic performance @xcite . as a result, they conflate abilities tested with specific instructions, as done for syntactic phenomena in @xcite , with latent abilities like producing coherent text or following instructions. as this entanglement makes it infeasible to draw definitive conclusions @xcite @xcite , recent studies call to assess lms' linguistic competence comprehensively and isolated @xcite . in this work, we introduce holmes (figure 2 ), a benchmark to assess the linguistic competence of lms (figure 7 ) regarding numerous linguistic phenomena. to disentangle lms' understanding of these phenomena from their linguistic performance, we assess the lms' internals using classifier-based probing @xcite @xcite . as illustrated in figure 1 for probing the part-of-speech (pos) tags for words, we first train linear models (probes) using the internal representations of text inputs from the last model layer to predict the specific phenomena aspects. we then approximate the lms' grasp of these phenomena using the probes' performance, rigorously verified using control tasks @xcite and from an information theory perspective @xcite . with this particular and comprehensive scope, we thoroughly address the initially raised questions as follows: meta-study ( § 3) the review of over 270 probing studies reveals a gap in comprehensively figure 1 : in holmes, we encode examples of probing datasets using frozen lms. then, we train probes (linear models) with labels representing the specific linguistic phenomenon under test. finally, we use the results of testing the probes to approximate the lms' linguistic competence regarding the tested phenomena. benchmark ( § 4) addressing these identified deficiencies, holmes offers a structured way to assess lms' english linguistic competence comprehensively. it features 208 distinct datasets covering morphology, syntax, semantics, reasoning, and discourse phenomena, including previously underrepresented ones like negation or rhetoric. results and analysis ( § 5) from assessing 59 lms, we find that no lm consistently excels over the others. further, linguistic competence is more pronounced for morphology and syntax than the other types of phenomena, and lms' linguistic competence is fundamentally affected by model size, model architecture, and instruction tuning. first, we generalize previous findings @xcite and show that lms' linguistic competence, particularly morphology and syntax, scales beyond 350 million parameters. second, contrary to the prompting evaluations @xcite and aligned with @xcite and @xcite , model architecture is critical. the linguistic competence of decoder-only lms lags behind encoder-only ones. not even 70 billion decoders produce representations for words with the same stability as encoders with 110 million parameters. third, while instruction tuning @xcite @xcite aims to align lms with human interactions, we focus for the first time on its effect on linguistic competence. we found that instruction tuning improves morphology and syntax but has mixed effects on other phenomena types, hinting at a superficial alignment. lastly, we compare holmes with other benchmarks. while lm rankings of reasoning-intense downstream tasks @xcite correlate with reasoning phenomena, explicitly prompting for linguistic phenomena @xcite leads to unreliable results. as these results show that holmes aligns with other benchmarks, its probing-based evaluation is indispensable for explicitly testing lms' linguistic competence disentangled from their linguistic performance. efficiency ( § 6) finally, to mitigate the heavy computational burden of evaluating a new lm on holmes, we form the streamlined version flashholmes by selectively excluding samples not significantly influencing overall rankings @xcite . specifically, flash-holmes approximates holmes rankings with high precision while requiring only ∼3% of the computation. contributions with holmes, we introduce a comprehensive and thorough benchmark to assess lms' linguistic competence, providing ground to evaluate them more holistically. extensive experiments on holmes reveal that lms' linguistic competence is manifold and more pronounced for phenomena targeting words and syntactic structure than semantic, reasoning, or discourse. lms properties like size or architecture crucially account for differences among lms. fostering further research, we provide interactive tools to explore holmes and straightforward evaluation code for upcoming lms with efficiency in mind. ## meta-study this section summarizes our survey of 274 studies ( § 3.1) probing lms' linguistic competence. we analyze them regarding their evolution, probing tasks and lms addressed ( § 3.2), and identify the need to consolidate existing resources ( § 3.3). ## holmes benchmark with holmes, we provide an extensive ground to tackle these identified deficiencies in the existing literature and comprehensively investigate the english linguistic competence of lms. specifically, holmes features 208 datasets addressing distinct aspects of 66 phenomena covering morphology, syntax, semantic, reasoning, and discourse. ## holmes results using holmes, we evaluate a diverse collection of 59 lms. 3 using the results of these extensive experiments, we first answer the research question: what is the linguistic competence of lms? in doing so, we discuss the reliability of results (i) and the linguistic competence of lms concerning the unique structure of holmes (ii). subsequently, we examine how linguistic competence varies among lms, as we find lms prevailing for different types of linguistic phenomena (figure 7 ) and delve into the effects of model architecture (iii), size (iv), and instruction tuning (v). finally, we show how holmes' results relate to the linguistic performance of lms by comparing them with the openllm benchmark (vi) and further experiments with the helm benchmark (vii). i) holmes results are reliable. figure 8 shows the reliability of probing-based evaluations using averaged results across random seeds and lms. single outliers are datasets that are too hard for all lms, either because the sample size is too small or the linguistic phenomena under test are too complex. first, a low average deviation (σ = 0.02) across five seeds underscores the reliability of probing-based measures. these results also highlight the stability of probing results over prompting-based evaluations, where prompt paraphrasing leads to deviations of σ = 0.07 reported in @xcite . next, substantial compression (average @xmath0.9) and selectivity (average @xmath1.31) further confirm the probes' reliability. note, for selectivity, we consider only base-sized model (10m-200m parameters) for computational efficiency. interestingly, two parallel trends emerge. more challenging datasets with many labels, like pos tagging, are arranged around a selectivity of 0.1 to 0.4 and a task metric of 0.3. in contrast, for easier binary classification tasks (such as linguistic applicability), we observe selectivity around 0.2 to 0.5 and a task metric of 0.6 to 0.9. furthermore, our analysis reveals a statistically significant positive correlation (@xmath2.05) between the task metrics and both compression (τ = 0.64) and selectivity (τ = 0.65). this finding provides strong evidence for the reliability of our task metric, thereby justifying its use as the primary evaluation measure in our study. this balanced influence of the five phenomena types is further visible when considering their ranking correlations (figure 10 , left). a high average correlation of 68.4±7.5 with the overall results (last column/row) hints that they are facets of a broader occurrence but share common characteristics. still, breaking into categories is meaningful, as the phenomena types (first five columns/rows) are medium correlated (average of 54.7 ± 13.9). analyzing the results of phenomena types further highlights the value of this distinction. while results of semantics and reasoning are similarly correlated with the overall results (73.9 and 75.6), their direct correlation (58.4) indicates their supplementary nature. further, discourse results show the lowest correlation with others (44.4 ± 14.7), indicating a particular scope. the top-ranked 13b lm for holmes and open-llm, vicuna, was trained on fewer instructions than others (125k) but of higher quality. this high quality seems important as lms with more instructions but lower quality (tülu with approx. 330k instructions) lose performance, the same for 70b versions. further and aligned with the previous comparison with openllm results, reasoning specialization (orca-2) is reflected in the corresponding phenomena. these insights show again that while providing a particular perspective, holmes shows apparent differences between lms and allows us to map them to methodological decisions. as truthfulqa shows lower correlations with the linguistic phenomena and other datasets within openllm, we presume this dataset captures distinctly different skills (possibly knowledge). ## efficiency seamless, easy, cost-effective integration of new lms is crucial to adopting benchmarks widely. as holmes covers many datasets and examples, it is computationally heavy in encoding text and training the probes. it takes ∼ 6 gpu days to encode the 70 million tokens (∼230k pages) and two days to run the 208 probes for a 70b model. to account for this issue, we introduce flash-holmes, a streamlined version of holmes, to evaluate new lms with a fraction of the compute while maintaining evaluation integrity. besides excluding licensed data (18 probing datasets), we analyze the effect of discarding training instances. as a result, we reduce the computation for encoding and the actual probing simultaneously. we follow @xcite and calculate the rank resolution, 95% ci of model rank difference. this measure indicates the maximum expected rank deviation from evaluating an lm on flashholmes compared to holmes. for example, a rank resolution of one means that an lm evaluated on flashholmes and holmes has the same rank or switch place with its neighbors with a probability of 95%. figure 13 shows the resulting rank resolution when training only on a fraction of the instances, from 1/2 to 1/512. solely focusing on efficiency (1/512) still provides a decent rank resolution of ∼2.6. in contrast, considering 1/2 of the training data results in the best reliability of ∼0.9. to balance benchmark reliability and efficiency, we compose flashholmes using 1/32 of the training instances. precisely, it reduces the computation expenses of evaluating lms to ∼3% of what holmes would have required while preserving a high rank-correlation of ∼1.5. assessing the linguistic competence of lms analyzing lms' linguistic competence started with static word vectors @xcite , sentence embeddings @xcite , the internals of translation models @xcite , or contextualized lms @xcite . other methodological work addressed the validity of obtained results with control tasks @xcite or from an information theory perspective @xcite , or studied causal effects @xcite . while further studies focus on whether lms follow human understanding of linguistic competence when solving downstream tasks @xcite @xcite , @xcite and @xcite found that downstream task fine-tuning hurts the understanding of linguistic phenomena. in contrast to prior studies, holmes assesses the linguistic competence of an extensive set of contemporary lms covering a comprehensive collection of linguistic phenomena. unlike other work evaluating linguistic phenomena @xcite using prompting leading to unreliable results @xcite , probing allows holmes to reliably and comprehensively compare lms regardless of architecture or pre-training. as a result, holmes can address recent calls to thoroughly and explicitly evaluate linguistic phenomena @xcite @xcite . ## conclusion holmes marks the most up-to-date and extensive consolidation of existing resources addressing the need to assess the linguistic competence of lms in isolation. our experiments demonstrate that lms' linguistic competence is pronounced regarding formal phenomena but lacks functional ones when information about broader textual contexts, such as rhetorical structure, is required. simultaneously, size, architecture, and instruction tuning are crucial factors for differences among lms. as lm and resources in linguistics constantly grow, we will actively extend holmes with new datasets and upcoming lms.