LM entry: A Language Model Benchmark of Elementary Language Tasks (2023)

Abstract: As the performance of large language models rapidly improves, benchmarks are getting larger and more complex as well. We present LMentry, a benchmark that avoids this “arms race” by focusing on a compact set of tasks that are trivial to humans, e.g. writing a sentence containing a specific word, identifying which words in a list belong to a specific category, or choosing which of two words is longer.LMentry is specifically designed to provide quick and interpretable insights into the capabilities and robustness of large language models. Our experiments reveal a wide variety of failure cases that, while immediately obvious to humans, pose a considerable challenge for large language models, including OpenAI’s latest 175B-parameter instruction-tuned model, TextDavinci002.LMentry complements contemporary evaluation approaches of large language models, providing a quick, automatic, and easy-to-run “unit test”, without resorting to large benchmark suites of complex tasks.

## introduction large language models are typically evaluated via perplexity or by performance on downstream tasks @xcite . perplexity, while associated with general model quality, does not provide specific or interpretable insights into model capabilities @xcite . downstream tasks can provide more isolated testbeds, but with large language models exhibiting evermore impressive results, many benchmarks primarily focus on complex tasks @xcite @xcite . while this trend has obvious merits, it also makes it harder to inspect more fundamental capabilities and to measure model robustness in a systematic manner. in addition, the sheer size of such benchmarks makes replication and analysis out of reach for many in the research community. we present lmentry, a benchmark that requires only very basic language abilities, designed to complement common approaches for evaluating large language models. lmentry consists of 25 tasks which humans are generally expected to perform perfectly, e.g. writing a sentence that contains a specific word, or choosing which of two words is longer. lmentry measures both a model's accuracy and its robustness to a variety of trivial input perturbations. in addition, lmentry provides an lmentry score, a single aggregate score combining accuracy and multiple aspects of robustness. lmentry is intended for zero-shot evaluation: it should be reasonable to expect proficiency from large general-purpose models on such trivial tasks without any task-specific training. we measure the lmentry performance of 16 language models, including gpt-3 @xcite , @xcite , t0 @xcite , and tk @xcite . while human performance on lmentry is measured at 100%, the best performing model on lmentry, the 175b parameter instruction-finetuned textdavinci002, has an lmentry score of 66.1%. the second best model, instructgpt 175b, scores 48.4%, and no other model scores higher than 41%. we also find that instruction finetuning significantly improves accuracy, with instruction-tuned models outperforming much larger (but not instruction-tuned) models. that being said, neither instruction finetuning nor scaling appears to increase robustness. our experiments reveal a wide variety of failure cases that, while immediately obvious to humans, pose a considerable challenge even for the strongest contemporary large language models. lmentry provides informative insights in a quick and automatic manner, thus complementing current evaluation approaches of large language models. 1 ## lmentry lmentry is a benchmark for measuring language model performance on tasks that are trivial to humans. it is designed to provide an interpretable assessment of large language models in a zero-shot setting, evaluating both accuracy and robustness to simple input perturbations. lmentry consists of 25 tasks which humans are generally expected to perform perfectly, e.g. writing a sentence containing a specific word, identifying which words in a list belong to a specific category, choosing which of two words is longer, or identifying which of two words rhymes with a third word. task creation to meet our goals of simplicity and interpretability, we specify a guideline for creating lmentry tasks. in short, lmentry tasks have to (1) be easily solvable by an elementary school student, (2) allow for quick automatic evaluation, (3) be short, and (4) seek to measure a specific aspect of language understanding in isolation. the guideline further details each requirement, and is available in appendix d. following the guideline, each author creates a set of tasks, which are then reviewed by the other authors for adherence to the guideline; every task without complete agreement argument order argument content template adjacent tasks figure 1 : the 4 aspects of robustness lmentry measures, illustrated by the more letters task. argument order: the accuracy gap between examples which differ by whether the answer appears first or second. argument content: the accuracy gap between examples created from different argument subsets. in this example, we test how the length difference between the two words affects accuracy. template: the accuracy gap between a task's templates (the arguments are the same). adjacent tasks: the accuracy gap between similar tasks with a slight difference; here, we measure the accuracy gap between more letters and less letters. lmentry score a model's performance on lmentry (its lmentry score) takes into account both accuracy and robustness. accuracy is calculated as the mean accuracy over the 25 lmentry tasks. robustness is calculated as the mean over four robustness aspects: (1) argument order, (2) argument content, (3) template, and (4) adjacent tasks, which are all illustrated in figure 1 . a model's lmentry score is its accuracy multiplied by its robustness, and is always a percentage. intuitively, the lmentry score can be viewed as a model's accuracy penalized by its brittleness. each robustness aspect is calculated as the mean over all the tasks to which it applies. 2 on an individual task, the robustness aspect is the largest accuracy gap between any two cases (c i and c j ) the aspect considers: for example, if the 3 templates (cases) of task t have accuracies of 60%, 50%, and 80%, then t 's template robustness is 70% (100 -|80 -50|). ## experimental settings we describe the models tested on lmentry and our evaluation protocol. ## results the best performing model is the 175b parameter instruction-finetuned textdavinci002 with a score of 66.1%, significantly below human performance, which is estimated at ~100%. instructgpt 175b scores 48.4%, and no other model scores higher than 41%. these underwhelming lmentry scores are a product of both limited accuracy and a lack of robustness. accuracy varies substantially between models, and in section 4.1 we investigate the factors impacting accuracy the most, particularly the efficacy of instruction finetuning. in section 4.2 we discuss robustness, examining the inconsistent effects of scale on different aspects of robustness. we also highlight many robustness failure cases, illustrating the need to account for robustness when measuring lmentry performance. ## related work the recent emergence of large language models raises the question of how to evaluate their capabilities. we survey three different approaches: perplex- task pair accuracy word after in sentence 90.6 word before in sentence 48.8 word containing letter 85.9 word not containing letter 38.5 word ending in letter 84.1 sentence ending with word 42.1 sentence starting with word 80.0 sentence ending with word 42.1 perplexity a commonly used approach for evaluating large language models is measuring perplexity. perplexity, while associated with overall model quality, cannot provide finer-grained insights (meister and cotterell, 2021). furthermore, the nexttoken prediction objective, used to optimize perplexity, does not always align with human behaviors, expectations, or values @xcite @xcite @xcite . large complex benchmarks large language models are typically evaluated on many downstream tasks @xcite @xcite @xcite @xcite . as the performance of large language models rapidly improves, recent benchmarks become larger and more complex @xcite @xcite . lmentry takes a different approach, avoiding this "arms race" between model and benchmark development by focusing on trivial tasks. while much simpler than the standard benchmarks for evaluating large language models, lmentry nonetheless provides relevant and interpretable insights into model capabilities and robustness, as shown throughout section 4. in addition, lmentry's relatively compact size makes it accessible to labs with fewer resources. simple tasks predating the introduction of large language models, several works try to evaluate models via simple tasks that crystallize a particular aspect of natural language. olmpics (talmor et al., 2020) consists of 8 short classification tasks designed for masked language models. in contrast, lmentry's tasks are all generative, as lmentry can reliably evaluate free-form generations (section 2). in addition, humans score between 85% and 91% on 5 of olmpics' 8 tasks, even when allowed to search the web, while humans score 100% on lmentry without any external resources. babi @xcite is a set of 20 tasks where humans are expected to achieve 100% accuracy without external resources. however, unlike lmentry, most of babi's tasks require multi-step reasoning and commonsense knowledge. hans @xcite also contains relatively simple tasks, but focuses on robustness to three specific permutation types pertaining to natural language inference. checklist @xcite is more of an evaluation framework; given a specific task (e.g. sentiment analysis) checklist assists the user in creating additional task examples for exposing potential failure cases. while highly customizable, creating examples via checklist's software still requires creativity and manual work, whereas lmentry is a concrete ready-to-use benchmark. ## discussion we present lmentry, a benchmark for evaluating large language models using tasks that are trivial for humans. results show that even 175b parameter models significantly underperform humans on these trivial tasks, and are highly brittle to a variety of inconsequential input perturbations. our findings suggest that informative insights on large language models do not always necessitate complex benchmarks, whose sheer size can make them out of reach for many in our community. compared to the incredible performance of large language models on many complex tasks, their performance on lmentry's trivial tasks is rather underwhelming. we hope that highlighting this discrepancy will encourage further research into the underlining principles governing the behavior of large language models, particularly making sure their performance on complex user-facing tasks stands on solid ground. looking forward, with the goal of building more robust models, lmentry can serve as a first step in verifying a model can walk before sending it running. ## limitations when building lmentry, an important criterion for a task is the ability to easily create many examples. one of the benefits of this approach is in handling model memorization: if a model achieves very good results on some task(s), one can quickly create additional examples to verify whether the success is indicative of task competence or originating from example memorization. that being said, a few lmentry tasks are inherently limited in this regard. for example, the number of possible examples of word starting with letter is limited by the number of letters in the english alphabet. we opt to include these tasks in lmentry as we judge their benefits (e.g. broader capability coverage) to outweigh this limitation. to mitigate this disadvantage, lmentry task data also includes a "canary string" (section 2). while a canary string helps in filtering out data from training corpora, it is not a catch-all solution. additionally, our experiments include some models whose exact implementation details are yet to be publicly available (e.g. textdavinci002). we include these results as these models have the best lmentry performance at the time of writing. to shed as much light as possible, we provide the predictions of these models along with all available metadata from the api. this includes the logits of the final layer for the top 5 tokens at each position. dan hendrycks, collin burns, steven basart, andy zou, mantas mazeika, dawn song, and jacob steinhardt. 2021. measuring massive multitask language understanding. proceedings of the international conference on learning representations (iclr). po-sen huang, huan zhang, ray jiang, robert stanforth, johannes welbl, jack rae, vishal maini, dani yogatama, and pushmeet kohli. 2020. reducing sentiment bias in language models via counterfactual evaluation. in findings of the association for computational linguistics: emnlp 2020, pages 65-83, online. association for computational linguistics. opher lieber, or sharir, barak lenz, and yoav shoham. 2021. jurassic-1: technical details and evaluation. technical report, ai21 labs. yao lu, max bartolo, alastair moore, sebastian riedel, and pontus stenetorp. 2022. fantastically ordered prompts and where to find them: overcoming fewshot prompt order sensitivity. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 8086-8098, dublin, ireland. association for computational linguistics. tom mccoy, ellie pavlick, and tal linzen. 2019. right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3428-3448, florence, italy. association for computational linguistics. clara meister and ryan cotterell. 2021. language model evaluation beyond perplexity. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 5328-5339, online. association for computational linguistics. sewon min, xinxi lyu, ari holtzman, mikel artetxe, mike lewis, hannaneh hajishirzi, and luke zettlemoyer. 2022. rethinking the role of demonstrations: what makes in-context learning work? swaroop mishra, daniel khashabi, chitta baral, and hannaneh hajishirzi. 2022. cross-task generalization via natural language crowdsourcing instructions. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3470-3487, dublin, ireland. association for computational linguistics.