MMLU - SR : A Benchmark for Stress-Testing Reasoning Capability of Large Language Models (2024)

Abstract: We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that “truly” understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.

## introduction large language models (llms) have achieved impressive quantitative performance on a wide range of benchmarks, natural language processing @xcite , general knowledge question-answering @xcite , and coding @xcite . additionally, by integrating with some advanced prompting techniques, such as chain-of-thought (cot) @xcite and its variants @xcite @xcite , llms seem to exhibit a certain level of reasoning abilities including mathematics @xcite and even causal inference/discovery @xcite @xcite . however, some studies @xcite ) have * visiting student at rutgers ml lab. a hallmark of human intelligence is the ability to handle abstract concepts and to associate them with arbitrary terms @xcite . with a few exceptions such as onomatopoeia, the connection between particular words and particular meanings is arbitrary, and identical concepts are invoked by different words in different human languages (e.g. dog vs chien). similarly, human reasoners are capable of analogizing structural relationships from one domain to another, meaning that conceptual equivalence can be retained even when details change @xcite . it follows that true human-like comprehension should be unimpaired when terms are substituted for synonymous terms, as long as the substitution is comprehensibly defined. we wondered whether llm peformance reflects true human-like comprehension in this sense, or whether it relies heavily on the specific terms used on training corpora. to assess this, we propose mmlu-sr, a new benchmark dataset that uses symbol replacement to remove some important terms from the questions and answers as shown in figure 1 . instead of relying on memorized terms, this approach tests whether llms can reason using the definitions and concepts of those terms, ensuring a more robust evaluation of their understanding. our evaluations on gpt-3.5/4, gemini, and llama3 families showed significantly lower performance on mmlu-sr compared to the original mmlu, demonstrating the effectiveness of our approach in preventing models from exploiting memorized data. mmlu-sr thus provides a more chal-figure 1 : illustration of our mmlu-sr testing scenarios. the red-colored and green-colored words represent the original symbols in the mmlu dataset showing in answers and questions, which are replaced in the mmlu-sr dataset with random words followed by their definitions, shown in orange text. the example question from the mmlu dataset is correctly answered by both gpt-3.5-turbo and chatgpt-4. however, the modified question from the mmlu-sr "question and answer" dataset is answered incorrectly by both models. our findings indicate that while current llms excel on traditional benchmarks, they face substantial difficulties when key terms are replaced, highlighting the need for benchmarks like mmlu-sr to ensure robust and comprehensive evaluation of language models. they preserve the original structure and intent of mmlu while enabling a broader examination of multilingual capabilities. reasoning capabilities benchmarks. several advanced reasoning benchmarks have been developed to evaluate the reasoning capabilities of language models. agieval @xcite includes standardized tests and civil service exams to assess reasoning and problem-solving skills in academic and professional scenarios. boolq @xcite comprises over 15,000 real yes/no questions paired with wikipedia passages to test the ability of models to infer answers from contextual information. gsm8k @xcite features 8.5k grade-school math problems requiring multistep operations, targeting the evaluation of basic to intermediate mathematical problem-solving abilities. drop @xcite , an adversariallycreated reading comprehension benchmark, challenges models to navigate references and perform question: suppose 'hat' means "the name given to any laboratory technique that measures enzyme activity within a sample." hat can be used to identify carriers of: answer: a. cystic fibrosis. b. fragile x syndrome. c. oculocutaneous albinism. d. tay-sachs disease. the hypothetical technique "hat," described as measuring enzyme activity within a sample. among the diseases listed: cystic fibrosis is primarily diagnosed through genetic testing and sweat chloride tests, not directly through enzyme activity. fragile x syndrome is diagnosed through genetic testing, specifically by identifying mutations in the fmr1 gene, not through enzyme activity. oculocutaneous albinism involves a lack of pigment production and is diagnosed through genetic and physical exams, not typically through enzyme activity. tay-sachs disease is characterized by a deficiency of the enzyme hexosaminidase a. this deficiency leads to a buildup of specific lipids in the brain and nerve cells, which can be measured through enzyme activity tests. given this, the best answer is: d. tay-sachs disease. ## conclusion we introduced mmlu-sr, a novel benchmark that challenges llms by replacing key terms in questions with random words followed by their definitions, aiming to test the models' reasoning and comprehension abilities rather than their memorization skills. our evaluation across multiple domains revealed that popular llms suffer from significant drops in performance with these modifications, highlighting their reliance on memorized terms. mmlu-sr's unique approach addresses concerns about overfitting to traditional benchmarks and provides a more rigorous measure of true language understanding. this dataset will enable researchers to better identify and address the reasoning limitations of current llms, fostering the development of more robust and genuinely intelligent models.