ALERT : Adapt Language Models to Reasoning Tasks (2023)

Abstract: Recent advancements in large language models have enabled them to perform well on complex tasks that require step-by-step reasoning with few-shot learning. However, it is unclear whether these models are applying reasoning skills they have learnt during pre-training , or if they are simply memorizing their training corpus at finer granularity and have learnt to better understand their context. To address this question, we introduce {pasted macro ‘OUR’}model, a benchmark and suite of analyses for evaluating reasoning skills of language models. {pasted macro ‘OUR’}model enables comparing pre-trained and finetuned models on complex tasks that require reasoning skills to solve. Our benchmark provides a test bed to asses any language model on fine-grained reasoning skills, which spans over 20 datasets and covers 10 different reasoning skills. By using {pasted macro ‘OUR’}model we further investigate the role of finetuning . Our extensive empirical analysis shows that language models learn more reasoning skills such as textual entailment, abductive reasoning, and analogical reasoning during the finetuning stage compared to pretraining stage. However, we also find that when language models are finetuned they tend to overfit to the prompt template, which hurts the robustness of models causing generalization problems.

## introduction large language models (llms) (e.g., gpt-3 @xcite , palm @xcite , opt @xcite ) have shown increasing in-context learning capabilities with scaling up the model and data sizes. despite this progress, even the largest of these models still struggle with tasks such as commonsense reasoning @xcite , and math word problems @xcite which require arithmetic reasoning or symbolic manipulation @xcite . the cafeteria had 23 apples. if they used 20 to make lunch and bought 6 more, how many apples do they have? the answer is 29 apples . isabella entered the hall. olivia entered the hall. the apple is in the blue_treasure_chest. olivia exited the hall. isabella moved the apple to the green_basket. question: where does isabella think that olivia searches for the apple? the answer is isabella thinks that olivia searches for the apple in the green_basket . the powerful llms (such as text-davinci-003 foot_0 and chatgpt foot_1 ) fail to make correct predictions. to improve large llms' performance on tasks that require multiple steps of reasoning, recent work used different prompting methods which included a rationale with the final answer in the form of: scratchpad for arithmetic and logical reasoning @xcite , chain-of-thought (cot) @xcite for practically any tasks, or adding let's think step-by-step @xcite to prompt models to generate explanations. other works such as @xcite integrated step-by-step explanations into the finetuning stage (cot-finetuning). while these techniques may improve the accuracy and interpretability, it is not well understood which reasoning skills they rely on or to what degree they require higher-order reasoning. it is also uncertain how frequently the stated reasoning steps actually contribute to the final task predictions. for instance, to correctly answer the questions in in this work, to gain a deeper understanding of llms reasoning abilities in in-context learning settings, we introduce alert, a new pipeline to benchmark different llms on various reasoning skills and provide analysis to assess reasoning abilities. unlike existing commonly used benchmarks (e.g., @xcite ; @xcite ; @xcite ), alert can evaluate llms' fine-grained reasoning skills. it spans over 20 datasets and covers 10 different reasoning skills including logical, causal, commonsense, abductive, spatial, analogical, argument and deductive reasoning as well as textual entailment, and mathematics (see figure 6 ). alert enables easy benchmarking of any lm (e.g., pre-trained, finetuned, cotfinetuned) on a rich set of new inference methods including zero-shot, few-shot and cot. using alert, we further investigate whether finetuning can improve lms' performance on downstream reasoning tasks. specifically, we are interested in diagnosing what actually improved when we observe a performance increase on reasoning tasks. is it because models have seen similar data in the finetuning stage? or is it because models have seen prompts in a specific template and memorize the template during finetuning such as definitions provided in the niv2 benchmark @xcite ? or does the llm actually acquired the required reasoning skill? we investigate these three possibilities. to study the above questions, we compare three different model types (as shown in figure 2 ): a pretrained model and two types of finetuned models. specifically: using these three types of models, we investigate the role of finetuning on three dimensions: reasoning skills datasets logical bigbench repeat copy logic, mmmlu answer generation causal plausable result generation, anli r2 entailment, anli r3 entailment, cb entailment commonsense piqa answer generation, commongen sentence generation, sciq answer generation, openbookqa question answering entailment nli r2 entailment, anli r3 entailment, cb entailment, lue entailment classification mathematics semeval closed vocabulary math, semeval geometric math, mmmlu formal logic abductive tellmewhy spatial babi t1 single supporting fact, piqa answer generation, toqa find location easy clean analogical commongen sentence generation, bard analogical reasoning causation argument argument stance classification, argument consequence classification deductive rocstories correct answer generation to verify this, we carefully divide the evaluation datasets into groups which require different reasoning skills. we compile held-out datasets as shown in figure 6 which require skills held-out from any of the training datasets. this way, we expect to see larger improvements on in-domain skills compared to held-out skills if reasoning skills can be transferred during finetuning stages. ## conclusion we introduce alert, a carefully curated benchmark for evaluating reasoning abilities of llms. it comprises over 20 datasets and covers 10 different reasoning skills. using this benchmark, we further investigate the impact of finetuning on these complex tasks. our experiments reveal that llms do not simply memorize training data, but are capable of learning various reasoning skills, such as textual entailment, abductive reasoning and analogical reasoning. while we found that finetuning generally leads to improved performance, we also discovered some negative effects. llms tend to memorize the data template representation and templates seen during finetuning, thus reducing the robustness of the model to generalized settings. cot-finetuning (opt-cot) can alleviate this issue to some extent, but it is still less robust compared to the vanilla pre-trained model.