AILS - NTUA at S em E val-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles (2024)

Abstract: In this paper, we outline our submission for the SemEval-2024 Task 9 competition: ‘BRAINTEASER: A Novel Task Defying Common Sense’. We engage in both sub-tasks: Sub-task A-Sentence Puzzle and Sub-task B-Word Puzzle. We evaluate a plethora of pre-trained transformer-based language models of different sizes through fine-tuning. Subsequently, we undertake an analysis of their scores and responses to aid future researchers in understanding and utilizing these models effectively. Our top-performing approaches secured competitive positions on the competition leaderboard across both sub-tasks. In the evaluation phase, our best submission attained an average accuracy score of 81.7% in the Sentence Puzzle, and 85.4% in the Word Puzzle, significantly outperforming the best neural baseline (ChatGPT) by more than 20% and 30% respectively.

## introduction in natural language processing (nlp), reasoning serves as the cognitive backbone, enabling systems to transcend mere language comprehension and delve into sophisticated understanding. despite the excellence of large language models (llms) in several linguistic tasks, their reasoning capabilities are still questionable to a non-negligible extend @xcite @xcite @xcite @xcite , often posing the fundamental concerns of whether they can indeed reason or memorize exhaustively @xcite . such limitations can be probed via well-crafted datasets and benchmarks, showcasing varying llm deficiencies at a time. as the core of the current paper, brainteaser @xcite incorporates problems that stress models to think "out-of-the-box"; to this end, the key novelty of brainteaser is that in order to answer correctly, models need to defy default senses of concepts and common associations. surprisingly, state-of-the-art (sota) llms, such as chatgpt can only exhibit a maximum accuracy of ∼60% when solving brain-teaser riddles, demonstrating an inherently limited reasoning ability in unconventional thinking. thus, assuming that large-scale training and prompting may not always serve as universally applicable solutions towards flexible reasoning, we move one step back and leverage transfer learning techniques starting from smaller models based on masked language modelling, such as bert @xcite and consequent bert-based encoders. then, we proceed with similar techniques on llms, aiming to showcase that significant performance advancements using a small set of in-domain data for parameter updating can be achieved in comparison to merely querying the model's prior knowledge via prompting. therefore, our contributions are: our code is available on github foot_0 . ## related work reasoning in nlp has enjoyed several advancements due to the surge of pre-trained language mod-els and especially llms @xcite . reasoning challenges incorporate commonsense reasoning @xcite , involving inference regarding everyday situations, mathematical reasoning @xcite , referring to the ability of solving mathematical problems, logical reasoning @xcite , which includes the systematic deduction of conclusions based on established principles and formal rules, causal reasoning @xcite , which studies cause-and-effect relationships explaining why an event leads to another, and several other sub-tasks @xcite @xcite . in terms of reasoning evaluation, bigbench @xcite comprises 204 reasoning tasks, targeting to explore the related capabilities of recent llms. several dedicated datasets have been developed to tackle different reasoning challenges, including commonsenseqa @xcite , @xcite , riddlesense @xcite and others; most of these datasets are incorporated in tasksource (sileo, 2023). especially riddlesense questions aspects of reasoning close to brainteaser @xcite . ## task and dataset description the brainteaser task at semeval-2024 @xcite features lateral thinking puzzles presented as multiple-choice questions (qas). each question offers four options, with one being the correct answer and the others serving as distractors. additionally, the final option is always "none of above". it consists of two sub-tasks, task a: sentence puzzle and task b: word puzzle. in addition to the original puzzles, the dataset includes adversarial subsets created by manually modifying the original brain teasers while preserving their reasoning paths. the original data were perturbed in two ways: first, there is semantic reconstruction of each original question without altering the answers or the distractors. second, the original data underwent context reconstruction, wherein the original reasoning path remains intact, but the brain teaser describes a new situational context. overall, the dataset used for training and evaluation consists of triplets of data: original, semantic, and context reconstruction. task a: sentence puzzle in this sub-task, the sentence pairs are crafted in a manner that makes it relatively easy for humans to discern the correct question choice original a peanut. what kind of nut has no shell? a doughnut. a walnut. none of above. semantic reconstruction a doughnut. which nut doesn't have a shell? a walnut. a peanut. ## methods we focus on tuning language models belonging into two categories. first, we fine-tune variations of encoder models, namely bert @xcite , roberta-large @xcite and debertav3base @xcite , to assess the impact of transfer learning using various datasets requiring similar reasoning abilities, apart from brainteaser. we study the problem using the provided multichoice setup, but we also transform it into a binary classification task. secondly, the encoders' results are compared with those obtained from finetuned llms using the brainteaser dataset. to achieve this, we fine-tune llama 2 @xcite , phi-2 @xcite and mistral-7b @xcite , which have already demonstrated enhanced reasoning abilities. in this regard, we examine the effect of the model size on our task, which has already been reported in the literature to significantly influence the reasoning abilities of the models @xcite , along with other tuning hyperparameters. model details are presented in app. a. ## experimental results our metrics for the sentence puzzle sub-task are presented in task a: sentence puzzle initially, it becomes apparent that pre-training encoders across various commonsense reasoning datasets results in substantial performance enhancements, as it enables the system to grasp domainagnostic features which prove advantageous for the subsequent task. additionally, several commonsense pre-trained encoders fine-tuned on brain-teaser data outperform llama 2 and phi-2. another noteworthy observation from task b: word puzzle in regarding encoders, it is evident that, especially vanilla roberta-large lacks robust commonsense reasoning and struggles to systematically handle ambiguity; in contrast, roberta-large pre-trained on winogrande presents competitive performance. this notable enhancement (over 40%) due to wino-grande pre-training suggests that this particular dataset effectively equips the model with the ability to understand word puzzle-related reasoning complexities, making its scores competitive with debertav3 in this sub-task, despite the higher debertav3-base performance over robertalarge in baseline reasoning benchmarks @xcite . other than that, pre-training on other commonsense reasoning datasets does not significantly improve the overall performance for encoders. conclusively, apart from winogrande the rest of the extra pre-training datasets do not hold reasoning cues close to brainteaser's word puzzles. regarding llms, mistral-7b notably outperformed all others by a significant margin, even surpassing the 8 times larger model tuned using the same hyperparameters (mixtral-8x7b). llama 2 example of correct prediction with explanation question choice the other end is not tied to anything. if a dog is tied to a piece of rope that is 6m long how can it reach a bone that is 7m away? the rope stretches proportionally, providing the extra length needed for the horse to reach the hay seven meters away. the dog dug a 1m tunnel underground to reach the bone that was 7m away while still tied to the 6m long rope. none of above. explanation: a dog cannot reach a bone that is 7m away if it is tied to a piece of rope that is 6m long. this is because the distance between the dog and the bone is greater than the length of the rope. therefore, the dog will not be able to reach the bone unless the rope is untied. ## conclusion in this study, we systematically evaluate pre-trained and fine-tuned encoders, along with instructiontuned large language models (llms), against two multi-class classification sub-tasks within the "brainteaser: a novel task defying common sense". we achieve competitive performance in both sub-tasks, accompanied by a plethora of insights regarding the influence of leveraging indomain data, the variability model scale and architecture introduce, as well as the examination of diverging reasoning paths. as future work, we will delve into further reasoning patterns llms tend to follow with regard to lateral thinking challenges.