A maz U tah_ NLP at S em E val-2024 Task 9: A M ulti C hoice Question Answering System for Commonsense Defying Reasoning (2024)

Abstract: The SemEval 2024 BRAINTEASER task represents a pioneering venture in Natural Language Processing (NLP) by focusing on lateral thinking, a dimension of cognitive reasoning that is often overlooked in traditional linguistic analyses. This challenge comprises of Sentence Puzzle and Word Puzzle sub-tasks and aims to test language models’ capacity for divergent thinking. In this paper, we present our approach to the BRAINTEASER task. We employ a holistic strategy by leveraging cutting-edge pre-trained models in multiple choice architecture, and diversify the training data with Sentence and Word Puzzle datasets. To gain further improvement, we fine-tuned the model with synthetic humor/jokes dataset and the RiddleSense dataset which helped augmenting the model’s lateral thinking abilities. Empirical results show that our approach achieve 92.5% accuracy in Sentence Puzzle subtask and 80.2% accuracy in Word Puzzle subtask.

## introduction the success of language models has inspired the natural language processing community to attend to tasks that require implicit and complex reasoning. human reasoning encompasses two types of reasoning: lateral and vertical thinking approaches. lateral thinking demand out-of-the-box thinking. it is a form of creative reasoning that deviates from traditional, logical processes and has received little attention from nlp community. vertical thinking on the other hand, relies on logical reasoning, and have been relatively popular in the past few years. the brainteaser dataset by @xcite stands as a crucial benchmark for evaluating question-answering systems. it particularly assesses these systems on their ability for lateral thinking -pushing them to transcend conventional * both authors contributed equally to this work. our system adopts a multifaceted strategy for this challenge, centering on the use of advanced pre-trained models like bert @xcite and debertav3 @xcite through hug-gingface's @xcite automodelformul-tiplechoice and automodelforsequenceclassification. this approach is enhanced by a diverse training regimen that mixes sentence and word puzzle datasets, ensuring a broad exposure to different types of lateral thinking challenges. additionally, the model is fine-tuned with a humor/jokes dataset generated by @xcite and the riddlesense @xcite dataset, which introduces elements of creativity, unconventional thinking, and complex puzzle-solving. this comprehensive strategy aims to equip the model with enhanced lateral thinking abilities, crucial for tackling the creative and nuanced demands of the brain-teaser task in semeval 2024. in our participation in the brainteaser task, we discovered that our system, particularly when finetuned with automodelformultiplechoice, outperformed the baseline instruction-tuned systems mentioned in the original paper. this approach demonstrated a significant advantage in handling multiple-choice tasks. however, we faced challenges with automodelforsequenceclassification, suggesting an area for improvement. the incorporation of additional synthetic data and open-source dataset like riddlesense positively influenced our performance. quantitatively, our system achieved a commendable 6th place in the sentence puzzle and 10th in the word puzzle, indicating stronger proficiency in sentence-based challenges and room for growth in word-based puzzles. our code and data will be available at https://github.com/soumyasmruti/ semeval-2024-brainteaser after cleaning and de-anonimization. ## background the task involves two types of brain teasers: sentence puzzle and word puzzle. in sentence puzzle, the input is a sentence-based question that defies commonsense, with multiple-choice answers. for instance, "a man shaves everyday, yet keeps his beard long." the choices include "he is a barber," "he wants to maintain his appearance," and so on. the word puzzle involves a word-based teaser, like "what part of london is in france?" with choices focusing on letters in the words (e.g., "the letter n"). the output in both cases is the selection of the correct choice that represents lateral thinking. in order to counter the potential for large language models (llms) memorizing solutions, brainteaser @xcite incorporates two novel methods of puzzle generation: semantic and context reconstruction. these techniques generate variations of puzzles that preserve the core challenge of overturning conventional commonsense reasoning without altering the fundamental nature of the puzzles. this approach is aimed at enhancing the robustness of the puzzles against the memorization capabilities of llms, ensuring that the puzzles continue to effectively test the models' ability to engage in lateral thinking by challenging ingrained commonsense assumptions. this is to ensure the model is evaluating reasoning ability rather than memorization. systems are evaluated based on two accuracy metrics: instance-based accuracy, considering each question (original and adversarial) as a separate instance, and group-based accuracy, where a system must correctly solve all questions in a group (original and its adversarial versions) to score. ## related work we can broadly categorize the reasoning landscape of language models into two groups. the first, is 'commonsense reasoning', also known as 'vertical reasoning'. this refers to the ability to make deductions based on everyday knowledge. the second category is 'lateral reasoning'; i.e. a creative problem-solving approach that involves looking at situations from unconventional perspectives. researchers have explored various approaches to endow llms with commonsense reasoning abilities @xcite . one prominent approach is the use of knowledge graphs, which represent structured knowledge in the form of entities and their relationships @xcite . authors in @xcite proposed a method for incorporating commonsense knowledge from conceptnet @xcite into language models, leading to improved performance on commonsense reasoning tasks. another approach involves fine-tuning pretrained llms on commonsense reasoning datasets. authors in this paper @xcite introduced the cosmos qa dataset, which consists of multiple-choice questions that require commonsense reasoning. they showed that fine-tuning pretrained llms on this dataset can significantly improve their commonsense reasoning capabilities. researchers have also investigated the use of prompting techniques to elicit commonsense reasoning from llms without explicit fine-tuning. @xcite proposed a method called "conditional prompt-tuning" that enables llms to perform commonsense reasoning by conditioning on carefully designed prompts. in another work @xcite , chain-of-thought prompting showed how to unlock llm's reasoning ability via effective prompting techniques. there hasn't been extensive research on 'lateral thinking' of llms. very recenlty, olagpt @xcite proposed a cognitive architecture framework in which they summarize various methods of human reasoning into chain-of-thought (cot) templates, to maximize the llms' reasoning effect. overall, while llms have shown flashes of nonlinear, exploratory thinking on some benchmarks, lateral thinking as a holistic cognitive process remains an open challenge. ## methodology in this section, we describe different methods and approaches we employed in solving the brain-teaser puzzle. ## results in in the table, "amsc" represents automod-elforsequenceclassification, and "ammc" represents automodelformultiplechoice. the models used are bert-base-uncased and microsoft/deberta-v3-base. the notation "train-data-wp+sp" indicates that the training data for this approach includes both sentence puzzle and word puzzle training data provided by the organizers of the task. "humor" represents the synthetic dataset generated by prompting gpt-4, and "riddlesense" refers to the open-source riddlesense dataset @xcite . the scores of human performance and the baseline system, as provided in the original paper @xcite , are depicted in gray. scores obtained by our system are shown in black, with the best performances for each task highlighted in bold. ## conclusion in this work, we present our novel system designed for the semeval 2024 brainteaser task, which notably achieved 6th place in the sentence puzzle and 10th in the word puzzle categories. our approach leverages advanced pre-trained models like bert and deberta, optimized through hug-gingface's automodelformultiplechoice and au-tomodelforsequenceclassification. this strategy was further enhanced by incorporating a diverse training regimen, blending sentence and word puzzle datasets with a unique humor/jokes dataset and the riddlesense dataset. this mix has been instrumental in equipping our model with the lateral thinking capabilities essential for this task. while our system excelled in the sentence puzzle, reflecting a stronger grasp in sentence-based lateral thinking, the performance in the word puzzle highlighted areas for improvement, particularly in wordbased lateral reasoning. the additional challenge posed by adversarial versions of puzzles, involving both semantic and context reconstruction, underscores the complexity of this task. our system's performance underscores the efficacy of our training approach in enhancing lateral thinking in language models, a significant step forward in nlp. future work will focus on refining our approach for word-based puzzles and further enhancing the model's ability to navigate complex, creative reasoning paths, thereby advancing the field's understanding of ai's potential in mimicking nuanced aspects of human cognition.