IIMAS at S em E val-2024 Task 9: A Comparative Approach for Brainteaser Solutions (2024)

Abstract: In this document, we detail our participation experience in SemEval-2024 Task 9: BRAINTEASER-A Novel Task Defying Common Sense. We tackled this challenge by applying fine-tuning techniques with pre-trained models (BERT and RoBERTa Winogrande), while also augmenting the dataset with the LLMs ChatGPT and Gemini. We achieved an accuracy of 0.93 with our best model, along with an F1 score of 0.87 for the Entailment class, 0.94 for the Contradiction class, and 0.96 for the Neutral class

## introduction the brainteasers are problems or puzzles, typically designed to be solved for amusement. to solve brainteasers is necessary the lateral and vertical think, so interpret the context itself contained in them. the semeval 2024 brainteaser: a novel task defying common sense task poses a set of brainteasers and their answers, divided into two types: sentence puzzles and word puzzles, both in the english language and require an understanding of common sense and the ability to overwrite them through unconventional thinking that distinguishes these defaults from fixed constraints. in sentence puzzles, a challenge is presented that defies common sense focused on sentence fragments. in word puzzles, the answer challenges the predefined meaning of the word and focuses on the letter composition of the target question @xcite . solving brainteasers requires an unconventional or out-of-the-box approach, which stimulates lateral thinking. this style of thinking is crucial for discovering ingenious solutions to complex problems and for considering situations from multiple perspectives. this type of thinking must be integrated into language models, as it enables them to provide diverse perspectives and apply them to more complex aspects of language, such as understanding metaphors, idioms, or ambiguities. this paper documents the participation of the iimas team at semeval-2024 task 9, where the resolution of brainteasers was approached using a classification framework. our strategy relied on fine-tuning techniques applied to pre-trained models using a transformer architecture. in addition to describing our approach, we also analyze the challenges encountered during the process and discuss potential areas for improvement in future research. this paper sheds light on the application of cutting-edge techniques in natural language processing to tackle comprehension and reasoning problems, such as brainteasers, and provides valuable insight into the performance and limitations of our approach in this specific context. during the evaluation phase, the results placed us at the 33th out of 50 participants. ## background we examine various methodologies for solving brain teaser challenges. in this overview, we present some of these approaches. @xcite focused on solving logic grid puzzles. initially, they identified keywords as entities and the relationships between them. subsequently, they constructed a pair of answer set programming rules. these rules served as inputs for a logic reasoner named logicia, equipped with a predefined set of predicates. their model demonstrated an impressive 85.05% accuracy in classifying constituents and successfully solved 71 out of 100 test puzzles. the riddlesense challenge, introduced by @xcite , aims to explore the task of answering riddles. this challenge presents participants with a multiple-choice question-answering scenario, where a model must select one answer from a set of five choices (one correct answer and four distractors) in response to a given riddle question. the dataset comprises 5.7k meticulously curated examples. in their experiments, researchers employed various approaches including fine-tuning pre-trained language models such as bert @xcite , roberta @xcite , and albert @xcite , alongside fine-tuning a text-to-text qa model @xcite . their methodology involved concatenating the question with the answer choices. during evaluation, three native english speakers achieved an average accuracy of 91.3%, with the best-performing model achieving 68.8% accuracy. current language models can be evaluated in what is known as vertical or convergent thinking and perform well; however, the existence of lateral or divergent thinking in the human mind leads to considering the option of evaluating these same models in this way of thinking. this idea is taken by @xcite to propose a way to evaluate large language models (llms) in lateral thinking puzzles, also known as situations puzzles. this type of puzzle involves a host who knows the complete truth but gives the player a story lacking certain information. the player, through questions that are only answered with yes or no, must deduce the whole truth. the gtp-4 model from openai had the best performance in this type of puzzle according to the proposed evaluation. tong et al. (2023) also identified the need for non-linear thinking in llms, so in their work, they proposed inferential exclusion prompting (iep) inspired by the method of elimination thinking. this proposal consists of, given a problem, the iep instructs the llms to plan different responses and then eliminate those options that are contradictory or irrelevant. the iep was evaluated for various problems: parajumbles, riddles, puzzles, brain teasers, and critical reasoning queries against chain-of-thought (cot) prompting. ## system overview the data used in this task were provided by @xcite , comprising a set of 507 brainteasers for sentence puzzles and 396 brainteasers for word puzzles. each of these brainteasers includes one correct answer alongside three distractors. this dataset showcases the complexity of the posed problems, suggesting that they can be effectively addressed through a natural language understanding (nli) approach. in this context, the bart model @xcite serves as an option for resolving multi-genre natural language inference (multinli) problems, where a model's ability to determine which of the proposed premises is true relative to a hypothesis is evaluated using a multi-choice approach. we apply zero-shot classification to the bart model, and as result, we got a low performance as we describe in to accomplish this, data transformation was necessary to operate under a classification approach, where each question serves as a value for the premise feature, and each answer is treated as a value for the hypothesis feature, these being the indicators: distractor1, distractor2, distractor(unsure), and correct answer. each pair of data is assigned a class label. for fine-tuning bertbase-uncased, three different classes are managed. for sentence pairs containing distractor1 and dis-tractor2, the corresponding label is contradiction; for distractor(unsure), it is neutral, and for the correct answer, it is entailment (see fig 1 ). for the roberta winogrande model @xcite , it is expected that the resulting sentence from concatenating the premise with the hypothesis will have a boolean value depending on the dependencies of the hypothesis concerning the premise. therefore, the labels used are false and true. both models utilize the following hyperparameter values: @xmath0, @xmath1, @xmath2, as well as a split of the dataset with 80% for training and 20% for evaluation purposes. in order to enhance the performance of the models, we leveraged the unique capabilities of large language models (llms). we employed chatgpt 3.5 foot_2 and gemini foot_3 to generate additional brainteaser instances. these instances were then incorporated into the fine-tuning process of pre-trained models. despite having more examples due to data transformation, additional examples were generated through language models such as chatgpt and gemini. the generated data underwent manual review to prevent errors regarding the correct answers to the brainteasers. with the expansion and transformation of the data, a total of 4,644 labeled pairs were obtained for fine-tuning the models with brainteasers from both tasks. ## experimental setup the evaluation results of the bert fine-tuning model are presented in consequently, the use of brainteasers generated for the word puzzle task was discarded, as is shown in after eliminating all synthetically generated data, a noticeable improvement in the evaluation metrics for entailment and contradiction classes was achieved, as we present in ## result during the evaluation phase of semeval-2024 task 9, administrators provided a dataset comprising 120 sentence puzzles and 96 word puzzles. the results, depicted in ## conclusions this work introduced a solution for semeval-2024 task 9: "brainteaser -a novel task defying common sense", leveraging pre-trained lan-