BAMO at S em E val-2024 Task 9: BRAINTEASER : A Novel Task Defying Common Sense (2024)

Abstract: This paper outlines our approach to SemEval 2024 Task 9, BRAINTEASER: A Novel Task Defying Common Sense. The task aims to evaluate the ability of language models to think creatively. The dataset comprises multi-choice questions that challenge models to think ‘outside of the box’. We fine-tune 2 models, BERT and RoBERTa Large. Next, we employ a Chain of Thought (CoT) zero-shot prompting approach with 6 large language models, such as GPT-3.5, Mixtral, and Llama2. Finally, we utilize ReConcile, a technique that employs a ‘round table conference’ approach with multiple agents for zero-shot learning, to generate consensus answers among 3 selected language models. Our best method achieves an overall accuracy of 85 percent on the sentence puzzles subtask.

## introduction evaluation methods in the nlp community predominantly emphasize vertical thinking, characterized by sequential, analytical processes based on rationality, logic, and rules. however, semeval-2024 task 9, brainteaser @xcite , which is based on the original brainteaser dataset @xcite , aims to introduce a task that promotes lateral thinking (or "thinking outside the box"), a divergent and creative process involving the exploration of new perspectives when addressing problems. the brainteaser qa task consists of two subtasks for the english language: sentence puzzles and word puzzles. this task is designed to challenge the common sense reasoning capabilities of nlp models and stimulate the development of models that can think laterally. we generate baselines for two attention-based models, bert @xcite and roberta-large @xcite , as selected in the task paper, to solve these types of multiple-choice problems. then we fine-tune them with the same configs. after achieving some accuracy through finetuning, we explore zero-shot prompting with various large language models (llms). to further improve results, zero-shot prompting is conducted using a chain of thought technique @xcite . as illustrated in figure 1 , we compel the model to analyze and provide step-by-step reasoning for its answer instead of simply providing a correct option alone. this approach helps the model focus more on details and answer questions with fewer errors. figure 2: an illustration of reconcile for initial round communication among multiple agents is fundamentally important in complex decision-making processes. therefore, as our main strategy, we use the reconcile technique (chen et al., 2023), which generates answers by gathering the consensus of multiple models based on their confidence levels, to address these problems. to make this technique compatible with this type of dataset, we extend the application of reconcile from yes/no questions to the brainteaser questions. in this system, we apply the concept of a society of minds to multiple agents. for round table decision-making, we execute three phases: initial response generation, multi-round discussion, and final answer generation on three language learning models: mix-tral8x7b @xcite , claude foot_0 , and gpt3.5 foot_1 . as illustrated in figure 2 , in each phase, we generate specific prompts (see appendix b) for models to answer the question, along with their confidence level between 0 and 1. we then use their answers for the next round and derive an overall consensus answer for each round. the method that gave us our best result achieved a rank of 11 out of 33 for the sentence puzzles in the task leaderboard. further details of our implementations are available through our github repository. foot_2 . ## conclusion in this paper, we present our approach to semeval 2024 task 9, brainteaser: a novel task defying common sense, which challenges models to think creatively beyond conventional reasoning. through fine-tuning bert and roberta models, as well as employing zero-shot prompting techniques using various large language models, we achieved notable performance improvements. particularly, microsoft copilot performs the best without being taught beforehand, showing it understands the tricky task really well. furthermore, our reconcile round