ROSHA at S em E val-2024 Task 9: BRAINTEASER A Novel Task Defying Common Sense (2024)

Abstract: In our exploration of SemEval 2024 Task 9, specifically the challenging BRAINTEASER: A Novel Task Defying Common Sense, we employed various strategies for the BRAINTEASER QA task, which encompasses both sentence and word puzzles. In the initial approach, we applied the XLM-RoBERTa model both to the original training dataset and concurrently to the original dataset alongside the BiRdQA dataset and the original dataset alongside RiddleSense for comprehensive model training.Another strategy involved expanding each word within our BiRdQA dataset into a full sentence. This unique perspective aimed to enhance the semantic impact of individual words in our training regimen for word puzzle (WP) riddles. Utilizing ChatGPT-3.5, we extended each word into an extensive sentence, applying this process to all options within each riddle.Furthermore, we explored the implementation of RECONCILE (Round-table conference) using three prominent large language models—ChatGPT, Gemini, and the Mixtral-8x7B Large Language Model (LLM). As a final approach, we leveraged GPT-4 results. Remarkably, our most successful experiment yielded noteworthy results, achieving a score of 0.900 for sentence puzzles (S_ori) and 0.906 for word puzzles (W_ori).

## introduction human reasoning involves two primary types of thinking: vertical and lateral. vertical thinking, synonymous with linear, convergent, or logical thinking, follows a sequential analytical process based on rationality and rules. conversely, lateral thinking, often referred to as "thinking outside the box," is a divergent and creative process that challenges preconceptions by approaching problems from new perspectives. despite the success of language models in tasks requiring implicit and complex reasoning, there is a notable lack of attention to lateral thinking puzzles within the nlp community. to address this gap, the brainteaser question answering task @xcite has been introduced, designed to evaluate a model's ability to exhibit lateral thinking and challenge default commonsense associations. semeval 2024 task 9, brainteaser @xcite comprises two subtasks, sentence puzzle and word puzzle, which require unconventional thinking to overcome commonsense "defaults" without violating hard constraints. an adversarial subset is included in both tasks, created by manually modifying original brain teasers without altering their underlying reasoning paths. in our initial series of experiments, our focus is on fine-tuning xlm-roberta @xcite in three variations: once on the original training data, once alongside the birdqa dataset @xcite , and once alongside the riddlesense dataset @xcite . additionally, we introduced an innovative approach involving the extension of each word in the birdqa dataset into a complete sentence. this method aims to enhance the contextual meaning of individual words during the training process for word puzzle (wp) riddles. to achieve this, we utilized chatgpt-3.5 to expand each word into a comprehensive sentence, applying this transformation to all options within each riddle. subsequently, our exploration extends to the application of reconcile (round-table conference) (chen et al., 2023), incorporating three substantial language models: gpt 3.5, gemini, and the mixtral-8x7b @xcite ) large language model (llm), a pre-trained generative sparse mixture of experts. noteworthy is the superior performance of the mixtral-8x7b model compared to llama 2 70b across various benchmarks. in the third set of experiments, we assess the zero-shot performance of gpt-4 using the copilot gui. our observations highlight a significant superiority of gpt-4 over alternative models and methods. furthermore, our findings underscore the collaborative utilization of large language models (llms) in a round-table format, showcasing substantial enhancements in overall performance. evaluation metrics are based on two accuracy measures: instance-based accuracy, treating each question (original/adversarial) as a distinct instance, and group-based accuracy, where each question and its associated adversarial instances form a group, and a system is awarded a score of 1 only if it correctly solves all questions within the group. our submission to the evaluation phase comprised xlm-roberta fine-tuned on the original training dataset and birdqa dataset. the resulting method ranked 25 out of 31 in sentence puzzles and 20 out of 23 in word puzzles. for a detailed implementation of our method, refer to our github repository. ## background the model's inputs consisted of the puzzle and its corresponding choices, provided as input to xlm-roberta. for alternative methods, we employed a prompt, feeding both the puzzle and choices to the model. all puzzles were written in english. to enhance the training of xlm-roberta, we augmented the primary training dataset with additional datasets, namely birdqa and riddlesense. in the context of word puzzles, we further enriched each choice by transforming it into a complete sentence using chatgpt. the output from all models and methods was expressed as a numerical representation, denoting the correct choice in a zero-based format. ## experimental setup we allocated 20% of the original dataset for our validation set, resulting in 80 samples for validation in the word puzzle (wp) domain and 102 samples for validation in the sentence puzzle (sp) domain. notably, when incorporating additional datasets into our training data, we maintained consistency by retaining the original validation dataset throughout the training process. this decision was driven by the recognition that the supplementary data introduced distinct variations compared to the original training and testing data. preserving the originality of the validation data aimed to uphold the quality and uniqueness of the final model. for fine-tuning using xlm-roberta, we utilized the hugging face platform and implemented a cosine annealing scheduler. ## results leveraging the birdqa and riddlesense datasets led to enhancements across all the metrics utilized for evaluating our model, surpassing the performance observed with the original dataset. gpt-4 consistently outperformed other models by a significant margin, demonstrating superior results across all metrics. ## conclusion this study explores various methodologies for tackling semeval 2024 task 9: "brainteaser: a novel task defying common sense." to enhance our model's performance in word puzzles, we incorporate additional datasets for fine-tuning. additionally, we introduce a modified round-table approach implemented over four rounds. we also evaluate the zero-shot performance of gpt-4 on this task,