ALF at S em E val-2024 Task 9: Exploring Lateral Thinking Capabilities of LM s through Multi-task Fine-tuning (2024)

Abstract: Recent advancements in natural language processing (NLP) have prompted the development of sophisticated reasoning benchmarks. This paper presents our system for the SemEval 2024 Task 9 competition and also investigates the efficacy of fine-tuning language models (LMs) on BrainTeaser—a benchmark designed to evaluate NLP models’ lateral thinking and creative reasoning abilities. Our experiments focus on two prominent families of pre-trained models, BERT and T5. Additionally, we explore the potential benefits of multi-task fine-tuning on commonsense reasoning datasets to enhance performance. Our top-performing model, DeBERTa-v3-large, achieves an impressive overall accuracy of 93.33%, surpassing human performance.

## introduction the semeval 2024 task 9, brainteaser, is a multiple-choice question-answering task, organized by @xcite and based on the brainteaser benchmark @xcite that aims to test the ability of nlp models to exhibit lateral thinking, a creative type of human reasoning process that often requires looking at problems from a new perspective. unlike similar benchmarks for computational creativity, such as riddlesense @xcite , which focus on problems resolvable through commonsense associations, the brainteaser benchmark comprises questions that challenge models to defy default commonsense associations and linear inference chains @xcite . the task includes two subtasks: sentence puzzle and word puzzle. while the puzzles in the first subtask focus on the meaning of sentences, the word puzzles concentrate on the letter composition of questions and their choices. the following are examples of questions in each subtask. we solely engage in the first subtask of brain-teaser (sentence puzzles) and, due to resource constraints, confine our experiments to models with fewer than one billion parameters. in the subsequent section (section 2), we provide a brief discussion of the models we fine-tuned. subsequently, we offer a more detailed introduction to the task in section 3. section 4 delves into the specifics of our experiments and their outcomes, while section 5 presents our results in the competition alongside a concise error analysis. ## system overview inspired by the recent progress in pre-trained language models, our work investigates the performance of fine-tuned language models on the brain-teaser task. specifically, we fine-tuned two groups of models, i.e., bert-based and t5-based models. ## conclusion in this study, we investigated the effectiveness of fine-tuning various language models (including bert-based and t5-based models) on the brain-teaser benchmark. we demonstrated the efficacy of multi-task fine-tuning on additional commonsense datasets and its impact on performance in brainteaser. although our best models achieved performance surpassing human levels, it's important to note that our study was limited to language models with fewer than one billion parameters and training sets comprising at most two datasets combined. future research could explore extending this study in these directions, as well as investigating other aspects of computational creativity and question-answering. we hope that our work inspires future research in these areas and contributes to the ongoing advancement of natural language understanding and reasoning.