S em E val-2024 Task 9: BRAINTEASER : A Novel Task Defying Common Sense (2024)

Abstract: While vertical thinking relies on logical and commonsense reasoning, lateral thinking requires systems to defy commonsense associations and overwrite them through unconventional thinking. Lateral thinking has been shown to be challenging for current models but has received little attention. A recent benchmark, BRAINTEASER, aims to evaluate current models’ lateral thinking ability in a zero-shot setting. In this paper, we split the original benchmark to also support fine-tuning setting and present SemEval Task 9, BRAINTEASER(S), the first task at this competition designed to test the system’s reasoning and lateral thinking ability. As a popular task, BRAINTEASER(S)’s two subtasks receive 483 team submissions from 182 participants during the competition. This paper provides a fine-grained system analysis of the competition results, together with a reflection on what this means for the ability of the systems to reason laterally.We hope that the BRAINTEASER(S) subtasks and findings in this paper can stimulate future work on lateral thinking and robust reasoning by computational models

## introduction vertical thinking requires logical and commonsense reasoning, i.e., making plausible sequential associations of different pieces of commonsense knowledge. as presented in figure 1 (top), we can easily infer that flooding a room requires filling it with water, based on common sense, and inanimate objects with five fingers are gloves in the riddle. in contrast, lateral thinking is a creative and divergent process that requires thinking out of the box and defying common sense. for example, as shown in figure 1 (bottom), one needs to overwrite the commonsense associations of man shaves to he 1 we use brainteaser to represent the original benchmark and brainteaser(s) to represent the data in se-meval task for clarity. ## source dataset we use our recently introduced brainteaser dataset @xcite as the basis for our evaluation. in this section, we briefly describe the data construction pipeline and we refer interested readers to @xcite for full details. the data construction pipeline has three stages. in the first stage, we collect lateral thinking puzzles from public websites such as riddles.com and rd. com and conduct filtering and deduplication. then, the remaining questions are manually verified to ensure that they fit in the sentence or word puzzle categories. since the collected puzzles are open-ended questions, which poses great challenges for evaluation. these open-ended puzzles are then converted to multiple-choice questions in the second stage. specifically, we leverage tools such as comet @xcite , wordnet and wikipedia to construct distractors for every question. for sentence puzzles, we collect distractors that overwrite non-central premises of the question, and for word puzzles, we collect distractors that are semantically similar to the correct answer to ensure they are challenging for systems. finally, in stage three, we construct additional data to mitigate the risk of memorization by large pretrained language models. in particular, for each question, we rephrase the original question using an open-source rephrasing tool without changing its answers or distractors. 5 this set is referred to as semantic reconstruction. additionally, we leverage gpt-4 to reconstruct each question into a new context such that the misleading question premise is kept. in this case, both the question and the correct answer become different, but the reasoning path remains the same. after reconstruction, the distractors are collected in the same way as described earlier. this set is referred to as context reconstruction. a strong reasoning model is expected to solve all variants of the question consistently, as their reasoning patterns are identical despite being phrased differently. in total, we construct 1,119 data samples, including reconstruction variants. we report the key statistics in ## discussion we start the discussion with the question: "is lateral thinking solved?" the best-performing systems reach 100% on both tasks, making it seem that the task is solved. however, there remain many questions to explore. our discussion targets 5 questions to provide overall insights: 1) what's the difference between the brainteaser(s) se-meval task and the original brainteaser benchmark? 2) what's the difference between the best systems for sentence puzzles and word puzzles? 3) are model predictions consistent with individual and group partitions? 4) what does finetuning mean for lateral thinking tasks? 5) what challenges still exist in the realm of lateral thinking? ## conclusions and future perspectives this paper summarizes semeval 2024 task 9, brainteaser(s), a novel task defying common sense. we present the motivation, data design, data construction, evaluation process, competition systems, participant results, result analysis, and discussion. brainteaser(s) was popular among participants and received 483 submissions from 182 teams during the competition, with various method adaptations and architecture selections demonstrating different advantages on each subtask and evaluation metric. the best-performing systems have impressive performance on both subtasks, which reach 100% accuracy on lateral thinking puzzles from the web. however, our finegrained analysis highlights the remaining questions and challenges for further research. importantly, brainteser(s) semeval result is evaluated over a subset (20%) of original brainteaser benchmark. even on this subset and despite the access to 80% of the data for training, models still strug-gle to reason consistently on semantic and context reconstruction. future work should investigate flexible ways to combine lateral and vertical thinking, construct better evaluation metrics for creative and open-ended generations, build connections within reconstruction based on analogical reasoning @xcite and explore a dynamic, multi-stage process where the model (or human) can request clarifications or obtain contextual hints. the brainteaser(s) semeval task, together with its source brainteaser task, is the first step toward injecting ai systems with lateral thinking ability. we hope that the competition results and analysis can inspire future research on developing and evaluating lateral thinking models.