i REL at S em E val-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers (2024)

Abstract: This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models’ lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default commonsense associations and exhibit unconventional thinking. We propose a unique strategy to improve the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dynamic few-shot prompting techniques and introduce a model-generated reasoning strategy that utilizes the LLM’s reasoning capabilities to improve performance. Our approach demonstrated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of performing as well as the human annotators, thus highlighting the efficacy of the proposed strategies.

## introduction human cognition is characterized by two distinct modes of thinking: vertical and lateral @xcite . vertical thinking, often called logical or convergent thinking, follows a structured analytical process based on reasoning and established rules. by contrast, lateral thinking, or "thinking outside the box," is a creative and divergent process that challenges conventional assumptions and explores novel perspectives. vertical thinking and lateral thinking are both complementary @xcite . vertical thinking is known for its selectivity and focus, while lateral thinking is known for its creativity and ability to generate alternative approaches and perspectives. it is crucial to recognize the value of both types of thinking and utilize them in a balanced manner to achieve optimal results. we work on two subtasks (sentence puzzle and word puzzle) introduced as part of the semeval-2023 task 9: brainteaser: a novel task defying common sense @xcite . the sentence-type brain teasers contain puzzle-defying common sense teasers centred around sentence snippets. for instance, the question "what has a bed but no head, a mouth but no teeth?" challenges the default association of beds with people and forces the solver to consider other possibilities, such as a river (which has a riverbed but no heads or teeth). in word-type brain teasers, the answer violates the default meaning of the word and focuses on the letter composition of the target question. for example, the question "what word becomes shorter when you add two letters to it?" challenges the assumption that adding letters to a word would make it longer and requires the solver to recognize that the word "short" becomes "shorter" when "er" is added to it. we used the gemini 1.0 pro model @xcite to evaluate the model's performance in zeroshot and few-shot settings. we also make notable enhancements in the few-shot setting. firstly, by employing contextualized question selection, we ensure that the model is exposed to more relevant examples by identifying questions from the training set that closely resemble those in the test set. secondly, we enable the model to generate explanations for correct answer choices during training through reason generation, thereby deepening its comprehension of the examples. these approaches have demonstrated improvements in the evaluation scores. and @xcite demonstrate recent advancements in multiple-choice question answering. they achieve this by developing new datasets and evaluating large language models (llms) on them, thus contributing significantly to the field's progress. @xcite presents olagpt, an innovative framework designed to enhance the reasoning capabilities of large language models (llms) by drawing inspiration from human cognitive architecture. olagpt integrates cognitive modules such as attention, memory, reasoning, and learning, emphasizing a reasoning module that simulates human-like thought processes. the module then enables olagpt to create multiple agents and utilize various thinking templates, including lateral and integrative thinking, to solve reasoning problems effectively. @xcite introduces a novel evaluation benchmark to assess a model's lateral thinking abilities in an interactive framework, utilizing lateral thinking puzzles as the context. ## data the primary dataset @xcite used in this study encompasses data pertinent to two subtasks: sentence puzzles and word puzzles. the puzzle is presented in a single correct mcq format, where each puzzle consists of a question and several options. among these options, only one is the correct answer to the puzzle. creating multiplechoice questions challenges balancing fairness and intellectual engagement @xcite . this necessitates carefully curating distractors that are not only incorrect but also sufficiently challenging. it is worth noting that within the training data for both puzzles, every brainteaser was accompanied by two distractor options alongside the correct option. please refer to ## results the results of our experiments and ablation studies conducted for the sentence puzzle and word puzzle subtasks are presented in tables 2 and 3 , respectively. the baseline scores from the bestperforming model in the instruction, commonsense, and human categories, as provided by @xcite , serve as benchmarks for comparison. notably, across all six columns of scores, the zero-shot approaches and few-shot approaches outperform the baseline chat-gpt 0 shot and roberta-l models by a significant margin. figure 2: few-shot prompting performance on the word puzzle subtask employing the zero-shot approach of providing the model with the definition of the sentence or word puzzle (definition prompting) yields superior performance across both subtasks compared to simply prompting it to indicate the correct option (direct prompting). the significant improvement contributing factor to the overall score is evident in the significant improvements observed in the context reconstruction scores(con). based on the results shown in figure 1 and contrary to expectations, the findings in figure 2 and ## conclusion in our work, our extensive experimentation demonstrates that the gemini pro 1 model can perform lateral thinking tasks and demonstrate significant improvements in both sentence puzzle and word puzzle subtasks by employing static and dynamic example selection coupled with self-generated reasoning strategies. we achieved notable enhancements over the baseline models and observed minor improvements as we increased the number of examples used for prompting. while our approach demonstrates notable progress, it still falls short of the performance of human annotators, indicating that further research and development are necessary to bridge this gap.