K now C omp at S em E val-2024 Task 9: Conceptualization-Augmented Prompting with Large Language Models for Lateral Reasoning (2024)

Abstract: Lateral thinking is essential in breaking away from conventional thought patterns and finding innovative solutions to problems. Despite this, language models often struggle with reasoning tasks that require lateral thinking. In this paper, we present our system for SemEval-2024 Task 9’s BrainTeaser challenge, which requires language models to answer brain teaser questions that typically involve lateral reasoning scenarios. Our framework is based on large language models and incorporates a zero-shot prompting method that integrates conceptualizations of automatically detected instances in the question. We also transform the task of question answering into a declarative format to enhance the discriminatory ability of large language models. Our zero-shot evaluation results with ChatGPT indicate that our approach outperforms baselines, including zero-shot and few-shot prompting and chain-of-thought reasoning. Additionally, our system ranks ninth on the official leaderboard, demonstrating its strong performance.

## introduction recently, the natural language processing (nlp) community has witnessed remarkable advancements driven by large language models, such as gpt-3.5 @xcite and gpt4 @xcite , that demonstrated impressive capabilities in tasks like text generation @xcite @xcite , translation @xcite @xcite , reasoning @xcite @xcite @xcite , complex reasoning @xcite , analogical understanding @xcite and sentiment analysis @xcite . however, these models predominantly rely on conventional sequential thinking, often struggling to exhibit the creativ-ity and innovative problem-solving abilities that humans possess. this limitation has spurred researchers to explore the realm of lateral thinking within the nlp domain @xcite . lateral thinking, a concept popularized by @xcite , refers to the ability to break free from established thought patterns and approach problems from unconventional angles. it encourages the exploration of unorthodox ideas, perspectives, and solutions, leading to breakthroughs and the discovery of new opportunities that may have otherwise remained hidden @xcite . harnessing the power of lateral thinking can significantly enhance the capabilities of language models, enabling them to tackle complex, non-linear challenges by thinking "outside the box." however, engaging in this type of reasoning presents a significant challenge, as it demands the ability to contradict common knowledge-a skill highly valued by cutting-edge language models like chatgpt (openai, 2022) and gpt4 @xcite . challenging traditional modes of commonsense reasoning poses a serious obstacle for these language models, as it requires them to set aside their inherent strengths and approach the problem from a different perspective. in light of this direction, @xcite have recently introduced brainteaser, a human-curated benchmark that evaluates the lateral thinking ability of language models. this benchmark encompasses sentence and word puzzles in a questionanswering format that challenge common sense, demanding language models to demonstrate innovative thinking in order to provide accurate and insightful responses. the findings of this study expose a significant disparity in the lateral thinking capacities of even large-scale language models, including those augmented with commonsense knowledge @xcite , when compared to human performance. this gap in accuracy exceeds 40%, emphasizing the necessity for novel approaches to enhance the reasoning capabilities of language models. we propose a new approach to enhance the lateral thinking capability of language models by applying conceptualization @xcite . conceptualization is the process of abstracting instances into high-level concepts, which introduces abstract knowledge associated with the concept for the instance @xcite . our method involves instructing chatgpt to perform conceptualization over the premises in the question via a step-by-step process that identifies instances, conceptualizes them into concepts, generates relevant abstract knowledge, and merges them back into the prompt. to make the judgment less biased among choices, we transform the questions into declarative formats. we test our framework with chatgpt (openai, 2022) in a zero-shot manner, where no training data is used. our experiment results show that our framework achieves an overall accuracy of 78.3% for sentence puzzles and 85.4% for word puzzles, ranking ninth and eighth in the official leaderboard, respectively. ## task definition and dataset we follow the identical task definition as proposed by @xcite , where each data entry can be viewed as a question-answering (qa) task. in each qa pair, the question describes a specific context or puzzle, and the answer serves as the lateral explanation or solution to the puzzle. the goal is to find an explanation that supports and does not contradict a given set of premises (p ), which includes explicitly stated clauses and implicitly derived clauses through default commonsense inferences or associations. the set of premises (p ) plays a crucial role in the puzzle. it encompasses the atomic premise set, which includes explicitly stated clauses (p 1 , p 2 , p 3 ) provided by the context, as well as implicit clauses (p 4 , p 5 ) obtained through default commonsense inferences or associations. these implicit premises can sometimes lead to incorrect assumptions or constraints that hinder finding the correct solution @xcite . the puzzle is presented in a multiple-choice format, where the answer choices represent potential explanations or solutions. this format is chosen to make the task more amenable to automated evaluation and facilitate human comprehension. we use the dataset presented by @xcite as our evaluation benchmark and follow the the original released split of data. since we approach this task by following a zero-shot manner, no training and validation data is used. as shown in ## method in this section, we introduce our proposed method. our method can be divided into three steps: (1) automatically identify instances in the premises in the question and conceptualize them; (2) transform the qa pair into declarative statements; and (3) prompt chatgpt in a zero-shot manner to obtain its prediction. ## experiments in this section, we present details of experiments we conducted on the brainteaser benchmark. ## conclusion in conclusion, this paper describes the solution by the knowcomp group to task 9 of semeval-2024. our method tackles the task of lateral thinking by leveraging the framework of conceptualization, which is a traditional reasoning method performed by humans, to assist large language models in answering brain teaser questions in a zero-shot manner. experiment results show the superiority of our method, outperforming all previous zero-shot baselines with the same large language model as the backbone.