yangqi at S em E val-2024 Task 9: Simulate Human Thinking by Large Language Model for Lateral Thinking Challenges (2024)

Abstract: This paper describes our system used in the SemEval-2024 Task 9 on two sub-tasks, BRAINTEASER: A Novel Task Defying Common Sense. In this work, we developed a system SHTL, which means simulate human thinking capabilities by Large Language Model (LLM). Our approach bifurcates into two main components: Common Sense Reasoning and Rationalize Defying Common Sense. To mitigate the hallucinations of LLM, we implemented a strategy that combines Retrieval-augmented Generation (RAG) with the the Self-Adaptive In-Context Learning (SAICL), thereby sufficiently leveraging the powerful language ability of LLM. The effectiveness of our method has been validated by its performance on the test set, with an average performance on two subtasks that is 30.1 higher than ChatGPT setting zero-shot and only 0.8 lower than that of humans.

## introduction human reasoning processes comprise two types of thinking: vertical and lateral. vertical thinking, also known as linear, convergent, or logical thinking, is a sequential analytical process that is based on rationality, logic, and rules. meanwhile, lateral thinking is a divergent and creative process that involves looking at a problem from a new perspective and defying preconceptions.the success of language models has inspired the natural language processing community to attend to tasks that require implicit and complex reasoning, relying on human-like common sense mechanisms. while such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. recently, the team led by yifan jiang proposed task 9 for semeval-2024, named "brainteaser: a novel task defying common sense," @xcite (jiang et al., * corresponding author. in this paper, we introduce our entries into two braintease subtasks. inspired by recent research on using llm to design agents @xcite , our approach leverages an llm to architect a system that adeptly simulates the intricacies of human divergent thinking processes. specifically, our model capitalizes on the advanced linguistic capabilities inherent within the llm, thereby obviating the need for supplementary training protocols. this strategy enables our system to demonstrate commendable performance across both targeted subtasks. furthermore, we also focus on the issue of hallucinations in llm. llm can sometimes generate erroneous or highly inaccurate responses. the tendency for llm to produce these hallucinations becomes particularly noticeable when confronted with such phenomena in our investigations. to solve this problem, we draw inspiration from rag @xcite strategies. we also discover that the performance of llm can vary greatly depending on the specific prompt words used. as a result, we develop several sets of prompt words and ultimately select the set that achieved the highest performance on our validation tests. our method achieves competitive results on semeval-2024 task 9, ranking well on all tasks, especially on more fine-grained classification tasks. our method far exceeds the performance of chat-gpt, and on the officially provided test set, there is only a slight gap with the results of human evaluation. question choice a man shaves everyday, yet keeps his beard long. he is a barber. he wants to maintain his appearance. he wants his girlfriend to buy him a razor. none of the above. ## methodology we propose a system that simulates human lateral thinking patterns, which consists of two stages. during the first stage, our system engages in a simulation of how humans typically read and interpret brainteaser question stems. the aim here is to check the question stems meticulously, intending to pinpoint specific elements that appear to contravene established common sense norms. the second stage is to combine the parts that violate common sense with four options for thinking, find the option that can "resolve" the parts that defy common sense, and use it as the final answer. the overall architecture of the system is shown in figure 1 . ## experimental setup in this section, we introduce our system settings, and baseline model. ## results and analysis the final results of our experiments are presented in ## conclusion in this paper, we introduce a lateral thinking system named shtl, designed to simulate human lateral thinking capabilities for solving brain teaser questions. the system is divided into two stages. the first stage focuses on common sense reasoning, primarily comprised of the rag module and the saicl module, which are interconnected through appropriate prompt words to generate instances of defying common sense. the second stage involves identifying the correct options to rationalize the defying common sense generated in the previous stage. this system achieves competitive results, significantly outperforming the chatgpt setting in a zero-shot scenario, and its performance on the test set is close to that of human evaluation.