Zero Shot is All You Need at S em E val-2024 Task 9: A study of State of the Art LLM s on Lateral Thinking Puzzles (2024)

Abstract: The successful deployment of large language models in numerous NLP tasks has spurred the demand for tackling more complex tasks, which were previously unattainable. SemEval-2024 Task 9 introduces the brainteaser dataset that necessitates intricate, human-like reasoning to solve puzzles that challenge common sense. At first glance, the riddles in the dataset may appear trivial for humans to solve. However, these riddles demand lateral thinking, which deviates from vertical thinking that is the dominant form when it comes to current reasoning tasks. In this paper, we examine the ability of current state-of-the-art LLMs to solve this task. Our study is diversified by selecting both open and closed source LLMs with varying numbers of parameters. Additionally, we extend the task dataset with synthetic explanations derived from the LLMs’ reasoning processes during task resolution. These could serve as a valuable resource for further expanding the task dataset and developing more robust methods for tasks that require complex reasoning. All the codes and datasets are available in paper’s GitHub repository.

## introduction in the domain of cognitive science, human reasoning is characterized by two distinct processes housed within the brain: 1) vertical thinking and 2) lateral thinking @xcite . vertical thinking, also known as linear, convergent, or logical thinking, is an analytical process that progresses in a sequential manner. it is rooted in rationality, logic, and established rules, and is typically associated with the left hemisphere of the brain. conversely, lateral thinking, colloquially referred to as "thinking outside the box", is a divergent and creative 1 github.com/erfanmoosavimonazzah/semeval2024-task9-brainteaser * authors contributed equally to this work process. it entails approaching a problem from a novel perspective and challenging pre-existing assumptions, and is linked with the right hemisphere of the brain @xcite . to solve a vertical puzzle, the model could follow a linear solution path and provide a step-by-step reasoning for the solution. however, the model was unable to provide a step-by-step solution for solving the lateral puzzle. instead, it offered a fresh perspective on the puzzle and explained why the answer might be unreasonable when considering common sense. with the expansion of the llms market and re-1 1889 model sp wp ours bing* 86.7 97.9 gemini @xcite 70.8 77.1 mixtral 8x7b @xcite search community, the demand for more complex benchmarks and tasks has increased as previous tasks become trivial for each new generation of models. as proposed by @xcite , examining the ability of current state-of-the-art llms to reason on lateral thinking puzzles is a crucial aspect of their reasoning capabilities, which is the primary focus of this paper. we investigate a set of llms in a single model zero-shot, multi-agent voting, and multi-agent debate setup (?). to diversify our experiments, we studied both state-ofthe-art production-level llms like chatgpt and gemini, and open ones like microsoft phi-2 and mixtral 8x7b. we also publish an extended brainteaser dataset, which contains the detailed answers from all the models and techniques we tested. this dataset could be used for further analysis of these models and help to expand this relatively small dataset into a comprehensive benchmark for lateral reasoning evaluation. interestingly, we found that current state-of-the-art models, despite having errors when it comes to lateral thinking, show very promising results. even smaller llms like phi-2 were able to demonstrate lateral thinking procedures in some cases. we submitted our best results which are produced by microsoft bing chat to the task leader-board and ranked 2nd and 10th for word puzzle and sentence puzzle, respectively. @xcite 2 system overview 2.1 zero-shot @xcite frames the puzzles as multiplechoice question-answering tasks. each question or riddle is accompanied by four potential answers, with the last option always being "none of the above". among the remaining three choices, one is invariably correct, while the others are guaranteed to be incorrect. empirical experiments reveal a significant challenge with the current formulation, specifically the last choice. for most of these riddles, the solution is not a logical expression but often involves wordplay or clever misconceptions. due to these factors, llms tend to favor the last choice "none of the above" since none of the other answers make sense in the conventional vertical thinking manner. for example when we asked mixtral model, "what animal keeps the best time? 1) a watchdog. 2) an elephant. 3) a whale. 4) none of above." the model responded "the correct answer is 'none of the above'. the reasoning behind this is that animals do not keep time in the sense of telling the exact hour and minutes like a clock or watch does. however, some animals such as certain bird species and insects are known for their keen sense of timing when it comes to activities such as migration and hibernation.". moreover, this option comprises a negligible portion of the train and validation dataset. assuming that this is also the case for the test dataset (since the la-2 1890 zero-shot prompt which option is the answer to this riddle, explain in a step by step manner: what clothing does a house wear? 1) address. 2) suit. 3) wedding dress. please place your answer in a json format: { "explanation": "option_number": } model response { "explanation": "the phrase 'what clothing does a house wear?' is intentionally misleading because houses do not physically wear clothes. the answer lies within option (1): an address. in essence, you can think of an address as the 'clothing' a house wears since it identifies and distinguishes each building from another.", "option_number": 1 } ## experimental setup we carry out experiments with five distinct llms. for more information about each of them, refer to ## results the majority of our models outperformed the paper baselines, as shown in in the third type, the models were unable to come up with any good explanation. it seems they got confused by the nature of the puzzle and started to generate nonsense. see ## conclusion although there is still a gap between the accuracy of llms and humans when it comes to solving challenging puzzles that require lateral thinking, they currently perform well considering the difficulty of this task. our results indicate a promising path for using an ensemble of large language models to collaborate and solve a problem together, whether they are fine-tuned for this collaboration, like mixtral, or we use prompting ideas like voting or debating. we believe this path still requires thorough research, specifically in the quality of the 4 1892 reasonings generated by each model.