BRAINTEASER : Lateral Thinking Puzzles for Large Language Models (2023)

Abstract: The success of language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, relying on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap, we devise BrainTeaser: a multiple-choice Question Answering task designed to test the model’s ability to exhibit lateral thinking and defy default commonsense associations. We design a three-step procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and generation of adversarial examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BrainTeaser based on a semantic and contextual reconstruction of its questions. Our experiments with state-of-the-art instruction- and commonsense language models reveal a significant gap between human and model performance, which is further widened when consistency across adversarial formats is considered. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.

## introduction human reasoning processes comprise two types of thinking: vertical and lateral @xcite . vertical thinking, also known as linear, convergent, or logical thinking, is a sequential analytical process that is based on rationality, logic, and rules, typically associated with the left-brain hemisphere. vertical thinking, as illustrated in figure 1 (top), is needed to create a reasoning path from flooding a room to filling it with water for physical reasoning, and from inanimate objects with five fingers to gloves in riddles. meanwhile, lateral thinking (or "thinking * work done when km was at carnegie mellon university how do you flood a room? ## related work we review prior work on computational creativity, commonsense reasoning, and model robustness. computational creativity computational creativity work includes a broader set of tasks, some of which have been relatively popular, including pun @xcite and humor @xcite detection. a particular class of creative challenges, called brain teasers @xcite , is designed to evaluate a wide range of human intelligence skills, including strategy development, planning, visual-spatial thinking, creativity, and memory @xcite . most similar to our task, lin et al. (2021) collects riddles from public websites to challenge current models. while in principle computational creativity puzzles and brain teasers combine vertical and lateral thinking, prior work has focused on the former category. our brainteaser task complements these works with word-and sentence-level lateral thinking puzzles. brainteaser can serve as a formal platform to evaluate the creative skills of llms, which have been partially explored in recent work @xcite @xcite . commonsense reasoning the task of commonsense reasoning has been popular in recent years @xcite @xcite , accompanied by the introduction of numerous challenging benchmarks @xcite @xcite and availability of largescale commonsense resources @xcite . while each of the existing datasets focuses on different dimensions of commonsense knowledge @xcite , most of them are constructed in the multiple-choice format, due to the ease of evaluation. some prior works have focused on generative commonsense reasoning @xcite . however, due to the vast plausible answer space, the evaluation has been challenging and a large amount of answer annotations have to be collected in order to ensure fairness @xcite . curiously, while possession of common sense has been a central goal of ai, its role in our brain-teaser task is as a distractor. namely, successful solutions of the lateral thinking puzzles in brain-teaser require the models to defy commonsense associations and linear inference chains. robustness studies as a novel benchmark, brainteaser relates to other works that evaluate the performance of llms. since these models are surpassing human performance on some existing benchmarks @xcite , the nlp community has shifted the focus towards robustness evaluation, i.e., whether the model can retain a similar performance to semantically perturbed or adversarially constructed questions @xcite . some recent works have adopted model adversarial approaches to generate datasets that are challenging for models to solve @xcite , while others combine multiple tasks to evaluate the model's behavioral consistency across semantic, logical, and factual categories @xcite . besides dataset construction, analysis studies have also shown that models easily learn shortcuts to solve the datasets @xcite and their performance heavily depends on the overlap of tokens between training and test data @xcite . different from prior works where associative resources are used to finetune the model to improve robustness, we expect that the lateral thinking puzzles in brainteaser require unique associations and creative reasoning paths. in this way, brainteaser is designed to minimize the impact of confounding factors like memorization in llms @xcite @xcite . ## construction of brainteaser in this section, we first provide a definition of lateral thinking puzzles in various granularities. we then present a three-stage pipeline for constructing the multiple-choice puzzles in the brainteaser dataset, consisting of data collection, distractor sampling, and reconstruction sample generation. finally, we present key data statistics and quality validation results. ## experimental setup we describe the models selected for our experiments and the metrics used to evaluate the reasoning accuracy and consistency of these models. ## results our experiments target five questions: 1) can llms reason on lateral thinking puzzles similar to humans? 2) how do llms perform on reconstruction variants? 3) are model predictions consistent across partitions? 4) does tuning on commonsense knowledge help to answer brainteaser puzzles better? 5) can llms do better in the few-shot setting with more demonstrations? overall performance the main results are shown in ## conclusions and outlook we defined the task of lateral thinking for llms, formulated as a multiple-choice qa with a sentence-and word-level puzzles. we developed brainteaser, a 1.1k lateral thinking benchmark that combines original puzzles and their reconstruction variants. our experiments showed that chat-gpt's performance on this task is halfway between random and humans, whereas other models often perform close to random. while scaling up model size improved performance, enriching with common sense or providing few-shot demonstrations yielded limited benefits. meanwhile, all models tend to solve the variants of the same puzzle inconsistently. our error analysis showed that the models' lateral thinking is often hindered by memorization and misleading commonsense associations. in the future, we intend to develop lateral thinking models, create additional lateral thinking evaluation tasks (e.g., relating to alteration @xcite ), and investigate flexible ways to combine lateral and vertical thinking.