M ason T igers at S em E val-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thought Prompts (2024)

Abstract: Our paper presents team MasonTigers submission to the SemEval-2024 Task 9 - which provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques. Zero-shot and few-shot prompting generate reasonably good results when tested with proprietary LLMs, compared to the open-source models. We obtain further improved results with chain-of-thought prompting, an iterative prompting method that breaks down the reasoning process step-by-step. We obtain our best results by utilizing an ensemble of chain-of-thought prompts, placing 2nd in the word puzzle subtask and 13th in the sentence puzzle subtask. The strong performance of prompted LLMs demonstrates their capability for complex reasoning when provided with a decomposition of the thought process. Our work sheds light on how step-wise explanatory prompts can unlock more of the knowledge encoded in the parameters of large models.

## introduction in recent years, llms have achieved impressive performance on several question answering and language understanding tasks when provided with appropriate prompting @xcite . however, complex reasoning abilities often present a challenge for these models. semeval-2024 task 9 @xcite introduces a novel dataset called brainteaser @xcite which includes a set of complex puzzles and brainteasers. such tasks involve solving word and sentence puzzles, which require multi-step inference and deduction. the dataset covers a diverse range of puzzle types including sequences, analogies, classification, mathematical reasoning, inferences about implicit relationships, and more. solutions frequently demand a chained application of knowledge and logic across multiple steps to uncover insights or concepts not directly stated in the problem description. solving these elaborate reasoning problems is a challenging scenario for nlp systems. we explore whether and how llms can succeed on this task. we employ proprietary models such as @xcite and claude 2.1 @xcite through apis. these models have shown promising few-shot reasoning ability. we also use mixtral @xcite , an open-source llm that shows state-of-the-results in several language reasoning tasks. the prompting paradigm involves providing models with natural language descriptions that encode the reasoning process step-bystep @xcite . we implement various prompting approaches for mapping puzzles to conditional text and systematically transforming reasoning into explanation chains. our core method, chain-of-thought prompting @xcite , iteratively breaks down the deduction into simplified logical steps. experiments reveal that while zero-shot performance lags due to a lack of grounding, multi-step prompts can unlock substantial reasoning ability in models. performance improves with more steps and more specificity in the prompt. while introducing few-shot prompting generates good results, we observed that models do significantly better with chain-of-thought prompting. we experiment with several chains of thought and achieve mostly similar results with each attempt. to make a more empirically confident guess towards solving the puzzles we adopt an ensemble of these chains based on majority voting. our approach achieves competitive performance, ranking 2nd on the word puzzle subtask and 13th on sentence puzzles. ## related work llms have been widely used for complex and challenging language processing tasks recently @xcite . they have shown good reasoning abilities in several tasks. the task of solving puzzles and the brainteaser dataset @xcite represent both a novel task and a novel dataset respectively. similarly to their multiple choice questions (mcqs) approach, a few datasets like mathqa @xcite , have been compiled. however, these are intended for specific tasks in which domain knowledge is usually enough thus they not requiring deep reason-ing. a similar work is done by @xcite where they investigate a task that combines natural language understanding and commonsense reasoning. they present deep learning architectures for distinguishing between sensible and nonsensical statements. pun detection by @xcite is a puzzlelike activity that is similar to brainteaser. it presents a method for joint pun detection and localization utilizing a sequence labeling perspective. this highlights the complexity of language comprehension, especially in detecting subtle wordplay. another dataset, lateval is curated by @xcite that delves further into lateral thinking and commonsense reasoning, highlighting the challenges faced by language models in tasks requiring unconventional thinking and creativity. @xcite presents rome, a dataset designed to assess vision-language models' capacity to reason beyond intuitive understanding, highlighting the shortcomings of existing models in understanding events that defy common sense. in the field of reasoning task, a chain-of-thought @xcite implies a logical sequence of connected ideas, fostering coherence and depth in responses. on the other hand, a tree-of-thought suggests branching out into various related ideas, offering a more comprehensive exploration of a topic. while few-shot prompting is effective for some tasks by providing examples to guide the model, it may have limitations in capturing the complexity of nuanced conversations. the optimal choice may involve a hybrid approach, where a few-shot prompt sets the initial @xcite context, and the model subsequently follows a chain or tree of thought to generate more contextually rich and coherent responses useful for reasoning tasks. tan (2023) shows the performance of llm's on the reasoning of arithmetic word problems. it states that higher degrees of realization are associated with better overall accuracy on arithmetic problems. and chain-of-thought is really helpful in this aspect as it covers a variety of prompts to strengthen the reasoning. similarly, @xcite presents a new reasoning framework for large language models by addressing a gap in prior tree-based reasoning methods which overlooked inherent uncertainties in intermediate decision points made by models. overall, the key innovation is leveraging uncertainty estimation locally within the models during tree reasoning to enable more precise problem-solving and reasoning. ## the brainteaser dataset the brainteaser dataset @xcite , introduced with the task @xcite ) is a question-answering benchmark designed to evaluate models' ability for lateral thinking, i.e., to defy default commonsense associations and reason creatively. the dataset contains 1,100 multiplechoice questions divided into two sub-tasks -627 sentence-based puzzles relying on narrative context and common phrases and 492 puzzles focused on the literal form and letters of words. for a fair comparison with human performance, the dataset also provides a separate human evaluation set with 102 randomly sampled questions. each question in brainteaser has one correct answer and three distractor choices, including the option "none of the above". to prevent memorization of training data, the dataset also contains semantically and contextually reconstructed variants for every question while preserving the original reasoning process and answers. the key statistics of the dataset are shown in during the semeval-2024 task 9 development phase, a total of 240 prompts (120 for both sentence and word puzzles) are provided. during the test phase, a total of 216 prompts (120 for sentence and 96 for word puzzles) are provided. ## experiments in our experiments, we focus on several prompting strategies by employing three state-of-the-art models including proprietary models like gpt-4 (openai, 2023) and claude 2.1 @xcite (accessed via api key) and one open-source model -mixtral @xcite . ## results we analyze the performance of the three models -including gpt4 turbo, claude 2.1, and mixtral. question 1: how do you spell cow in thirteen letters? -choices 1: 1. see oh dereford. 2. see o double you. 3. cowcowcowcoww. finally, the results highlight the effectiveness of chain-of-thought prompting in boosting the performance of llms. this approach, especially when combined with more examples and the ensemble method, greatly improves models' abilities to process and generate more accurate responses. gpt4 turbo's top performance is likely due to its advanced design, which makes the most of these strategies. on the other hand, claude 2.1's results point to the importance of model-specific adjustments. ## conclusion and future work in this paper, we presented masontigers' approach to semeval-2024 task 9 on solving puzzles using llms. we explored various prompting strategies to guide the models, including zero-shot, few-shot, and chain-of-thought prompting. our key method involved iteratively breaking down reasoning into simplified logical steps to decompose the complex