OUNLP at S em E val-2024 Task 9: Retrieval-Augmented Generation for Solving Brain Teasers with LLM s (2024)

Abstract: The advancement of natural language processing has given rise to a variety of large language models (LLMs) with capabilities extending into the realm of complex problem-solving, including brainteasers that challenge not only linguistic fluency but also logical reasoning. This paper documents our submission to the SemEval 2024 Brainteaser task, in which we investigate the performance of state-of-the-art LLMs, such as GPT-3.5, GPT-4, and the Gemini model, on a diverse set of brainteasers using prompt engineering as a tool to enhance the models’ problem-solving abilities. We experimented with a series of structured prompts ranging from basic to those integrating task descriptions and explanations. Through a comparative analysis, we sought to determine which combinations of model and prompt yielded the highest accuracy in solving these puzzles. Our findings provide a snapshot of the current landscape of AI problem-solving and highlight the nuanced nature of LLM performance, influenced by both the complexity of the tasks and the sophistication of the prompts employed.

## introduction the pursuit of creating artificial intelligence models with advanced reasoning and problem-solving capabilities has led researchers down the path of deploying brainteasers as a benchmark for ai systems' linguistic and reasoning prowess. these brainteasers are more than trivial or recreational challenges; they are testaments to the complexity of human cognition, embedding layers of semantics, pragmatics, and world knowledge that remain elusive to ai systems. the gulf between the operational logic of current ai models and the intricate understanding displayed by the human mind is significant, particularly in domains necessitating advanced reasoning and a robust common sense foundation. this disparity is not only observed but keenly felt in the context of ai systems' interaction with human language and thought @xcite . the limitations of pattern recognition as the mainstay of ai systems' learning mechanisms have been critically examined, sparking a discourse that emphasizes the imperative for ai systems to transcend these confines. rigorous benchmarks that challenge ai systems to demonstrate inferential reasoning are essential to catalyze this evolution @xcite . brainteasers emerge as one medium through which ai systems' competencies can be evaluated. they are not simply puzzles to be solved but are reflective of the complex, often ambiguous nature of human communication and problem-solving. the brainteaser task introduced at se-meval 2024 @xcite is part of this evolution of ai system assessment, standing at the center of linguistic analysis and computational intelligence. it is designed to evaluate what machines can understand and how they can apply this understanding in a manner similar to human thought processes. language models, such as gpt-3.5 foot_1 and gpt-4 (openai, 2023), are increasingly being subjected to these tests to gauge their mastery over language and logic, as demonstrated in recent comparative analyses @xcite . the brainteaser task's format, which intertwines linguistic cues with logical conundrums, requires systems to not only comprehend the text at a superficial level but to delve into the implied, the inferred, and the intuitive aspects that are second nature to human beings. by benchmarking language models against brainteasers within the framework of the brain-teaser task, we are able to learn more about the current capabilities of popular llms. this work can help to provide a direction for future research by pinpointing where current models fall short and where the next wave of innovation is urgently needed. our approach involved testing different llm models. we web-scraped example riddles and used them as an example for the model. additionally, we tested with the closest riddle and the most different riddle and found that gpt-4.0 oneshot with a similar riddle worked best for the sentence puzzle and gpt-4.0 oneshot with a different riddle worked best for the word puzzle. the word puzzle turned out to be a significantly harder task than the sentence puzzle. ## related work the exploration of reasoning abilities in large language models (llms) has been the focus of several studies in recent years. notably, work by openai provides foundational insights into the capabilities of gpt-3, especially highlighting its potential in solving reasoning tasks through few-shot learning @xcite . this work is particularly relevant as it demonstrates how providing a few examples can significantly improve an llm's ability to solve reasoning problems, akin to the one-shot and few-shot techniques examined in our study. furthering the discussion on reasoning, work has been done that discusses the 'chain-of-thought' (cot) prompting method, where models are guided to articulate intermediate steps when solving complex tasks @xcite . this process is similar to the explanation method in solving the brainteasers, which encourages models to elaborate on their reasoning, leading to improved performance. the brainteasers in the training data provided often require making analogies and similarities in reasoning. work has been done that offers an analysis of how word embeddings capture semantic relationships, which can be fundamental in retrieving similar examples to aid reasoning @xcite . this is directly linked to our one-shot similar and few-shot in-context learning approaches, where the ability of an llm to use analogous examples influences its problem-solving effectiveness. moreover, the strategies for solving brainteasers with ai systems have been enriched by incorporating external knowledge bases. an investigation has been conducted into the inherent knowledge within language models and their ability to function as knowledge bases @xcite . the integration of external knowledge is particularly pertinent to tasks requiring common sense and realworld information, underscoring the importance of knowledge retrieval in the context of a brainteaser. lastly, a pivotal study has been done that introduces a dataset designed to probe ai systems' common sense reasoning capabilities @xcite . this study aligns with our aim in solving brainteasers to evaluate the capacity of llms to handle questions that necessitate an understanding of the world as humans perceive it. ## methodology our experimental design relies on prompt engineering to explore the effectiveness of language models in solving brainteasers. in this study, we experimented with different prompt structures to determine their impact on the model's performance. the primary prompt format tested was structured as follows: "please pick the best choice for the brain teaser. each brain teaser has only one possible solution, including the choice 'none of the above.' the answer should only provide the choice text." this directive was chosen to explicitly instruct the model to select a single, most appropriate answer from a set of given options. to ensure a controlled variable, we explicitly presented the model with the choices, observing how it navigates the selection process when options are directly provided. an interesting observation was made regarding the specification of the type of brainteaser. initially, it was hypothesized that indicating whether the puzzle was a 'word puzzle' or a 'sentence puzzle' would aid the models in narrowing down their reasoning scope, thereby improving accuracy. however, the results indicated that such specifications did not significantly affect the models' performance. this finding suggests that the models possess a level of task generalization, wherein they apply similar reasoning processes to both types of puzzles without the need for explicit differentiation. furthermore, we explored the effect of including choices within the prompt. by contrasting scenarios with and without provided options, we aimed to assess whether the presence of choices would guide the model to a correct answer more efficiently. prompts structured to request the model to pick from provided choices explicitly did not significantly alter the success rate compared to when no choices were given. this aspect of the study aimed to discern the degree to which the models rely on contextual clues versus intrinsic problem-solving capabilities. for example, in a test with a bad prompt on gpt-3.5 without choice, a 27.60% accuracy was achieved, while with the same prompt with choices, a 28.80% accuracy was achieved. due to the slight increase in accuracy, we decided to include choices for the rest of the prompts used. we also implemented explanation and chain of thought (cot) reasoning to guide the language model toward a more structured and reasoned approach when tackling brain teasers. explanation prompts encouraged the model to articulate the rationale behind its chosen answers. similarly, cot prompts aimed to simulate a step-by-step reasoning process, mirroring how humans might approach problem-solving. to further enhance the accuracy of our language model in solving brain teasers, we adopted a oneshot in-context learning approach, leveraging a large dataset of riddles as context for the model. we extracted a comprehensive collection of 3,899 riddles by downloading texts from the riddles.com website, including both the questions and their corresponding answers. this dataset was a reference for the model to draw upon when presented with new puzzles. we employed two distinct strategies for selecting a relevant riddle from this dataset to use as an example in our one-shot method. the first strategy aimed to identify the riddle most similar to the brain teaser in question, believing that a similar context might prime the model more effectively for the task at hand. conversely, the second strategy sought out the riddle most dissimilar to the brain teaser, hypothesizing that a contrasting example could stimulate a broader range of the model's reasoning capabilities. to facilitate the rapid identification of the most similar or dissimilar riddle, we encoded each riddle question into a vector representation using the all-minilm-l6-v2 model (reimers and gurevych, 2019) in the sentencetransformers 3 python package. this allowed us to compute the cosine similarity between the vector representation of the new brain teaser and those of the riddles in 3 https://sbert.net our dataset. using this similarity score, we could efficiently identify the riddle that was either closest or farthest in semantic space from the brain teaser presented to the model. the highest cosine similarity is used as the similar riddle, while the lowest cosine similarity is used as the dissimilar riddle. this approach significantly improved the efficiency of selecting an appropriate example riddle, enabling a more streamlined integration of the oneshot learning method into our experimental setup. the rationale behind using cosine similarity was to leverage the high-dimensional space in which language representations reside, making it possible to quantify the semantic proximity between different textual inputs effectively. a depiction of the different elements of different prompts is depicted in figure 1 . in an effort to enhance the accuracy of solutions provided by the language model for word puzzles, we also experimented with an innovative approach modeled after collective human problemsolving dynamics. this method involved simulating a "council" of three hypothetical individuals engaged in a discussion about a puzzle, with the aim of reaching a consensus on the answer. 4 the intent was to emulate the collaborative approach often used in human group problem-solving, where different perspectives and thought processes can lead to more accurate solutions. the implementation of this method required the model to generate three distinct responses, each purportedly from a different "council member," who would then "discuss" their reasoning and perspectives on the puzzle. following this simulated deliberation, the model was prompted to synthesize the viewpoints into a single, collective answer (see example in appendix a). despite the creative nature of this technique, the results were not as promising as anticipated. the accuracy of the word puzzles did not show significant improvement using the council-based discussion method. this outcome suggests that while the approach mirrors human group interactions, it may not translate effectively within the constraints of a single ai model's processing capabilities. ## results we ran and tested the llms on the training set @xcite with each combination of model and prompt. this allows us to get a comprehensive view of the performance of the llms across different prompts. it is important to increase the general accuracy across all llms by adding more information about the question to the prompt, along with examples. the models we evaluated included gpt-3.5, gpt-4, gemini pro @xcite , and a suite of memory-efficient language models from the languagemodels repository. 5 the models used from this package included neural-chat-7b-v3-1, flan-alpaca-xl, flan-alpaca-gpt4-xl, flan-t5-xl, fastchat-t5-3b-v1.0, lamini-flan-t5-783m, flan-t5-large, lamini-flan-t5-248m, flan-alpaca-base, flan-t5-base, dialogstudio-t5-base-v1.0, lamini-flan-t5-77m, flan-t5-small, phi-1_5, lamini-gpt-774m, and lamini-gpt-124m. this set explores sixteen models, yet they never outperformed the other llms (gpt and gemini series). therefore, we only report the results from the best of these models for each prompt and task in the final result tables, it was found that the one-shot method consistently had the top 2 accuracy in the prompts studied, proving the efficiency of one-shot methods for llms in general. gemini's accuracy when only using the basic prompt was very high for the sentence puzzle task, which shows gemini's versatility and adaptability to different questions with high accuracy without needing examples to perform well. the gpt-4 system with the basic prompt with a chain-of-thought method also proved to be highly accurate. the chain of thought approach has been shown to improve accuracies for llms. despite these efforts, we observed that this strategy did not lead to a measurable increase in accuracy. this outcome suggests that while such prompts can often lead to more interpretable answers, they do not necessarily enhance the model's ability to deduce the correct solution in the context of brain teasers. further research may explore whether the complexity of the puzzles or the inherent limitations of the models' understanding contributed to this result. despite the overall increase in accuracy observed with more informative prompts, gpt-4 did not always outperform gpt-3.5 with prompts like the task description + explain. this highlights that while advancements in model architecture contribute to enhanced performance, they do not guarantee superior outcomes in every scenario, particularly in specialized tasks like puzzle-solving. since gpt-4 is trained on a larger range of data sources to improve general performance across a broad range of tasks, the generalized training approach may lead gpt-4 not performing as well in this specific task. furthermore, we found that employing chainof-thought and explanation methodologies did not significantly improve performance in this context. this deviation from expected outcomes may indicate that for certain types of puzzles, these approaches do not align with the models' strengths or the nature of the problem-solving process required. the performance of open-source packages like languagemodels was notably lower compared to their commercial counterparts. this gap underscores the developmental distance that open-source models need to traverse to reach the sophistication level of models like gpt-4 or gemini, suggesting that access to extensive datasets, computing resources, and proprietary algorithms plays a significant role in model performance. typically the models from this set that worked best were the lamini-flan-t5 class, which was always the case for the one-shot setting. the main exceptions to this were in the zero-shot scenario, specifically with the basic prompt (neuralchat worked best for sentence puzzle and dialogstudio worked best for word puzzle) and the task description prompt (phi-1.5 for sp and neuralchat for wp). these cases provide positive examples of situations in which lightweight, open models are more competitive with proprietary, closed models. additionally, the increased difficulty of word puzzles presents a notable challenge, potentially due to their reliance on nuanced understanding, cultural context, and semantic associations that can be challenging even for human solvers. this complexity is reflected in the lower accuracy rates across all models for word puzzles when compared to sentence puzzles, implying that word puzzles may represent a closer analog to human-level problemsolving and, as such, provide a more stringent test of ai reasoning and language capabilities. the three submission prompts submitted are gpt-4 one-shot different for sentence puzzle and gpt-4 one-shot similar for word puzzles, gpt-4 one-shot similar for sentence puzzle and gpt-4 one-shot different for word puzzles, and gemini basic for sentence puzzle and gpt-4 basic + cot for word puzzle. the first submission received an accuracy score of 0.925 for the sentence puzzle and 0.9375 for the word puzzle, the second a score of 0.95 for the sentence puzzle and 0.78125 for the word puzzle, and the third a score of 0.625 for the sentence puzzle and 0.46875 for the word puzzle. not that these accuracies are from the "original" riddles. the second submission was the highest and ranked us 14th in terms of average score, 11th on the word puzzle task, and 10th on the sentence puzzle task under the name vspuzzler. our submitted system performed exceptionally well on the "original" version of the brainteasers (ranking 3rd overall for sentence puzzle and 7th for word puzzle within this subcategory) but underperformed on the "context reconstruction" variations of the brainteasers in which the original reasoning path was used within a new situational context. ## conclusion this study's evaluation of llms across a range of prompt types provides insights into the strengths and limitations of current ai systems in solving brain teasers. our findings revealed that in-context learning methods are highly effective, particularly for the gemini model and gpt-4, when solving sentence puzzles. however, gpt-4 did not consistently outperform gpt-3.5 across all prompt types, which suggests that the latest models do not always guarantee an improvement in task-specific performance. the chain-of-thought and explanation strategies, while enhancing interpretability, did not necessarily translate into higher accuracy, indicating the need for further research into how these models process complex language tasks. the performance gap between proprietary and open-source models highlights the significant role of resources and proprietary technology in developing llms. the increased difficulty of word puzzles suggests that tasks requiring a nuanced understanding and cultural context remain challenging for ai systems, closely mirroring the complexity of human cognition. this study underscores the importance of tailored prompting strategies to leverage the capabilities of llms and the potential for future advancements in ai-based problem-solving. {"response": "b", "explanation": "each daughter has one brother, which means there is only one brother shared among all six daughters. if mr. and mrs. mustard have six daughters and one son, together with the two of them, that makes a total of nine family members. therefore, the correct answer is choice b." } a council method was also used as one of the prompts. below is an example of it in use: please pick the best choice for the sentence puzzle brain teaser (sentence-type brain teaser where the puzzle-defying commonsense is centered on sentence snippets). each brain teaser has only one possible solution including the choice none of above, answer should only provide the choice. imagine three different experts are answering this question. all experts will write down 1 step of their thinking, then share it with the group. then all experts will go on to the next step, etc. if any expert realises they're wrong at any point then they leave. question:mr. and mrs. mustard have six daughters and each daughter has one brother. but there are only 9 people in the family, how is that possible? choice: