HW - TSC at S em E val-2024 Task 9: Exploring Prompt Engineering Strategies for Brain Teaser Puzzles Through LLM s (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive performance on many Natural Language Processing (NLP) tasks. However, their ability to solve more creative, lateral thinking puzzles remains relatively unexplored. In this work, we develop methods to enhance the lateral thinking and puzzle-solving capabilities of LLMs. We curate a dataset of word-type and sentence-type brain teasers requiring creative problem-solving abilities beyond commonsense reasoning. We first evaluate the zero-shot performance of models like GPT-3.5 and GPT-4 on this dataset. To improve their puzzle-solving skills, we employ prompting techniques like providing reasoning clues and chaining multiple examples to demonstrate the desired thinking process. We also fine-tune the state-of-the-art Mixtral 7x8b LLM on ourdataset. Our methods enable the models to achieve strong results, securing 2nd and 3rd places in the brain teaser task. Our work highlights the potential of LLMs in acquiring complex reasoning abilities with the appropriate training. The efficacy of our approaches opens up new research avenues into advancing lateral thinking and creative problem-solving with AI systems.

## introduction in recent years, the advent of advanced language models has revolutionized the field of nlp, steering research towards challenges that necessitate intricate and implicit reasoning processes akin to human commonsense reasoning. such tasks often require vertical thinking, an analytical and methodical approach to problem-solving. this paradigm has enjoyed substantial popularity and success within the nlp community. however, lateral thinking puzzles, which demand creative reasoning and the ability to perceive indirect or non-obvious solutions, have not been equally explored. lateral thinking involves breaking away from conventional patterns to reveal novel insights, a feat that models based on rigid commonsense associations often struggle with. recognizing this disparity, we introduce llms in "brainteaser," a meticulously curated multiple-choice question answering (qa) task in order to evaluate llms' capabilities for lateral thinking. the dataset @xcite ) contains two subtasks: word and sentence brain teasers. word puzzles are word-type brain teasers where the answer deviates from the typical meaning of the word and instead focuses on the letter composition. sentence puzzles are sentence-type brain teasers centered around nonsensical or illogical snippets of text. the key characteristics of the dataset are described in in our approach, we employ the formidable gpt-4 language model to address brainteaser's questions under both zero-shot and few-shot conditions, thereby assessing its inherent reasoning capabilities without and with limited context. additionally, we leverage prompt engineering strategies and incorporate a chain of thought (cot) prompting technique to enhance gpt-4's comprehension of the task requirements. this innovative methodology not only facilitates clearer demonstration of the problem-solving process but also aligns the model's reasoning with human-like thought patterns. examples of the word and sentence puzzle samples are provided in tables 2 and 3 , respectively. semantic reconstruction (sr) rephrases the original question without altering the correct answer or distractor. context reconstruction (cr) maintains id question choice list wp-0 how do you spell cow in thirteen letters? see oh dereford see o double you. cowcowcowcoww. none of above. wp-0_sr in thirteen letters, how do you spell cow? see oh dereford see o double you. cowcowcowcoww. none of above. wp-0_cr how do you spell cob in seven letters? cobcobb cobbler see o bee. none of above. the reasoning path but changes both the question and answer to reflect a new situational context. the results of our experiments are both promising and insightful. our model achieved commendable rankings, securing 2nd and 3rd places in the task, which underscores the potential of llms in mastering complex, creative problem-solving tasks that extend beyond the scope of traditional commonsense reasoning. these outcomes not only validate the efficacy of our methods but also pave the way for further explorations into the untapped potential of lateral thinking in ai-driven language understanding. subsequently, pretrained language models (plm) like bert @xcite , bart @xcite , and gpt2 @xcite are proposed. these models represent milestones in the development of language models, as they are based on the classical transformer architecture @xcite and significantly increase the text generation capabilities of models. initially, most of these models have relatively small sizes. research has shown that even by solely increasing model size while keeping model architecture similar, abilities on difficult tasks can substantially improve @xcite . this phenomenon of emerging abilities with scale is referred to as emergent behavior @xcite . this has led to the development of llms which have profoundly impacted research and society. for example, the release of llm has created much interest due to its strong text generation abilities like abstract writing and logical reasoning. this has catalyzed further research into llms, with models like llama @xcite , llama 2 @xcite , mistral 7b @xcite , gpt 4 @xcite , and mixtral 8x7b @xcite demonstrating impressive performance on various tasks. ## conclusion in conclusion, we demonstrate our prompt design method to enhance creative problem-solving in llms, enabling strong performance on brain teaser puzzles. through prompting strategies and model fine-tuning, our methods attain 2nd and 3rd place rankings on this lateral thinking task. these results validate our techniques and highlight the potential for developing multifaceted reasoning skills in ai. our work provides promising pathways toward more human-like language understanding and flexible thinking in natural language models. in summary, we take steps toward training ai systems capable of creative problem-solving.