R iddle M asters at S em E val-2024 Task 9: Comparing Instruction Fine-tuning with Zero-Shot Approaches (2024)

Abstract: This paper describes our contribution to SemEval 2023 Task 8: Brainteaser. We compared multiple zero-shot approaches using GPT-4, the state of the art model with Mistral-7B, a much smaller open-source LLM. While GPT-4 remains a clear winner in all the zero-shot approaches, we show that finetuning Mistral-7B can achieve comparable, even though marginally lower results.

## introduction traditionally, the natural language processing (nlp) community has focused on tasks that require objective and complex reasoning. on the other hand, puzzles that defy traditional ways of reasoning have been less explored. brainteaser, a task at semeval 2024 @xcite , aims to fill this gap by investigating the abilities of large language models (llms) in more abstract and creative thinking. this competition consists of two sub-tasks: sentence puzzle (sp) and word puzzle (wp). according to the task description , sentence puzzles are brainteasers where the entire sentence snippet defies common sense. similarly, word puzzles are puzzles where the answer violates the default meaning of the word and focus on the letter composition of the question. in this work, we investigate a set of zero-shot approaches and compare them with a fine-tuned version of mistral-7b, an open source 7 billion llm @xcite . for the zero-shot approaches, we compare mistral-7b with gpt-4, the state of the art transformer model, across various prompts. we find that one-shot approaches using gpt-4 produces the best results across both our tasks. however, tweaking the prompts results in significant accuracy increases for mistral-7b. we also find that fine-tuned mistral-7b is the second best model in the sentence puzzle sub-task, indicating that instruction fine-tuning may be a way to get better results with smaller models. ## background the nlp task most related to this competition is question answering (qa), as all riddles consists of a question and multiple potential answers. question answering has been the focus of extensive prior work (soares and parreiras, 2020). typically, question answering systems consist of three main components: (1) question processing, (2) document processing, and (3) answer processing @xcite . the main goal of the question processing is to extract the keywords from the query so they can be parsed to the document processing component, as well as to identify the type of answer that we need to return @xcite . the goal for the document processing system is information retrieval (ir), based on the keywords collected from the previous component. typically, the ir system's job is to identify a subset of documents relevant to the keywords identified previously @xcite . as the desired output needs to be accurate and succinct, the ir system needs to further break down the relevant documents into smaller units such as passages, paragraphs, or sentences. the final stage of question answering is answer processing, which involves formulating the desired answer based on the knowledge previously retrieved, using a process called span labeling @xcite : given a passage, identifying the span of text that can be used to answer the question. these components are largely suitable for answering questions in a way that utilizes straightforward information processing and logical thinking, but struggle against question answering tasks that require creative responses or common-sense reasoning, such as solving puzzles and brainteasers @xcite . with the recent breakthroughs of llms such as bert @xcite or gpt-3 @xcite , we have seen exceptional capabili-ties of these language models in solving qa tasks, as well as their exhibition of complex reasoning abilities @xcite . however, when it comes to creative thinking and common sense reasoning, large language models achieve limited results @xcite @xcite . as such, @xcite generate a dataset of brainteasers to benchmark the performance pf stateof-the-art llms in answering puzzles and brainteasers, as a way to test their lateral thinking capabilities. our work aims to contribute to this domain, by evaluating the performance of multiple zero-shot approaches and our version of fine-tuned mistral-7b language model @xcite on the same dataset @xcite . dataset. the authors generated the initial brainteaser dataset by crawling the puzzles from the internet. however, recent work has shown that memorization is a common problem with llms @xcite . to evaluate lateral thinking instead of memorization, the authors used two reconstruction strategies (semantic and context) to create variants of each puzzle. semantic reconstruction rephrases the original question and was created via an opensource rephrasing tool @xcite ). in contrast, context reconstruction was achieved through a combination of gpt-4 prompts and human annotators @xcite . ## system overview in this section, we first describe the train and test datasets. later, we describe our proposed approaches, detailing the prompts used in the zeroshot approaches and the fine-tuning methodology. ## results the results for all the experiments are included in ## limitations and future work while we tried to explore various prompts for our zero-shot approaches, there is a possibility that further experiments might reveal more effec-tive techniques. future work could explore additional prompts as well as look into automating the prompt-search process. another area of potential improvements would be the exploration of additional datasets, especially those that include similar riddles based on lateral thinking. lastly, future work could explore additional fine-tuning techniques and discover if the accuracy can be further improved. ## conclusions we present a comparison of zero-shot approaches with instruction fine-tuning within semeval-2024 task 9. our experiments applied a variety of best practices for prompt engineering to explore the potential of zero-shot approaches in tasks that require lateral thinking and reasoning. we find that upon iterating over multiple prompts, zero-shots approaches using gpt-4 remain the solution that results in higher accuracy. however, instruction fine-tuned mistral-7b provides a second best alternative in the sentence puzzle sub-task.