BITS Pilani at S em E val-2024 Task 9: Prompt Engineering with GPT -4 for Solving Brainteasers (2024)

Abstract: Solving brainteasers is a task that requires complex reasoning prowess. The increase of research in natural language processing has leadto the development of massive large languagemodels with billions (or trillions) of parameters that are able to solve difficult questionsdue to their advanced reasoning capabilities.The SemEval BRAINTEASER shared tasks consists of sentence and word puzzles along withoptions containing the answer for the puzzle.Our team uses OpenAI’s GPT-4 model alongwith prompt engineering to solve these brainteasers.

## introduction there are two different types of thinking processes, vertical and lateral @xcite . vertical thinking refers to the form of linear thinking thinking we are conditioned to. it is based on rationality and logic. lateral thinking, or "out-of-the-box" thinking is a more creative way of thinking from different perspectives. this is contrary to first method. the recent advancements of natural language processing models, more specifically large language models have achieved great progress in reasoning capabilities and therefore vertical thinking tasks @xcite . this lateral, creative form of thinking has multiple use cases in the real world since rapid innovation and out of the box thinking are key functionalities of blooming institutions. innovations are crucial to solve global scale problems like climate change and are very important to big tech companies to keep their consumers happy and engaged. therefore an interesting part of language models are their abilities to show lateral thinking and defy default commonsense associations. for the semeval 2024 task 9: brainteaser: a novel task defying common sense @xcite on codalab @xcite , we aim to solve the brainteasers as a multiple-choice question answering (qa) tasks. our team proposes a system for this where we use prompt engineering with gpt-4 to solve these brainteasers. all of our code can be found on github at https://github.com/dipsivenkatesh/ semeval-2024-task-9 2 background ## experimental setup we load the brainteaser test datasets @xcite provided to us by the brainteaser shared task organizers using the huggingface datasets library @xcite . for the sentence puzzle we have 120 puzzles with 4 options corresponding to each puzzle and for the word puzzle we have 96 question s and for each question we have 4 options. the test set is unlabeled, it doesn't specify the correct option, and our systems must evaluate the correct option for each brainteaser. we generate the prompts for each question with the methods specified above and pass them to the gpt-4 turbo chat completions api for solving the brainteasers. ## results for evaluation, the organizers rank the system based on accuracy of the answers on the questionanswering task. the gpt-4 with prompt engineering system that we have provided achieves 9 th place on the leaderboard in the evaluation phase foot_1 . the performance of the system on all the different evaluation components can be found in