D a V inci at S em E val-2024 Task 9: Few-shot prompting GPT -3.5 for Unconventional Reasoning (2024)

Abstract: While significant work has been done in the field of NLP on vertical thinking, which involves primarily logical thinking, little work has been done towards lateral thinking, which involves looking at problems from an unconventional perspective defying existing conceptions and notions. Towards this direction, SemEval 2024 introduces the task of BRAINTEASER, which involves two types of questions – Sentence Puzzle and Word Puzzle that defy conventional common-sense reasoning and constraints. In this paper, we tackle both the questions using few-shot prompting on GPT-3.5 and gain insights regarding the difference in the nature of the two types of questions. Our prompting strategy placed us 26th on the leaderboard for the Sentence Puzzle and 15th on the Word Puzzle task.

## introduction the human brain consists of two hemispheres -left and right. both of them are responsible for different kinds of thinking strategies. the left hemisphere is involved in vertical thinking, and the right hemisphere is involved in lateral thinking @xcite . @xcite thinking is a more sequential analytical process. in contrast, in lateral (outside the box, divergent, creative) thinking, we look at the problem from a new point of view, ignoring the expected associations with items. in the field of nlp, much research has been done around vertical thinking and significant progress has been made. the recent work around large language models (llms) @xcite has achieved great performance in solving complex reasoning tasks @xcite @xcite . this performance is consistent in both cases when no task examples have been provided to the model * equal contribution during inference (zero-shot) @xcite and when the model is introduced with the task during inference time (few-shot) @xcite . however, lateral thinking has been overlooked when training nlp models like llms. when creating datasets for various models, texts that involve lateral thinking are mostly considered noise and filtered out from the data because researchers want their models to perform better at traditional reasoning tasks and not get confused by lateral thinking. the task brainteaser @xcite tries to bridge this gap that exists between vertical and lateral thinking for llms and other nlp models. they formulated a set of multi-choice question answers containing puzzles that can be solved only using lateral thinking. the benchmark dataset contains two types of lateral thinking puzzles -sentence puzzles and word puzzles. this has been constructed by designing a data collection procedure that crawled relevant puzzles from many websites that were publicly available performing semi-automatic filtering of irrelevant questions. ## system overview our architecture uses gpt-3.5 (brown et al., 2020) (specifically gpt-3.5-turbo) with few-shot prompting to answer the question. ## results and analysis the results are detailed in tab.1. for comparison, we also list the zero-shot prompting results reported in jiang et al., 2023. as we can see, the two-shot performance on word puzzle improved over the zero-shot setting for all the categories, while the same worsened in case of the sentence puzzle. this is because of the very nature of the two problems. sentence puzzle involves deeper nonconventional semantic understanding of the question and the choices, which despite conveying reasoning behind the answers in the few-shot examples, cannot be generalized as easily with just 2 examples. on the other hand, the only tricky component of the word puzzle is that the structural aspect of certain words needs to be taken instead of the actual surface meaning of the said words. this can be much more easily generalized through just as few as two examples in the prompt. further, adding the examples in the sentence puzzle that don't generalize very well for other questions in the testing set might have acted as noise for the model, which led to poorer performance. we also note that using five-shot prompt instead of two-shot prompt hugely increases the performance. this is to be expected, as providing more examples would help the model generalize even better towards solving the task. this is specially true for word puzzle questions, where adding more examples allows the model to generalize the task much better. however, in sentence puzzle we still notice a drop in the overall performance as compared to the zero-shot model. this is because of a drop in the performance of the context reconstruction questions, and a marginal increase in comparison to zero-shot in other types of questions. however, group based accuracy increases in five-shot, which might indicate that with five examples, the model is able to handle the variations in reconstructions better, albeit with performance of contextual reconstruction taking a dip. these observations are in line with the drop observed in two-shot prompt in comparison to the zero-shot prompt, highlighting the difficult nature of the task of sentence puzzle questions and the inability of the model to generalize using few sentence puzzle examples. however, we do note that the performance on sentence puzzle also does improve with additional examples between two-shot and five-shot prompting. ## conclusion in conclusion, we explored the effectiveness of few-shot prompting for llms for complex and unconventional tasks. further, it demonstrates that few-shot prompting is helpful only in scenarios where the examples convey enough information that can be better generalized, as the results worsened in the sentence puzzle while improved in the word puzzle. in future, better prompting strategies like chain of thought prompting @xcite can be utilized to improve the performance. additionally, finetuning the pre-trained llms might also help in the task further. also, increasing the number of training examples might help in further improving the model performance, as observed in the gains of performance in the five-shot prompt in comparison to the two-shot prompt.