u T e BC - NLP at S em E val-2024 Task 9: Can LLM s be Lateral Thinkers? (2024)

Abstract: Inspired by human cognition, Jiang et al. 2023 create a benchmark for assessing LLMs’ lateral thinking—thinking outside the box. Building upon this benchmark, we investigate how different prompting methods enhance LLMs’ performance on this task to reveal their inherent power for outside-the-box thinking ability. Through participating in SemEval-2024, task 9, Sentence Puzzle sub-task, we explore prompt engineering methods: chain of thoughts (CoT) and direct prompting, enhancing with informative descriptions, and employing contextualizing prompts using a retrieval augmented generation (RAG) pipeline. Our experiments involve three LLMs including GPT-3.5, GPT-4, and Zephyr-7B-beta. We generate a dataset of thinking paths between riddles and options using GPT-4, validated by humans for quality. Findings indicate that compressed informative prompts enhance performance. Dynamic in-context learning enhances model performance significantly. Furthermore, fine-tuning Zephyr on our dataset enhances performance across other commonsense datasets, underscoring the value of innovative thinking.

## introduction human cognition provides the foundational framework for understanding large language model (llm) development, incorporating two critical thinking modes: vertical and lateral @xcite . vertical thinking, synonymous with logical reasoning, relies on structured analysis and established principles. conversely, lateral thinking, known for its creativity, challenges conventions and fosters innovative perspectives, enriching language processing capabilities. recognizing and leveraging the synergy between vertical and lateral thinking are essential in maximizing llms' cognitive potential. a man shaves everyday, yet keeps his beard long. ## experiments and results in this section, we report and discuss our results. ## conclusion in conclusion, our study emphasizes the crucial role of prompting methods in augmenting the lateral thinking capabilities of llms. through diverse cot-based strategies, prompt refinements, and rag techniques for in-context learning, we showcase the efficacy of well-structured prompts and thinking styles in elevating llm performance. additionally, fine-tuning models on a lateral thinking dataset proves advantageous, leading to improved performance on various commonsense tasks. this underscores the significance of integrating out-ofthe-box thinking in model training, opening promising avenues for future research to enhance llms' reasoning abilities. advances in neural information processing systems, volume 35, pages 24824-24837. curran associates, inc. shitao xiao, zheng liu, peitian zhang, and niklas muennighoff. 2023. c-pack: packaged resources to advance general chinese embedding. michihiro yasunaga, xinyun chen, yujia li, panupong pasupat, jure leskovec, percy liang, ed h. chi, and denny zhou. 2024. large language models as analogical reasoners. in the twelfth international conference on learning representations. rowan zellers, yonatan bisk, roy schwartz, and yejin choi. 2018. swag: a large-scale adversarial dataset for grounded commonsense inference. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 93-104, brussels, belgium. association for computational linguistics. zhuosheng zhang, aston zhang, mu li, and alex smola. 2022. automatic chain of thought prompting in large language models. arxiv preprint arxiv:2210.03493. anni zou, zhuosheng zhang, hai zhao, and xiangru tang. 2023. meta-cot: generalizable chain-ofthought prompting in mixed-task scenarios with large language models. arxiv preprint arxiv:2310.06692.