Abdelhak at S em E val-2024 Task 9: Decoding Brainteasers, The Efficacy of Dedicated Models Versus C hat GPT (2024)

Abstract: This study introduces a dedicated model aimed at solving the BRAINTEASER task 9 , a novel challenge designed to assess models’ lateral thinking capabilities through sentence and word puzzles. Our model demonstrates remarkable efficacy, securing Rank 1 in sentence puzzle solving during the test phase with an overall score of 0.98. Additionally, we explore the comparative performance of ChatGPT, specifically analyzing how variations in temperature settings affect its ability to engage in lateral thinking and problem-solving. Our findings indicate a notable performance disparity between the dedicated model and ChatGPT, underscoring the potential of specialized approaches in enhancing creative reasoning in AI.

## introduction the brainteaser task @xcite aims to challenge the lateral thinking abilities of models, setting it apart from traditional tasks focused on vertical logical reasoning. it introduces lateral thinking puzzles in the form of multiple-choice questions to test the models' ability to think creatively and challenge common sense associations. the goal is to identify the gap between human and model performances in creative thinking, highlighting the need for progress in ai's creative reasoning abilities. nlp (natural language processing) transformer models have revolutionized text understanding and generation with their architecture capable of processing word sequences more efficiently. for multiple-choice questions, these models utilize their ability to understand context and language nuances to select the most appropriate answer from several options. thanks to deep learning and attention mechanisms, they excel in various nlp tasks, significantly improving the accuracy and relevance of responses generated in complex contexts. the integration of nlp transformer mod-els into the brainteaser task aims to explore their ability to solve lateral thinking puzzles in the form of multiple-choice questions. this approach highlights the challenges posed by deep language understanding and the creativity required to surpass traditional logical reasoning. it emphasizes the importance of advancing in the development of models capable of navigating beyond common sense associations, encouraging innovation in the interpretation and generation of complex and nuanced responses. in our study, we will explore the ability of language models to handle this task, with the following main contributions of this paper : the brainteaser shared task 9 is a question answering (qa) task based on evaluating the capacity of language models to engage in lateral thinking and to solve puzzles that require unconventional thinking. brainteaser comprises two distinct subtasks: sentence puzzle and word puzzle, both of which involve defying commonsense "defaults" but through different methodologies. both tasks include an adversarial subset, created by manually modifying the original brain teasers without changing their latent reasoning path. they construct adversarial versions of the original data in two ways: the brainteaser paper reveals a significant gap between human performances and ai models, and underscores the need to enhance lateral reasoning in language models. ## related work the task of commonsense reasoning has long been a challenge for deep learning and has been the subject of research for several years, accompanied by various benchmarks such as @xcite , which introduces a new large-scale nli benchmark dataset created through an adversarial process involving humans and models. this improves nli models' performance on popular benchmarks and reveals their weaknesses, offering a dynamic framework for continuous improvement in natural language understanding. a study demonstrated a simple and unsupervised method for commonsense reasoning using language models trained on vast text corpora, significantly outperforming state-of-theart methods on pronoun disambiguation problems and the winograd schema challenge without the need for annotated knowledge bases or manually engineered features @xcite . transformer models like bert @xcite , gpt @xcite , and their variants have revolutionized natural language understanding, including question answering @xcite . their architecture captures semantic and contextual nuances @xcite , proving exceptionally effective in comprehending and responding to complex inquiries. by training on extensive text corpora, they develop a deep understanding, enabling them to identify the most plausible answers among multiple choices @xcite . large pretrained language models (plms) can achieve near-human performance on commonsense reasoning tasks by generating contrastive explanations that highlight the key attributes needed to justify correct answers. this approach not only improves performance on commonsense reasoning benchmarks but also produces explanations judged by humans as more relevant and understandable @xcite recent studies reveal that chatgpt has notable capabilities to effectively solve a variety of problems in several languages, including the task of answering questions. moreover, its performance improves with each new version. chatgpt excels in certain areas but also has its limitations in terms of consistency and complex reasoning tasks. @xcite . ## conclusion our research underscores the significance of dedicated models in advancing ai's capability to solve complex lateral thinking tasks, as exemplified by our model's top-ranking performance in the brainteaser sentence puzzles. the comparative analysis with chatgpt highlights the limitations of general-purpose models in specific creative reasoning challenges, despite their overall versatility. the study also reveals the nuanced role of temperature settings in modulating chatgpt's performance, offering insights into optimizing ai models for enhanced creativity and lateral thinking. future work should focus on bridging the gap in word puzzle performance and further refining the balance between creativity and logical reasoning in ai systems.