Deja Vu at S em E val 2024 Task 9: A Comparative Study of Advanced Language Models for Commonsense Reasoning (2024)

Abstract: This research systematically forms an impression of the capabilities of advanced language models in addressing the BRAINTEASER task introduced at SemEval 2024, which is specifically designed to explore the models’ proficiency in lateral commonsense reasoning. The task sets forth an array of Sentence and Word Puzzles, carefully crafted to challenge the models with scenarios requiring unconventional thought processes. Our methodology encompasses a holistic approach, incorporating pre-processing of data, fine-tuning of transformer-based language models, and strategic data augmentation to explore the depth and flexibility of each model’s understanding. The preliminary results of our analysis are encouraging, highlighting significant potential for advancements in the models’ ability to engage in lateral reasoning. Further insights gained from post-competition evaluations suggest scopes for notable enhancements in model performance, emphasizing the continuous evolution of the models in mastering complex reasoning tasks.

## introduction the reasoning ability of the human brain demonstrates a dualistic problem-solving approach, which integrates both vertical and lateral methodologies @xcite . vertical thinking places emphasis on a methodical and logical examination, executed in a sequential fashion, guided by established norms and regulations. on the contrary, lateral thinking @xcite promotes innovation and fosters the ability to perceive challenges from distinct, frequently unusual observation points, thereby encouraging individuals to go beyond conventional limitations. over the past few years, significant progress has been made in the booming domain of natural language processing (nlp), as novel technologies aim to mimic the the complicated ways in which human think @xcite . this undertaking overcomes conventional logical reasoning and dives into the domain of creative cognition, wherein machines are engineered to creatively navigate and interpret the complex nature of human language and thought processes. out of these efforts, the brainteaser task is particularly noteworthy for its pioneering aspect @xcite . the challenge, which is part of the semeval 2024, has been carefully constructed to evaluate a model's capacity for lateral thinking and its aptitude for questioning and redefining conventional commonplace assumptions. brainteaser @xcite takes a significant progress in the direction of bridging the divide that exists between the cognitive flexibility of humans and that of machines @xcite , exceeding the bounds of study. by selecting puzzles that require perception at both the sentence and word levels, this task highlights the significant technological advancement towards machines capable of creative thinking and logical conclusions that beat the apparent. the task encourages participants overcome the limitations of natural language processing (nlp) models by evaluating their capacity to interpret and decode language in a manner that emulates the creative reasoning of humans. motivated by the unique characteristics of the assignment and the potential it provides to advance the domain of nlp technology, our group wholeheartedly accepted the challenge presented by brainteaser. our approach was experimental, leveraging a variety of transformer models to explore their capacity for creative and lateral thinking @xcite . these models, known for their effectiveness @xcite in understanding and generating human language, were put to the test to see if they could indeed mimic the thought processes traditionally attributed to humans. we ranked at 20th in each of the subtasks and in average the rank was 31. the experience was rich with learning opportunities, offering us valuable insights into the capabilities and limitations of current technologies when faced with tasks that require a departure from conventional reasoning. ## task and data description the task @xcite ) is making a system which is evaluated on understanding sentences and words in ways that defy usual expectations. it has two main challenges: the task employs specific tests to make sure the system analyzes information deeply instead of merely remembering answers. these tests involve altering the phrasing or setting of questions without changing the basic problem, known as semantic and context reconstruction. the systems are evaluated based on two primary factors: their ability to address single questions, referred to as instance-based performance, and their consistency in answering groups of related questions, known as group-based performance. the goal of this task is to advance the system's abilities in problem-solving and creative thinking the dataset @xcite provided for the task includes two distinct types of files, one for sentence puzzles and another for word puzzles. each file is rich with essential elements such as the posed question, the correct answer, three alternative options (distractors), labels, a list of choices, and the sequence in which these choices are presented. during the training phase, the dataset comprises 507 sentence puzzles and 396 word puzzles, demonstrating a comprehensive range of scenarios for model training. for the testing phase, the dataset narrows down to 120 sentence puzzles and 96 word puzzles, aimed at rigorously evaluating the models' understanding and reasoning capabilities in both puzzle types 3 system description ## result our research explored how different transformer models performed when tasked with solving two types of puzzles: sentence and word puzzles. initially, we observed encouraging results from all models on the sentence puzzles, which were more abundant in our dataset. this success highlighted the models' proficiency in contexts where narrative clues guide the solution process. however, when we combined sentence and word puzzles into a single dataset, we noticed a significant drop in accuracy across all models (as shown in the better performance on sentence puzzles can be attributed to the models' inherent strengths. both bert and xlnet are designed to excel in understanding and processing complex narrative contexts, benefiting from extensive pre-training across diverse text types. this foundation enables them to navigate the intricate language of sentence puzzles more adeptly. on the other hand, word puzzles often rely on subtle wordplay and linguistic nuances less represented in the models' training data, posing a greater challenge.. the disparity in performance between puzzle types underscores a crucial insight: transformer models, despite their advanced capabilities, exhibit varying degrees of adaptability to different linguistic tasks. the initial high accuracy rates with sentence puzzles showcase their potential, while the subsequent drop in performance upon introducing word puzzles highlights areas for improvement, particularly in enhancing the models' versatility and ability to generalize across diverse language tasks. our findings indicate a clear path forward-further refining these models to better capture and interpret the breadth of human language, extending their applicability beyond structured narrative contexts to include the nuanced, often unpredictable areas of word puzzles. ## limitation and error analysis our study had some challenges and places where error analysis showed that things could be done better. one big problem with the models is that they are skewed because they were trained on datasets that might not fully show how people use words in different situations. this might make it harder for the models to generalise to new types of data, especially when they move from sentence puzzles to word puzzles (see tables 2 and 3 ). the models' different results on sentence puzzles versus word puzzles also points out a need for error analysis and better training strategies or model architectures that can handle the complex nature of both types of puzzles equally. the fact that accuracy went down when datasets were combined suggests overfitting, an important problem that needs more research to make models more reliable. these new ideas help us plan future research, like looking into bigger and more varied training datasets and making models that are specifically made to deal with the problems that come up in different language tasks. ## conclusion our research into using transformer models like bert, xlnet, and bart for question-answering tasks shows both their strengths and weaknesses when trying to understand words like humans do. the results point to a potential way to improve system's ability to interpret, but they also show that more progress needs to be made. in the future, action should be put into improving these models so that they understand context better, are more clear, and can be used in more areas.