D e BERT a at S em E val-2024 Task 9: Using D e BERT a for Defying Common Sense (2024)

Abstract: The widespread success of language models has spurred the natural language processing (NLP) community to tackle tasks demanding implicit and intricate reasoning, drawing upon human-like common-sense mechanisms. While endeavors in vertical thinking tasks have garnered considerable attention, there has been a relative dearth of exploration in lateral thinking puzzles. To address this gap, we introduce BRAINTEASER: a multiple-choice Question Answering task meticulously crafted to evaluate the model’s capacity for lateral thinking and its ability to challenge default common-sense associations. At the SemEval-2024 Task 9, for the first subtask (i.e., Sentence Puzzle) the organizers asked the participants to develop models able to reply to multi-answer brain-teasing questions. For this purpose, we propose the application of a DeBERTa model in a zero-shot configuration. Our proposed approach is able to reach an overall score of 0.250. Suggesting a significant room for improvements in future works.

## introduction human reasoning encompasses two fundamental types of cognitive processing: vertical and lateral thinking. vertical thinking is marked by its sequential and analytical approach, drawing upon principles of rationality, logic, and rule-following, often attributed to the left-brain hemisphere @xcite . this mode of thinking is essential for creating logical pathways, such as understanding physical scenarios or solving riddles based on direct associations. in contrast, lateral thinking, often referred to as "thinking outside the box," is a creative cognitive process. it entails exploring problems from unconventional perspectives and challenging ingrained assumptions. lateral thinking, associated with the right-brain hemisphere, is crucial for resolving unconventional puzzles by defying common-sense associations and considering alternative perspectives. while natural language processing (nlp) models have made significant strides in vertical thinking tasks, particularly in the field of large language models (llms). their performance in lateral thinking remains largely unexplored. llms have demonstrated remarkable performance across various reasoning tasks, even when provided with minimal or no training examples. these models excel in tasks requiring vertical thinking abilities, such as reasoning over physical interactions and social implications @xcite , showcasing strong common-sense association and inference capabilities. however, prior research has largely overlooked the evaluation of llms' lateral thinking abilities, as creative thinking problems are often filtered out during data preprocessing, and only those aligned with common-sense associations are retained. to address this gap, a novel benchmark called brainteaser @xcite to evaluate the lateral thinking abilities of state-of-the-art llms is proposed at semeval-2024 task 9 @xcite . the organizers frame lateral thinking puzzles as multiple-choice question answering (qa) tasks, a format that is intuitive for humans to engage with and straightforward to assess automatically. the brainteaser benchmark comprises two tasks: sentence puzzles and word puzzles, designed to assess lateral thinking at different levels of granularity. to develop the dataset, organizers employ a data collection pipeline that retrieves relevant puzzles from publicly available websites, filters out irrelevant question categories, and ensures high data quality. additionally, to mitigate concerns regarding llm memorization and consistency, the organizers enhance brainteaser with two reconstruction strategies: semantic reconstruction and context reconstruction. these strate-gies aim to promote deeper understanding and reasoning rather than mere memorization of patterns. to meet these objectives, there is a growing demand for automated tools capable of understanding data using recent advancements in nlp models. the emergence of machine and deep learning architectures has sparked increased interest in nlp, prompting substantial efforts to develop techniques for automated identification and understanding of textual content available on the internet. in the literature, various strategies have been proposed so far. over the past fifteen years, some of the most successful approaches have included support vector machines (svm) @xcite , convolutional neural networks (cnn) @xcite , graph neural networks (gnn) @xcite , ensemble models @xcite , and transformers @xcite . the sections of this paper are structured as follows: section 2 offers background information on task 9, held at semeval-2024. in section 3, we outline the approach introduced in this study. section 4 delves into the specifics of the experimental setup employed to reproduce our findings. the outcomes of the official task and relevant discussions are presented in section 5. finally, section 6 concludes our study and suggests avenues for future research. we make all the code publicly available and reusable on github foot_0 . ## background the increasing adoption of transformer-based architectures in academic research has also been bolstered by various methodologies showcased at se-meval 2024. these methodologies tackle diverse tasks and yield noteworthy findings. for instance, at the task 2 @xcite , where to address the challenge of identifying the inference relation between a plain language statement and clinical trial reports is used t5 @xcite ; task 4 @xcite where is employed a mistral 7b model to detect persuasion techniques in memes @xcite ; and task 8 @xcite , that utilizes a distilbert model to identify machine-generated text @xcite . the task 9 hosted at semeval-2024, is based on the human reasoning processes comprising the two already-mentioned types of thinking: vertical and lateral. specifically, the brainteaser qa task consists of two subtasks: the sentence and the word puzzle ones, that require awareness of commonsense "defaults" and overwriting them through unconventional thinking that distinguishes these defaults from hard constraints. in detail, for the sentence puzzle one, the puzzle defying common-sense is centred on sentence snippets. on the other hand, for the word puzzle subtask, the response diverges from the conventional interpretation of the word and concentrates on the letter arrangement within the target question. both subtasks incorporate an adversarial subset, crafted by manually altering the original brainteasers while preserving their underlying reasoning paths. an example from the official codalab page foot_1 takes as example the following original sentence: "a man shaves everyday, yet keeps his beard long." the four possible explanations are: ## system overview despite evidence suggesting that transformers may not always yield optimal results for every text classification task @xcite , various strategies, such as domain-specific fine-tuning @xcite and data augmentation @xcite @xcite , have proven to be advantageous depending on the task's objectives. however, to address the task 9 hosted at semeval-2024 we made use of a zero-shot learning approach @xcite , making use of the deberta transformer @xcite . our approach is zero-shot @xcite and make use of the above-mentioned de-berta model. specifically, we employed the multilingual version 3 fine-tuned on the squad2.0 dataset 3 . deberta improves upon the bert and roberta models by introducing disentangled attention mechanisms and an enhanced mask decoder. leveraging these enhancements, deberta outperforms roberta across most natural language understanding (nlu) tasks when trained on a dataset of 80gb in size. in deberta v3, efficiency is further enhanced by integrating electra-style pre-training with gradient disentangled embedding sharing. comparative analysis against de-berta reveals notable enhancements in model performance across downstream tasks in the v3 version. further elaboration on the novel techniques employed in this new model can be found in the original paper. the version of deberta utilized in our experiments is mdeberta, a multilingual variant of deberta. it maintains an identical architecture while being trained on cc100 multilingual data. the mdeberta v3 base model comprises 12 layers with a hidden size of 768. it encompasses 86 million backbone parameters and a vocabulary of 250,000 tokens, resulting in 190 million parameters in the embedding layer. this model underwent training using 2.5 trillion tokens of cc100 data, akin to the xlm-r model. for the experimental settings, we started evaluating several prompt engineering strategies @xcite to optimize the model replies and to obtain satisfactory results guided by the labelled samples in the training set. for example, we included in the prompt/question to the model, the premise that the given question is a brain-teaser one. furthermore, we also evaluated the performance of the model on the training set using a few-shot learning setup. in this case, we provided as input (included in the prompt) ten questions indicating the correct answer. also in 3 https://rajpurkar.github.io/squad-explorer/ this case we did not obtain satisfactory results. more specifically, given the task hosted at semeval-2024, we asked the model: "what is the correct answer to the brain teaser question from the following choices? (pick only one option (a)-(d)". to this request, the model replied with one or more words that we parsed to extract one of the choices. for example, given the context: "romeo and juliet are discovered dead on the bedroom floor. glass shards and some water were on the floor when they were found. a bookcase and a bed are the sole pieces of furniture in the space. other than the neighboring railroad track, the house is located in a rural area. how is that even doable? " ## experimental setup we implemented our model on google colab. the library we used come from hugging face and as already mentioned is a multilingual version of de-berta foot_2 . the dataset provided for all the phases are available on the official competition page. we did not perform any additional fine-tuning on the model. to run the experiment, a t4 gpu from google has been used. after the generation of the predictions, we exported the results on the format required by the organizers. as already mentioned, all of our code is available on github. ## results participants in brain teaser may participate in any or all of the two subtasks. the organizers created two adversarial questions, semantic and context reconstruction, for each brain-teaser (examples can be found on the task home page). the evaluation metrics applied are instance-based accuracy and group-based accuracy, defined as follows: in ## conclusion this paper introduces the utilization of a deberta model for addressing task 9 at semeval-2024. in our submission, we opted for a zero-shot learning approach, leveraging a pre-trained and finetuned transformer model without further adaptation. through various experiments, we found it advantageous to construct a prompt containing the question for the model. subsequently, we provided the context, question, and answer candidates as the prompt, prompting the model to discern the correct candidate answer. despite the task's inherent complexity, as evidenced by the final ranking, there remains ample room for improvement. potential alternative methodologies include leveraging the few-shot learning capabilities of the model or exploring alternative models such as gpt and t5. additionally, integrating additional data or incorporating samples from training and development sets could yield performance enhancements. further refinements could be achieved through finetuning and framing the problem as a text classification task. moreover, given the promising results observed across various tasks, the adoption of fewshot learning or data augmentation strategies could also be explored for improved outcomes @xcite @xcite @xcite @xcite . while our straightforward approach demonstrates potential for refinement, it is noteworthy that it required no additional pre-training. moreover, the computational resources needed to address the task were manageable, utilizing the free online resources provided by google colab.