NIMZ at S em E val-2024 Task 9: Evaluating Methods in Solving Brainteasers Defying Commonsense (2024)

Abstract: The goal and dream of the artificial intelligence field have long been the development of intelligent systems or agents that mimic human behavior and thinking. Creativity is an essential trait in humans that is closely related to lateral thinking. The remarkable advancements in Language Models have led to extensive research on question-answering and explicit and implicit reasoning involving vertical thinking. However, there is an increasing need to shift focus towards research and development of models that can think laterally. One must step outside the traditional frame of commonsense concepts in lateral thinking to conclude. Task 9 of SemEval-2024 is Brainteaser (Jiang et al.,2024), which requires lateral thinking to answer riddle-like multiple-choice questions. In our study, we assessed the performance of various models for the Brainteaser task. We achieved an overall accuracy of 75% for the Sentence Puzzle subtask and 66.7% for the Word Puzzle subtask. All the codes, along with the links to our saved models, are available on our GitHub.

## introduction with recent advancements in deep learning and especially language models, extensive research has been conducted about reasoning in various natural language processing tasks, including question answering. these reasoning methods adopt vertical thinking. however, lateral thinking is another type often associated with creativity. in the 9th task of semeval, brainteaser @xcite , a task of answering multiple-choice riddle-like questions is defined. to answer these questions, the model needs to employ lateral thinking. this method of thinking differs from vertical thinking in that the reasoning process is not linear. to arrive at a conclusion, one must examine the subject from a perspective beyond the usual conventional thinking 1 https://github.com/z-rahimi-r/ nimz-at-semeval-task-9-brainteaser paradigms @xcite . an example of a comparison between the two types of thinking is provided in figure 2 in the appendix. lateral thinking demands a mind that is open, flexible, and creative. equipping ai models with cognitive abilities such as lateral thinking can enhance problem-solving, adaptability, and coping with new situations and challenges. in this work, we have evaluated the performance of three categories of models on answering brainteaser questions. we trained and evaluated two language models, bert @xcite , and roberta @xcite , the model presented in @xcite (qa-gnn), and a t5 @xcite model for sentence puzzle and word puzzle subtasks. in the qa-gnn method, the conceptnet knowledge graph @xcite is used as the source of commonsense knowledge. through the brainteaser task, we gained insights into two types of thinking -vertical and lateral. we also learned the significance of implementing lateral thinking in ai systems to bridge the gap between human and ai performance. furthermore, this task piqued our interest in the captivating subject of creativity in artificial intelligence models. we achieved an overall accuracy of 75% and ranked 20th for the sentence puzzle subtask. for the word puzzle subtask, we ranked 19th and achieved an overall accuracy of 66.7%. all the codes, along with the links to our saved models, are available on our github. ## background the goal and dream of the artificial intelligence field has long been the development of intelligent systems or entities with human-like behavior and thinking. according to existing research, there are two types of thinking in humans: vertical and lateral. most of the existing research focuses on vertical thinking. vertical thinking involves a logi-cal and sequential approach, while lateral thinking requires creativity and flexibility to explore problems from unique and unconventional perspectives @xcite . the brainteaser dataset @xcite ) contains 1100 riddle-like english questions requiring lateral thinking. the nature of questions often defies commonsense when approached with vertical thinking. the brainteaser task includes two subtasks: sentence puzzle and word puzzle. the details of the dataset are presented in the most research focuses on vertical thinking, using commonsense for implicit and explicit reasoning tasks such as commonsense question answering. commonsense intelligence is intuitively reasoning about everyday situations and events, which requires knowledge of how the world works @xcite . in the task of commonsense question answering, two popular methods are fine-tuning language models and using graph neural network (gnn) models. in recent years, the use of knowledge graphs, the primary sources of commonsense knowledge, has increased. commonsense knowledge stored in language model parameters is mainly descriptive and taxonomic knowledge, often explicitly stated in the language content that these models have been trained on @xcite . the method presented in comet @xcite can be a means to teach language models other types of knowledge. the success of comet can be attributed to the combination of neural and symbolic representations of knowledge, as well as the use of language to represent symbolic knowledge @xcite . the comet model is finetuned on the atomic knowledge graph @xcite . this knowledge graph serves as a customized textbook for language models to learn commonsense knowledge and how the world works @xcite . in the second popular category of methods, a knowledge graph is used as the complementary source of knowledge with the help of graph neural networks as the medium to harvest this knowledge @xcite @xcite . one advantage of using graph neural networks is their interpretability. in qa-gnn @xcite , the roberta lm is used with graph neural networks. each answer option is checked independently in their method to determine if it is the answer. for each answer option, a subgraph is extracted from the conceptnet. this subgraph consists of the entities in question and the answer option, all the entities within two hops from question and answer entities on the concept-net graph, and the relations between them. in the presented method, the question and the answer option are concatenated and encoded using roberta lm @xcite , then placed as a context node in the subgraph. since some nodes in the subgraph are more related to the question and its answer, the roberta lm is used to calculate a score for each node in the subgraph. this score is used as an additional feature to the node embeddings to increase the influence of more related entities. training is done through the messagepassing method. finally, the score of each option being the answer is calculated and the answer option with the highest score will be the final answer to the question. the approach described in @xcite is similar to qa-gnn but with one key difference. while qa-gnn evaluates each answer option independently using a local graph, this method also includes a global graph that allows for simultaneous evaluation and comparison of all answer options, leading to refined probabilities. refining the probabilities of each answer option in this way can produce a more accurate result. they consider this method similar to how humans eliminate less likely options. the most similar available study to the brainteaser task is riddlesense @xcite , where a riddle dataset is presented. to solve the riddles, one needs advanced natural language understanding, commonsense, and counterfactual reasoning skills, which are complex cognitive processes. they have trained and evaluated several language models, gnn-based models, and text-to-text models on the riddlesense dataset. ## system overview in this section, we will present the systems used to tackle the brainteaser task. the three main approaches in question-answering tasks are finetuning language models, graph neural networks, and text-to-text transformers. so, we decided to evaluate the performance of these models on the brainteaser task. although the role of commonsense in this task is as a distractor @xcite , we decided to evaluate the impact of using commonsense knowledge through concept-net knowledge graph and graph neural networks. while the answer may challenge commonsense in the brainteaser questions, it does not violate it. all the models are trained for sentence puzzles and word puzzles separately. the general sketch for each type of system is presented in figure 1 . ## experimental setup we have trained and evaluated base and large sizes of bert, roberta, and t5 models using the hugging-face transformers library, with different hyperparameters to find the best setting. to train the qa-gnn model, we followed the procedure provided by the code available on the github of @xcite . after preprocessing the brainteaser dataset, the qa-gnn models were trained for 100 epochs with early-stopping. in the inference phase of the t5 model, in some cases, the extracted span was incomplete and did not include the letter of the answer option, in these cases the "none of above" option was selected. the code for the in-house train-dev split and the hyperparameters used for training the best-performing models are available in the notebooks on our github foot_0 . ## results we evaluated models from different categories on this task. due to the riddle-like and unique nature of the questions, it was difficult for the models to generalize to new questions of the test set. we achieved an overall accuracy of 75% and ranked 20th for the sentence puzzle subtask. for the word puzzle subtask, we ranked 19th and achieved an overall accuracy of 66.7%. the qa-gnn model performed best for the sentence puzzle in the evaluation phase. still, for the word puzzle, the bertbase model had the best performance, and qa-gnn performed poorly, which could be due to the absence of reasoning paths on the knowledge graph between the concepts of the answer option and the question. the results of the two phases, evaluation and post-evaluation, are presented in the ## conclusion in this study, we evaluated the performance of three main categories of popular methods in the questionanswering task on the two subtasks of sentence puzzle and word puzzle of the semeval-task 9 brainteaser. we have achieved an overall accuracy of 75% for the sentence puzzle subtask and 66.7% for the word puzzle subtask. the nature of the brainteaser questions is such that they challenge commonsense and require lateral thinking and intellectual creativity to be solved. models other than llms tend to perform poorly in generalizing to new and different examples, especially when it comes to tasks that require creativity, such as puzzles and brainteasers. while llms tend to perform better, they still have limited capability when it comes to being creative. regarding the suggestions for future work, we believe utilizing the chain-of-thought @xcite ) method and teaching llms to reason step by step with the incontext-learning method can be effective. another idea is to develop two modules for llms or ai agents. the first module will aid in the creative production of knowledge, while the second module will check the rationality of the produced knowledge and its consistency concerning the context of the desired problem. as mentioned earlier, the autoregressive nature of current llms and reliance on probabilistic solutions have limited their ability to produce creative content. so, there is a need to design new architectures and different training methods to overcome this limitation. this can be a helpful step towards enhancing creativity and lateral thinking in ai systems.