S t FX - NLP at S em E val-2024 Task 9: BRAINTEASER : Three Unsupervised Riddle-Solvers (2024)

Abstract: In this paper, we explore three unsupervised learning models that we applied to Task 9: BRAINTEASER of SemEval 2024. Two of these models incorporate word sense disambiguation and part-of-speech tagging, specifically leveraging SensEmBERT and the Stanford log-linear part-of-speech tagger. Our third model relies on a more traditional language modelling approach. The best performing model, a bag-of-words model leveraging word sense disambiguation and part-of-speech tagging, secured the 10th spot out of 11 places on both the sentence puzzle and word puzzle subtasks.

## introduction riddles often exploit the commonsense of the solver to lead them astray, subverting expectations with it's answer. for example, the riddle "a young girl fell off of a 20 foot ladder but wasn't hurt. how? she fell off of the bottom rung." leads the solver astray by including the height of the ladder in the initial question, tricking one into latching onto misleading information. task 9: brainteaser @xcite presents riddles to a predictive model and asks the model to choose one of four answers to the riddle, in the hopes of bridging the gap between vertical and lateral thinking @xcite within language models. the data provided for the task is written in english and was obtained from public websites by utilizing web crawlers @xcite . the three models we employ to solve this task all apply an unsupervised learning approach, with two of the three models leveraging word senses and part-of-speech tagging to aid in their predictive capabilities. we wanted to leverage the senses of the nouns in the question and in each possible answer as we hypothesized that the senses present in the question and each answer may aid our models in piercing the proverbial commonsense veil that makes brainteasers and riddles difficult to begin with. our best approach, the bag-of-words model, landed us in 10 th place out of 11 places in the "overall" results of both subtasks. while 13 teams competed, two teams tied for both 2 nd and 4 th place in the sentence subtask, two teams also tied for both 1 st and 11 th place in the word subtask results. our code can be found on github foot_0 . ## background brainteaser places emphasis on the ability of a predictive model to use vertical and lateral thinking. vertical thinking leverages logic and rationality to perform a sequential analysis of a problem, whereas lateral thinking (or "thinking outside the box") leverages creativity to solve problems. the task is divided into two subtasks -sentence puzzles and word puzzles. we applied our models to both, with each subtask requiring vertical and lateral thinking to solve. figure 1 breaks down how sentence and word puzzles can be solved with lateral thinking. the train of thought labeled with a red "x" demonstrates logical thinking based on the information available at the time, whereas the alternate thought process -the line of thinking that allows the solution to be derived -displays how lateral thinking can affect the answer to a riddle as more context is provided. the dataset associated with the task presents each sample as a question and four possible answers. word sense disambiguation (wsd) is a natural language processing (nlp) task that involves determining the correct meaning or sense of a word within a given context @xcite . many words in natural language have multiple senses, and wsd aims to identify the intended sense of a word in a specific sentence or context. this is used in various language processing applications, such as machine translation, information retrieval, and text summarization. we employ wsd by leveraging sensembert @xcite , coupled with wordnet @xcite to disambiguate the sense of a token in a particular context. sensembert is a knowledge-based approach to wsd that produces high-quality sense embeddings. wordnet is a large lexical database that organizes words and their meanings into sets of interlinked synonyms called synsets. we leverage part-of-speech (pos) tagging in order to determine which tokens in each question and answer are nouns we can determine the sense of. we employ the english version of the stanford log-linear pos tagger 2 @xcite -which leverages dependency networks to aid in tagging tokens -in this work. for the purposes of our work, we only work with nouns -tokens whose tag begins with "nn". ## system overview the following is a description of each approach we took in an attempt to solve the task. we implemented a bag-of-words, language modelling, and a sense comparison approach. the language model at the core of all three of our approaches is bert-large-cased @xcite , the same model leveraged by @xcite in the creation of sensembert. ## experimental setup as previously mentioned, we only use the test set in our experiments. due to the unsupervised nature of algorithm 2: sense comparison 1 input: question, list of four possible answers 2 output: predicted answer 3 q_senses ← wordnetsenseextraction(question) 4 q_cls ← [embedding for sense in q_senses] // calculated by bert-large-cased 5 answers ← [choice 1 , choice 2 , choice 3 , choice 4 ] 6 answer_similarity ← [] 7 for a in answers do 8 a_senses ← wordnetsenseextraction(a) 9 a_cls ← [embedding for sense in a_senses] // calculated by bert-large-cased 10 total_similarity ← 0; 11 for q_cls_embedding in q_cls do 12 for a_cls_embedding in a_cls do 13 similarity_score ← cos_sim(q_cls_embedding, a_cls_embedding); 14 total_similarity ← total_similarity + similarity_score; 15 answer_similarity[i] ← total_similarity len(q_cls)•len(a_cls) 16 max_index ← index of max element in answer_similarity 17 return answers[max_index] ## results the performances of our models, the provided baseline models, and the best performing models submitted to this task are found in our bow-wsd model (section 3.1), the best performing of our three approaches, was able to surpass the roberta-l baseline in 2 of 6 of the sentence puzzle categories, and outperforms the same baseline on 5 of 6 of the word puzzle categories. bow-wsd outperforms or comes very close to outperforming the roberta-l baseline in both "overall" categories. the performance of our unsupervised models didn't approach the chatgpt or human baselines in any category. the closest our models got to the chatgpt baseline was in the original word puzzle category with a difference of 0.155, whereas the closest our models got to the human baseline was in the context sentence puzzle category with a difference of 0.469. the numbers achieved by our bow-wsd model netted us 10 th place overall in the sentence puzzle subtask. we suspect the relationship between the tokens in the question senses and the tokens in the correct answer's senses allowed our bow-wsd model to outperform our other approaches. using the sentence puzzle in ## conclusion whilst the best of our unsupervised models surpassed only one of the established baselines, we have been able to show that word sense disambiguation may have a place in riddle-solving models. our bow-wsd model performed better on the sentence puzzles, but our language modelling approach performed better on the word puzzle subtask. the inherent logical reasoning large language models obtain through the copious amount of train-ing data they're trained on can be led astray by the information provided by a riddle. leveraging word sense disambiguation we attempt to isolate the meaning of each noun and compare and contrast said meanings to those present in each possible answer. in the future, we will explore other means of incorporating wsd models within our riddleanswering model along with an ensemble method. while our unsupervised approaches didn't perform well compared to other submitted models on the task leaderboard, the senses of the nouns in each question and answer held information valuable enough to allow our models to surpass one of the three proposed baselines. regarding our bag-ofwords model, we will add a metric that penalizes an answer if the senses it displays are wildly different to those of the initial question. this penalty could reduce the impact red herrings typically found in riddles have on the bow-wsd model's predictive abilities.