IUST - NLPLAB at S em E val-2024 Task 9: BRAINTEASER By MPN et (Sentence Puzzle) (2024)

Abstract: This study addresses a task encompassing two distinct subtasks: Sentence-puzzle and Word-puzzle. Our primary focus lies within the Sentence-puzzle subtask, which involves discerning the correct answer from a set of three options for a given riddle constructed from sentence fragments. We propose four distinct methodologies tailored to address this subtask effectively. Firstly, we introduce a zero-shot approach leveraging the capabilities of the GPT-3.5 model. Additionally, we present three fine-tuning methodologies utilizing MPNet as the underlying architecture, each employing a different loss function. We conduct comprehensive evaluations of these methodologies on the designated task dataset and meticulously document the obtained results. Furthermore, we conduct an in-depth analysis to ascertain the respective strengths and weaknesses of each method. Through this analysis, we aim to provide valuable insights into the challenges inherent to this task domain.

## introduction the remarkable efficacy of language models in navigating complex reasoning tasks, particularly in the realm of vertical thinking, has prompted their exploration in lateral thinking problem domains @xcite . one such domain, exemplified by the brainteaser task @xcite , entails a multiple-choice question answering framework comprising two distinct subtasks: sentencepuzzle and word-puzzle. this paper directs its focus toward the sentence-puzzle subtask, which hinges on unraveling the intricate nuances of common sense embedded within sentence fragments @xcite . initially, we adopted a zero-shot approach, followed by experimentation with three distinct finetuning methodologies tailored for language model (llm) architectures as the backbone. additionally, we have made our code openly accessible on github foot_0 to facilitate reproducibility and further research endeavors. a primary challenge we encountered pertained to the constraint imposed by the dataset size, posing impediments to both fine-tuning procedures and model training from scratch. to mitigate this challenge, we employed various strategies, including the utilization of k-fold cross-validation techniques, to enhance the robustness and generalizability of our approach. ## background in the implementation of the zero-shot method, we employed chatgpt-3.5, utilizing a consistent prompt template throughout. conversely, for the fine-tuning process, we adopted a pre-trained sentence embedding technique to map input sentences into meaningful numerical vectors, facilitating subsequent decision-making regarding the provided questions and options. furthermore, in our fine-tuning methodologies, we integrated two distinct types of loss functions: binary cross-entropy loss and triplet loss. the binary cross-entropy loss function operates on the premise of determining whether two sentences coherently match or not. conversely, the triplet loss function aims to optimize the proximity between the question and the correct answer while concurrently ensuring a clear distinction between the question and unrelated options. ## dataset the task dataset comprises 507 samples designated for training purposes, with an additional 120 samples allocated for the test set. notably, the evaluation set encompasses a subset of the training samples, necessitated by data scarcity. consequently, for two out of the three fine-tuning methods, no data from the training set were utilized for evaluation. conversely, the third method, employing the k-fold technique, leveraged the training samples for both the training and evaluation phases. ## results we conducted evaluations of the aforementioned methods on the test set, and the results are presented in original sentences. however, its efficacy diminishes notably when applied to other categories such as semantic, showcasing a significant disparity compared to its performance on the original sentences. conversely, all of our fine-tuning methods demonstrate comparable performance across all categories. among our fine-tuning methodologies, the triplet loss approach stands out with the most impressive performance, achieving the highest overall score among all methods. the uniformity in scores observed with the triplet loss method suggests that it does not exhibit bias towards specific words or segments of the sentence; rather, it considers the entire sentence holistically. this is in contrast to the zeroshot method, where significant discrepancies exist among its scores. however, it's worth noting that the performance of our fine-tuning models could potentially improve with a larger volume of data. ## conclusion in this paper, we have presented four distinct methodologies for the brainteaser task, a novel challenge involving common sense reasoning and sentence puzzle solving. we have evaluated our methods on the task dataset and compared their performance across different categories. our results show that the zero-shot approach, based on gpt-3.5-turbo, achieves the highest score on the original sentences, but fails to generalize well to other categories. on the other hand, our fine-tuning methods, based on mpnet and various loss functions, demonstrate more consistent and robust performance across all categories, with the triplet loss approach achieving the best overall score. we have also employed the k-fold technique to mitigate the data scarcity issue and enhance the generalization capability of our models. through our analysis, we have provided valuable insights into the strengths and weaknesses of each method, as well as the challenges inherent to this task domain. we hope that our work will inspire further research on this novel and intriguing problem of common sense reasoning and sentence puzzle solving.