YNU - HPCC at S em E val-2024 Task 9: Using Pre-trained Language Models with L o RA for Multiple-choice Answering Tasks (2024)

Abstract: This study describes the model built in Task 9: brainteaser in the SemEval-2024 competition, which is a multiple-choice task. As active participants in Task 9, our system strategically employs the decoding-enhanced BERT (DeBERTa) architecture enriched with disentangled attention mechanisms. Additionally, we fine-tuned our model using low-rank adaptation (LoRA) to optimize its performance further. Moreover, we integrate focal loss into our framework to address label imbalance issues. The systematic integration of these techniques has resulted in outstanding performance metrics. Upon evaluation using the provided test dataset, our system showcases commendable results, with a remarkable accuracy score of 0.9 for subtask 1, positioning us fifth among all participants. Similarly, for subtask 2, our system exhibits a substantial accuracy rate of 0.781, securing a commendable seventh-place ranking. The code for this paper is published at: https://github.com/123yunnandaxue/Semveal-2024_task9.

## introduction the human reasoning process includes two types of thinking: vertical and horizontal. vertical thinking is a sequential analysis process based on rationality, logic, and rules. horizontal thinking is a divergent and creative process. the success of language models has inspired the natural language model (nlp) community to focus on tasks that require implicit and complex reasoning. although this type of vertical thinking task is widespread, horizontal thinking puzzles have received little attention @xcite . task 9 in the semeval-2024 competition: brainteaser is a multiple-choice task that tests the model's ability to demonstrate horizontal thinking and challenge default common sense associations. the task consists of two subtasks, sentence and word puzzles @xcite . in recent years, machine learning models have garnered significant attention. traditionally, these models have employed a two-step process involving the extraction of hand-crafted features from documents followed by classification using algorithms like naïve bayes @xcite , svm @xcite , hmm @xcite , or random forests @xcite . however, this approach presents limitations, such as the need for meticulous feature engineering and reliance on domain knowledge for feature design. to address these shortcomings, neural approaches have emerged. early attempts, such as latent semantic analysis (lsa) @xcite and neural language models, initially underperformed compared to classical models but paved the way for developing more powerful embedding models. significant advancements were made with the introduction of word2vec @xcite , elmo (peters et al., 1802), roberta @xcite , gpt @xcite , bert @xcite , and subsequent models like gpt-3 @xcite and gshard @xcite , which boast increasingly more significant parameters and training datasets @xcite . this paper proposes a deep learning system for task 9 in semeval-2024, titled brainteaser. we use the decoding-enhanced bert (deberta) @xcite model with disentangled attention as the base model and use lora @xcite to fine-tune the model. focal loss was used to address the issue of label imbalance. the back-translation method is used to enhance the original dataset, and the processed dataset is used to train the model. the experimental results of this paper were ultimately presented in task 9 of the semeval-2024 competition. on the original dataset, the accuracy of task 1 was 0.9, ranking fifth; the accuracy of subtask 2 was 0.78, ranking seventh. the rest of this paper is organized as follows. in section 2, we provided a detailed description of the proposed system and model. the experiment and results are discussed in section 3. finally, section 4 presents the conclusion. ## deberta transformer @xcite has become the most effective neural network architecture for neural language models. unlike recurrent neural networks (rnns) @xcite that process text sequentially, transformers apply self-attention functionality to parallelly calculate the attention weight of each word in the input text. therefore, compared to rnns, they can perform large-scale model training in parallel. in this paper, the de-berta model we use is a new transformer neural language model that improves the bert model using two novel techniques: a disentangled attention mechanism and an enhanced mask decoder. figure 1 shows the structure of the system. ## conclusions this paper describes a deep learning model for a multiple-choice task (task 9: brainteaser in the semeval-2024 competition), using deberta as the base model and achieving good results, ranking fifth in accuracy in subtask 1 and ranking seventh in accuracy in subtask 2. however, there is still considerable room for improvement in the model. therefore, we will try more methods to improve the model's efficiency in the future.