Team Enigma at A rg M ining- EMNLP 2021: Leveraging Pre-trained Language Models for Key Point Matching (2021)

Abstract: We present the system description for our submission towards the Key Point Analysis Shared Task at ArgMining 2021. Track 1 of the shared task requires participants to develop methods to predict the match score between each pair of arguments and key points, provided they belong to the same topic under the same stance. We leveraged existing state of the art pre-trained language models along with incorporating additional data and features extracted from the inputs (topics, key points, and arguments) to improve performance. We were able to achieve mAP strict and mAP relaxed score of 0.872 and 0.966 respectively in the evaluation phase, securing 5th place on the leaderboard. In the post evaluation phase, we achieved a mAP strict and mAP relaxed score of 0.921 and 0.982 respectively.

## introduction the quantitative summarization -key point analysis (kpa) shared task requires participants to identify the keypoints in a given corpus. formally, given an input corpus of relatively short, opinionated texts focused on a particular topic, kpa aims to identify the most prominent keypoints in the corpus. hence the goal is to condense free-form text into a set of concise bullet points using a welldefined quantitative framework. in track 1, given a debatable topic, a set of keypoints per stance, and a set of crowd arguments supporting or contesting the topic, participants must report for each argument the corresponding match score for each keypoint under the same stance towards the topic. in track 2, we are required to build a language model that would generate keypoints given a set of arguments and a topic and finally find the match score of that particular keypoint with the argument. we mainly focused on the first track. we frame the task of identifying the most prominent keypoints as a sentence similarity task, obtaining the most similar keypoints corresponding to a given argument. since the argkp-2021 dataset @xcite contains crowd arguments for or against a particular stance, naturally, we expect some paraphrasing in the arguments put forth by different people. this indicates that semantic similarity would be an appropriate measure of similarity. however, we observe the problem of semantic drift @xcite in keypoint -argument pairs. hence, we add additional lexical overlap and syntactic parse based features to improve performance (details on the features can be found in section 4). ## dataset description the argkp-2021 dataset @xcite which was the main dataset used for the shared task consists of approximately 27,520 argument/keypoint pairs for 31 controversial topics. each of the pairs is labeled as matching or nonmatching, along with a stance towards the topic. the train data comprises of 5583 arguments and 207 keypoints, the validation data comprises of 932 arguments and 36 keypoints and the test data comprises of 723 arguments and 33 keypoints. additionally, since external datasets were permitted, we experimented with two more datasets i.e., the ibm rank 30k dataset @xcite and the semantic textual similarity or sts dataset @xcite (described in section 4.5) to train our model before fine-tuning on the argkp-2021 dataset. the sts dataset comprises of 8020 pairs of sentences, whereas the ibm rank 30k dataset comprises of 30497 pairs of arguments and keypoints. ## implementation details in this section, we elaborate on our experiments and methodology to find the best-performing models. the section is organized to describe the addition of dependency parsing features in section 4.2, parts of speech features in section 4.3, tf-idf features in section 4.4, and the use of external datasets in section 4.5. ## results and discussions after we had concluded our experiments, a new evaluation method was proposed by organizers, which removes the positive bias towards a system that predict less true positives in high confidence. in the default evaluation metric a perfect recall is attained only when all positive ground truth labels are predicted, whereas the new method allows a perfect recall score when the top 50% of the predictions (ranked by confidence) are positive. however, since we had completed all our experiments at this point, it was not feasible to rerun all our experiments in the given time frame. hence we have reported all our results according to the default evaluation method. among all the transformer models without the use of external datasets, we found bart-large to perform best, along with deberta-large with tfidf as additional features, achieving the best map strict and map relaxed score of 0.909, 0.982 and 0.911, 0.987 respectively. all the reported results are averaged over three seeds. ## ablation study we designed different settings to compare and validate our approach and its performance. this section consists of results on excluding of topics from input in section 6.1, incorporating average of hidden states before feeding to dense layers in section 6.2, and boosting in section 6.3. since we obtain best results with bart-large and debertalarge with tf-idf features, thus the following ablation study is done with these class of models. ## conclusion in this work, we used pre-trained language models (plms) to predict the match score for each argument and keypoint pair under the same stance towards the topic. we observed the state-of-theart plms such as bart and deberta perform the best compared to other models. we further improve the performance with additional datasets (ibm rank 30k and sts) to perform additional pre-training (with sentence similarity) before finetuning on argkp-2021 dataset. we experimented with pos, dependency and tf-idf features to evaluate the addition of extra syntactic features. we support the selection of our final models with various ablation studies. it would be a good future direction to generate appropriate explanations from concatenated input and propose methods to use explanations in the training process.