Modern Talking in Key Point Analysis: Key Point Matching using Pretrained Encoders (2021)

Abstract: We contribute to the ArgMining 2021 shared task on Quantitative Summarization and Key Point Analysis with two approaches for argument key point matching. For key point matching the task is to decide if a short key point matches the content of an argument with the same topic and stance towards the topic. We approach this task in two ways: First, we develop a simple rule-based baseline matcher by computing token overlap after removing stop words, stemming, and adding synonyms/antonyms. Second, we fine-tune pretrained BERT and RoBERTalanguage models as aregression classifier for only a single epoch. We manually examine errors of our proposed matcher models and find that long arguments are harder to classify. Our fine-tuned RoBERTa-Base model achieves a mean average precision score of 0.913, the best score for strict labels of all participating teams.

## introduction arguments influence our decisions in many places of our daily life @xcite . but with the increasingly larger amount of information found on the web 1 and more effective argument mining, people often need to summarize arguments @xcite . @xcite see matching key points to arguments as an intermediate step towards automatically generating argumentative summaries (section 2). the argmining 2021 shared task on quantitative summarization and key point analysis @xcite is the first task on key point matching, which is an important step towards summarizing arguments. by matching arguments with a pre-defined set of key points, an argumentative text can be summarized using the prevalence of the key points in it. different approaches of matching argument key point 1 https://internetlivestats.com/ pairs, here called matchers, should be proposed and discussed. given an argument and a key point, a matcher should return a real value between 0 and 1 which represents the extent to which the argument matches the key point. 2 for evaluating different argument key point matchers, the shared task organizers uses mean average precision evaluation as a metric @xcite to evaluate the approaches and publish the argkp-2021 benchmark dataset (section 3) to compare matchers @xcite . pretrained language models like bert and roberta are nowadays becoming standard approaches to tackle various natural language processing tasks @xcite . because of their extensive pretraining, often fine-tuning these language models with even a small task-specific dataset can achieve state-ofthe-art performance @xcite . as the argkp-2021 dataset @xcite used in the argmining 2021 shared task on quantitative summarization is relatively small (24 083 labelled pairs), we decide to fine-tune bert and roberta language models rather than train a neural classifier from scratch (section 4). contrasting this neural approach, we introduce a simple rule-based baseline matcher that compares preprocessed tokens of each argument to the tokens of each key point (section 4). for the baseline, we compute token overlap after removing stop words, adding synonyms and antonyms, and stemming the tokens from both argument and key point using the nltk toolkit @xcite . our fine-tuned roberta-base matcher achieves a mean average precision score of up to 0.967 and ranks second in the shared task's leaderboard (section 5). in a manual error analysis, we find that the imbalanced argkp-2021 dataset causes neural models to predict non-matching argument key point pairs more precisely than matching pairs (section 6). we further observe a tendency that large length differences between arguments and key points can cause errors. to encourage researchers to train more robust argument key point matchers, we release our source code under a free license. key point analysis is the task of matching a given argument with one or more pre-defined key points @xcite . to develop models for the task, @xcite introduced a dataset (argkp-2021) which contains 24 093 argument key point pairs on 28 topics. each argument and key point is labeled manually as match or no-match. the authors experimented with several unsupervised and supervised approaches to perform the task in a cross-topic experimental setting. bert @xcite performed the best in their experiments by reaching an f1 score of 0.68. in a later work, bar-haim et al. (2020b) develop a summarization approach for online discussions that uses key point analysis. the summrization approach takes as input a set of comments on a given topic and extracts a set of representative key points from them. the output of the summarization approach is the set of extracts key points together with the count of matched comments for each key point. in its essence, the summarization approach uses a matching model that gives a score for a given comment and key point or a pair of key points. for matching models, @xcite compare different variants of bert @xcite . among the tested models, albert @xcite performed the best with an f1 score 0.809, but roberta @xcite were chosen for key point extraction at the end, which is 6 times faster than albert and still achieves an f1 score of 0.773. our approaches for the key point analysis are based on bert and roberta. bert stands for bidirectional encoder representations from transformers and is an open-source bidirectional language representation model published by google @xcite . bert is pre-trained over unlabeled text to learn a language representation and can be fine-tuned on downstream tasks. during pre-training, bert is trained on two unsupervised tasks: masked language model and next structure prediction. roberta is an improved variant of bert that is introduced by facebook in 2019 @xcite . @xcite modified bert by using a larger training data size of 160gb of uncompressed text, more compute power, larger batch-training size, and optimized hyperparameters. in comparison to bert, pre-training tasks for roberta were done with full-length sentences and include only masked language model while applying different masks in each training epoch (dynamic masking). roberta outperforms bert on all 9 glue tasks in the single-task setting and 4 out of 9 tasks in the ensembles setting @xcite . ## data the dataset used in the argmining 2021 shared task on quantitative summarization and key point analysis is the argkp-2021 dataset (bar-haim et al., 2020a) which consists of 24 083 argument and key point pairs labeled as matching/nonmatching. they all belong to one of 28 controversial topics, for example: "assisted suicide should be a criminal offence". every key point and argument pair is annotated with its stance towards the topic. the training split of the argkp-2021 dataset has 5 583 arguments belonging to 207 key points within 24 topics. this leaves the validation split with 932 arguments and 36 key points for 4 topics. friedman et al. (2021) complement the argkp-2021 dataset's training and validation split with a test split that is used to evaluate submissions to the shared task. the test split contains 723 arguments with 33 key points from 3 topics. ## approach to match key points to arguments, we propose two different approaches. first, we discuss a simple yet effective baseline measuring token overlap be-tween key points and arguments. second, to improve upon this simple baseline, we introduce an approach based on bert and roberta @xcite . we fine-tune both language models in standard configuration with only minor changes highlighted below. ## results submissions to the argmining 2021 shared task on quantitative summarization and key point analysis are evaluated with respect to mean average precision @xcite . the organizers calculate the score by pairing each argument with the best matching key point according to the predicted matching probabilities. within each topic-stance combination, only 50 % of the arguments with the highest predicted matching score are then considered for evaluation. the task organizers claim that this removal of 50 % of the pairs is necessary because some arguments do not match any of the key points, which would influence mean average precision negatively @xcite . for the remaining argument key point pairs in each topic-stance combination, the average precision is calculated and the final score is computed as the mean of all average precision scores. the task organizers consider two evaluation settings: strict and relaxed. both settings are based on the ground-truth labels from the argkp-2021 dataset @xcite . the two evaluation settings are created to account for argument key point pairs in the argkp-2021 with undecided labels (i.e. not enough agreement between annotators). in the strict setting, the shared task organizers consider those undecided pairs as no-match. in the relaxed setting, however, the shared task organizers consider the undecided pairs as match @xcite . the mean average precision score is then calculated in the two settings based on the ground-truth labels and the derived labels for the undecided pairs. we stress that in this complex evaluation setup, the mean average precision score in the relaxed setting would favor assuming matches in case of model uncertainty. in comparison, in the strict setting mean average precision would favor assuming no-match between an argument and key point. however, we find that because only the most probable matching key point is being considered for evaluation, this effect is minor. the evaluation score in general favors matchers that can match a single key point for each argument with high precision. it is however not important if a matcher does predict non-matches with high certainty. ## conclusion and future work we approach the practical problem of matching arguments with short key points with the goal of summarizing arguments. although our token overlap baseline approach is very simple, it achieves a mean average precision of up to 0.575 on the test set, nearly double the score of a random matcher. the baseline approach is straightforward to implement but can not eliminate the problem of context understanding. roberta-base and bert-base have achieved good performance, because they can overcome the context understanding challenge. our fine-tuned roberta-base model also performed better than bert-base in this task and scores a mean average precision of up to 0.967. with strict ground truth labels it achieves a mean average precision score of 0.913 on the test set, which is the best score of the participating teams in the shared task. this again shows the importance of architecture, training objectives, and hyperparameter selection.