Overview of the 2021 Key Point Analysis Shared Task (2021)

Abstract: We describe the 2021 Key Point Analysis (KPA-2021) shared task on key point analysis that we organized as a part of the 8th Workshop on Argument Mining (ArgMining 2021) at EMNLP 2021. We outline various approaches and discuss the results of the shared task. We expect the task and the findings reported in this paper to be relevant for researchers working on text summarization and argument mining.

## introduction key point analysis (kpa) was introduced in @xcite as a challenging nlp task with tight relations to computational argumentation, opinion analysis, and summarization, and with many practical applications @xcite . given a potentially large collection of relatively short, opinionated texts focused on a topic of interest, the goal of kpa is to produce a succinct list of the most prominent key-points (kps) in the input corpus, along with their relative prevalence. thus, the output of kpa is a bullet-like summary, with an important quantitative angle. successful solutions to kpa can be used to gain better insights from public opinions as expressed in social media, surveys, and so forth, giving rise to a new form of a communication channel between decision makers and people that might be impacted by the decision. various requirements govern the value of the kpa output. kps are expected to be succinct, nonredundant, capturing points that are central to the topic of interest, and reflecting a clear stance towards that topic. ideally, they should be at the right granularity for summarising the input data -not too specific and yet still informative and not overly general. in addition, accurate mapping of input texts to kps is obviously essential. first, to ensure a reliable estimate of the prevalence of each key * these authors equally contributed to this work. point. and second, to enable the user to drill-down, to gain a deeper understanding of the issues underlying each key point, as expressed by the input texts mapped to that key point. the goal of the kpa-2021 shared task was to further increase the attention of the nlp community to this emerging task, while enriching the space of existing kpa solutions. since providing a complete kpa solution is challenging, we divided the task into two tracks, enabling teams to participate only in the first, relatively simpler track. specifically, in the first track, referred to as matching track, kps are given as part of the input, and the task is focused on mapping input text to these kps. in contrast, in the second track, referred to as generation track, no kps are provided and the task requires to further generate the kps. the data being considered for evaluation on both tracks are crowd sourced arguments on three debatable topics. the nature of the kpa task requires to consider various evaluation measures to estimate the quality of the results across different dimensions. in this paper we report and analyze the results of 22 models submitted by 17 teams to the kpa-2021 shared task across these different evaluation measures. we also discuss lessons learned from these multiple submissions, that can guide future development of new kpa algorithms and associated evaluation methods. ## related work our shared task is based on @xcite , which introduced the problem of kpa and proposed an extractive approach to address the task. in particular, we focus on benchmarking two sub tasks: (1) key point matching; and (2) key point generation and matching. for the matching task bar-haim et al. (2020b) uses roberta @xcite based classifier, trained on argument-kp pairs. for the generation task @xcite employs the follow-ing steps: first, compiling a list of high quality arguments that serve as kp candidates. next, a match confidence score is calculated between all kp candidate-argument pairs, as well as between all the kp candidates to themselves. finally, several filtering and ordering strategies are employed to maintain the most salient, yet non-overlapping kp candidates to serve as kps. below we briefly review the work closely related to kpa in a few fields. ## submitted models in total, 17 models were submitted to matching track, and 5 models to generation track. note that some teams did not provide description for their model. still, we show their results in the results section, but we cannot provide their descriptions. ## results in addition to the submitted models, we evaluate the state of the art roberta based model described in @xcite and in section 2, termed hereafter barh. ## additional analysis accurate matching of arguments to kps aims to properly reflect the prevalence of each kp in the data to facilitate more informed decision making. in this section we analyze how well different models were able to reproduce the presumably true distribution of kps in the input data. to that end, we focus on the matching track, and consider the distribution of the expert kps as the ground truth distribution. as can be seen in the top row of figure 2(a), the five kps provided by the expert for this topic and stance are not equally represented in the argument set. nearly 40% of the arguments were matched by the annotators to "social media regulation is required to deal with malicious users" (kp-1-9); around 30% were matched to "social media regulation can help to deal with negative content" (kp-1-8); and so on. correspondingly, for each model submitted in this track, we compare the distribution of kps as predicted by its matching to the manually obtained ground truth distribution. to that end, we use the jensen shannon divergence (jsd). the smaller the divergence, the more accurate is the summary generated by a model (based on the 50% of the arguments that pass the threshold). we find that the average topic-stance jsd for the top 5 ranking models ranges from 0.152 to 0.238 (see in the generation track, there is no single set of kps, since each model generates its own set. it is topic id text 0 routine child vaccinations should be mandatory 1 social media platforms should be regulated by the government 2 ## discussion we presented kpa-2021, the first shared task focused on key point analysis. the shared task received 22 submissions from 17 teams, covering both the matching track and the generation track. we presented the submitted models, their performance on different evaluation measures, and further analyzed the results. as expected by its simpler nature, matching track received more submissions. however, ev-idently, success in this intermediate matching task, does not guarantee success in the overall kpa task, which also requires generating the kps. future work should determine whether future kpa shared tasks should focus on the generation track, or perhaps modify the evaluation measures of the intermediate matching track, such that they better reflect the model performance in the full kpa task. given the inherent subjective nature of matching arguments to kps, we opted for a ternary label, allowing argument-kp pairs to receive an "undecided" label, if the annotators votes were inconclusive. these undecided pairs are eventually considered either as positive or as negative matches, resulting with two gold standards, one potentially too strict, while the other perhaps too lenient. in future tasks, it may be valuable to consider a binary labeling scheme that will give rise to a single gold standard. such labeling may be created conditionally, such that an undecided pair is marked as a positive match if and only if a minimum number of annotators marked it as positive and no other kp was labeled as matching to that argument. a further point for future consideration, is the choice of a pre-defined threshold of 50%, that guided our evaluation. although, this has the advantage of not requiring submissions to tune a confidence threshold, it has the limitation that it complicates the evaluation, since the ground truth coverage depends on the arguments, topic and stance. a possible alternative would be to require each model to set its own minimum confidence threshold to determine if an input argument should not be matched to any kp. finally, in this task, we used comparative evaluations to determine the quality of the sets of generated kps. obviously, this results with rankingbased evaluation. establishing an absolute evaluation metric in this context merits further investigation. we thank all participating teams for taking the time to participate in this challenging new shared task. we believe that these contributions, along with the data collected and shared in this report, will be valuable to further advance the research around kpa and related tasks.