Key Point Matching with Transformers (2021)

Abstract: This work aims at describing a solution for the Track 1 of the KPA 2021 shared task, analyzing different methodologies for the specific problem of key point matching. The analysis focuses on transformer based architectures, experimentally investigating the effectiveness of variants specifically tailored to the task.

## introduction in the context of natural language processing, text summarization is the act of creating a concise summary of a given text. this is particularly useful when the text at hand represents arguments, i.e. opinions, about a certain topic. an example is that of political debates, in which politicians express opinions concerning the subject matter. another scenario is that of e-commerce sites, where users can write reviews about different items. in both cases, a third party can greatly benefit from a summary of the arguments in order to make effective decisions. in recent years, there have been several attempts to automatize the summarization of arguments in a meaningful way. the first works focused on clustering, by which arguments are assigned to clusters based on different notions of relatedness (e.g. semantic similarity) between pairs of arguments @xcite @xcite . unfortunately, with this methodology the actual summary of the arguments had to be compiled manually by analyzing the contents of the clusters. more recently, @xcite proposed to transform the summarization problem into what has been named key point analysis (kpa). given a certain topic of discussion, arguments are summarized by matching them to a short list of high-level summaries, called key points. this allows a quantitative approach to summarization, in which the saliency of a key point (i.e. the degree of matched arguments) gives a hint of the underlying nature of the arguments. even though @xcite have shown that a domain expert can create meaningful key points for a topic even without looking at the arguments, the final goal of this field would be to generate the key points and match them with the respective arguments without human intervention. with this goal in mind, the kpa 2021 shared task foot_0 @xcite poses two problems to its participants: a key point matching subtask (track 1) and a key point generation subtask (track 2). this paper describes a solution to the key point matching subtask, in which key points are given and the goal is to match them with the appropriate arguments. the problem is framed as a classification task and the paper will focus on analyzing different transformer based models with different possible variants. all experiments are executed on a nvidia tesla p100 with limited access. the computational limitations have influenced many design choices and results of the paper. ## kpa 2021 shared task: data and evaluation metrics as training data, the proponents shared the argkp dataset @xcite divided into training and development sets, containing respectively 24 and 4 debate topics of various nature and arguments mined from the internet. the dataset contains 24,093 <argument, key @xmath0. each pair is associated with the concerned topic, presented as a short-sized text, and the stance towards the topic (+1 for the pro side, -1 for the con side). a binary label (1 for a match, 0 otherwise) is assigned only @xmath1, key @xmath2, noting that arguments and key points having opposite stance automatically constitute a mismatch. at the end of the competition, the task organizers have also released an external test set containing 3 topics, totalling 3426 pairs, to be used for the final evaluation and ranking of participants. not every feasible pair appears in the labeled data. a pair appears in the dataset as a match or a mismatch if, respectively, more than 60% or less than 15% of the original annotators labeled it as matching. all the other pairs are not included in the dataset. intuitively, this ensures that the pairs are not labeled ambiguously whenever there was no net agreement between the annotators. even if the dataset is constructed like so, in the evaluation phase of the task the final model will be queried on all possible pairs including the unlabeled ones, testing the system's ability to deal with uncertainty. in fact, the model will have to output, for each argument, a match score in [0, 1] for each of the key points under the same topic and in the same stance towards the topic. the task is evaluated by the average rank of the strict and the relaxed mean average precisions (map), where the ranks are intended as the final positions of the two values in the rankings of the competition. the map, which is the mean of the precision values of the model at different recalls, is chosen as a metric since it gives a summary of the recall-precision curve without having to choose a specific classification threshold. given the match scores output by a model on a certain dataset, the strict and relaxed map values are computed as follows. first, each argument is paired with the highest scoring key point. then, half of the lowest matching pairs are discarded. this is done to avoid evaluating the model on arguments with no matching key points. at this point, precision values and therefore map values can be computed on the remaining pairs based on the labeled data. note, however, that a pair can have a high match score but no labels in the dataset because of its ambiguity. in this case, the pair can arbitrarily be considered a match or a mismatch. this results in the computation of two different map scores: a strict map, for which absent pairs are considered a match, and a relaxed map, for which the pairs are considered a mismatch. the final values are computed by taking the macro-average of strict and re-laxed map values for each topic/stance combination. ## methodological survey assigning a match score to @xmath3, key @xmath4. in recent years, transfer learning through fine-tuning of pretrained transformers such as bert @xcite have dramatically improved results on a variety of nlp tasks, including classification. furthermore, transformer based architectures are particularly suited for the task of sentence pair classification. in fact, many of these models are pretrained on tasks such as next sentence prediction (nsp) and sentence order prediction (sop), which require the model to take in pairs of sentences or segments and to classify them accordingly. this implies that the argkp dataset can be easily adapted to these models. moreover, since the models have already learned how to process and internally represent sentences, one would expect to see good results also in the context of key point matching. for these reasons, this work focuses on transformers. this is not a limiting decision, since there are many transformers architectures to test and many possible variants for each model. in literature, previous works related to @xcite already report results for 4 transformer based models, namely bert @xcite , xlnet @xcite , roberta @xcite and albert @xcite , considering only their largest versions. for each model, only classification metrics such as accuracy, precision, recall and f1 scores are reported. this work aims at expanding the analysis by studying the behavior of the models under the metrics of the competition and by experimentally evaluating new variants. in particular, to allow a comparison with the state of the art, the same 4 transformers are analyzed also here, extending the investigation to smaller versions of the models and to other variants that will be described later on. smaller models are not expected to be superior to the larger versions. nonetheless, they are included in the study to give a reference point regarding their performance on the task of key point matching. key point matching can also be considered as an instantiation of a natural language inference (nli) task @xcite , where the problem is to determine whether the key point logically follows from the argument (or viceversa). however, fine-tuning models pretrained on nli datasets already showed no improvement in literature @xcite . because of this and of the additional computational burden caused by pretraining on large nli datasets, this variant is avoided. besides basic transformers, this work includes sbert @xcite , a sentence embedding transformer model. sbert produces semantically meaningful sentence embeddings, which means that semantically similar sentences are close in the vector space. sbert has been used alone, without fine-tuning, computing the cosine similarity between arguments and key points embeddings. furthermore, it has also been used in one of the variants, feeding its outputs to the linear top-layer of the transformer based models to see if adding information about semantically meaningful sentence embeddings yields an increase in the performance. bert, xlnet, roberta and albert have been downloaded through the huggingface library @xcite , while sbert has been downloaded through the sentence-transformers python framework, choosing the paraphrase-mpnet-base-v2 model since it has the best performance on a variety of different tasks. bert is uncased. xlnet and roberta are cased since no uncased versions have been released for them. for each transformer other than sbert and albert, both base and large versions have been tested. for what concerns albert, albert-base-v2 has been used for the smaller version, while albert-xxlarge-v1 has been chosen for the larger version since it is generally better than albert-xxlarge-v2 on downstream tasks. a thorough analysis should have included also rnns and cnns. in fact, arguments and key points are generally short-sized and simpler models may actually work better in this context. however, given the non negligible computational limitations of this work, discarding these and other valuable models allowed for the exploration of a greater number of transformer based architectures and variants. ## data preprocessing and analysis since the training data shared by the organizers is split into a training (tr) and a validation set (vl), an internal test set (ts) split had to be extracted from the tr in order to assess the performance of the final model. in order not to allow @xmath5, key @xmath6, the ts has been extracted from the tr by doing a topic based division. the final internal ts contains 3 topics with a total of 3164 pairs, while the final tr and vl contain respectively 21 and 4 topics with a total of 17471 and 3458 pairs. arguments are mined from the internet and even if they have been originally filtered, their quality can vary a lot throughout the dataset. they sometimes contain misspellings and an informal use of symbols (e.g. slash between words). even if there is evidence of the fact that bert based models can suffer from natural adversarial instances such as typos @xcite , the quality of the dataset has not been improved for the sake of simplicity. as mentioned earlier in the paper, for @xmath7, key @xmath8. adding the topic to the input of the model is useful especially when neither the argument nor the key point contains enough information about the context of discourse. in previous works, the topic has not been included in the inputs and therefore its utility will be tested in one of the variants presented later. when the topic is considered, it is simply prepended to the concerned argument and the input to the model therefore @xmath9, kp>. notice, however, that topics are always written in the affirmative form as in "kids should be wearing uniforms at school". when the stance towards the topic is negative, in order not to fool the model by giving it contradictory information, the topic should be expressed in its negative form. one way to do this would be to encode the stance as a binary input to the linear output layers. here, instead, affirmative topic are simply transformed into their negative form in order to exploit the encoding capabilities of the transformers. transforming an affirmation to a negation is not a trivial task, but fortunately the topics contained in the dataset and in the external ts are all in the form subject + should or is + rest. this allows to just add not after should or is in order to complete the transfor-mation, noting that, in order to make the system generalizable, one would have to develop a fully working affirmative-to-negative module. the alternative to this would have been to encode the stance as a binary input to the models. arguments appear in the dataset in all lower case, all caps (a very small portion) and a mixture of both. arguably, in the context of key point matching, a cased analyis of the text would not be much useful. for this reason, uncased models have been used when possible, transforming the text to lowercase before processing it. all text is tokenized and transformed to lowercase by relying on the different tokenizers provided by each transformer. finally, @xmath10, @xmath11: [cls] arg [sep] kp [sep]. ## models and variants in the model selection phase, bert, albert, roberta, xlnet and sbert are all tested with the following modalities. for each model except sbert, the representations computed by the transformers are fed to a single linear layer with two output logits. a softmax layer is then used at evaluation time to compute the final match probability. each model except sbert can have multiple possible configurations corresponding to different variants of the basic models. this work focuses on three main variants. first, one can decide whether to inform or not the model about the topic with the modalities shown in the previous section. second, the final contextualized representations output by the transformers can be computed in a number of different ways as in @xcite . this was allowed since the best representation usually depends on the particular task at issue. one can decide to just take the [cls] token representation or to get a final vector by applying a pooling operation to the hidden representations of the transformer. three pooling options are available: average of the last layer (avg last), average of the second last layer (avg 2ndlast), average of the sum of the last four layers (avg sumlast4). finally, one can choose to concatenate the 768dimensional sbert embeddings of the two input segments to the representation output by the transformer. this was allowed to see if adding information about semantically meaningful sentence embeddings would help the models in the classification task. even if the task does not require the models to compute the actual classification labels, for the sake of completeness, classification metrics such as accuracy, precision, recall and f1 scores have been computed both on the tr and the vl. computing classification labels from match scores can be done in multiple ways as shown in @xcite . it can be done locally, considering just the single score of a pair, or globally, taking into account all the match scores between an argument and the key points. since this problem was not the focus of the paper, only two simple local methods have been evaluated. in the first one, a match score is trivially transformed using a threshold of 0.5. in the second one, the threshold is learned and is chosen to maximize the f1 score on the tr. since the learned threshold often generalizes worse than the fixed one, it is not considered in the discussion and related results are shown in appendix a. ## test set results and analysis having chosen the best model on the vl, the last phase consisted of its assessment on the internal and external ts. results on both internal and external ts with a reference to previous results on vl are shown in ## error analysis to complete the study, errors on the external ts have been analyzed manually in order to try to understand the weaknesses of the model. to avoid choosing a particular threshold for the classification labeling, only the pairs which the model was most confident about have been considered for the study. since the final model has a low precision, most of the analysis focused on the pairs labeled with 0 that have been wrongly matched. what emerged was that, at first glance, many of these pairs seemed to be matching also for a human. for humans, a reasonable match score can be assigned only after seeing all the possible key points and the general distribution of the arguments. to show an example, consider the following topic, argument and key point in order: "routine child vaccinations should be mandatory", "child vaccination should be mandatory to prevent children from spreading the virus", "routine child vaccinations are necessary to protect others". the argument seems to be matching the key point to a high degree. let us now introduce a key point that actually matches the argument: "routine child vaccinations should be mandatory to prevent virus/disease spreading". the argument seems to be matching this second key point more than the first one. the score assigment can be also influenced by the nature of the other arguments. if the arguments generally happen to match key points in a strict way, one may end up assigning lower scores to looser but still matching pairs. this indicates that, perhaps, the model should be informed about the existence of other key points and arguments to assign the scores in a reasonable way. other more subtle errors are caused by the model wrongly associating different terms coming from the same semantic field. consider for example the following topic, argument and key point: "the usa is not a good country to live in", "the usa is a bad country to live in because some people are aggressive", "the us is xenophobic/racist". here, the model is fooled by the semantic similarity between aggressiveness and xenophobia/racism. more specifically, since transformer based models are pretrained focusing only on the co-occurrence of words, the model is not aware of the actual logical relationship between them. consequently, it can not know that someone who is aggressive is not necessarily xenophobic or racist. lastly, some errors depend on the bad quality of the arguments and of the labeling. consider the following triple: "the usa is not a good country to live in", "not because it is a country with many restrictions for immigrants.", "the us is xenophobic/racist". the model correctly assigns a low score to the input. however, the pair is labeled with a 1, indicating that the annotator interpreted the initial "not" as a mining mistake. ## conclusions key point analysis is a promising way to summarize a large number of arguments about a certain topic. in this paper, which describes a solution for the track 1 of the kpa 2021 shared task, the specific problem of assigning match scores @xmath12, key @xmath13, focusing on different state of the art transformerbased architectures and variants specifically tailored to the problem. the analysis included a number of different transformers, namely bert, xlnet, roberta and albert, as well as sentence embedding transformers, such as sbert. after a lightweight model selection procedure, roberta managed to beat the other models under the metrics imposed by the competition. informing the model about the topic (appropriately modified based on the stance) has been beneficial for the overall performance. on the other hand, supplying the model with information about the semantic similarity between arguments and key points through sbert embeddings gave no particular advantage. similarly, changing the type of pooling of the transformer's hidden representations did not result in any notable difference. after the evaluation on the external ts, the model maintained a good behavior under the specified metrics, which take into account ambiguous nonlabeled data, while getting a low precision on nonambiguous data, showing that the metrics may favor imprecise models. lastly, the final error analysis showed how the classification problem as formulated in the paper may be underspecified and that the model should also be informed about the existence of other key points and arguments. moreover, particular errors highlighted the intrinsic weaknesses of transformer based models in the task of key point matching. many choices and results of this work have been determined by computational limitations. with more computational power, one could improve the model selection by employing a robust crossvalidation procedure and by testing other valuable architectures. finally, as suggested by the error analysis, an interesting direction in the key point matching related research would be to experiment with models that take information regarding multiple key points and arguments when computing a match score for a @xmath14, key @xmath15.