LAST at S em E val-2020 Task 10: Finding Tokens to Emphasise in Short Written Texts with Precomputed Embedding Models and L ight GBM (2020)

Abstract: To select tokens to be emphasised in short texts, a system mainly based on precomputed embedding models, such as BERT and ELMo, and LightGBM is proposed. Its performance is low. Additional analyzes suggest that its effectiveness is poor at predicting the highest emphasis scores while they are the most important for the challenge and that it is very sensitive to the specific instances provided during learning.

## introduction this paper reports on the participation of the laboratoire d'analyse statistique des textes (last) in the semeval-2020 task 10 emphasis selection for written text in visual media. this task requires automatic systems to select tokens 1 for emphasis in english famous quotes (e.g., no matter how hard you work, someone else is working harder. 2 ) and other short written text such the ones used in flyers, advertisements or social media posts (e.g., stay positive). its importance is growing continuously given the explosion in the number of slogans that web users want to display on smaller and smaller screens of more and more devices. writing brief, attention-grabbing and engaging message has become paramount. one of the characteristics of this task which makes it particularly complex resides in the brevity of the texts which means that almost all words, with the notable exception of grammatical words, are important otherwise they would have been omitted. so, it is more about identifying the most important words on a scale than making a binary decision between important vs. not important. this task has not attracted attention until last year when @xcite simultaneously highlighted its interest, collected from adobe spark 3 a dataset annotated by human judges, and proposed an automatic system for performing it. this system uses a deep sequence labeling network composed of two stacked bidirectional lstm layers and relies on elmo embeddings. @xcite observed that it outperformed a conditional random fields model @xcite with hand-crafted features. however, they did not evaluate other classical supervised learning techniques, such as gradient boosting decision tree @xcite , or the combination of these techniques with precomputed embedding models. the main objective of this study is to try to determine whether the use of such procedures can be competitive for the emphasis selection task. the remainder of this report describes the datasets made available for this challenge and the evaluation criteria, the systems developed, and the results obtained as well as a few analyses performed to get a better idea of the factors that affect the system performance. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## data and evaluation settings the materials for this task consisted of 3,876 brief pieces of text (or items) from 2 to 38 words foot_0 . the challenge organizers recruited raters, through amazon mechanical turk, for annotating these items. for each item, nine of them had to decide whether each token should be emphasized, a difficult decision for human judges since @xcite reported an agreement (fleiss' kappa) below 0.65. these items were randomly split up into three sets (see @xcite for details): the training set (train: 2,741 items and 32,399 tokens), the development set (dev: 392 items and 4,385 tokens) and the test set (test: 743 items and 8,192 tokens). the true labels for the train and dev sets were available from the beginning of the training phase while those for the test set still remain hidden. systems proposed to this shared task had to submit for each token of each item a predicted score of emphasis. on the basis of @xcite , the task organizers decided to use for assessing performances an ad hoc measure called match m, roughly foot_1 defined in @xcite , but implemented in a python function. it assesses the accuracy with which the tokens with the highest emphasis scores according to the submission are also those with the highest emphasis scores in the gold standard. this calculation is carried out successively for the first, first two, first three and first four tokens with the highest scores in the gold standard. the final match m for a system is the unweighted average of the average of the four match m scores over all of the items. being an average of accuracy scores, its maximum value is 1. the minimum value for a given item depends on both its length in tokens and the number of ties among the highest scores foot_2 . an important property of this evaluation measure is that it is calculated by item on the ranked data. it follows that it treats in the same way two tokens ranked first in their respective item, but one selected by all the annotators and the other by less than half while it can be considered that identifying the former is more important. ## system overview the proposed approach treats the prediction of the token emphasis scores as a regression problem and uses lightgbm @xcite to perform this prediction. the first analyzes carried out indicated that using classical features in text categorization such as token, lemma and postag n-grams were far from achieving an effectiveness comparable to that of the approach proposed by @xcite , which was chosen by the task organizers as the baseline. one potential explanation is that the majority of tokens in the materials appear only once. to take this difficulty into account, i chose to represent each token by means of precomputed embedding models and to use the corresponding vectors as features for lightgbm. it was expected that the embeddings will make rare tokens similar to more frequent ones in the training materials. the first analyzes carried out with this approach showed that adding to these embedding features the n-grams features tested first was not useful. so, i decided to focus on the embeddings. however, a limitation of the approach is that no information is extracted from the context (i.e., the whole item). an extended version of the system was therefore developed. it was based on the emphasis score predicted by the base system to which contextual features were added, some of which being obtained after processing the items by means of the stanford corenlp (manning et al., 2014). ## results as a reminder, the base system uses only traits from precomputed embedding models and the extended system uses the prediction from the base system (one feature) to which contextual features are added. these two systems were submitted for the challenge using the combination of train and dev sets for learning (3,133 items and 36,784 tokens). the best of the two was the extended system, which achieved a mean match m of 0.756 (see the full model in in order to assess the usefulness of the different feature sets, an ablation procedure was used. it must be noted that all the compared models include the binary feature which codes whether the embeddings of several bert tokens were added to get the embeddings for an original token. the values obtained on the official test set are given in in order to determine if the observed differences were statistically significant, a monte-carlo permutation test for related samples @xcite ) was used to compare selected conditions at a threshold level of 0.01, using 1,000 random permutations. these analyzes were performed by means of figure 2 : bubble plot of true and predicted emphasis scores for the 7-fold cv a cross-validation procedure since the gold standard for the test materials is not available. as bestgen (2020) observed that performing a single k-fold cross-validation provided sufficient information to assess the reproducibility of observed differences, the 7-fold distribution used during the development of the system was employed. figure 1 shows that the default parameters are significantly less efficient than the optimized ones while the extended model is significantly better than the other models. to get a better idea of the successes and failures of the extended system, figure 2 presents a bubble plot of the true and predicted scores of all tokens in the 7-fold cv, the size of a bubble being proportionally related to the number of observations in this area. it clearly shows that the system effectiveness is low at predicting the highest scores (see the large variability of the predicted scores for true scores of 8 and 9), which are the most important for obtaining a high match m in the challenge. in this figure, three areas deserve special attention. the bottom left corner contains a large number of effective predictions. these are very largely punctuation marks and grammatical words correctly predicted by the model as having to obtain (in general) low scores. for example, out of the 289 that in the combined train and dev sets, 96.5% have an emphasis score less than or equal to 2 and none have a score higher than 5. you 're never a loser until you quit trying . the two areas highlighted in figure 2 contain tokens for which the errors are the greatest. they correspond among others to grammatical words considered by the annotators to be emphasized because they are part of a chunk such as university of life. there is also a somewhat strange case (see ## conclusion the system proposed for task 10 of semeval-2020 to select tokens to be highlighted in short texts was mainly based on precomputed embedding models and lightgbm. its performance was mediocre since it did barely better than the baseline. it would not be correct to draw an argument from the fact that the human annotators themselves do not agree with each other in this task since other teams have proposed systems capable of achieving a much better performance. the performance of the extended system might have been improved if a weighting function for the features had been used such as the bi-normal separation feature scaling @xcite or bm25 which has proved useful in the vardial challenge @xcite ). yet, it seems to me that the major limitation of the proposed system is to take into account very little contextual information.