MIDAS at S em E val-2020 Task 10: Emphasis Selection Using Label Distribution Learning and Contextual Embeddings (2020)

Abstract: This paper presents our submission to the SemEval 2020 - Task 10 on emphasis selection in written text. We approach this emphasis selection problem as a sequence labeling task where we represent the underlying text with various contextual embedding models. We also employ label distribution learning to account for annotator disagreements. We experiment with the choice of model architectures, trainability of layers, and different contextual embeddings. Our best performing architecture is an ensemble of different models, which achieved an overall matching score of 0.783, placing us 15th out of 31 participating teams. Lastly, we analyze the results in terms of parts of speech tags, sentence lengths, and word ordering.

## introduction emphasis selection is an emerging research problem @xcite in the natural language processing domain, which involves automatic identification of words or phrases from a short text that would serve as good candidates for visual emphasis. this research is most relevant to visual media such as flyers, posters, ads, and motivational messages where certain words or phrases can be visually emphasized with the use of different color, font, or other typographic features. this type of emphasis can help with expressing an intent, providing more clarity, or drawing attention towards specific information in the text. automatic emphasis selection is therefore useful in graphic design and presentation applications to assist users with appropriate choice of text layout. prior works in speech processing @xcite have modeled word-level emphasis using acoustic and prosodic features. understanding emphasis in speech is critical to many downstream applications such as text-to-speech synthesis @xcite , speech-to-speech translation @xcite , and computer assisted pronunciation training @xcite . in computational linguistics, emphasis selection is very closely related to the problem of keyphrase extraction @xcite . keyphrases typically refer nouns and noun-phrases that capture the most salient topics in long documents such as scientific articles @xcite @xcite , news articles @xcite , web pages @xcite , etc. in contrast, emphasis selection deals with very short texts (e.g. social media posts), and also emphasis could be applied to words belonging to various parts of speech. the goal of semeval 2020 -task 10 is to design methods for automatic emphasis selection in short texts. to this end, the organizers @xcite provided a dataset consisting of over 3,000 sentences annotated for token-level emphasis by multiple annotators. the authors employed the standard i-o tagging schema, which is widely used in annotation of token-level tags. we approached emphasis selection as a sequence labeling task solved using a bidirectional long short-term memory (bilstm) model, where the individual tokens are represented using various contextual embedding models. we also employ label distribution learning (ldl) @xcite approach, which elegantly accounts for disagreements between the annotators. ## methods let @xmath0, w 2 , ..., w n } be the input text, where w i is the i th token. the problem of emphasis selection is to assign each token w i one of two possible labels @xmath1, e o }, where e i denotes emphasis on the token and e o means otherwise. we approach this problem as a sequence labeling task solved using a bilstm model. we first represent each token w i with a dense vector x i of a fixed size. to this end, we explore three different embedding architectures: bert @xcite , roberta @xcite , and xl-net @xcite . thus the given input text d is transformed into a sequence of vectors {x 1 , x 2 , ..., x n }. we then feed these vectors to a bilstm model which captures the sequential relations between the tokens. the hidden state of the bilstm h i is associated with the token w i . thus h i provides a fixed-size representation for token w i while incorporating information from the surrounding tokens. in standard sequence prediction problem, we can apply an affine transformation to map h i to the class space. however, in this paper, as with @xcite , we employ ldl @xcite which transforms the output space into a distribution over the labels e. namely, the objective of the model is not to just assign one label for a token but a real-valued vector. this vector is a distribution over the labels e, where the values are proportional to the number of annotations. to achieve this objective, we use kl-divergence between the predictions and ground truth as the loss function for the model. the above equation demonstrates the loss for one sample, where p(e j ) is the ground truth distribution and p(e j ) is the model prediction. note that the above equation reduces to negative log-likelihood in case of standard sequence prediction. the entire architecture is described in figure 1 . ## conclusion in this paper, we present our submission to the semeval 2020 -task 10 on emphasis selection in written text. our best performing model achieved an overall matching score of 0.783, placing us 15th out of 31 participating teams. we approached emphasis selection as sequence prediction problem solved using bilstms. our experimental work demonstrates the effect of model architectures, trainability of layers, and embeddings on the performance. we analyze the results in terms of parts of speech tags and sentence lengths. our analysis provides some interesting insight into some of the shortcomings of the models and also the challenges with emphasis selection.