T ext L earner at S em E val-2020 Task 10: A Contextualized Ranking System in Solving Emphasis Selection in Text (2020)

Abstract: This paper describes the emphasis selection system of the team TextLearner for SemEval 2020 Task 10: Emphasis Selection For Written Text in Visual Media. The system aims to learn the emphasis selection distribution using contextual representations extracted from pre-trained language models and a two-staged ranking model. The experimental results demonstrate the strong contextual representation power of the recent advanced transformer-based language model RoBERTa, which can be exploited using a simple but effective architecture on top.

## introduction visual communication aims at conveying intended information from the author to the audience with the help of visual elements. one of its essential design principles is to reduce the ambiguity and increase the effectiveness of communication. visual communication usually consists of two elements: text and image. for text, the author often emphasizes some parts of the text by changing the visual representation of that part to better convey the intention. visual communication design tools often fail in understanding the meaning of text and user intention. they rely solely on visual attributes when suggesting emphasized textual parts and their visual representations, which often leads to the wrong text emphasis @xcite . therefore, a system that could help design tools to understand the semantic meaning and common human interpretation of the text would better assist the user to design effective visual communication. semeval 2020 task 10: emphasis selection for written text in visual media is a shared task that invites participants to propose methods that can model human emphasis selection for short written english text @xcite . the task contains several challenges. first, emphasis selection is highly subjective; different annotator may emphasize different parts of the text. second, there is a lack of additional context information; the model therefore has to rely only on the text. third, there is a relatively small amount of training data; the training set provided by the organizers contains only 2742 instances of short written text. our strategy to tackle these three challenges is to model the emphasis selection distribution of annotators by a combination of contextual representations of text extracted from a pre-trained language model and a two-staged ranking model to suggest possible tokens for emphasis. from experimental results, we demonstrate that our simple two-staged ranking model built on top of roberta (large model, fine-tuned on mnli) @xcite outperforms the lstm-based baseline system @xcite , which indicates transformer-based pre-trained language models such as roberta have a strong contextual representation power and the combination of such language models with a simple ranking model is beneficial to the problem of emphasis selection. ## system we propose a two-staged emphasis selection system. the first stage of this system functions as the initial top k word selection to select candidate words for emphasizing. the second stage then re-ranks these k words by re-assigning an emphasis score to each of them. figure 1 depicts the overall system architecture. ## experiments this section describes the experiments on selecting the embedding layer for the first stage and examining the different compositions of contextual word pairs in improving the performance of the re-ranking model at the second stage. ## conclusion in this paper, we presented our emphasis selection system for semeval 2020 task 10: emphasis selection for written text in visual media. the system is based on an embedding layer to extract contextual representation from the input sentence, followed by a two-staged ranking model to learn the emphasis selection distribution. the experimental results show that the transformer-based language model roberta provides rich contextual representation to support the proposed two-staged ranking system in outperforming the lstm-based baseline system @xcite and successfully coping with this task.