S em E val-2020 Task 10: Emphasis Selection for Written Text in Visual Media (2020)

Abstract: In this paper, we present the main findings and compare the results of SemEval-2020 Task 10, Emphasis Selection for Written Text in Visual Media. The goal of this shared task is to design automatic methods for emphasis selection, i.e. choosing candidates for emphasis in textual content to enable automated design assistance in authoring. The main focus is on short text instances for social media, with a variety of examples, from social media posts to inspirational quotes. Participants were asked to model emphasis using plain text with no additional context from the user or other design considerations. SemEval-2020 Emphasis Selection shared task attracted 197 participants in the early phase and a total of 31 teams made submissions to this task. The highest-ranked submission achieved 0.823 Matchm score. The analysis of systems submitted to the task indicates that BERT and RoBERTa were the most common choice of pre-trained models used, and part of speech tag (POS) was the most useful feature. Full results can be found on the task’s website.

## introduction in visual communication, emphasis is an intentional focus on one or more components to create a main focal point or center of interest with the composition. it has been found that it only takes the human eye 50 milliseconds to form an opinion on a visual composition @xcite . therefore, it is important for a visual element to deliver a clear message by calling attention to specific information. whether on flyers, posters, ads, social media posts or motivational messages, emphasis is usually designed to grab a viewer's attention by being distinct from the rest of the design elements. thanks to various online platforms, a massive amount of digital text is being generated by users every day. these media are filled with content competing for attention and are usually highly designed to engage viewers' attention to convey their message in the most efficient way. for textual content, word emphasis is used as a powerful tool to better convey the desired meaning of the written text to the audience. utilizing emphasis techniques can potentially add another dimension to the text through visualization. emphasis on textual content can be done with colors, backgrounds, or fonts, or with styles like italic and boldface to clarify or even change the meaning of a sentence by drawing attention to some specific information. figure 1a shows an example that is aesthetically appealing but fails to effectively communicate its intent. understanding the text would allow the system to propose a different layout that emphasizes words that contribute more to the communication of the intent, as shown in figure 1b . in the last few years, we have observed many significant improvements in various platforms for generating, formatting, and editing digital text. for example, some graphic design applications such as adobe spark foot_1 perform automatic text emphasis using templates that include images and text with different design effects. however, the used layout algorithms are often inflexible in that they rigidly emphasize words based on the visual attributes (e.g., word length) of those words, rather than the semantics of the text. as a result, the outcome may fail to accurately communicate the meaning of the written text, resulting in unintended emphasis and the wrong message to the audience. however, an emphasis selection model can potentially make better suggestions by having a better understanding of the input text. task characteristics this emphasis selection task poses new challenges associated with the nature of the task: (1) no additional context from the user or the rest of the design such as background image is provided. therefore the proposed task requires a computational understanding of the written text. expected impact of the task the ultimate goal is to enable design assistance for authors by suggesting words that are good candidates to emphasize. the typical applications of this task include, but are not limited to, creating flyers, posters, drawings, advertisements and other visual material one may find online and across social media platforms such as pinterest, instagram and snapchat. moreover, emphasis selection models have applications in many design programs such as adobe spark, apache openoffice impress, gimp, or microsoft powerpoint. ## related work we firstly introduced and formulated the task of emphasis selection in @xcite in which an end-to-end label distribution learning (ldl) model in a sequence tagging architecture is proposed to model emphasis. we evaluated the model against different baselines on the spark dataset (introduced in section 3). keyword or key-phrase detection may be the closest topic to emphasis selection. keywords can capture the main topics described in a given document @xcite . modeling keywords or key-phrases has been widely addressed in different domains such as news articles @xcite , scientific publications (nguyen and @xcite and twitter data @xcite . keyword detection mainly focuses on finding important nouns or noun phrases @xcite . in contrast, emphasis could be applied to a subset of words with different roles in a sentence. generally, word emphasis may use to express emotions, show contrast, capture a reader's interest or clarify a message. moreover, emphasis selection in social media posts deals with very short texts and the prediction needs to be made based on a single instance. in the context of expressive prosody generation, emphasis has been addressed based on acoustic and prosodic features that exist in spoken data. for example, @xcite predicted emphasized accent phrases from advertisement text information and @xcite modeled word emphasis on audience-addressed speeches. ## data collection the data used for this shared task is the integration of two datasets from different sources, which are created from scratch based on texts collected from the adobe spark and wisdom quotes website. the dataset used for this task can be found in the task's data repository foot_2 . the following are the descriptions of the two datasets. the spark dataset is collection of 1,195 instances from adobe spark foot_3 . it contains a variety of subjects featured in flyers, posters, advertisements or motivational memes on social media. the quotes dataset is a collection of quotes from well-known authors collected from wisdom quotes foot_4 with 2,681 instances. the data is split up randomly between training, development and test sets. a training data set of 2,741 instances, development set of 392 instances, and test set of 743 instances were released to the participants. fleiss' kappa score @xcite of 24.60 was observed on the data set. such a kappa score indicates the existence of multiple perspectives about emphasis in the dataset. ## data analysis many systems reported performance gain by using part of speech tags (pos) tags in their models. in this section, we analyze the effectiveness of this feature by closely examining the top 20 pos tags in our dataset. we used the stanford part-of-speech tagger @xcite to obtain pos tags for all tokens in our dataset. we divide the emphasis probabilities to four intervals (0-0.25, 0.25-0.50, 0.50-0.75 and 0.75-1.00) and compute how the pos tags are distributed in these four intervals. figure 3 shows the occurrence of the top 20 pos tags in four emphasis probability intervals for all token labels in our training set. pos tags like ",", ".", "dt" and "prp" are more favored to have low emphasis probabilities (0-0.25). interestingly, words with the highest probabilities (0.75-1.00) are usually from "nnp", "nn" and "jj" word types. as we expected, there are some general trends for emphasized words with respect to the type of words in sentences, which make pos tags a useful feature for modeling emphasis. ## evaluation metric the evaluation was performed on the test set. participants were asked to provide a real value (greater and equal to zero) for each token in the test set that indicates the probability of the token being emphasized. all models were evaluated with match m metric and ranked based on the averaged values of scores for @xmath0, 2, 3, 4. match m for each instance x in the test set d test , we selected a set s (x) m of @xmath1. . . 4} words with the top m probabilities according to the ground truth. analogously, we selected a prediction set ŝ(x) m for each m, based on the predicted probabilities. we defined the metric match m as follows: finally, we computed the average value of match m for all @xmath2. . . 4} and ranked the submitted systems based on this averaged value (rank). to better handle word duplicates, the computation is based on the position of words in a sentence rather than the actual words. note that there were many cases where two or more tokens have the exact same probability. in this case, if the model predicts either one of the labels, we considered it as a correct answer. ## baseline model we provided a baseline model for this task. this model (dl-bilstm-elmo) is a sequence-labeling model that essentially utilizes elmo contextualized embeddings @xcite as well as two bilstm layers to label emphasis. during the training phase, the kullback-leibler divergence (kl-div) @xcite is used as the loss function. more analysis and the complete description of this model is provided in @xcite . ## systems and results this task attracted 197 participants and a total of 31 teams made submissions to this task. the teams that submitted papers for the semeval-2020 proceedings are listed in the base models used in the task submissions ranged from elmo (peters et al., 2018), bert @xcite and roberta @xcite , to state-of-the-art pre-trained models such as ernie 2.0 @xcite , xlm-roberta @xcite , t5 @xcite and albert @xcite . figure 4 shows different pre-trained models used in this task. among them, bert and roberta were used most often. ensemble transformer-based models were one of the most popular approaches (26% of submissions). all submissions applied deep neural network techniques to model emphasis. moreover, some teams did explore hand-crafted features, such as part-of-speech tags, named entities, valence, arousal, dominance scores to enhance the performance of their models. ## conclusion this paper summarizes the insights gained from organizing task 10 at semeval-2020. given a short piece of text, the task consists of selecting candidate words to emphasize. we received a good number of system submissions, with 13 teams submitting a system description paper. while there were many differences between individual systems, we observed a strong trend favoring the use of transformer based models as key ingredient in the proposed architectures. many description papers present valuable analyses of the data and task. we encourage readers interested in this task to take a careful look at these papers for additional inspiration on how to improve results further.