T ë X tmarkers at S em E val-2020 Task 10: Emphasis Selection with Agreement Dependent Crowd Layers (2020)

Abstract: In visual communication, the ability of a short piece of text to catch someone’s eye in a single glance or from a distance is of paramount importance. In our approach to the SemEval-2020 task “Emphasis Selection For Written Text in Visual Media”, we use contextualized word representations from a pretrained model of the state-of-the-art BERT architecture together with a stacked bidirectional GRU network to predict token-level emphasis probabilities. For tackling low inter-annotator agreement in the dataset, we attempt to model multiple annotators jointly by introducing initialization with agreement dependent noise to a crowd layer architecture. We found our approach to both perform substantially better than initialization with identities for this purpose and to outperform a baseline trained with token level majority voting. Our submission system reaches substantially higher Match m on the development set than the task baseline (0.779), but only slightly outperforms the test set baseline (0.754) using a three model ensemble.

## introduction emphasis selection is the task of choosing individual words or phrases of short written texts to emphasize. while emphasis selection has also been used in previous work for more natural articulation in text-tospeech systems @xcite , in the semeval shared task emphasis annotations are intended for use in visual communication such as in posters or advertisements @xcite . word emphasis helps readers conveniently scan for important information in a text. capital letters, bold letters, italics and underlines can help emphasize certain facts and drive the message home; however, overuse of these features can lead to confusion and create an unattractive aesthetic. usually advertisers focus on one or two key words or phrases that will pique the readers' interest @xcite . as with many tasks based on sentence modeling, substantial improvements have been gained in emphasis selection by leveraging recent breakthroughs in language modeling such as the elmo architecture @xcite . following this trend, our submission uses contextualized word representations from a pretrained model of the state-of-the-art bert architecture @xcite together with a stacked bidirectional gru network for learning task specific information. a major challenge when developing a system for automatic emphasis selection are very low agreements between annotators about which parts of a sentence should be emphasized which impacts how models are trained and evaluated @xcite . to tackle annotator subjectivity, both majority voting @xcite and regression with a kl divergence objective @xcite have been used for training in previous work. in our approach, we apply a crowd layer architecture adapted from @xcite and improve its effectiveness for the task by introducing agreement dependent noise during initialization. ## data a combination of two english datasets was provided for this task @xcite . in total, the training and development sets comprise of 3,106 short text instances (sentences). both datasets consist of tokenized sentences with annotations by nine different annotators for each token using an inside-outside-beginning (iob) tagging scheme. this is a scheme whereby tokens are tagged as either i when inside an emphasized chunk, b when at the beginning of a chunk or o when outside of a chunk @xcite . the frequencies of the iob tags are then counted and the sum of the b and i tags is divided by the number of annotators for each token. this then indicates the emphasis probability which is used for evaluation @xcite . the tokens with the highest probability score are more likely to be the appropriate candidates for emphasis. the first dataset is from adobe spark, and contains 960 short text instances from flyers, posters, advertisements or motivational memes on social media. it contains 5,940 tokens and the average number of tokens per instance is 6.18, ranging from 1 to 27 tokens. the average emphasis probability per token is 0.38. the second dataset is a collection of quotes from well-known authors collected from wisdom quotes containing 2,174 short texts. it contains 30,845 tokens and the average number of tokens per instance is 14.18, ranging from 3 to 38 tokens. the average emphasis probability per token is 0.26. the light's kappa @xcite between all raters in the task training set is 0.23. in comparison, the kappa agreement between the four professional labelers in the speech emphasis dataset used by @xcite for an expressive text-to-speech system is only slightly higher at 0.35. this suggests that emphasis annotations are very subjective and that this is a common issue across domains which is important to tackle in order to achieve good system performance. figure 1: schematic of our architecture with its three variants differing in the final layers: majority voting, crowd layer and an optional attention layer before the crowd layer. in the variants using a crowd layer, the output of the shared softmax is used directly at inference time. our neural network architectures make use of state-of-the-art language modeling of the input sentences by using pretrained bert embeddings @xcite followed by a bi-directional gru layer @xcite with a feedforward network for each time step for modeling task specific information. the architecture and its variants are illustrated in figure 1 . since the bert model we use was trained on wordpiece tokens @xcite , token boundaries in the pre-tokenized training data are very different in some cases. while token boundaries are generally similar for highly frequent words, wordpiece splits up words which are rare or were not present in the training data into smaller subword units @xcite . to accommodate for this, we construct word vectors for the original tokens in the data by further tokenizing each token separately into its subword units following the same wordpiece segmentation rules as in the pretrained model. the subword units for each of the original tokens are then concatenated to form the entire sentence and passed into the bert embedding model where contextualized word representations are extracted from the final hidden states. finally, we compute a single vector representation for the original tokens by applying max pooling over the subword vectors which we found to work slightly better than mean pooling in preliminary experiments. for modeling task specific information we then feed the resulting vectors into a component-level dropout layer @xcite for regularization after which they are passed into a bidirectional rnn layer. while lstms are the preferred rnn cells in previous work @xcite , we achieved slightly higher scores using gru cells instead which also have the added benefit of being faster to train due to their simplified gating mechanism @xcite . we use a stack of two bidirectional gru (bigru) layers, apply dropout between layers as regularization and use learned matrices as the forward and backward initial states instead of using zero states for minor performance improvements. the forward and backward hidden states of the last gru layer are then concatenated into single vectors for each token in a sentence and passed into a feedforward network. our feedforward layers are pre-activated by the tanh function. for regularization, all feedforward layers are preceded by layer normalization followed by dropout inspired by ic-layers @xcite . the final feedforward layers are additionally followed by another pair of layer normalization and dropout and a relu activation. similarly to @xcite , we also tested an architectural variant which includes an attention layer over final feedforward layer outputs where scores are computed via bahdanau-style attention @xcite between each output vector and the mean of all output vectors in the sentence: here, f i is the output of the feedforward network at time step i, n is the number of words in a sentence, w and u are learned weights of single vectors and the mean of all vectors in the sentence respectively, and c i is the product of an output vector f i with its corresponding attention weight α i . we primarily experimented with two techniques for dealing with labels from different annotators. as a baseline, we used a single softmax activated projection layer trained on labels obtained via token level majority voting and optimized with a cross-entropy objective. in our second approach, we attempt to model consensus between annotators in the dataset explicitly in a shared representation by parameterizing the annotators using a crowd layer @xcite . ## evaluation our models were implemented in python with the pytorch framework foot_1 @xcite . we used the pre-trained cased bert model ("bert-cased") from @xcite as implemented in python in the "transformers" library foot_2 @xcite with which we compute 768-dimensional contextualized word embeddings together with our pooling approach described in section 3. in all models we used two bigru layers with 250 neurons each and 2 feedforward layers with 150 neurons each. for the crowd layer we tested three different initializations of the annotator weights which vary by the standard deviation (σ) of normal noise added to identity matrices to investigate the effectiveness of our approach. we trained models using identities without noise (σ = 0) as in @xcite , a standard normal distribution (σ = 1) and our agreement dependent approach (σ = 0.77). for regularization, we used dropout rates of 0.1 on the input and 0.2 for all other dropout layers between bigru and feedforward layers. our networks were trained using the adam optimizer (kingma and ba, 2014) with a learning rate of 0.001 and training was stopped after two epochs of no improvement on development set loss. in addition to single models, we also evaluated some simple ensembles of different architectural variants. scores from ensembles were obtained by averaging output probabilities of the constituent models for each token. we compare scores using the match m metric with @xmath0. . . 4} which was used as the evaluation metric for the shared task @xcite . ## conclusion we presented a gru-based recurrent architecture for emphasis selection using state-of-the-art language modeling with bert and introduced agreement dependent noise initialization to adapt a crowd layer to modeling annotators for a dataset with very low inter-annotator agreement which we showed to outperform initializations with only identities or addition of standard normal noise. while our architecture outperformed the task baseline substantially on the development set only one of our ensemble systems