EL - BERT at S em E val-2020 Task 10: A Multi-Embedding Ensemble Based Approach for Emphasis Selection in Visual Media (2020)

Abstract: In visual media, text emphasis is the strengthening of words in a text to convey the intent of the author. Text emphasis in visual media is generally done by using different colors, backgrounds, or font for the text; it helps in conveying the actual meaning of the message to the readers. Emphasis selection is the task of choosing candidate words for emphasis, it helps in automatically designing posters and other media contents with written text. If we consider only the text and do not know the intent, then there can be multiple valid emphasis selections. We propose the use of ensembles for emphasis selection to improve over single emphasis selection models. We show that the use of multi-embedding helps in enhancing the results for base models. To show the efficacy of proposed approach we have also done a comparison of our results with state-of-the-art models.

## introduction the semeval-2020 task 10 @xcite challenge focuses on emphasis selection in visual media. emphasis is the process of giving importance to some parts of communication to convey the message in a better way. it is used to draw the attention of readers to a specific section of the information. it is also used for removing the ambiguity in the message. in vocal communication, the emphasis is generally conveyed by giving stress on the specific word. in visual communications like flyers, posters, advertisements, the emphasis is conveyed by using different fonts, colors, or backgrounds for the text. text designing systems like adobe spark 1 can automatically provide template-based layouts for text. however, these algorithms generally rely on the visual features of the text like word length and suggest designs based on those features. sometimes, this type of method does not emphasize on the proper words and might not help in conveying important information or may even convey wrong information. in fig 1a , we show the automatic design provided by adobe spark. even though figure 1a is aesthetically pleasing, it is not giving emphasis to essential words and might fail in conveying the message. instead, figure 1b uses a different layout and emphasizes on vital words. given only the text and not the intent of the message, there can be multiple valid emphases, and different authors can prefer to emphasize different words. therefore, we cannot use a single label to say whether a word should be emphasized or not. to tackle this, we use learning label distribution (ldl) @xcite . ldl is used for assigning a real number to each word showing the probability of a word being emphasized. our main contributions can be summarized as: ## related work in text data, the majority of works focus on important keyword identification from long texts. there are mainly two methods for keyword extraction: supervised and unsupervised. supervised methods generally treat the keyword identification as a classification problem and classify words as either keyword or not @xcite @xcite . unsupervised methods usually utilize tf-idf (hasan and ng, 2010) scores or clustering methods @xcite for keyword identification. recently, @xcite also proposed a model using rnns for keyword identification. different methods are proposed for the emphasis selection in audio. most works use acoustic features like loudness, pitch to detect emphasized words in audio data @xcite . recently, some works are proposed to predict word emphasis in text to improve text-to-speech (tts) systems @xcite . @xcite proposed ensemble-based models for emphasis selection in audio. @xcite proposed the use of label distribution learning (ldl) for emphasis selection in short texts in visual media. here, we show that the use of ensemble models trained with different embeddings performs better than base models. ## model we use the dl-bilstm model proposed by @xcite as our base model, to the best of our knowledge this model provides state-of-the-art results for emphasis selection in visual media. each word of the input sequence is represented with word embedding. part-of-speech (pos) and sentence embedding are added as additional information. two bidirectional lstm @xcite layers are used to capture the sequence information. the last layer of the model is fully-connected and assigns probability using the hidden state of lstm. figure 2 shows the overall architecture of the dl-bilstm model. we have used different sets of embedding to train the models. wordbert and sentbert represent word, and sentence embedding generated using pre-trained bert @xcite , respectively. posembd represents one-hot-encoded pos tag. in model-4, model-5 and model-6, we use elmo @xcite as word embedding. below is the list of models and used embeddings: model-1: dl-bilstm + wordbert model-2: dl-bilstm + wordbert + posembd model-3: dl-bilstm + wordbert + posembd + sentbert model-4: dl-bilstm + elmo model-5: dl-bilstm + elmo + posembd model-6: dl-bilstm + elmo + posembd + sentbert ## experimental settings we have used bert-as-service framework foot_1 and 1024 dimension pre-trained cased bert embedding for encoding the words and sentences. we have used universal pos tagger from nltk foot_2 to get the pos tags for sentences, and then have used a bag of word embedding to encode pos tag information. we have also used 2048 dimension elmo embedding for encoding word information. the size of the bilstm layer in the model depends on the set of embeddings used, for training the dl-bilstm model, we have used kl-divergence loss, and for training sl-bilstm model, we have used negative log-likelihood loss. we have also used different ensembles of the above models, the final score of an ensemble is the average of scores provided by all the models. ## results as proposed by @xcite , we have used match m score for evaluation of our models. ## conclusion & future work we show that the use of multiple embeddings for encoding different information (i.e., word and sentences) can help in improving model performance. we also show that ensemble models perform better for emphasis selection then single base models. in the future, we will work on ensembles with different model architectures and methods for emphasis selection. we will also work on generating task-specific word embedding for emphasis selection.