IITK at S em E val-2020 Task 10: Transformers for Emphasis Selection (2020)

Abstract: We propose an end-to-end model that takes as input the text and corresponding to each word gives the probability of the word to be emphasized. Our results show that transformer-based models are particularly effective in this task. We achieved an evaluation score of 0.810 and were ranked third on the leaderboard.

## introduction visual communication relies on images and short texts, e.g., in flyers, posters, etc. the purpose of these is to convey the message effectively and without ambiguity. moreover, they should be able to attract the reader's attention within the first few seconds. for text, this can be achieved by laying emphasis on particular words to convey the intent better. the emphasis selection task is about designing automatic methods for choosing candidate words to be emphasized in short written texts, to enable automated design assistance in authoring (figure 1 ). the dataset provided for this task is in english, and task description paper @xcite provided by the organizers describes the task, data, evaluation, results, and a summary of participating systems. we tried two different approaches for the task. our bilstm + attention approach is inspired by the baseline paper @xcite . in this approach, we tweaked the bilstm layers @xcite , tried other layers like gru @xcite , and used character embeddings along with word embeddings @xcite . our transformers approach involves transfer learning using transformer based models @xcite . this approach involves two types of models, the first one being a transformer-based model with the bilstm layer, the attention layer @xcite , and fully connected layers on top. the second type of model involves transformer-based models with fully connected layers. we used bert @xcite , roberta @xcite , xlnet @xcite , and gpt-2 @xcite as transformer-based models. in the end, we tried the homogeneous and heterogeneous ensemble of these models. due to the nature of the problem, the systems which incorporate the whole context of a sentence would perform better. further, the small size of the dataset was a bottleneck that can be countered by using transfer learning via the pre-trained models. using transformer-based models accounts for both of these observations. our best submission achieved match m score of 0.810 (match m evaluation metric is described in section 2.2) and was ranked third on the leaderboard. our submissions under-performed in score 1 (match 1 as defined in section 2.2) as compared to the other top-performing teams. our code is available on github 2 . 2 background ## system overview our system takes as input the words in the text and corresponding to each word, gives the probability of the word to be emphasized. we tried two different types of sequence labeling approaches to learn emphasis patterns. ## experimental setup our implementation uses pytorch foot_2 library for deep learning models and the transformers library by hugging face foot_3 for the pre-trained transformer models and their tokenizers. in bilstm + attention approach, for obtaining character-level embeddings, a pair of bilstm layers with hidden size of 300 is used, whose output is passed through a highway layer and then concatenated with the glove or elmo embeddings. this concatenated vector is then passed through another pair of bilstm layers with hidden dimension of 512. for the classifier, a pair of fully connected layer is used with relu activation function (agarap, 2018) and hidden dimension of 20. finally, the output of the last fully connected layer is passed through a sigmoid layer, which gives the probabilistic score of each word. to avoid overfitting, dropout layers with a probability of 0.3 were used. in the transformers approach, the pre-trained transformer models were used without freezing the layers, and the outputs of all the layers were concatenated. for the classifier, three fully connected layers are used with relu activation function and hidden dimension of 900 and 40 for larger transformer models and 300 and 20 for normal models. dropout layers with a probability of 0.3 are also added to avoid overfitting. in both approaches, binary cross-entropy loss is used for training the model, whereas match m score is used as the performance metrics for validation (as described in section 2.2). we used adam optimizer (kingma and ba, 2014) with the learning rate set to 0.001 for bilstm + attention approach and 2e-5 for the transformers approach. the model is fine-tuned for 100 epochs in bilstm + attention approach and 30 epochs in the transformers approach, and the reported test result corresponds to the best score obtained on the validation set. ## results we attempted numerous small changes to our models in the bilstm + attention as well as the transformers approach to enhance the performance. this included hyper-parameter tuning like changing size and dimensions for different layers with some specific attempts particular to each approach. for the bilstm + attention approach, we tried training the model separately on quotes and spark dataset. we experimented using highway layers along with bilstm layers to attain a better outcome. we also tried using gru layers @xcite instead of bilstm layers. the word embeddings used in this approach are glove and elmo embeddings. we also tried incorporating a language model to the character embedding model as done in this paper @xcite , where they predict the next word using forward and backward character rnn layers whenever they encounter space in the sentence. in ## conclusion we described the systems used for submission in the task of emphasis selection for written text in visual media. the task was similar to the sequence labeling task, and hence, similar approaches can be used for this task. our main approach used transformer-based models like bert, roberta, xlnet, and their ensembles. these models are pre-trained and hence, perform better after fine-tuning on our small dataset. future work includes making an application that automates the process of poster or advertisement making. a short written text will be the input to the app. it then predicts the probability of various words to be emphasized based on predictions from our model and also adds an appropriate image as the background either using gans @xcite or using some other api. finally, it will output a poster, flyer, or advertisement.