ERNIE at S em E val-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model (2020)

Abstract: This paper describes the system designed by ERNIE Team which achieved the first place in SemEval-2020 Task 10: Emphasis Selection For Written Text in Visual Media. Given a sentence, we are asked to find out the most important words as the suggestion for automated design. We leverage the unsupervised pre-training model and finetune these models on our task. After our investigation, we found that the following models achieved an excellent performance in this task: ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. We combine a pointwise regression loss and a pairwise ranking loss which is more close to the final Match m metric to finetune our models. And we also find that additional feature engineering and data augmentation can help improve the performance. Our best model achieves the highest score of 0.823 and ranks first for all kinds of metrics.

## introduction emphasis selection for written text in visual media is proposed by @xcite and @xcite . the purpose of this shared task is to design automatic methods for emphasis selection, i.e. choosing candidates for emphasis in short written text, to enable automated design assistance in authoring. for example, @xcite mentions that such a technique can be applied to some graphic design applications such as adobe spark to perform automatic text layout using templates that include images and text with different fonts and colors. the major challenge is that given only thousands of annotated short text data without any context about the text or visual background images, we are asked to learn the author-or domain-specific emphatic about the short text. besides, these short text data are annotated by crowd-sourcing workers. and we find that different annotators have different standards, which increases the difficulty of this task. to identify the most important words, we model the task as a sequential labeling problem. our base models leverage different unsupervised language model such as ernie 2.0 @xcite , xlm-roberta @xcite , roberta @xcite and albert @xcite . these large unsupervised models are pre-trained on a large amount of unannotated data and carry valuable lexical, syntactic, and semantic information in training corpora. our approach is as follows: firstly, the word-level output representations for the sentence are computed by pre-trained models and then fed into a designed downstream neural network for word selections; secondly, we finetune the downstream networks together with the pre-trained models on the annotated training data; thirdly, we investigate several different objective functions to learn our model; and finally, we apply feature engineering and several data augmentation strategies for further improvement. the rest of the paper is organized as follows. in section 2, we will briefly overview some related works to our system. section 3 shows the details of our approach. our experiments will be shown in section 4, and section 5 concludes. ## related work recently pre-trained models have achieved state-of-the-art results in various language understanding tasks such as bert @xcite , xlm-roberta @xcite , roberta @xcite , albert @xcite and ernie 2.0 @xcite . @xcite first introduced a bidirectional encoder representation from transformers called bert. they developed several pretraining strategies such as masked language models and a next sentence prediction task. since then, many studies about pretraining strategies come out and most of them share similar neural network architectures but with different pretraining schemes. for example, roberta @xcite finds that dynamically changing the masking for the masked language model and removing the next sentence prediction task can help with improving the performance in downstream tasks. albert @xcite adds sentence order prediction task and optimizes the memory usage of original bert to achieve better results. xlm-roberta @xcite trains its models over one hundred languages with multilingual settings and gets the first single large model for all languages. ernie 2.0 @xcite ) is an improvement of ernie 1.0 @xcite and the world's first model to score over 90 in terms of the macro-average score on glue benchmark @xcite . ernie 1.0 @xcite introduces knowledge masking strategies. it gains a large benefit from entity-level and phrase-level masked language models. ernie 2.0 @xcite proposes a continual pre-training framework that incrementally builds pre-training tasks and then learns pre-trained models on these constructed tasks via continual multi-task learning. ernie 2.0 constructs three kinds of tasks including word-aware tasks, structure-aware tasks and semantic-aware tasks. all of these pre-training tasks rely on self-supervised or weak-supervised signals that could be obtained from massive data without human annotation. a continual multi-task learning method is proposed to improve the model's memory over different pre-training tasks. the researchers of ernie 2.0 released a new version recently which made a few improvements on knowledge masking and application-oriented tasks, aiming to advance the model's general semantic representation capability. in order to improve the knowledge masking strategy, they proposed a new mutual information-based dynamic knowledge masking algorithm. they also constructed specific pretraining tasks for different applications. for example, they added a coreference resolution task to identify all expressions in a text that refer to the same entity. details can be found on the blog foot_1 . following the current trends of pre-training and fine-tuning paradigm for natural language processing, our system adopts these models as our base word and sentence representation. ## experimental results all experiments are executed on an nvidia v100 gpu. each model runs 10 epochs with early stopping strategies based on the performance in the validation set. since the training and validation data are both small, we find that models have a large variant performances among each run. 2 so we combine all the provided training and validation sets, then split them in a random 8-fold settings. for each model in each fold, we run five times and we will report the average score on our 8-fold settings to get a much more stable analysis. ## conclusion in this paper, we present our system that ranks first in semeval-2020 task 10. our solution contains several strategies and we provide detailed experiments to analyze which of them are effective. our experiments show that models empowered by pre-trained language models are most effective, especially for ernie 2.0. besides, lexical features, pairwise loss, and data augmentation can also bring improvement for some of our models.