FPAI at S em E val-2020 Task 10: A Query Enhanced Model with R o BERT a for Emphasis Selection (2020)

Abstract: This paper describes the model we apply in the SemEval-2020 Task 10. We formalize the task of emphasis selection as a simplified query-based machine reading comprehension (MRC) task, i.e. answering a fixed question of “Find candidates for emphasis”. We propose our subword puzzle encoding mechanism and subword fusion layer to align and fuse subwords. By introducing the semantic prior knowledge of the informative query and some other techniques, we attain the 7th place during the evaluation phase and the first place during train phase.

## introduction the semeval-2020 task 10 focuses on the emphasis selection in visual text media, i.e. choosing candidates for emphasis in short written text, to enable automated design assistance in authoring @xcite . in this paper, we propose a novel model enhanced by an informative query with roberta @xcite as the backbone to select the emphasis in a sentence. we are inspired by @xcite who introduced a mrc approach to improve the performance of the ner task by querying differently for diverse named entities. we reform the mrc query and formalize the emphasis selection task as a simplified question answering task in machine reading comprehension (mrc). that is, given a natural language query, emphasis tokens are extracted to answer the query. for example, the task of selecting emphasis tokens "dangerous", "evolve", "not", "to" in "what 's dangerous is not to evolve" is formalized as an mrc task of answering a fixed query, "find candidates for emphasis". additionally, we propose subword puzzle encoding and the subword fusion layer to solve the token alignment problem gained from roberta. our work shows that the informative query is beneficial in the case of emphasis selection. ## background the purpose of this task is to design automatic methods for selecting emphasis on visual media such as flyers, posters and ads. the selected emphasis, written in english, is highlighted to grab a viewer's attention. the competition dataset consists of 3,143 sentences with labels and 743 sentences without labels. details about the dataset are introduced in @xcite and @xcite . most of sentences in the dataset contain a few words, usually less than 20. ## model figure 1 depicts our proposed model: given a sentence, we apply concatenation to the uniform query and the sentence to obtain the input. then we apply subword puzzle encoding to store the affiliation information of subwords. after that, we apply roberta to get the representation of each token, i.e. the embeddings from the last two hidden layers in roberta. they are then fused in the subword fusion layer and output by a fully connected layer to produce regression score. the model is fed with chunks of sentences and a uniform query is combined with each sentence in a sentence pair format foot_0 at the entrance. then it is processed by the subword puzzle and fed into roberta for encoding. the fused embedding, @xmath0. in the end, the aligned embeddings are sequentially passed through a dropout layer and a full connected layer with the sigmoid activation to produce a logit score for each token. ## experiments in the following, we show the parameters we used for replication and discuss how to select hidden layers and queries. we show the improvement using pseudo labels and deliver an introduction about an attempt on a man-revision. ## conclusion we reformulate the task of emphasis selection as an mrc task with a fixed query, "find candidates for emphasis" and apply roberta-base as the backbone language model. by adopting the techniques of subword fusion, pseudo labels, we attain an average score of 0.796 (match-1: 0.690, match-2: 0.780, match-3: 0.840, match-4: 0.873) and achieve the 7 th place in the evaluation phase and the first place in the train phase. our implementation shows that 1) reformulating the task into an mrc task with the subword fusion technique is effective; 2) embeddings of the last two layers of roberta absorb most of the information; 3) pseudo labels can be applied to improve the model performance; 4) the man-revision does not help with the model performance improvement.