Hitachi at S em E val-2020 Task 10: Emphasis Distribution Fusion on Fine-Tuned Language Models (2020)

Abstract: This paper shows our system for SemEval-2020 task 10, Emphasis Selection for Written Text in Visual Media. Our strategy is two-fold. First, we propose fine-tuning many pre-trained language models, predicting an emphasis probability distribution over tokens. Then, we propose stacking a trainable distribution fusion DistFuse system to fuse the predictions of the fine-tuned models. Experimental results show tha DistFuse is comparable or better when compared with a naive average ensemble. As a result, we were ranked 2nd amongst 31 teams.

## introduction this paper presents our strategy for semeval-2020 task 10, emphasis selection for written text in visual media @xcite . the task is aimed at emphasis selection, choosing candidates for emphasis in short written text in visual media @xcite . rather than predicting emphasis spans or using images, the task involves the prediction of an emphasis distribution over a short text without any image inputs. ## background the modeling of word emphasis has been widely tackled in some contexts. @xcite proposed a model for extracting key phrases from twitter text. in the context of accent, @xcite val1 fold2 fold3 fold1 fold3 fold1 fold2 val1 val2 val3 model (fold1) model (fold2) model (fold3) val2 val3 predict train folds ・・・ valid fold distfuse predict & train train with hyperparameter set 2 @xmath0. in this shared task, however, the form of emphasis is different. @xcite provided a corpus with emphasized tokens based on bio labels. the authors kept inter-subjectivity in the annotator as well as the ambiguity of the input rather than deciding gold spans. to this end, how many annotators marked a token emphasized was recorded, producing an emphasized probability over tokens. ## conclusion in this paper, we proposed a model for the task of emphasis selection. we employed seven pre-trained language models and fused them with the distribution fusion (distfuse) system. experimental results suggested that distfuse is promising in terms of boosting performance. we estimate that the effectiveness of distfuse would be further validated by additional analyses @xcite , which is future work. as additional future work, we will examine more effective ways of computing distributions.