IDS at S em E val-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize? (2020)

Abstract: We propose a novel method that enables us to determine words that deserve to be emphasized from written text in visual media, relying only on the information from the self-attention distributions of pre-trained language models (PLMs). With extensive experiments and analyses, we show that 1) our zero-shot approach is superior to a reasonable baseline that adopts TF-IDF and that 2) there exist several attention heads in PLMs specialized for emphasis selection, confirming that PLMs are capable of recognizing important words in sentences.

## introduction in terms of visual communication such as social media posts (posters, flyers, and ads) and motivational messages, text emphasis is crucial in that it facilitates the comprehension of written text and that it helps convey the author's intent. therefore, it is expected that the development of an automatic system that recommends which part to emphasize in the text of visual media will bring significant advantages; e.g., it is possible to accelerate the making process of posters and videos for advertisement. in this paper, we attempt to devise such a system solely with the aid of pre-trained language models (plms) without any task-specific laborious training. recently, there has been a substantial amount of work in the literature to figure out what knowledge transformer @xcite based plms such as bert @xcite contain and why they perform surprisingly well on various downstream tasks @xcite @xcite . among these, a group of studies has focused on analyzing plms' self-attention distributions to find some evidence that supports the existence of linguistic knowledge within the pre-trained weights of plms. specifically, @xcite investigated bert's individual attention heads to probe its ability to parse dependency trees, while @xcite proposed an unsupervised constituency parsing method applicable on top of the distributions. following the same philosophy shared by the work mentioned above, we propose a zero-shot emphasis selection method, assuming that during pre-training, some attention heads in plms are inspired to recognize which words are more important than others. we test our method on a carefully designed dataset @xcite , and it records the ranking score 0.690 on the validation set, which outperforms an intuitive baseline adopting the term frequency-inverse document frequency (tf-idf) strategy @xcite . furthermore, our method operates in a fully zero-shot manner, not leveraging gold-standard annotations provided by the dataset at all, implying its universal applicability in a low/zero-resource regime. in detail, as shown in for evaluation, we use the m atch m (@xmath0, 2, 3, 4]) score which is calculated as: ## results in here we mention several takeaways from our results. first, particular plms (gpt-2, xlnet) report comparably lower ranking scores against the other plms and the tf-idf baseline. since gpt-2 adopts a transformer decoder to its architecture, its attention distribution is leaned on the first word of a sentence @xcite . this results in a sub-optimal solution where each first word for all 390 sentences in the validation set becomes the top-1 e f req high score word. on the other hand, xlnet model tends to focus more on punctuation marks such as period (.) and comma (,). specifically, the xlnet-base model predicts that a period should be one of the top-4 high e f req words for 289 sentences in the validation set, even though the prediction is correct only for 48 cases. ## conclusion we have proposed a zero-shot emphasis selection method, focusing on investigating whether pre-trained language models contain enough knowledge to select proper words to be emphasized. we have found that some plms report comparable performance, confirming that some specialized attention heads of plms have ability to detect meaningful words.