Sentence Structure and Word Relationship Modeling for Emphasis Selection (2021)

Abstract: Emphasis Selection is a newly proposed task which focuses on choosing words for emphasis in short sentences. Traditional methods only consider the sequence information of a sentence while ignoring the rich sentence structure and word relationship information. In this paper, we propose a new framework that considers sentence structure via a sentence structure graph and word relationship via a word similarity graph. The sentence structure graph is derived from the parse tree of a sentence. The word similarity graph allows nodes to share information with their neighbors since we argue that in emphasis selection, similar words are more likely to be emphasized together. Graph neural networks are employed to learn the representation of each node of these two graphs. Experimental results demonstrate that our framework can achieve superior performance.

## introduction emphasis selection recently proposed by @xcite aims to select candidate words for emphasis in short sentences. by emphasizing words, people's intent can be better conveyed, which is useful in a variety of applications. for example, it can be used in spoken language processing to generate more expressive sentences and be used to enable automated design assistance in authoring, i.e., labeling important parts in a paragraph or in a poster title. although it seems that this task is highly similar to the task of keyword extraction @xcite , these two tasks are fundamentally different. the first difference is that keyword extraction focuses on a paragraph which is composed of multiple sentences while emphasis selection aims to choose words from a short sentence. this difference implies that modeling sentence structure is more effective in emphasis selection. the second difference is that many global word statistics methods employed in keyword extraction such as tf-idf and word co-occurence frequency will not work in this task, because for short sentences, it is meaningless to count word frequency and whether the word should be emphasized has nothing to do with the frequency of the word. in addition, keyword extraction requires that the collected keywords are diverse, which means that if two words have similar meaning, only one should be kept. however, in emphasis selection, similar words tend to be emphasized together. emphasis selection also shares some resemblances with entity recognition @xcite . but one major difference is that the parts of speech of emphasized words are more diverse and the relation of adjacent words is weaker in the emphasis selection task. generally speaking, emphasis selection can be modeled as a sequence classification task where the input is a sentence and the output is each word's probability to be emphasized. @xcite propose a model which is based on the recurrent neural network @xcite and kl-divergence loss function. despite the fact that it looks like a straightforward task, there still exist some challenges. the first challenge is about how to incorporate sentence structure information into the model. sentence structure information includes what role (subject, predicate, object, etc.) the word plays as well as the position of the word in a sentence. obviously, this kind of information is very useful. existing works @xcite fail to model the global structure of a sentence. the second challenge is that there is no given context except a short sentence, so it requires the model to be able to capture some common patterns or regularities of most people. more concretely, if two words are similar, they are more likely to be emphasized together. for example, in figure 1 , persistence and victory are more likely to be emphasized together. this observation can also be found in the second example: never and impossible. moreover, we analyze the training dataset and get a more concrete understanding of this phenomenon through the following procedures: for each training sentence, we consider the most popular emphasized word called word a. then, we identify the most similar word called word b to the word a based on glove embedding @xcite . we find that the word b is also emphasized with a higher probability than other words in this sentence and this phenomenon occurs in about 26% of the training dataset. therefore, modeling this kind of relationship between words definitely can help improve the performance of models. in this paper, we propose a sentence structure graph to handle the sentence structure issue. specifically, the sentence structure graph is derived from the parse tree of a sentence which contains useful information for this task. for example, as illustrated in figure 2 , when the path is @xmath0, the word i is not inclined to be emphasized since this path indicates that this word is a subject. however, when the path is @xmath1, the word basketball is likely to be emphasized since this word is a noun in a verb phrase. generally, such sentence structure graph can reveal the role of words in a sentence which is beneficial for emphasis selection. another important information -word relationship information is captured by a word similarity graph. through the word similarity graph, words can share information with their neighbours, resulting in similar emphasized probabilities of similar words. next, graph neural networks @xcite @xcite @xcite which has been demonstrated effective in modeling graph structure data are employed to learn the representation of each node of these two graphs. we conduct extensive experiments based on different word embeddings, i.e., glove @xcite , elmo @xcite , roberta @xcite , 2019) and the experimental results show that our model can achieve superior performance. ## related work emphasis selection is a new task proposed by @xcite which aims to choose a subset of words to emphasize in a sentence. shirani et al. @xcite propose a model which is based on the recurrent neural network @xcite . kl-divergence loss function is adopted to conduct the label distribution learning (ldl) @xcite . this method achieves competitive performance over the sequence labeling model: crf @xcite . in recent years, graph neural networks @xcite @xcite @xcite have demonstrated superiority in modeling the structure of graphs. @xcite propose a graph convolutional network which is based on the fourier theory. one drawback of this model is that the edge weight of the graph needs to be known in advance. to overcome this shortcoming, petar et al. @xcite use a masked self-attention layer to calculate the weight of node's neighbours dynamically and then aggregate information by conducting a weighted addition operation. currently, graph neural networks are applied to various tasks. feria et al. @xcite construct a word graph by calculating the word embedding similarity and apply the community detection algorithm to find different communities. through the graph, they can find named entities for a bilingual language base in an unsupervised manner. sun et al. @xcite put forward a diverse graph pointer network for keyword extraction. they first construct a word graph based on the distance of two words and then use the graph convolutional network as an encoder to obtain each node's representation, finally a pointer network decoder and the diverse mechanism are employed to generate diverse keywords. the graph encoder can capture document-level word salience and overcome the long-range dependency problem of rnn. ## methodology we follow the same problem setting given by @xcite . suppose a sentence is composed of n words @xmath2, x 2 , ..., x n ). our goal is to obtain a subset s of words in c as selected words for emphasis where 1 ≤ |s| ≤ n. we model this task as a prediction problem: figure 3 depicts the architecture of our proposed model which is composed of three parts: (i) the middle part -sequence encoder (ii) the left partword similarity graph encoder (iii) the right part -sentence structure graph encoder. next, we will provide a detailed description of each part. ## conclusions the sentence structure graph and the word similarity graph are proposed to solve two issues found in emphasis selection. the sentence structure graph helps to model the structure information of the sentence and the word similarity graph is useful in modeling relationships between words. with the development of graph neural network, the two graphs can be properly encoded and integrated into existing models. experimental results demonstrate that our framework can achieve superior performance.