Fine-grained Information Status Classification Using Discourse Context-Aware BERT (2020)

Abstract: Previous work on bridging anaphora recognition (Hou et al., 2013) casts the problem as a subtask of learning fine-grained information status (IS). However, these systems heavily depend on many hand-crafted linguistic features. In this paper, we propose a simple discourse context-aware BERT model for fine-grained IS classification. On the ISNotes corpus (Markert et al., 2012), our model achieves new state-of-the-art performances on fine-grained IS classification, obtaining a 4.8 absolute overall accuracy improvement compared to Hou et al. (2013). More importantly, we also show an improvement of 10.5 F1 points for bridging anaphora recognition without using any complex hand-crafted semantic features designed for capturing the bridging phenomenon. We further analyze the trained model and find that the most attended signals for each IS category correspond well to linguistic notions of information status.

## introduction information structure @xcite @xcite @xcite @xcite studies structural and semantic properties of a sentence according to its relation to the discourse context. information structure affects how discourse entities are referred to in a text, which is known as information status @xcite @xcite . specifically, information status (is henceforth) reflects the accessibility of a discourse entity based on the evolving discourse context and the speaker's assumption about the hearer's knowledge and beliefs. for instance, according to @xcite , old mentions 1 refer to entities that have been referred to previously; mediated mentions have not been mentioned before but are accessible to the hearer by reference to another old mention or to prior world knowledge; and new mentions refer to entities that are introduced to the discourse for the first time and are not known to the hearer before. in this paper, we mainly follow the is scheme proposed by @xcite and focus on learning fine-grained is on written texts. a mention's semantic and syntactic properties can signal its information status. for instance, indefinite nps tend to be new and pronouns are likely to be old. moreover, referential patterns of how a mention is referred to in a sentence also affect this mention's is. in example 1, "friends" is a bridging anaphor even if we do not know the antecedent (i.e., she); while the information status for "friends" in example 2 is mediated/worldknowledge. section 3.1 analyzes the characteristics of each is category and the relations between is and discourse context. in this work, we propose a simple yet effective discourse context-aware self-attention model based on bert @xcite for fine-grained is classification. we find that the sentence containing the target mention as well as the lexical overlap information between the target mention and the preceding mentions are the most important discourse context when assigning is for a mention. with the self-attention mechanism, our model can capture important signals within a mention and the interactions between the mention and its context. on the isnotes corpus @xcite , our model achieves new state-of-theart performance on fine-grained is classification, obtaining a 4.8 absolute overall accuracy improvement compared to @xcite . more importantly, we also show an improvement of 10.5 f1 points for bridging anaphora recognition without using any sophisticated hand-crafted semantic features. furthermore, to gain additional insights into our model's predictions, we analyze the attention mechanisms of our trained model. we find that the most attended tokens for each is category correspond well with linguistic features of information status. for instance, for the old is category, the most attended token list includes pronouns such as "she", "her", and "it". while for the new category, the model pays more attention to indefinite determiners such as "a" and "an". section 6 provides a detailed analysis of the attention map for each is category. to summarize, the main contributions of our work are as follows: another work on is classification was carried out by @xcite . they assumed that the distribution of is classes within sentences tends to have certain linear patterns, e.g., @xmath0. under this assumption, they trained a crf model with syntactic and surface features for fine-grained is classification on the german dirndl radio news corpus @xcite . recently, @xcite adapted eight rules from @xcite to recognize bridging anaphors and find their antecedents in the improved annotations of the extended dirndl corpus @xcite . different from the above-mentioned work, we do not use any complicated hand-crafted features, and our model improves the previous state-of-the-art results on both overall is classification accuracy and bridging recognition by a large margin on the isnotes corpus. our model also achieves competitive results for fine-grained is classification on switchboard compared to the approach in @xcite that uses the stanford coreference resolver and an svm classifier that explores 18 carefully designed hand-crafted rules. fine-tuning with contextual word embeddings. recent studies @xcite have shown that a range of downstream nlp tasks benefit from fine-tuning task-specific parameters with pre-trained contextual word representations. our work belongs to this category and we fine-tune our model based on bert representations @xcite . the novelty of our approach is that we create a "pseudo sentence" for each mention that encodes the most effective local and global discourse context for predicting the mention's is. the self-attention mechanism in transformer's self-attention encoder @xcite allows our model to attend to both the context and the mention itself for clues that are helpful to predict the mention's is. model probing. recently, there have been a number of studies exploring the types of knowledge encoded in the bert model. @xcite found that internal vector representations in bert encode rich linguistic information, with surface information at the bottom layers, syntactic information in the middle layers, and semantic information at the top layers. @xcite showed that certain attention heads in bert correspond well to the linguistic knowledge of syntax and coreference. in our work, we demonstrate that the attention patterns in our trained model embed linguistic notions of information status. ## experimental results on the switchboard is corpus in this section, we apply our discourse context-aware self-attention model to the switchboard dialogue is corpus @xcite . the corpus contains around 63k mentions annotated with is types (i.e., old, mediated, and new) and subtypes. note that the is scheme in this corpus is different from the one in isnotes in terms of fine-grained is classes. in general, bridging in this corpus includes non-anaphoric, syntactically linked part-of and set-member relations (e.g., the house's door), as well as comparative anaphors that are marked by surface indicators such as "other" or "different". 7 nevertheless, we think the ## attention to linguistic features in order to gain additional insights into our model's predictions, we analyze the attention maps in our best model (self-attention with bert large ) that is trained on isnotes. we aim to check to what extent the most attended tokens correspond to the linguistic features for each is class. specifically, we randomly choose one fold and apply the trained model to the testing dataset. since the "[cls]" token is used for prediction, we analyze the attention weights assigned to other tokens from "[cls]" for each testing instance. the weight of each token is normalized by the sequence length. the final attended score for each token is calculated by aggregating normalized attended weights across all testing instances in all 16 heads from the last layer. this is because previous work suggests that the last layer usually encodes the task-specific features in fine-tuning @xcite . we notice that a lot of the attended tokens in it is interesting to note that the model seems to learn the internal syntactic/semantic structure for a few is classes. for instance, "of" and "'s" are strong signals for m/syntactic mentions that have a prepositional structure or a possessive structure. also m/aggregate mentions usually contain the tokens "and/or" that indicate the coordination structure. similarly, for m/comparative category, the model learns to focus on a few premodifiers (e.g., "more", "other", and "higher") that indicate the comparison between two entities. finally for m/function mentions, the model learns to mostly focus on numbers. surprisingly, the model also learns to attend the verb "fell", which corresponds well with the definition of this is class (see section 3.1). for the most difficult category m/bridging, it seems that the model attends to some relational nouns (e.g., "friends" or "demand") that are likely used as bridging anaphors. ## conclusions we propose a simple discourse context-aware self-attention model for is classification based on the bert fine-tuning framework. we cast the is classification problem as a sentence classification task by creating a novel "pseudo sentence" for each mention. we design the "pseudo sentence" based on the linguistic intuitions about is and it contains most indicative context information to predict a mention's information status. such design allows the model to capture both clues from the mention and its context when predicting is. our model does not contain any complex hand-crafted semantic features and achieves the new state-ofthe-art results for is classification and bridging anaphora recognition on isnotes that contains written news articles. in another domain that consists of conversational dialogues (switchboard), our model also achieves competitive performance for fine-grained is classification compared to previous work @xcite . finally, in order to better understand our model's predictions, we probe our best model (self-attention with bert large ) on isnotes. we find that our model learns to pay more attention to signals that correspond well to the linguistic features of each is class.