A ttending to L ong- D istance D ocument C ontext for S equence L abeling (2020)

Abstract: We present in this work a method for incorporating global context in long documents when making local decisions in sequence labeling problems like NER. Inspired by work in featurized log-linear models (Chieu and Ng, 2002; Sutton and McCallum, 2004), our model learns to attend to multiple mentions of the same word type in generating a representation for each token in context, extending that work to learning representations that can be incorporated into modern neural models. Attending to broader context at test time provides complementary information to pretraining (Gururangan et al., 2020), yields strong gains over equivalently parameterized models lacking such context, and performs best at recognizing entities with high TF-IDF scores (i.e., those that are important within a document).

## introduction many of the main datasets used in nlp are comprised of relatively short documents: english ontonotes @xcite , for example, contains an average of 223 tokens per document, the wsj portion of the penn treebank @xcite averages 501 tokens, the imdb dataset @xcite averages 272 tokens, and squad 2.0 @xcite ) contains an average of 134 tokens per passage. this focus has, in turn, led to the development of models specifically optimized for the characteristics of short documents, including a pervasive focus on the sentence as the atomic unit of analysis for such tasks as ner and parsing, and influencing the maximum context * work completed while at uc berkeley. length of contextual language models like bert @xcite to be limited to 512 tokens. at the same time, however, longer documents are increasingly the objects of empirical study in areas as diverse as computational social science and the digital humanities-including novels @xcite , scientific articles @xcite and political manifestos @xcite . these long documents present not only challenges for nlp (such as any task, like coreference resolution, whose computational complexity is superlinear in the size of the document) but opportunities as well, since the longer document context presents greater opportunity for learning better representations. recent work in nlp has begun exploring this link between longer documents and representation learning. first, while contextualized models (e.g. @xcite generally consider the context of a few sentences, several recent advancements have enabled significantly longer input sequences (e.g. @xcite @xcite ; most, however, are either incapable of processing book-level documents or prohibitively resource-intensive for standard use. second, domain-and task-adaptive pretraining has proven especially effective for adapting the weights of general-purpose language models to the distribution of a particular domain or task @xcite @xcite . while longer documents are able provide more context for these models to adapt to, pretraining operates at the broad level of a domain, and is unable to exploit new con-text at evaluation time in unseen test documents. to highlight the value of considering document context at test time, consider the following sentence from e.m. forster's a room with a view (1908): "mr. beebe!" said the maid, and the new rector of summer street was shown in; he had at once started on friendly relations, owing to lucy's praise of him in her letters from florence. from the context of this sentence alone, it is unclear if florence refers to "a city in italy" or "a person named florence"; this local contextual ambiguity might lead an ner system to classify florence as either a person or location. however, examining the broader document context clarifies this entity type: other mentions of florence within the text more clearly indicate that it refers to the city: this work makes the following contributions: ## doc-arc the core idea behind doc-arc is to leverage nearby representations of the same word when generating a representation for a given token. rather than representing florence above through a contextual representation scoped only over one sentence, we represent it through a weighted combination of that token itself and other instances of florence in the document. by attending over multiple instances of the same word, we are able to preserve the importance of the specific local context of a token, while also reasoning about its broader use in the rest of the document. while this model has application to a wide range of nlp tasks, we focus on the sequence labeling problem of ner. ## datasets we evaluate our model on three named entity recognition (ner) datasets: litbank @xcite , jnlpba @xcite and ontonotes @xcite . litbank. the litbank dataset @xcite is comprised of relatively long documents drawn from 100 english novels, with each document containing annotations for roughly 2,000 words. this dataset contains annotations for nested entities using six of the ace 2005 @xcite categories (per, loc, fac, gpe, org, veh). we convert that hierarchy into a flat structure suitable for ner by preserving only the outermost layer for any nested structure (using the same process used by jnlpba for genia, described below); all annotations nested within another are removed. we use the same training, development and test splits reported in @xcite . while the labeled documents in litbank are already quite long, they represent less than 2% of the novels they are drawn from-the average full text novel in this collection is approximately 133,000 words. we draw on this broader context by treating the remainder of the novel as unlabeled document context that we can exploit. ## experiments we evaluate our static and dynamic doc-arc models on litbank, jnlpba, and ontonotes 1000 . to enable a fair comparison of the specific contribution of document-level attention, each doc-arc model is compared to a baseline which lacks contextual inputs and has a comparable number of trainable parameters. ## analysis doc-arc was designed to ( 1 ) improve the performance of ner systems for rare, but important entities by (2) leveraging rich contextual information in long documents. in this section, we characterize the extent to which these goals were met using both quantitative and qualitative analysis. ## conclusion we present in this work a new method for reasoning over the context of long documents by attending over representations of identical word types when generating a representation for a token in sequence labeling tasks like ner. we show that when comparing equivalently parameterized models, incorporating attention over the entire document context leads to performance gains over models that lack that contextual mechanisms; further, the gains are asymmetric, with a substantial increase in accuracy for important entities within a document (defined as those with high tf-idf scores). in the context of long documents, our approach presents a novel alternative to established methods such as long sequence modeling and task-adaptive pretraining. our work's main contribution is a computationally tractable method for attention in long documents, employing exact word match as a complexity-reducing heuristic. though our attention mechanism is ostensibly simple, doc-arc 's strong performance in comparison to noncontextual baselines demonstrates both the value of the exact-match heuristic and the general utility of our framework. this work leaves open several natural directions for future research, including incorporating document attention within a fully trainable task-tuned bert model, and broadening the focus of attention beyond identical word types to words that bear other forms of similarity (such as similarity in subword morphology and meaning). code and data to support this work can be found at https://github.com/mjoerke/doc-arc .