This Table is Different: A W ord N et-Based Approach to Identifying References to Document Entities (2016)

Abstract: Writing intended to inform frequently contains references to document entities (DEs), a mixed class that includes orthographically structured items (e.g., illustrations, sections, lists) and discourse entities (arguments, suggestions, points). Such references are vital to the interpretation of documents, but they often eschew identifiers such as “Figure 1” for inexplicit phrases like “in this figure” or “from these premises”. We examine inexplicit references to DEs, termed DE references, and recast the problem of their automatic detection into the determination of relevant word senses. We then show the feasibility of machine learning for the detection of DE-relevant word senses, using a corpus of human-labeled synsets from WordNet. We test cross-domain performance by gathering lemmas and synsets from three corpora: website privacy policies, Wikipedia articles, and Wikibooks textbooks. Identifying DE references will enable language technologies to use the information encoded by them, permitting the automatic generation of finely-tuned descriptions of DEs and the presentation of richly-structured information to readers.

## introduction it is rare that communication in a written document is a simple linear endeavor. writers make use of orthographic, paralinguistic, and discursive structures to augment and enhance what they write. these structures commonly include figures, tables, sections, subsections, extended quotations, examples, arguments, summaries, and other means of organizing the communication channel. such document entities (des, for brevity) may be linguistic or pictorial, and they may be well-delineated or vaguely bounded. additionally, they may be entirely distinct from the prose or embedded in it. des are necessarily connected to the text that they appear with (or subsume) in a document. although the relationship may be implicit, a referring expression is often used to make a local connection. when style permits, these referring expressions may use identifiers for des such as " notably, the referents of table and section in the above examples differ from those below: to understand (1) or (2) (in contexts with referents), the reader must realize that table and section refer to des rather than entities in another class of referents, as in (3) or (4). the presence or absence of potential referents may help; however, the (1)/(3) and (2)/(4) distinctions are clear even out of context. this suggests that differing word senses are responsible. references to des (de references, for brevity) are frequent in text written to inform, and they profoundly affect the referential structure and practical value of passages that contain them. entity linking and coreference resolution address similar phenomena, but systems for those tasks are unsuitable for de references (as explained in section 3). little has been done to empirically understand de references or automatically identify them in text, which would allow language technologies to exploit links between des and discourse context. this would enable the tagging of des with precise descriptive information from referring text, enabling (for example) relevancebased caption generation for des, automatic document layout generation, and tools to help readers quickly skim documents for specific resources or explanations of those resources. this paper presents results on developing a method to automatically label noun word senses that represent references to des. this was done using logistic regression and a selection of features from synsets in the english wordnet @xcite , from which word senses were sampled. to give the task a practical focus, word senses were selected for words in deictic phrases from three corpora: the set of featured textbooks from wikibooks, a random selection of articles from wikipedia, and a selection of privacy policies from popular websites. wikibooks was selected because prior work has noted a high density of de references. wikipedia was selected for the informative value of its text, which differs in style and purpose from wikibooks. the domain of privacy policies was chosen as a strong contrast with the other two domains, and for the potential benefits of downstream research to reduce reader confusion @xcite . the diversity of these corpora also provided an opportunity for cross-domain evaluation. the contributions of this work are threefold: the remainder of this paper is structured as follows. section 2 summarizes a prior study of de reference, with examples of the phenomenon and differences from the present work. several related topics are reviewed in section 3. section 4 details the collection of word senses and the manual annotation process. in sections 5 and 6, the procedure for the automatic labeling of synsets is presented, along with results for intradomain and cross-domain labeling. we conclude with a discussion of the significance of these results and some directions for future work. ## background the present work builds upon findings from a prior study of word senses relevant to de reference @xcite . there, the set of 122 english wikibooks 1 textbooks with printable versions was selected as a corpus. the set contained eleven subject areas, including computing, humanities, sciences, and languages. this corpus was chosen for several reasons. among the alternatives, it provided the largest volume of text with a reuse-friendly license. it addressed a diverse set of topics with text written to inform, thus implying a diverse set of des. additionally, 1 http://en.wikibooks.org/ category examples structural many of the resources listed elsewhere in this section have… in this chapter, we will show you how to draw… illustrative consider these sentences: [followed by example sentences] [following a source code fragment] …the first time the computer sees this statement, 'a' is zero, so it is less than 10. discourse utilizing this idea, subunit analogies were invented… in this case, you've narrowed the topic down to "badges." non-de reference devices similar to resistors turn this energy into light, motion… what type of things does a person in that career field know? the corpus represented the collaboration of a large number of writers. phrase templates were used to gather candidate instances of de reference. these templates consisted of noun phrases beginning with the demonstratives this, that, these, and those. a subset of the candidates was read and annotated with categories, shown in directly labeling large numbers of candidate instances proved to be time-consuming, and instead work focused on labeling the word senses (from wordnet) of the 27 most frequent nouns in candidate instances. these senses were manually labeled by reading their definitions to judge their ability to refer to des. by fitting the labeled de senses into the wordnet ontology, observations became possible on the kinds of entities that served as des. for example, des were more likely to be abstractions than physical entities. the word sense annotations from the prior study showed that, for 15 of the 27 examined nouns, the first (most common) word sense of the noun was able to refer to a de. they also illustrated a permeable boundary between des thought of as discourse entities and des that reside outside of the prose. for example, a question raised for consideration or solution (the definition of problem.n.02) could refer to a question embedded in informative prose or an orthographically-distinct exercise in a problem set. ## related work prior studies showed the communicative value of multiple representations and their tight integration, motivating the present work. @xcite presented the cognitive theory of multimedia learning and explored how pictorial des augment and enhance textual artifacts. similarly, @xcite argued that learning materials should be presented so that "disparate sources of information are physically and temporally integrated". @xcite argued for "abstract document structure as a separate descriptive level in the analysis and generation of written texts", further motivating our work. the aggregation of word senses discussed in the present work has a precedent in supersense tagging @xcite , especially for wikipedia text @xcite . notably, one of wordnet's lexicographer files is noun.communication, which contains "nouns denoting communicative processes and contents" ("wordnet 3.0 reference manual", 2012). however, the set of senses in this file is a poor match for current purposes, as it includes many senses that do not fit a written or document-oriented context (for example, a word sense for airwave is included in the file). the present work also identifies several de senses outside of this lexicographer file. overall, the meta-communicative focus of the present work is novel compared to prior efforts. the task of automatically identifying instances of de reference bears some similarity to coreference resolution. however, coreference resolvers are not suited for the present task; those tried by the researchers include corenlp @xcite , arkref (o' @xcite and the work of @xcite . one problem is that many des are partly pictorial or are not recognized by nlp tools as cohesive entities. many des are distinguished by their non-linguistic aspects (i.e., diagrams) or stylistic markup (bulleted lists, quotations delimited by quote marks). the task at hand also has commonalities with entity linking @xcite and wikification, the process of linking named entities in text with corresponding wikipedia pages @xcite . however, des differ markedly from named entities. des vary widely in their representation and they often reside in the same communication medium as references to them. references to des often incorporate pragmatic information: for example, the referent of "this figure" may be the closest figure or the one most recently referred to. the potentially non-textual nature of des also separates them from mentioned language @xcite , although the phenomena share a metalinguistic quality. shell nouns are nouns used anaphorically to refer to complex concepts such as points, assumptions, acts, or feelings @xcite . their referents intersect with des, although neither set subsumes the other: schmid's taxonomy of shell nouns does not include typical dereferential nouns like section, figure, or list, yet it does include non-des like fury, miracle, and pride. @xcite have automatically detected referents of some shell nouns, but their methods share the limitations of coreference resolvers, as described above. the prior study of de senses provided groundwork for the study of de reference, but the dataset it created lacked the size and diversity for appreciable machine learning results. this section describes a procedure used to collect and label more word senses. a processing pipeline collected promising lemmas from three corpora, and a manual labeling procedure resulted in synset labels agreed upon by multiple annotators. ## automatic labeling of synsets the present work substantially increased the number of de-labeled synsets available, but the intensity of the labeling task still constrained the volume of new labels generated. this limitation partly shaped the experimental procedure, and it also reinforced the motivation for automatic, domain-independent labeling of de synsets. ## results we first discuss the results of the classifier trials, and then add observations on a potential performance ceiling and the most valuable features. ## discussion the difficulty in identifying de synsets is substantial; specifically, recall poses a challenge for the current prediction scheme. however, training on one corpus' high rank set and testing on a different corpus' set produced results that were not consistently better or worse than loocv, which suggests that labeling synsets gathered for a new domain (or all of wordnet) is no less feasible. these observations answer q1 and q2. toward q3, some variation seemed to exist: for intra-corpus runs (t1 and t2), wikibooks synsets produced the highest score and wikipedia synsets produced the lowest. however, this ordering may be the result of differing positivenegative label ratios, and it did not hold for cross-training. the answer to q3 may be a nominal affirmation: the label ratio, which varies by corpus, naturally affects classifier performance. finally, q4 is simpler to answer: evaluating on broad rank sets generally produced worse performances than evaluating on high rank sets. the greater prevalence of discourse and psychological entities in broad rank sets seemed to be responsible. excluding discourse entities from the class of des may appear to be an effective ad hoc solution, but it causes a new problem: many des appear interchangeably as orthographic or prose-embedded entities (e.g., lists, which may appear in bullet form or in a sentence). since phrases that refer to des do not distinguish between the two, the exclusion of discourse entities would create further artificial distinctions. ## conclusion and future work in this paper we presented a method for automatically identifying word senses that refer to document entities. evidence suggests that identifying non-discourse de senses was attainable with high precision and recall, but the ambiguities of discourse des-which were in some ways inseparable-poses a problem. we also introduced a corpus of de-labeled word senses from three domains and a method for extracting orthographically-structured des from web documents. these contributions enable future work on the auto-matic detection of de reference and the development of associated applications. the use of these results toward de supersense tagging and referent identification is a clear next step. the researchers have experimented with a prototype de reference tagger, and preliminary results suggest that integrating tagging and referent identification may be advantageous. a lowprecision high-recall de reference tagger will produce many false positives, but the availability of (or lack of) referents for each instance may serve as a sieve to eliminate those false positives.