Text-based NP Enrichment (2022)

Abstract: Understanding the relations between entities denoted by NPs in a text is a critical part of human-like natural language understanding. However, only a fraction of such relations is covered by standard NLP tasks and benchmarks nowadays. In this work, we propose a novel task termed text-based NP enrichment (TNE), in which we aim to enrich each NP in a text with all the preposition-mediated relations—either explicit or implicit—that hold between it and other NPs in the text. The relations are represented as triplets, each denoted by two NPs related via a preposition. Humans recover such relations seamlessly, while current state-of-the-art models struggle with them due to the implicit nature of the problem. We build the first large-scale dataset for the problem, provide the formal framing and scope of annotation, analyze the data, and report the results of fine-tuned language models on the task, demonstrating the challenge it poses to current technology. A webpage with a data-exploration UI, a demo, and links to the code, models, and leaderboard, to foster further research into this challenging problem can be found at: yanaiela.github.io/TNE/.

## introduction a critical part of understanding a text is detecting the entities in the text, denoted by nps, and determining the different semantic relations that hold between them. some semantic relations between nps are explicitly mediated via verbs, as in (1): much work in nlp addresses the recovery of such verb-mediated relations (srl) @xcite , either using pre-specified role ontologies such as propbank or framenet @xcite , or, more recently, using natural-languagebased representations (qa-srl) @xcite ; * equal contribution. @xcite . another well-studied kind of semantic relations between nps is that of coreference @xcite , where two (or more) nps refer to the same entity. such np-np relations, which are either mediated by verbs (as in srl or relation extraction) or form coreference relations, represent only a subset of the np-np relations that are naturally expressed in texts. consider, for instance, the following sentences: all of the above cases contain examples of np-np relations, where the type of relation can be expressed via an english preposition. the preposition may be explicit in the text, as in (2), where the relation a person with blue eyes is explicitly expressed, or they may be implicit and left to the reader to infer, as in (3)-(4); in (3) readers easily infer that the roots are of the plant. likewise, in (4) readers infer that the window is in the room. 1 properly understanding the text means knowing that these relations hold, even when they are not explicitly stated in the utterance. figure 1 shows additional examples. these relations, both explicit and implicit, are indispensable for understanding the text. while human-readers infer these relations intuitively and spontaneously while reading, machine-readers generally ignore them. in this work, we thus propose a new nlu task in which we aim to recover all the preposition-mediated relations-whether explicit or implicit-between nps that exist in a text. we call this task text-based np enrichment or tne for short. the short examples (2)-(4) that illustrate the phenomenon may not look challenging to infer using current nlp technology. however, when we go beyond sentence level to document level, things become substantially more complicated. as we demonstrate in §6, a typical 3-paragraph text in our dataset has an average of 35.8 nps, which participate in an average of 186.7 preposition-mediated relations, the majority of which are implicit. figure 2 shows a complete annotated document from our dataset. the type of information recovered by the np enrichment task complements well-established core nlp tasks such as entity typing, entity linking, coreference resolution, and semantic-role labeling @xcite . we believe it serves as an important and much-needed building block for downstream applications that require text understanding, including information retrieval, relation extraction and event extraction, question answering, and so on. in particular, the np enrichment task neatly encapsulates much of the long-range information that is often required by such applications. take for example a system that attempts to extract reports on police shooting incidents @xcite , with the following challenging, but not uncommon, passage: 2 police officers spotted the butt of a handgun in alton sterling's front pocket 2 we thank katherine keith for this example. and saw him reach for the weapon before opening fire, according to a baton rouge police department search warrant filed monday that offers the first police account of the events leading up to his fatal shooting. considering this shooting-event passage, an ideal coreference model will resolve his to alton sterling's, making the entity being shot local to the shooting event. on top of that, an ideal np enrichment model as we propose here will also recover: fatal shooting @xcite ] making the shooter identity local to the shooting event as well, ready for use by a downstream event-argument extractor or machine reader. of course, one could hope that a dedicated, endto-end-trained shooting-events extraction model will learn to recover such information on its own. however, it will require pre-defining the frame of shooting events, and it will require a substantial amount of training data to get it right (which often does not happen in practice). focusing on text-based np enrichment provides an opportunity to learn a core nlu skill that does not focus on a pre-defined set of relations, and is not specific to a particular benchmark. finally, beyond its potential usefulness for downstream nlp applications, the text-based np enrichment task serves as a challenging benchmark for reading comprehension, as we further elaborate in §3. in what follows we formally define the textbased np enrichment task ( §2) and its relation to reading comprehension ( §3), we describe a largescale high-quality english tne dataset we collected ( §4) and its curation procedure ( §5). we analyze the dataset ( §6) and experiment with pretrained language model baselines ( §7), achieving moderate but far-from-perfect success on this dataset ( §8). we also conduct an analysis of the best model, showcasing the strengths, weaknesses, and open challenges of the best model ( §9). we then discuss the relation of tne to other linguistic concepts, such as bridging @xcite , relational nouns @xcite @xcite , and implicit arguments @xcite @xcite ) ( §10). we finally conclude that, in contrast to those linguistic tasks, the text-based np enrichment task is more exhaustive, sharply scoped, easier to communicate, and substantially easier to consistently annotate and use by non-experts. despite the task's apparent simplicity, the underlying linguistic phenomena are quite complex, and range from simple syntactic relations to relations that require pragmatics, world knowledge, and common-sense reasoning. performing well on the task suggests a human-like level of understanding. notably, human readers detect most of the relations almost subconsciously when reading, while some of the relations require an extra conscious inference step. example consider the following text: in this utterance, there are four non-pronominal base-nps: ''adam'', ''father'', ''the teacher'' and ''his school'', which makes 12 possible pairs: (adam, father), (adam, the teacher), . . . , (his school, the teacher). the preposition-mediated relations to be recovered in this example are: 5 i (father, of adam) ii (the teacher, of, adam) iii (the teacher, at, his school) iv (his school, of, adam) the first items are anchors, and the latter ones are the complements. ## tne as a reading comprehension benchmark while reading comprehension (rc) and question answering (qa) are often used interchangeably in the literature, measuring the reading comprehension capacity of models via question answering, as implemented in benchmarks such as squad @xcite , boolq @xcite and others, has several well-documented problems @xcite . we argue that the tne task we propose herein has properties that make it appealing for assessing rc, more than qa is. first, benchmarks for extractive (span-marking) qa are sensitive to the span-boundary selection, on the other hand, benchmarks for yes/no, multiple choice, or generative questions can in principle be answered in a way that is completely divorced from the text. on a more fundamental level, all qa benchmarks are very sensitive to lexical choices in the question and its similarity to the text. furthermore, qa benchmarks rely on human authored questions that are easy to solve based on surface artifacts. finally, in many cases, the existence of the question itself provides a huge hint towards the answer @xcite . the underlying cause for all of these issues is that qa-based setups do not measure the comprehension of a text, but rather comprehending a (text, question) pair, where the question adds a significant amount of information, focuses the model on specific aspects of the text, and exposes the evaluation to biases and artifacts. the reliance on the human-authored questions makes qa a bad format for measuring ''text understanding''-we are likely measuring something else, such as the ability of the model to discern patterns in human question-writing behavior. the tne task we define side-steps all the above issues. it is based on the text alone, without revealing additional information not present in the text. the exhaustive nature of the task entails looking both at positive instances (where a relation exists) and negative ones (where it doesn't), making it harder for models to pick up shallow heuristics. we don't reveal information to a model, beyond the information that the two nps appear in the same text. finally, the list of nps to be considered is pre-specified, isolating the problem of understanding the relations between nps in the text from the much easier yet intervening problem of identifying nps and agreeing on their exact spans. @xcite thus, we consider tne a less biased and less gameable measure of rc than qa-based benchmarks. of course, the information captured by tne is limited and does not cover all levels of text understanding. yet, performing the task correctly entails a non-trivial comprehension of texts, which human readers do as a byproduct of reading. ## text-based np enrichment dataset we collect a large-scale tne dataset, consisting of 5.5k documents in english (3,988 train, 500 dev, 500 in-domain test, and 509 out-of-domain test). it covers about 200k nps and over 1 million np relations. the main domain is wikinews articles, and the out-of-domain (ood) texts are split evenly between reviews from imdb, fiction from project gutenberg, and discussions from reddit. each annotated document consists of a title and 3 paragraphs of text, and contains a list of non-pronominal base-nps (most identified by spacy @xcite foot_4 but some added manually by the annotators), a list of coreference clusters over the nps, and a list of np-relations that hold in the text. each relation is a triplet consisting of two nps from the np list, and a connecting element which is one of 23 prepositions (displayed in ## dataset statistics and analysis we report statistics of the resulting np enrichment dataset, and summarize them in ## deterministic baselines we explore multiple deterministic baselines, which should expose regularities in the data that models may use (and therefore may result in an easy to solve dataset), and provide further insights about our data. in these baselines we focus on detecting valid anchor/complement pairs, without considering the preposition's identity. title link this baseline considers one of the title's nps as the complement for each np in the text. we experiment with three variants: title-first, title-last, and title-random, which use the first, last, and a random np in the title, respectively. ## neural models next, we experiment with three neural models based on a pre-trained masked language model (mlm), specifically, spanbert @xcite . we also experiment with an additional baseline with uncontextualized word embeddings. architecture at a high level, our models take the encoding of two nps-an anchor and a complement-and predict whether they are connected, and if so, by which preposition. to encode an anchor-complement pair, we first encode the text using the mlm and then encode each np by concatenating the vectors of its first and last tokens. the resulting anchor and complement vectors are then each fed into a different mlp, each with a single 500-dimensions hidden-layer. the concatenation of the mlp outputs results in the anchorcomplement representation. this representation is then fed into the prediction model, which has two variants. the architecture resembles the end-toend architecture for modeling coreference resolution @xcite . a schematic view of the architecture is presented in figure 5 . ## conclusions we propose a new task named text-based np enrichment, or tne, in which we aim to annotate each np with all its relations to other nps in the text. this task covers a lot of implicit relations that are nonetheless crucial for text understanding. we introduce a large-scale dataset enriched with such np links, containing 5.5k documents and over 1m links-enough for training large neural networks-and provide high-quality test sets, both in and out of domain. we propose several baselines for this task and show that it is challenging-even for state-of-the-art lm-based models-and that there is a big gap from human performance. we release the dataset, code, and models, and hope that the community will adopt this task as a standard component of the nlp pipeline.