Predicting Document Coverage for Relation Extraction (2022)

Abstract: This paper presents a new task of predicting the coverage of a text document for relation extraction (RE): Does the document contain many relational tuples for a given entity? Coverage predictions are useful in selecting the best documents for knowledge base construction with large input corpora. To study this problem, we present a dataset of 31,366 diverse documents for 520 entities. We analyze the correlation of document coverage with features like length, entity mention frequency, Alexa rank, language complexity, and information retrieval scores. Each of these features has only moderate predictive power. We employ methods combining features with statistical models like TF-IDF and language models like BERT. The model combining features and BERT, HERB, achieves an F1 score of up to 46%. We demonstrate the utility of coverage predictions on two use cases: KB construction and claim refutation.

## introduction motivation and problem relation extraction (re) from text documents is an important nlp task with a range of downstream applications @xcite . for these applications, it is vital to understand the quality of re results. while extractors typically provide confidence (or precision) scores, this paper puts forward the notion of re coverage (or recall). given an input document and an re method, coverage measures the fraction of the extracted relations compared to the complete ground-truth that holds in reality. we consider this on a per-subject and perpredicate basis-for example, all memberships of bill gates in organizations or all companies founded by elon musk. document coverage for re highly varies. consider the three text snippets about tesla as shown in figure 1 . the first text contains all five founders of tesla, while the second text contains only two of them, and the third has just one. analogously, for the entity tesla and the relation founded-by, we see that text 1 has coverage 1, text 2 has coverage 0.4, and text 3 has coverage 0.2. when applying re at scale, for example, to populate or augment a knowledge base (kb), an re system may have to process a huge number of input documents that differ widely in their coverage. as state-of-the-art extractors are based on heavy-duty neural networks @xcite @xcite , processing all documents in a large corpus may be prohibitive and inefficient. instead, prioritizing the input documents by identifying the best documents with high coverage could be effective. this is why coverage prediction is crucial for large-scale re, but the problem has not been explored so far. this problem would be easy if we could first run a neural re system on each document and then assess the yield, either by comparison to withheld labeled data or by sampling followed by human inspection. however, this is exactly the computational bottleneck that we must avoid. the challenge is to estimate document coverage, for a given entity and relation of interest, with inexpensive and lightweight techniques for document processing. approach and contributions this paper presents the first systematic approach for analyzing and predicting document coverage for relation extraction. to facilitate extensive experimental study on this novel task, we introduce a large document coverage (doco) dataset of 31,366 web documents for 520 distinct entities spanning 8 relations, along with automated extractions and coverage labels. we employ a classifier architecture that we call herb (for heuristics with bert), based on a document's lightweight features and additionally incorporates pretrained language models like bert without any costly re-training and fine-tuning. the best configuration of this classifier achieves an f1-score of up to 46%. the classifier provides scores for its predictions and thus also supports ranking documents by their expected yield for the re task at hand. we evaluate our approach against a range of state-of-the-art baselines. our results show that heuristic features like text length, entity mention frequency, language complexity, alexa rank, or information retrieval scores have only moderate predictive power. however, in combination with pre-trained language models, the proposed classifier gives useful predictions of document coverage. we further study the role of coverage prediction in two extrinsic use cases: kb construction and claim refutation. for kb construction, we show that coverage estimates by herb are effective in ranking candidate documents and can substantially reduce the number of web pages one needs to process for building a reasonably complete kb. for the task of claim refutation (e.g., tim cook is the ceo of microsoft), we show that coverage estimates for different documents can provide counter-evidence that can help to invalidate false statements obtained by re systems. the salient contributions of this work are: our data, models and code is publicly available. foot_0 knowledge base construction (kbc) re plays a crucial part in the more comprehensive kbc task: identifying instances of entity pairs that stand in a given relation in order to construct a knowledge base @xcite @xcite . the input is typically a set of documents, often assumed to be fixed and given upfront. this disregards the critical issue of benefit/cost trade-offs, which mandates identifying high-yield inputs for resource-bounded kbc. identifying relevant, expressive, and preferable sources for kbc is often referred to as source discovery. source discovery can be performed via ir-style ranking of documents or can be based on heuristic estimators of the yield of relation extractors @xcite document ranking in ir information retrieval (ir) ranks documents by relevance to a query with keywords or telegraphic phrases. relevance judgments are based on the perception of informativeness concerning the query and its underlying user intent. standard metrics for assessment, like precision, recall, and ndcg @xcite , are not applicable to our setting. the notion of coverage pursued in this paper refers to the yield of structured outputs by re systems rather than document relevance. for example, a query-topic-wise highly relevant document that contains few extractable facts about named entities would still have low re coverage. relevance of coverage estimates understanding and incorporating document coverage prediction into nlp-based information extraction is essential for several reasons. for resourcebounded kb construction, it is crucial to know which documents are most promising for extraction with limited budgets for crawling and re processing and/or human annotation @xcite . for claim refutation, coverage estimates can help to assess statements as questionable if documents with high coverage do not support them. so far, claim evaluation systems mostly rely on textual cues about factuality or source credibility @xcite @xcite . for question answering over knowledge bases, it is important to know whether a kb can be relied upon in terms of complete answer sets @xcite @xcite . current coverage estimation techniques for kbs do this analysis only post-hoc after the kb is fully constructed @xcite , losing access to valuable information from extraction time. ## coverage prediction we take an entity-centric perspective, and view re methods as functions mapping documententity-relation tuples onto the set of objects found in the document. formally, given a document d, figure 2 : dataset construction pipeline. there are two main phases: 1) corpus collection to create gt web , and 2) coverage calculation. phase 1 involves: i) for each entity e i , n websites are collected using the bing search api, ii) text is scraped from each website, iii) re tuples from documents are extracted via rosette/diffbot, and iv) re tuples are deduplicated and consolidated to form gt web . the scraped documents are stored as inputs for phase 2 which consists of: i) for each document d i , previously extracted relations are collected, and ii) based on the choice of gt, coverage is calculated to create the final doco dataset. an entity e, a relation r, a ground truth gt of objects that stand in relation r with e, and a relation extraction method extr, the document coverage of d for (e, r) applying extr is defined as: ## dataset construction a thorough study of document coverage prediction requires a corpus with two characteristics: (i) relation diversity (i.e., documents containing enough automatically extractable relations) and (ii) content diversity (i.e., multiple documents with varying content per entity). existing text corpora, like the popular nyt @xcite and newsroom dataset @xcite , contain ample numbers of articles that mention newsworthy entities; however, the articles are primarily short, mentioning only very few relations. on the other end, machine-translated multilingual versions of wikipedia articles @xcite allow extraction of many relations but lack diversity. for the novel task of predicting document information coverage, we thus built the doco (document coverage) dataset, consisting of 31,366 web documents for 520 distinct entities, each with its coverage value. figure 2 illustrates the dataset construction. entity selection first, well-known entities of two types, person (per) and organization (org), were selected from popular ranking lists by time 100 foot_3 and forbes 5,6 (''influential people around the globe'', ''most valuable tech companies''). these entities covered 12 diverse sub-domains, including politicians, entrepreneurs, singers, sports figures, writers, and actors, for per, and technology, automobile, retail, conglomerate, pharmaceuticals, and financial corporations, for org. popular and long-tail entities for per, companies across demographics and with differing net worth for org, were chosen to further obtain documents with varying content. websites and content we aimed to collect diverse 100 urls per entity by issuing a set of search engine queries per entity, for example, ''about per'', ''per biography'', ''org history''. a total of 6 set of queries for per and 10 for org was designed. since the urls returned over the set of queries were not always unique, we retained the duplicated url only once. extracting textual content without noisy headers, menus, and comments required a labor-intensive scraping step. we handled the multi-domain content scraping task through a combination of libraries like newspaper3k, 7 readability, 8 and online scraping services like import.io foot_8 and parsehub. 10 we ensured highquality scraped content by applying rule-based filters to remove noisy elements like embedded ads and reference links. the scraped documents covered a range of website domains, including biographical sites, news articles, official company profiles, newsletters, and so on. relation tuples each document in doco was processed by two relation extraction apis, rosette and diffbot. to annotate each document with coverage, we focused only on the entity queried initially to obtain the document. for our experimental study, we selected the following frequently occurring relations: member-of, family, edu-at, and position-held, for per, and partner-org, founded-by, ceo, and board-member, for org. for more accurate coverage calculation, the re tuples were deduplicated, for example, (gates, member-of, microsoft corp.) would become (bill gates, member-of, microsoft), via alignment to wikidata identifiers returned by the apis. the relations extracted by the apis are finegrained like person-member-of, person-employeeof, org-acquired-by, and org-subsidiary-of. we combined the first two as member-of for per and the last two as partner-org for org as coarse-grained relations. ## approach we aim to model coverage prediction by processing unstructured document text by inexpensive lightweight techniques. this is crucial for identifying promising documents before embarking on heavy-duty re. heuristics we devise several simple heuristics involving textual features for document coverage. ## applications to demonstrate the importance of coverage prediction, we evaluated its utility in two use cases, knowledge base construction and claim refutation. for the former, we discuss the importance of ranking documents by re coverage (section 8.1) and a practically relevant setting where re is constrained by resource budgets (section 8.2). ## conclusion this paper introduces the new task of document coverage prediction and a large dataset for experimental study of the task. our methods show that heuristic features can boost the performance of pre-trained language models without costly fine-tuning. moreover, we demonstrate the value of coverage estimates for the use cases of knowledge base construction and claim refutation. our future research includes developing a user-friendly tool to support knowledge engineers.