W eb IE : Faithful and Robust Information Extraction on the Web (2023)

Abstract: Extracting structured and grounded fact triples from raw text is a fundamental task in Information Extraction (IE). Existing IE datasets are typically collected from Wikipedia articles, using hyperlinks to link entities to the Wikidata knowledge base. However, models trained only on Wikipedia have limitations when applied to web domains, which often contain noisy text or text that does not have any factual information. We present WebIE, the first large-scale, entity-linked closed IE dataset consisting of 1.6M sentences automatically collected from the English Common Crawl corpus. WebIE also includes negative examples, i.e. sentences without fact triples, to better reflect the data on the web. We annotate ~25K triples from WebIE through crowdsourcing and introduce mWebIE, a translation of the annotated set in four other languages: French, Spanish, Portuguese, and Hindi. We evaluate the in-domain, out-of-domain, and zero-shot cross-lingual performance of generative IE models and find models trained on WebIE show better generalisability. We also propose three training strategies that use entity linking as an auxiliary task. Our experiments show that adding Entity-Linking objectives improves the faithfulness of our generative IE models.

## introduction information extraction (ie) is the task of extracting structured information from unstructured text, usually in the form of @xmath0, relation, object>. it is essential for many natural language processing applications such as knowledge base population, question answering, faithful summarisation, and fake news detection @xcite @xcite . typically, two pieces of information are needed for training closed ie foot_0 systems: (i) the entities mentioned in the text and (ii) the relations that exist between each pair of entities. obtaining such information requires expensive annotations, therefore most existing ie datasets, such as wikinre @xcite or rebel @xcite , are built using wikipedia, as entity information is available through hyperlinks and relation information can be automatically extracted via distant supervision (ds) approach @xcite using a knowledge base (kb) such as wikidata. the ds approach assumes that if two entities are connected through a relation in a kb, then the sentences that mention both entities together express the relation. while models trained only on this fact-rich domain foot_1 have shown to be useful for ie applications, they have limited capacity when applied to extracting information in other web domains, which often contains noisy text or text without any factual information. take allenai's c4 dataset 4 , an open-sourced version of google's c4 @xcite dataset based on common crawl, as an example. our analysis using the ds approach reveals that less than 15% of the sentences contain triples ( §2.1), whereas we observe that a state-ofthe-art (sota) generative ie model, genie @xcite , which is trained on rebel, the largest ie dataset to date (which includes only positive examples), tends to generate triples for every sentence, resulting in a high rate of false positives and issues with hallucination. to address these issues and facilitate future work on ie on the web, we present webie, the first large-scale, entity-linked closed ie dataset collected from web sources. the webie dataset is bart lm -h ea d 1 l m -h e a d 2 + 2lm-heads + artificial-prompt [entity] obama # barack obama | honolulu # honolulu | hawaii # hawaii [triple] <@xmath1, hawaii. <#el#> obama was born in honolulu, hawaii. obama # barack obama | honolulu # honolulu | hawaii # hawaii bart obama was born in honolulu, hawaii. obama was born in honolulu, hawaii. <@xmath2, hawaii. for entity-prompt, the target is the el output ( §3.2) concatenated with the linearised triples. in artificial-prompt, we prepend an artificial token to the input to indicate the desired output: el (yellow box) or linearised triples. for 2lm-heads, we add an additional task-specific lm head to the decoder for the el task (grey box). previous works have shown that compared to discriminative pipelines which often suffer from accumulative errors due to separate entity linking and relation extraction (re) steps @xcite @xcite , generative models achieve superior performance in many closed ie tasks. therefore we primarily benchmark webie with generative, transformerbased encoder-decoder models, bart @xcite and mbart @xcite . the latter is used to evaluate the zero-shot cross-lingual transfer performance on mwebie. we further propose three training strategies ( §3.2) that use entity linking as an auxiliary task for generative ie, namely joint generation with the linked-entity prompt (entity-prompt), multitask learning with distinguished artificial prompt tokens (artificial-prompt), and training with an additional task-specific language model (lm) head (2lm-heads). we find that training with el as an auxiliary task overall leads to better and more faithful ie results. an illustration of these training strategies is provided in figure 1 . our experiments show that compared to models trained only on wikipedia datasets, models also trained on webie are more robust and generalisable, achieving a new sota performance on rebel ( §5) and competitive zero-shot performance on wikinre. we demonstrate that we-bie serves as a complementary dataset to existing datasets based on wikipedia, and show that including negative examples is crucial for addressing false positives in generative ie. ## (m)webie in this section, we provide a detailed explanation of the dataset collection process for (m)webie. ## generative information extraction this section describes the training strategies that we use for benchmarking (m)webie. ## experiments in this section, we explain the datasets used in the experiments and the detailed modelling setup. ## results and analysis we now present the main results of (m)webie and compare different training strategies. ## conclusions we present (m)webie, the first large-scale, entitylinked closed ie dataset on the web. a subset of the dataset is further annotated by humans and translated into four other languages, french, spanish, portuguese, and hindi, via crowd-sourcing. we benchmark webie with generative models and compare the models trained on webie and rebel (wikipedia-domain). our results show that models trained on webie have competitive zero-shot performance when applied to rebel and wikinre, whereas models trained only on rebel have 0% accuracy on the negative examples in webie. this highlights the importance of including negative examples for training more robust models and reducing hallucination in genera-tive ie on the web. models trained on both rebel and webie achieve the best performance on both datasets, as well as zero-shot results on wikinre, showing that webie serves as a complementary dataset to existing wikipedia-domain datasets. investigating the approaches with entity linking as an auxiliary task, we find that adding an additional task-specific lm head achieves the overall best performance. the entity-prompt approach shows the most significant improvement on we-bie with the constraint trie. we primarily benchmark transformer-based encoder-decoder models on webie, but future work could also explore pipeline frameworks and larger language models for few-shot performance.