IM o JIE : Iterative Memory-Based Joint Open Information Extraction (2020)

Abstract: While traditional systems for Open Information Extraction were statistical and rule-based, recently neural models have been introduced for the task. Our work builds upon CopyAttention, a sequence generation OpenIE model (Cui et. al. 18). Our analysis reveals that CopyAttention produces a constant number of extractions per sentence, and its extracted tuples often express redundant information. We present IMoJIE, an extension to CopyAttention, which produces the next extraction conditioned on all previously extracted tuples. This approach overcomes both shortcomings of CopyAttention, resulting in a variable number of diverse extractions per sentence. We train IMoJIE on training data bootstrapped from extractions of several non-neural systems, which have been automatically filtered to reduce redundancy and noise. IMoJIE outperforms CopyAttention by about 18 F1 pts, and a BERT-based strong baseline by 2 F1 pts, establishing a new state of the art for the task.

## introduction extracting structured information from unstructured text has been a key research area within nlp. the paradigm of open information extraction (openie) @xcite uses an open vocabulary to convert natural text to semi-structured representations, by extracting a set of (subject, relation, object) tuples. openie has found wide use in many downstream nlp tasks @xcite like multi-document question answering and summarization @xcite , event schema induction @xcite and word embedding generation @xcite . traditional openie systems are statistical or rule-based. they are largely unsupervised in nature, or bootstrapped from extractions made by earlier systems. they often consist of several components like pos tagging, and syntactic parsing. to bypass error accumulation in such pipelines, end-to-end neural systems have been proposed recently. recent neural openie methods belong to two categories: sequence labeling, e.g., rnnoie @xcite and sequence generation, e.g., copyattention @xcite . in principle, generation is more powerful because it can introduce auxiliary words or change word order. however, our analysis of copyattention reveals that it suffers from two drawbacks. first, it does not naturally adapt the number of extractions to the length or complexity of the input sentence. second, it is susceptible to stuttering: extraction of multiple triples bearing redundant information. these limitations arise because its decoder has no explicit mechanism to remember what parts of the sentence have already been 'consumed' or what triples have already been generated. its decoder uses a fixed-size beam for inference. however, beam search can only ensure that the extractions are not exact duplicates. in response, we design the first neural openie system that uses sequential decoding of tuples conditioned on previous tuples. we achieve this by adding every generated extraction so far to the encoder. this iterative process stops when the endofextractions tag is generated by the decoder, allowing it to produce a variable number of extractions. we name our system iterative memory joint open information extraction (imojie). copyattention uses a bootstrapping strategy, where the extractions from openie-4 @xcite are used as training data. however, we believe that training on extractions of multiple systems is preferable. for example, openie-4 benefits from high precision compared to clausie @xcite , which offers high recall. by aggregating extractions from both, imojie could potentially however, simply concatenating extractions from multiple systems does not work well, as it leads to redundancy as well as exaggerated noise in the dataset. we devise an unsupervised score-and-filter mechanism to automatically select a subset of these extractions that are non-redundant and expected to be of high quality. our approach scores all extractions with a scoring model, followed by filtering to reduce redundancy. we compare imojie against several neural and non-neural systems, including our extension of copyattention that uses bert @xcite instead of an lstm at encoding time, which forms a very strong baseline. on the recently proposed carb metric, which penalizes redundant extractions @xcite , imojie outperforms copyattention by about 18 pts in f1 and our strong bert baseline by 2 pts, establishing a new state of the art for openie. we release imojie & all related resources for further research 1 . in summary, our contributions are: • we propose imojie, a neural openie system that generates the next extraction, fully conditioned on the extractions produced so far. imo-jie produce a variable number of diverse extractions for a sentence, • we present an unsupervised aggregation scheme to bootstrap training data by combining extractions from multiple openie systems. • imojie trained on this data establishes a new 1 https://github.com/dair-iitd/imojie sota in openie, beating previous systems and also our strong bert-baseline. ## related work open information extraction (openie) involves extracting (arg1 phrase, relation phrase, arg2 phrase) assertions from a sentence. traditional open extractors are rule-based or statistical, e.g., textrunner @xcite , reverb @xcite , ollie @xcite , stanford-ie @xcite , clausie @xcite , openie-4 @xcite , openie-5 @xcite , props @xcite , and minie @xcite . these use syntactic or semantic parsers combined with rules to extract tuples from sentences. recently, to reduce error accumulation in these pipeline systems, neural openie models have been proposed. they belong to one of two paradigms: sequence labeling or sequence generation. sequence labeling involves tagging each word in the input sentence as belonging to the subject, predicate, object or other. the final extraction is obtained by collecting labeled spans into different fields and constructing a tuple. rnnoie @xcite ) is a labeling system that first identifies the relation words and then uses sequence labelling to get their arguments. it is trained on oie2016 dataset, which postprocesses srl data for openie @xcite . senseoie @xcite , improves upon rn-noie by using the extractions of multiple openie systems as features in a sequence labeling setting. however, their training requires manually annotated gold extractions, which is not scalable for the task. this restricts senseoie to train on a dataset of 3,000 sentences. in contrast, our proposed score-and-filter mechanism is unsupervised and can scale unboundedly. @xcite 2019 ) is another labeling system that better calibrates extractions across sentences. spanoie @xcite ) uses a span selection model, a variant of the sequence labelling paradigm. firstly, the predicate module finds the predicate spans in a sentence. subsequently, the argument module outputs the arguments for this predicate. however, spanoie cannot extract nominal relations. moreover, it bootstraps its training data over a single openie system only. in contrast, imojie overcomes both of these limitations. sequence generation uses a seq2seq model to generate output extractions one word at a time. the generated sequence contains field demarcators, which are used to convert the generated flat sequence to a tuple. copyattention @xcite is a neural generator trained over bootstrapped data generated from openie-4 extractions on a large corpus. during inference, it uses beam search to get the predicted extractions. it uses a fixed-size beam, limiting it to output a constant number of extractions per sentence. moreover, our analysis shows that copyattention extractions severely lack in diversity, as illustrated in we compare against logician's coverage attention as one of the approaches for increasing diversity. sequence-labeling based models lack the ability to change the sentence structure or introduce new auxiliary words while uttering predictions. for example, they cannot extract (trump, is the president of, us) from "us president trump", since 'is', 'of' are not in the original sentence. on the other hand, sequence-generation models are more general and, in principle, need not suffer from these limitations. evaluation: all neural models have shown improvements over the traditional systems using the oie2016 benchmark. however, recent work shows that the oie2016 dataset is quite noisy, and that its evaluation does not penalize highly redundant extractions @xcite . in our work, we use the latest carb benchmark, which crowdsources a new evaluation dataset, and also provides a modified evaluation framework to downscore near-redundant extractions @xcite . ## sequential decoding we now describe imojie, our generative approach that can output a variable number of diverse extractions per sentence. the architecture of our model is illustrated in figure 1 . at a high level, the next extraction from a sentence is best determined in context of all other tuples extracted from it so far. hence, imojie uses a decoding strategy that generates extractions in a sequential fashion, one after another, each one being aware of all the ones generated prior to it. this kind of sequential decoding is made possible by the use of an iterative memory. each of the generated extractions are added to the memory so that the next iteration of decoding has access to all of the previous extractions. we simulate this iterative memory with the help of bert encoder, whose input includes the [cls] token and original sentence appended with the decoded extractions so far, punctuated by the separator token [sep] before each extraction. imojie uses an lstm decoder, which is initialized with the embedding of [cls] token. the contextualized-embeddings of all the word tokens are used for the copy @xcite and attention @xcite modules. the decoder generates the tuple one word at a time, producing rel and obj tokens to indicate the start of relation and object respectively. the iterative process continues until the endofextractions token is generated. the overall process can be summarized as: 1. pass the sentence through the seq2seq architecture to generate the first extraction. 2. concatenate the generated extraction with the existing input and pass it again through the seq2seq architecture to generate the next extraction. 3. repeat step 2 until the endofextractions token is generated. imojie is trained using a cross-entropy loss between the generated output and the gold output. ## error analysis we randomly selected 50 sentences from the carb validation set. we consider only sentences where at least one of its extractions shows the error. we identified four major phenomena contributing to errors in the imojie model: (1) missing information: 66% of the sentences have at least one of the relations or arguments or both missing in predicted extractions, which are present in gold extractions. this leads to incomplete information. ## conclusions and discussion we propose imojie for the task of openie. imo-jie significantly improves upon the existing ope-nie systems in all three metrics, optimal f1, auc, and last f1, establishing a new state of the art system. unlike existing neural openie systems, imo-jie produces non-redundant as well as a variable number of openie tuples depending on the sentence, by iteratively generating them conditioned on the previous tuples. additionally, we also contribute a novel technique to combine multiple ope-nie datasets to create a high-quality dataset in a completely unsupervised manner. we release the training data, code, and the pretrained models. 9 imojie presents a novel way of using attention for text generation. @xcite showed that attending over the input words is important for text generation. @xcite showed that using a coverage loss to track the attention over the decoded words improves the quality of the generated output. we add to this narrative by showing that deep inter-attention between the input and the partially-decoded words (achieved by adding previous output in the input) creates a better representation for iterative generation of triples. this general observation may be of independent interest beyond openie, such as in text summarization.