O pen IE 6: I terative G rid L abeling and C oordination A nalysis for O pen I nformation E xtraction (2020)

Abstract: A recent state-of-the-art neural open information extraction (OpenIE) system generates extractions iteratively, requiring repeated encoding of partial outputs. This comes at a significant computational cost. On the other hand,sequence labeling approaches for OpenIE are much faster, but worse in extraction quality. In this paper, we bridge this trade-off by presenting an iterative labeling-based system that establishes a new state of the art for OpenIE, while extracting 10x faster. This is achieved through a novel Iterative Grid Labeling (IGL) architecture, which treats OpenIE as a 2-D grid labeling task. We improve its performance further by applying coverage (soft) constraints on the grid at training time. Moreover, on observing that the best OpenIE systems falter at handling coordination structures, our OpenIE system also incorporates a new coordination analyzer built with the same IGL architecture. This IGL based coordination analyzer helps our OpenIE system handle complicated coordination structures, while also establishing a new state of the art on the task of coordination analysis, with a 12.3 pts improvement in F1 over previous analyzers. Our OpenIE system - OpenIE6 - beats the previous systems by as much as 4 pts in F1, while being much faster.

## introduction open information extraction (openie) is an ontology-free information extraction paradigm that generates extractions of the form (subject; relation; object). built on the principles of domainindependence and scalability @xcite , openie systems extract open relations and arguments from the sentence, which allow them to be *equal contribution 1 https://github.com/dair-iitd/openie6 used for a wide variety of downstream tasks like question answering @xcite , event schema induction @xcite and fact salience @xcite . figure 1 : the extractions (rome; [is] the capital of; italy) and (rome; is known for; it's rich history) can be seen as the output of grid labeling. we additionally introduce a token [is] to the input. end-to-end neural systems for openie have been found to be more accurate compared to their nonneural counterparts, which were built on manually defined rules over linguistic pipelines. the two most popular neural openie paradigms are generation @xcite and labeling @xcite . generation systems generate extractions one word at a time. imojie @xcite ) is a state-of-the-art openie system that re-encodes the partial set of extractions output thus far when generating the next extraction. this captures dependencies among extractions, reducing the overall redundancy of the output set. however, this repeated re-encoding causes a significant reduction in speed, which limits use at web scale. on the other hand, labeling-based systems like rnnoie @xcite are much faster (150 sentences per second, compared to 3 sentences of imojie) but relatively less accurate. they label each word in the sentence as either s (subject), r (relation), o (object) or n (none) for each extraction. however, as the extractions are predicted independently, this does not model the inherent dependencies among the extractions. ## iterative grid labeling for openie given a sentence with word tokens {w 1 , w 2 , . . . , w n } the task of openie is to output a set of extractions, say {e 1 , e 2 , . . . , e m }, where each extraction is of the form (subject; relation; object). for a labeling-based system, each word is labeled as s (subject), r (relation), o (object), or n (none) for every extraction. we model this as a 2-d grid labeling problem of size (m, n ), where the words represent the columns and the extractions represent the rows (figure 2 ). the output at position (m, n) in the grid (l m,n ) represents the label assigned to the n th word in the m th extraction. we propose a novel iterative grid labeling (igl) approach to label this grid, filling up one row after another iteratively. we refer to the openie extractor trained using this approach as igl-oie. igl-oie is based on a bert encoder, which computes contextualized embeddings for each word. the input to the bert encoder is {w 1 , w 2 , . . . , w n , [is], [of], [from]}. the last three tokens (referred as st i in figure 3 ) are appended because, sometimes, openie is required to predict tokens that are not present in the input sentence. foot_0 e.g., "us president donald trump gave a speech on wednesday." will have one of the extractions as @xcite ; us). the appended tokens make such extractions possible in a labeling framework. the contextualized embeddings for each word or appended token are iteratively passed through a 2-layer transformer to get their il embeddings at different levels, until a maximum level m , i.e. a word w n has a different contextual embedding il m,n for every row (level) m. at every level m, each il m,n is passed though a fully-connected labeling layer to get the labels for words at that level (figure 3 ). embeddings of the predicted labels are added to the il embeddings before passing them to the next iteration. this, in principle, maintains the information of the extractions output so far, and hence can capture dependencies among labels of different extractions. for words that were broken into word-pieces by bert, only the embedding of the first word-piece is retained for label prediction. we sum the cross-entropy loss between the predicted labels and the gold labels at every level to get the final loss, denoted by j ce . openie systems typically assign a confidence value to an extraction. in igl, at every level, the respective extraction is assigned a confidence value by adding the log probabilities of the predicted labels (s, r, and o), and normalizing this by the extraction length. we believe that igl architecture has value beyond openie, and can be helpful in tasks where a set of labelings for a sentence is desired, especially when labelings have dependencies amongst them. 3 we showcase another application of igl for the task of coordination analysis in section 5. ## grid constraints our preliminary experiments revealed that igl-oie has good precision, but misses out important extractions. in particular, we observed that the set of output extractions did not capture all the information from the sentence ( to describe the constraints, we first define the notion of a head verb as all verbs except light verbs (do, be, is, has, etc.). we run a pos tagger on the input sentence, and find all head verbs in the sentence by removing all light verbs. 4 for example, for the sentence, "obama gained popularity after oprah endorsed him for the presidency", the head verbs are gained and endorsed. in order to cover all valid extractions like (obama; gained; popularity) and (oprah; endorsed him for; the presidency), we design the following coverage constraints: notation: we now describe the penalty terms for these constraints. let p n be the pos tag of w n . we define an indicator x imp @xmath0, v, jj, rb}, and 0 otherwise. similarly, let x hv @xmath1. at each extraction level m, the model computes y mn (k), the probability of assigning the n th word the label @xmath2, r, o, n}. we formulate the penalties associated with our constraints as follows: ideally, no constraint violations of hvc and hve would imply that ec would also never gets violated. however, as these are soft constraints, this scenario is never materialized in practice. we find that our model performs better and results in fewer constraint violations when trained with posc, hvc, hve and ec combined. the full loss function is @xmath3, where λ are hyperparameters. we refer to the openie extractor trained using this constrained loss as constrained iterative grid labeling openie extractor (cigl-oie). the model is initially trained without constraints for a fixed warmup number of iterations, followed by constrained training till convergence. ## coordination boundary detection coordinated conjunctions (cc) are conjunctions such as "and", "or" that connect, or coordinate words, phrases, or clauses (they are called the conjuncts). the goal of coordination analysis is to detect coordination structures -the coordinating conjunctions along with their constituent conjuncts. in this section we build a novel coordination analyzer and use its output downstream for openie. sentences can have hierarchical coordinations, i.e., some coordination structures nested within the conjunct span of others @xcite . therefore, we pose coordination analysis as a hierarchical labeling problem, as illustrated in figure 4 . we formulate a 2-d grid labeling problem, where all coordination structures at the same hierarchical level are predicted in the same row. specifically, we define a grid of size (m, n ), where m is the maximum depth of hierarchy and n is the number of words in the sentence. the value at (m, n) th position in the grid represents the label assigned to the n th word in the m th hierarchical level, which can be cc (coordinated conjunction), conj (belonging to a conjunct span), or n (none). using igl architecture for this grid gives an end-to-end coordination analyzer that can detect multiple coordination structures, with two or more conjuncts. we refer to this coordination analyzer as igl-ca. coordination analyzer in openie: conjuncts in a coordinate structure exhibit replaceability -a sentence is still coherent and consistent, if we replace a coordination structure with any of its conjuncts @xcite . following calmie's approach, we generate simple (non-conjunctive) sentences using igl-ca. we then run cigl-oie on these simple sentences to generate extractions. these extractions are de-duplicated and merged to yield the final extraction set (figure 4 ). this pipelined approach describes our final openie system -openie6. for a conjunctive sentence, cigl-oie's confidence values for extractions will be with respect to multiple simple sentences, and may not be calibrated across them. we use a separate confidence estimator, consisting of a bert encoder and an lstm decoder trained on (sentence, extraction) pairs. it computes a log-likelihood for every extraction w.r.t. the original sentence -this serves as a better confidence measure for openie6. ## experimental setup we train openie6 using the openie4 training dataset used to train imojie foot_3 . it has 190,661 extractions from 92,774 wikipedia sentences. we convert each extraction to a sequence of labels over the sentence. this is done by looking for an exact string match of the words in the extraction with the sentence. in case there are multiple string matches for one of the arguments of the extraction, we choose the string match closest to the other arguments. this simple heuristic covers almost 95% of the training data. we ignore the remaining extractions that have multiple string matches for more than one argument. we implement our models using pytorch lightning (falcon, 2019). we use pre-trained weights of "bert-base-cased" foot_4 for openie extractor and "bert-large-cased" 6 for coordination analysis. we do not use bert-large for openie extractor as we observe almost same performance with a significant increase in computational costs. we set the maximum number of iterations, @xmath4. we use the spacy pos tagger foot_5 for enforcing constraints. the various hyper-parameters used are mentioned in appendix b. ## error analysis we examine extractions from a random sample of 50 sentences from carb validation set, as output by openie6. we identify three major sources of errors in these sentences: grammatical errors: (24%) we find that the sentence formed by serializing the extraction is not grammatically correct. we believe that combining our extractor with a pre-trained language model might help reduce such errors. noun-based relations: (16%) these involve introducing additional words in the relation span. although our model can introduce [is], [of], [from] in relations (section 3), it may miss some words for which it was not trained. e.g. [in] in (first security; based [in]; salt lake city) for the phrase salt lake city-based first security. lack of context: (10%) neural models for ope-nie including ours, do not output extraction context @xcite . e.g. for "she believes aliens will destroy the earth", the extraction (context(she believes); aliens; will destroy; the earth) can be misinterpreted without the context. we also observe incorrect boundary identification for relation argument (13%), cases in which coordination structure in conjunctive sentences are incorrectly split (11%), lack of coverage (4%) and other miscellaneous errors (18%). ## conclusion we propose a new openie system -openie6, based on the novel iterative grid labeling architecture, which models sequence labeling tasks with overlapping spans as a 2-d grid labeling problem. openie6 is 10x faster, handles conjunctive sentences and establishes a new state of art for ope-nie. we highlight the role of constraints in training for openie. using the same architecture, we achieve a new state of the art for coordination parsing, with a 12.3 pts improvement in f1 over previous analyzers. we plan to explore the utility of this architecture in other nlp problems. openie6 is available at https://github.com/dair-iitd/ openie6 for further research.