Syntactically Robust Training on Partially-Observed Data for Open Information Extraction (2022)

Abstract: Open Information Extraction models have shown promising results with sufficient supervision. However, these models face a fundamental challenge that the syntactic distribution of training data is partially observable in comparison to the real world. In this paper, we propose a syntactically robust training framework that enables models to be trained on a syntactic-abundant distribution based on diverse paraphrase generation. To tackle the intrinsic problem of knowledge deformation of paraphrasing, two algorithms based on semantic similarity matching and syntactic tree walking are used to restore the expressionally transformed knowledge. The training framework can be generally applied to other syntactic partial observable domains. Based on the proposed framework, we build a new evaluation set called CaRB-AutoPara, a syntactically diverse dataset consistent with the real-world setting for validating the robustness of the models. Experiments including a thorough analysis show that the performance of the model degrades with the increase of the difference in syntactic distribution, while our framework gives a robust boundary.

## introduction open information extraction (openie) involves converting natural text to a set of n-ary structured tuples of the form (arg 1 , predicate, arg 2 , ..., arg n ), composed of a single predicate as well n arguments. with the advantages of domain independence and scalability, openie serves as a backbone in natural language understanding and fosters many applications such as text summarization @xcite and question answering @xcite . tremendous efforts have been devoted to build models that can better fit the extractions from texts @xcite @xcite * corresponding author: xubin@tsinghua.edu.cn the left and right vertical axes represent the f1 scores of the model and the distance between the training set and the clustering center of each subset, respectively. @xcite @xcite . however, a major issue remaining in openie is the syntactic partial observability -the syntactic distribution on the existing training set is only based on partial observations, and it is far from covering the entire syntactic hypothesis space in the real world. this issue creates a challenge that the models rely heavily on the syntactic forms during training, and degrade significantly when the syntactic distribution changes in the real world. an evaluation is shown in figure 1 . we cluster the carb @xcite samples based on the hw-syntactic distance (introduced in sec. 3.5), which is an effective metric that measures the syntactic difference between two sentences, and evaluate the state-of-the-art model trained on the ope-nie4 data @xcite on them. a frustrating result shows that the model performance exhibits a significant degradation as the syntactic similarity between the training set and clustering centers of subsets decreases. the biased performance comes from the inconsistency of the syn-tactic distributions among data. for example, in figure 1 , the model achieves a depressing f1 score of 0.47 on the subset 5 with the lowest average syntactic similarity to the training set. therefore, to build robust openie systems, we need to train the models on a sufficient syntactic distribution. however, it is not trivial to obtain data that are both diverse and accurate to satisfy the distribution assumption. first, it is extremely expensive and almost impossible for human annotators to provide a large corpus with diverse syntactic expressions. second, existing distant supervision-based methods are not applicable to openie due to the uncertainties of both the type and form of arguments and predicates. humans learn syntactic grammar by paraphrasing the same meaning into different expressions. for example, the following two sentences convey the same meaning in different syntactic forms. the diverse paraphrases of normal-scale training data can guarantee sufficient syntactic distribution. however, an intrinsic problem that hinders the efficiency of this approach is the knowledge deformation. in the following example, it is difficult to reveal the source object earth in the target paraphrase b as it has been transformed into the form of the name of the planet with different syntax. in this paper, we propose a syntactically robust training framework that enables openie models to be trained on a syntactic-abundant distribution based on the diverse paraphrase generation. specifically, we first generate a large-scale syntactically diverse paraphrase candidates set for the training data based on an off-the-shelf paraphrase generator. then, we propose two adaptive algorithms to recover the deformed arguments of the original knowledge, a semantic similarity-based matching method to locate the disordered arguments and a syntactic tree walking-based method to complete the consecutive spans. we further employ the generative t5 @xcite model to restore the deformed predicates as there are potential tense and voice changes in the target paraphrase. finally, a simple but effective denoising method is utilized to prevent the impact of false positives in training. to exhaustively validate the syntactic robustness of openie models in the real-world setting, an additional evaluation set including diverse paraphrases and knowledge triples has been built on the basis of carb. we conduct experiments on the standard and our proposed evaluation sets based on the division of different syntactic categories, and a comprehensive analysis shows that the model performance decreases with increasing the difference in the syntactic distributions, while our training framework gives a robust boundary. ## experiment this work proposes a syntactically robust training framework including two knowledge restoration strategies. therefore, our experiments are intended to demonstrate the effectiveness as well as the robustness of the proposed framework on test sets. ## related work open information extraction is a fundamental nlp task with a long research history @xcite . traditional models adopt rulebased or statistical methods incorporating syntactic or semantic parsers to extract knowledge tuples @xcite @xcite @xcite @xcite . recently, neural models that either adopt sequence label-ing strategies @xcite @xcite @xcite , or leverage sequence generative paradigms @xcite @xcite have achieved promising result. to alleviate the problem that neural models rely heavily on labor-intensive annotated data, @xcite proposes an unsupervised method that pretrains the model on synthetic data automatically labeled by patterns and then refines it using the rl process. paraphrase generation has proven to be useful for adversarial training and data augmentation @xcite . early methods adopt hand-crafted rules @xcite , synonym substitution @xcite , machine translation @xcite , and deep learning @xcite to improve the quality of generated sentences. to acquire syntactic diverse samples, recent studies involve reinforcement learning @xcite or syntactic constrains @xcite @xcite into the models. ## conclusion in this paper, we focus on solving the problem of partially observable of syntactic distribution on training data, and propose a syntactically robust training framework that enables openie models to be trained on a syntactic-abundant distribution based on diverse paraphrase generation. we propose a knowledge restoration algorithm to recover the deformed triples in syntactically transformed sentences based on semantic similaritybased matching and syntactic tree walking. to investigate the syntactic robustness of models, we build a syntactically diverse evaluation set that is consistent with the real-world setting. the experimental result with extensive analysis demonstrated the efficiency of our framework.