Zero-Shot Information Extraction as a Unified Text-to-Triple Translation (2021)

Abstract: We cast a suite of information extraction tasks into a text-to-triple translation framework. Instead of solving each task relying on task-specific datasets and models, we formalize the task as a translation between task-specific input text and output triples. By taking the task-specific input, we enable a task-agnostic translation by leveraging the latent knowledge that a pre-trained language model has about the task. We further demonstrate that a simple pre-training task of predicting which relational information corresponds to which input text is an effective way to produce task-specific outputs. This enables the zero-shot transfer of our framework to downstream tasks. We study the zero-shot performance of this framework on open information extraction (OIE2016, NYT, WEB, PENN), relation classification (FewRel and TACRED), and factual probe (Google-RE and T-REx). The model transfers non-trivially to most tasks and is often competitive with a fully supervised method without the need for any task-specific training. For instance, we significantly outperform the F1 score of the supervised open information extraction without needing to use its training set.

## introduction information extraction refers to the task of automatically extracting structured information from unstructured resources, benefiting a wide range of applications such as information retrieval and knowledge base population. information extraction covers a great variety of tasks in natural language processing (nlp), such as open information extraction and relation classification. for example, given a sentence "born in glasgow, fisher is a graduate of the london opera centre", open information extraction seeks to extract (fisher; born in; glasgow), and "city_of_birth" is predicted as the relation between a given pair of entities "fisher" and "glasgow" for relation classification. most current approaches design task-specific pipelines for different information extraction tasks. yet, this presents two limitations for information extraction. first, since most of the approaches employ a task-specific model, it is difficult to leverage a single pipeline to solve many tasks or adapt a model trained on one task to another without changing any task-specific modules. second, those supervised state-of-the-arts are trained on task-specific corpora to predict from a fixed set of task-specific categories, which restricts their usability since additional labeled data is needed to specify any other classes. such task-specific labeled data is scarce in information extraction. for example, the largest training set for open information extraction contains only 3,200 sentences @xcite . motivated by this, we aim to solve information extraction tasks within the same framework in a task-agnostic setting. in this paper, we propose a unified framework for information extraction. the basic idea is to treat every information extraction problem as a "text-totriple" problem, i.e., translating input text to output triples. we successfully apply our framework to three information extraction tasks, greatly improving zero-shot performance on many datasets and sometimes even reaching competitiveness with the current state-of-the-art fully supervised approaches. figure 1 shows how different information extraction tasks are handled within our framework. the framework encodes task priors in the input text and decodes the output triples to finally produce task predictions. we achieve this by leveraging the same translation process on all tasks, the only difference among tasks being the input encoding. this is in contrast with previous approaches using task-specific models and datasets. the design of the common translation module for all tasks is important: by leveraging the task priors encoded in the input text, we enable the zero-shot transfer of the general knowledge that a pre-trained lm has about the task. we demonstrate that a simple pretraining task of predicting which relational triple goes with which text on a task-agnostic corpus further enhances the zero-shot capabilities on all tasks. to the best of our knowledge, this is the first framework to handle a variety of information extraction tasks in a zero-shot setting. our contributions are summarized below. ## method we cast a suite of information extraction tasks into a text-to-triple translation framework. as shown in figure 1 , input and output are designed in a format that is appropriate for a given task. the translation process takes the input text and produces triples as output. the decoding step generates task predictions from the output. in this section, we describe the input and output format, the translation, and the decoding process. we use open information extraction (oie) as a running example in this section. for oie, we are given a sentence and asked to extract triples. ## experiments in this section, we show that deepex framework solves the different information extraction tasks considered and outperforms the task-specific stateof-the-art results on multiple datasets. to keep the framework as simple as possible, most settings and hyperparameters are shared across all experiments. for example, we use bert large @xcite for the beam search of the generating stage (sec. 2.2) for all tasks. the details of the experimental setup, datasets, and comparison methods are described in appendix a. ## related work relation classification aims to identify the correct relation type from a pre-defined set of relations between two given entities. language models (lm) @xcite pre-trained with self-supervised @xcite objectives, e.g., bert @xcite , gpt @xcite @xcite , roberta @xcite , transfer well to relation classification datasets in fine-tuning @xcite or few-shot regime @xcite with architecture modifications. sequenceto-sequence models, such as t5 @xcite , bart @xcite and glm @xcite , are adapted to the task based on data augmentation and fine-tuning @xcite . besides relation classification, @xcite generalize t5 to some more structured prediction tasks as well, e.g., semantic role labeling and event extraction. however, deepex enables zero-shot relation classification that does not require any taskspecific training. many open information extraction (oie) systems, e.g., stanford openie @xcite , ollie @xcite , reverb @xcite , and their descendant open ie4 leverage carefully-designed linguistic patterns (e.g., based on dependencies and pos tags) to extract triples from textual corpora without using additional training sets. recently, supervised oie systems @xcite @xcite formulate the oie as a sequence generation problem using neural networks trained on additional training sets. similar to our work, @xcite use the parameters of lms to extract triples, with the main difference that deepex not only improves the recall of the beam search, but also uses a pre-trained ranking model to enhance the zero-shot capability. lms are used in factual probing tasks, by using the outputs alone @xcite to answer the relation-specific queries in cloze statements. @xcite additionally feed sentences expressing the facts to the lms and shows improved results. other than template-based queries, learning trigger-based @xcite and continuous prompts @xcite are helpful in recalling the facts. the main difference is that deepex explores the internal parameters of the lms rather than the outputs, and the results are more interpretable. overall, in contrast to the existing approaches, deepex unifies the open information extraction, relation classification, and factual probe under the same framework in zero-shot settings. ## conclusion we have demonstrated that our unified approach can handle multiple information extraction tasks within a simple framework and shows improvements in zero-shot settings. unlike previous approaches designing complicated task-specific pipelines, deepex enables conducting all considered information extraction tasks with only input and output design. therefore, deepex is flexible and can be adapted to a variety of tasks. different from previous approaches that target pre-defined categories (e.g., fixed relation types for relation classification), deepex generalizes better to unseen classes as the generating stage leverages the transfer of latent knowledge that a pre-trained language model has about the tasks. besides, the ranking stage pre-trains on a large-scale task-agnostic dataset. deepex exhibits strong zero-shot capabilities in low-resource tasks without the need of any task-specific training set. deepex also exploits the in-depth information of the language models, i.e., parameters, rather than the outputs alone, which en-hances the explainability through enhanced model transparency. based on our findings, we believe that the unified approach advances the research in understanding natural language semantics (e.g., structure prediction tasks) using deep learning models. we hope our results will foster further research in this direction.