Exploiting Asymmetry for Synthetic Training Data Generation: S ynth IE and the Case of Information Extraction (2023)

Abstract: Large language models (LLMs) have great potential for synthetic data generation. This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in task difficulty makes it possible to produce large-scale, high-quality data for complex tasks. We demonstrate the effectiveness of this approach on closed information extraction, where collecting ground-truth data is challenging, and no satisfactory dataset exists to date. We synthetically generate a dataset of 1.8M data points, establish its superior quality compared to existing datasets in a human evaluation, and use it to finetune small models (220M and 770M parameters), termed SynthIE, that outperform the prior state of the art (with equal model size) by a substantial margin of 57 absolute points in micro-F1 and 79 points in macro-F1. Code, data, and models are available at anonymous.

## introduction large language models (llms) have demonstrated the ability to generate highly fluent and coherent textual data. one promising application of this capability is the generation of large amounts of high-quality synthetic data for training and evaluating smaller models. this becomes particularly useful for tasks where high-quality datasets are not readily available or access to real data is limited or expensive. however, in many complex natural language processing (nlp) tasks, the textual input x is mapped to a structured (rather than free-text) output y, and in such cases, llms may perform poorly as synthetic-data generators, since pretraining did not gear them to produce the specific required output format (even with in-context learning). here x task of interest, hard inverse task, easy for llm figure 1 : exploiting asymmetry for sdg. for hard tasks of interest with input x and output y , the reverse task (from y to x) may be much easier for an llm. if so, we can generate high-quality training pairs (x,y ) by prompting an llm to generate plausible inputs x from outputs y . this often holds true for tasks with structured y , as in closed information extraction, where x would be the input text and y would be the list of (subject, relation, object) triplets expressed in the input text. furthermore, this ensures full control over the sampling distribution p(y ), and thus balanced datasets. a good example of such a task, on which we focus in this work, is closed information extraction (cie). in cie, a model must extract a set of disambiguated triplets (i.e., facts) y from a given text x. these triplets are of the format @xcite , with the subject and object being entities in a knowledge base (e.g., wikidata) and the relation being specified in the knowledge base schema. collecting datasets for this task is time-consuming and expensive, as it requires annotators to know the entire entity and relation catalogs and reason about all possible facts expressed in the text x. as a result, only small or noisy datasets exist to date. the largest dataset available, rebel @xcite , suffers from several problems @xcite : (i) noise: it is collected with a mix of heuristics, and for many data points, the target output y does not contain all the facts expressed in the input x or is (partially) incorrect. (ii) skewness: most relations appear only rarely in the dataset, which results in models that ignore most of the information when the data is used for training and in poor performance estimates when it is used for evaluation. importantly, cie is also not solvable directly by llms such as gpt3.5, as such models are unaware of the entity and relation identifiers of interest (for examples of failures, see appendix a). we show that, although the llm cannot be used to produce output structures directly, we can use it to generate high-quality input texts by reversing the task. ## exploiting asymmetry for synthetic data generation in this section, we demonstrate how to exploit asymmetry for synthetic data generation with cie as an example task. our pipeline, depicted in fig. 2 , comprises three primary components: (i) construction of a knowledge graph (kg) containing the entities and relations of interest; (ii) sampling of coherent triplet sets from the kg with comprehensive coverage of the entities and relations, and (iii) generation of high-quality text, expressing the triplets without any additional triplets. next, we describe these three components in turn. ## synthetic data in action in this section, we evaluate the benefits of training on synthetically generated data. ## discussion implications for cie. the lack of a large, balanced, high-quality dataset has been a significant obstacle for cie. the synthetic data generated by the methodology proposed in this work satisfies all of the desiderata in terms of size, coverage, and quality (cf. sec. 3.4 and sec. 3.5). as an alternative to rebel which greatly suffers from both false positives and negatives, wiki-cie text enables a substantially more accurate evaluation. similarly, wiki-cie code enables the training of models performing well across all relations: (i) in terms of macro performance, synthie's predictions are of higher quality than rebel's gold annotations; (ii) on the highest quality test set (wiki-cie text), synthie pushes the macro-f1 score of 14%, for genie, to 93% (cf. while synthie's performance on rebel clean is better than the original gold annotations, it is still lower than the performance on wiki-cie text. analyzing the errors committed by our models and looking at the rebel data allowed us to uncover an interesting property of cie that, to the best of our knowledge, has not been identified by prior work. assume that a model was trained on exhaustively annotated data (the holy grail which is now within reach) and presented with text containing central information that cannot be linked to the kb, (i) either in theory (e.g., "he plays the guitar") or (ii) under the output constraints assumed by the model. this would place the model in an undesired space of the likelihood distribution where it attempts to express information it cannot express, and is therefore bound to make mistakes. fortunately, mistakes of this type can be avoided by (i) generating training data covering such scenarios and/or (ii) expanding the expressivity of models by extending the catalogs or developing novel output formats beyond subject, relation, object triplets. the latter goes towards exhaustive ie, whereas the former relaxes this desideratum. overall, we believe that sdg could bring us closer to practical cie systems. implications for sdg. this study highlights the efficacy of leveraging asymmetries in difficuly for sdg by developing a specific pipeline for cie. however, the idea can be readily applied to different pipelines for cie, or any other ie or parsing task, such as entity linking, oie, or abstract meaning representation parsing. with cie, which necessitates the most information external to the input (i.e., knowledge of entities, relations, and their ids), arguably being the hardest task, we are confident that applying our approach to these tasks would translate to similar results. finally, the proposed methodology is not limited to tasks with structured output spaces. rather, any problem with an inverse formulation that can be addressed more effectively by the llm can benefit from our approach.