Generating Data for Symbolic Language with Large Language Models (2023)

Abstract: While large language models (LLMs) bring not only performance but also complexity, recent work has started to turn LLMs into data generators rather than task inferencers, where another affordable task model is trained for efficient deployment and inference. However, such an approach has primarily been applied to natural language tasks, and has not yet been explored for symbolic language tasks with complex structured outputs (e.g., semantic parsing and code generation). In this paper, we propose SymGen which utilizes LLMs for generating various annotation-expensive symbolic language data. SymGen consists of an informative prompt to steer generation and an agreement-based verifier to improve data correctness. We conduct extensive experiments on six symbolic language tasks across various settings. Compared with the LLMs, we demonstrate the 1%-sized task model can achieve comparable or better performance, largely cutting inference and deployment costs. We also show that generated data with only a few human demonstrations can be as effective as over 10 times the amount of human-annotated data when training the task model, saving a considerable amount of annotation effort. SymGen takes a step toward data generation for annotation-expensive complex tasks, and we release the code at URL.

## introduction in the natural language processing (nlp) literature, the march of scaling language models has been an unending yet predictable trend, with new models constantly surpassing previous ones in not only performance but also complexity @xcite @xcite . such large language models (llms), however, incur a large computational cost in practice, especially when deployed in resource-restricted systems and inference in low-latency applications (bommasani spider [sql query] geoquery [prolog command] mtop [top representation] break [qdmr] input: what is the population of montana? output: answer(a,(population(b,a), const(b,stateid(montana)))) input: resume the timer in 10 seconds output: [in:resume_timer [sl:method_timer timer ] [sl:date_time in 10 seconds ] ] input: list the type of bed and name of all traditional rooms. output: select roomname,bedtype from rooms where @xmath0 input: how many large metallic items are there? output: 1#) return items 2#) return #1 that are large 3#) return #2 that are metallic 4#) return number of #3 figure 1 : sample symbolic language datasets with complex structured outputs. the names of the symbolic languages are shown in square brackets. @xcite . instead of treating llms as edge task inferencers, a recent line of work leverage llms as data generators, with the generated data being used to train more affordable task-specific models for efficient deployment and inference @xcite @xcite . with only a few or even without demonstration examples, the llms can generate highquality data via in-context learning @xcite or prompting @xcite . the task models trained on these generated data can achieve comparable or even better performance than the llms and enjoy a low inference cost at the same time. however, previous work mainly focuses on generating natural language data. to what extent this approach works for complex structured data, such as meaning representation and codes (figure 1 ), remains an open question. the investigation of data generation via llms in the context of such symbolic language tasks is also extremely intrigu-ing for two reasons: 1) the human annotation procedure for these tasks requires expensive domain expert efforts @xcite and carefullydesigned strategies @xcite @xcite ; 2) conventional data augmentation methods aiming at enriching datasets for these tasks require handcrafted rules, a considerable number of expert demonstration examples, and are mostly task-specific @xcite @xcite . to address these issues, we propose symbolic data generation (symgen) for various annotationexpensive symbolic language tasks. symgen works with an llm trained on code (i.e., codex-175b; chen et al. 2021) and optional task-specific structure knowledge (e.g., database for sql; @xcite through prompting or in-context learning. symgen also comprises an agreement-based verification module, in which the outputs are verified by execution (e.g., programs, logical forms) or formatting (e.g., pseudo-logical forms), to ensure high-quality generations. with the generated data, we train efficient task models with around 1% size of codex for task inference (e.g., t5 with size 770m and 3b; @xcite . we experiment on six symbolic languages which are sql, bash, python, prolog, task oriented parsing representation (top-representation; @xcite and question decomposition meaning representation (qdmr; @xcite ). we consider generating data in zero-shot, few-shot, and full data settings. our key research findings include the following: ## symgen to automatically curate numerous data for various annotation-expensive symbolic language tasks, we propose a unified pipeline, named symgen. symgen comprises data generation by prompting llms and data verification by executing or formatting. ## conclusion in this work, we treat llms as data generators rather than task inferencer for complex symbolic language tasks, with the generated data being used to train much affordable model for depolyment and inference. we demonstrate that a 1%-sized model trained under symgen can achieve superior performance to the llm inferencers. we especially show the effectiveness in low-resource scenarios, which is a common situation for symbolic language tasks due to the annotation-expensive characteristics. additionally, we also reveal the possibility of obtaining a well-performed task model through symgen even without any human annotations.