Language-to-Code Translation with a Single Labeled Example (2024)

Abstract: Tools for translating natural language into code promise natural, open-ended interaction with databases, web APIs, and other software systems. However, this promise is complicated by the diversity and continual development of these systems, each with its own interface and distinct set of features. Building a new language-to-code translator, even starting with a large language model (LM), typically requires annotating a large set of natural language commands with their associated programs. In this paper, we describe ICIP (In-Context Inverse Programming), a method for bootstrapping a language-to-code system using mostly (or entirely) unlabeled programs written using a potentially unfamiliar (but human-readable) library or API. ICIP uses a pre-trained LM to assign candidate natural language descriptions to these programs, then iteratively refines the descriptions to ensure global consistency. Across nine different application domains from the Overnight and Spider benchmarks and text-davinci-003 and CodeLlama-7b-Instruct models, ICIP outperforms a number of prompting baselines. Indeed, in a “nearly unsupervised” setting with only a single annotated program and 100 unlabeled examples, it achieves up to 85% of the performance of a fully supervised system.

## introduction even with the growing capability and flexibility of language models (lms), many tasks in nlp are best solved by translating language into a program or another formal representation. the program might query a database @xcite , carry out a request @xcite , call an external tool @xcite , or assist with a programming task @xcite . modern lms can predict programs from natural language amazingly well * correspondence to kaj@cs.utexas.edu. ## background translating natural language into code (or other formal representations) is a key sub-task in natural language processing, with applications spanning computational linguistics, natural language interfaces, and dialog systems. historically, many languageto-code systems were based on explicitly structured, compositional models @xcite @xcite . more recent systems have generally used a combination of neural sequence models, large-scale pre-training, and constrained decoding @xcite . large datasets of natural language text and code are widely available, but paired language-program data is comparatively rare. as a result, there has been long-standing interest in building languageto-code systems with less data. one prominent line of work focuses on weak supervision-using natural language questions paired with answers to automatically infer programs that produce the right result when executed @xcite @xcite . most relevant to the this paper, another line of research focuses on semi-supervised approaches, which learn from a mixture of paired language and program data as well as unpaired data in one or both modalities. these approaches typically combine supervised learning with a cycle-consistency objective, alternating between optimizing a language-tocode model on annotated programs, then assigning unlabeled programs natural language descriptions that produce the right code when translated @xcite . icip extends these approaches along several axes-it generalizes semantic parsing with cycle consistency to the incontext learning setting, and our best-performing model variant (maxrt below) employs a filtering criterion not used in previous work. together, these enable icip to obtain significantly improved sample efficiency. outside the domain of language-driven program synthesis, semi-supervised learning procedures based on enforcing cycle-consistency have been used (for pre-training, fine-tuning, and prompting) in tasks spanning machine translation, instruction following, and open-ended text generation @xcite @xcite . we expect that future work might apply icip-type approaches to these problems as well. in a similar spirit to the present work, @xcite also leverage lms' ability to perform an "easy" recognition problem (labeling text with control codes) to bootstrap supervision for a hard inverse problem (controllable text generation). ## experimental setup datasets. in our main experiments, we evaluate semantic parsing on two english-language benchmark datasets: spider @xcite , a text-to-sql task, and overnight @xcite , a benchmark evaluating semantic parsing for question answering in eight application domains spanning calendaring, restaurant search, spatial reasoning, and more. . both datasets contain several thousand labeled program-utterance pairs. the spider and overnight test splits contain 1,034 and 2,740 examples respectively. all of our test numbers report average accuracy over the entire test split. 3 to create the small training sets required to run our method and baselines, we sample m + n examples from the training split, without replacement, using the first m and last n respectively for the labeled set d lπ and the unlabeled set d π . for each condition (m + n ) that we report on, there is some variance in the results due to the particular draw of (d lπ , d π ); thus metrics. we report two measures of program recovery performance: (1) program accuracy, the proportion of predicted programs that exactly match the reference programs associated with a set of test utterances, and (2) answer (or "denotation") accuracy, the proportion of predicted programs that return the same result as their utterance's reference program when executed. in overnight, execution occurs against a fixed model; in spider, each query is executed against a query-specific database. ## analysis we conclude with a qualitative analysis of the predictions and labels generated by icip. we focus on overnight blocks, as it contains a large number of programs in a single domain from which aggregate statistics can be computed, and the maxrt filtering method, which achieves the best overall performance in this domain. ## conclusion we have presented in-context inverse programming (icip), a "nearly unsupervised" procedure for building language-to-code models for new libraries, data sources, or languages starting with a collection of unlabeled programs and as few as one labeled example. icip works by iteratively predicting natural language commands for unlabeled programs that enable reconstruction of the original program with high probability. experiments on multiple datasets and models show that this procedure is effective, substantially outperforming one-and many-shot prompting methods in low-data regimes. our results highlight the fact that pre-trained lms' ability to perform easy tasks, like labeling programs with descriptions, can be used to bootstrap models for harder tasks by encouraging both local consistency (via round-trip constraints) and global consistency (via in-context learning from inferred demonstrations).