TSTR : Target Similarity Tuning Meets the Real World (2023)

Abstract: Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs) to improve performance. Its goal is to adapt a sentence embedding model to have the similarity between two NL inputs match the similarity between their associated code outputs. In this paper, we propose different methods to apply and improve TST in the real world. First, we replace the sentence transformer with embeddings from a larger model, which reduces sensitivity to the language distribution and thus provides more flexibility in synthetic generation of examples, and we train a tiny model that transforms these embeddings to a space where embedding similarity matches code similarity, which allows the model to remain a black box and only requires a few matrix multiplications at inference time. Second, we how to efficiently select a smaller number of training examples to train the TST model. Third, we introduce a ranking-based evaluation for TST that does not require end-to-end code generation experiments, which can be expensive to perform.

## introduction code generation from natural language utterances is an important and useful ability of large language models (llms). experienced developers can save time, and less experienced users can use natural language to perform data transformation tasks that they would otherwise have to carry out manually @xcite . improving the code generation capabilities of llms is a popular research area @xcite . target similarity tuning (tst) was proposed as a method for selecting relevant examples to exploit the in-context learning ability of llms and improve performance @xcite . ## related work code generation from natural language is a popular area of research @xcite . instead of starting from scratch, fine-tuning a pretrained natural language understanding model to generate code is a popular approach, for example, codebert @xcite was trained from bert, codet5 @xcite was trained from t5 and codex @xcite was trained from gpt-3. a powerful method to generate code from natural language is prompting large language models @xcite . as opposed to fine-tuning, this does not require large training datasets and expensive compute. few-shot prompting consistently improves the performance of llms across a variety of tasks @xcite . besides helping the model pick the correct programming language @xcite the provided few-shots can teach the model about specific functions or parameters and contextualization. one way of selecting relevant examples is by using sentence embeddings @xcite . in some cases, however, similar natural language does not correspond to similar code, and vice versa. synchromesh @xcite introduced target similarity tuning (tst) to address this challenge and fine-tunes the sentence embedding similarity to match the associated code similarity. this work builds on the concept of tst and improves on important implementation details for training (selecting examples and allowing synthetic data generation), evaluating (cheap evaluation with a proxy metric) and deploying (api call and small fcnn do not have hardware requirements) tst in practice. ## tst r we briefly recap tst, how that transfers to tst r and how examples closer to the relevant decision boundary are selected to improve training ## evaluation setup this section describes the datasets, metrics, models and hyperparameters used in our experiments. foot_0 ## evaluation we perform experiments to compare tst r against baseline embedding retrieval methods (5.1), we show that ranking relevant examples serves as a proxy metric to optimize hyperparameters without llms (5.2), we evaluate how embeddings from large models (ada) are more robust with respect to variations in language (5.3), and we show the effect of selecting relevant examples to train tst r (5.4). ## conclusion we introduce tst r as a practical improvement of tst for selecting relevant examples in code generation from natural language. tst r replaces a fine-tuned sentencebert model with a small, trainable transformation on top of a frozen embedding model, and provides a strategy for selecting better training examples. additionally, we show that tst can be evaluated on pairs of utterances from the training set that are ranked with respect to a reference utterance from the testing set, which does not require end-to-end code generation. our experiments show that tst r outperforms classical tst when the language distribution of the example bank does not match that of the tests, that selecting examples closer to the relevant decision boundary improves performance, and that a pairwise ranking evaluation correlates to end-to-end code generation performance.