Functionality learning through specification instructions (2024)

Abstract: Test suites assess natural language processing models’ performance on specific functionalities: cases of interest involving model robustness, fairness, or particular linguistic capabilities. This paper introduces specification instructions: text descriptions specifying fine-grained task-specific behaviors. For each functionality in a suite, we generate an instruction that describes it. We combine the specification instructions to create specification-augmented prompts, which we feed to language models pre-trained on natural instruction data.We conduct experiments to measure how optimizing for some functionalities may negatively impact functionalities that are not covered by the specification set. Our analyses across four tasks and models of diverse sizes and families show that smaller models struggle to follow specification instructions. However, larger models (> 3B params.) can benefit from specifications and—surprisingly—even generalize certain desirable behaviors across functionalities.

## introduction test suites @xcite @xcite have been proposed as an evaluation framework to test for specific functionalities in natural language processing (nlp) models. each functionality is a set of test cases, generally input-output pairs, relating to a particular aspect of a task. for example, a test suite for hate speech detection can assess distinct expressions of hate (e.g., implicit derogation), while a sentiment analysis suite can measure how well a model handles specific phenomena (e.g., negation). suites complement the standard practice of evaluating on a held-out test set from the same distribution of the training set @xcite . if the latter is representative of the underlying task distribution, it is a good measure of average correctness; suites, on the other hand, allow for in-depth evaluation of relevant phenomena that may be underrepresented in general data. though test suites can point to failure cases, there are no clear guidelines on how to act upon their feedback to develop more robust and trustworthy models. data augmentation has been suggested as a potential avenue for improvement @xcite by including additional training cases that correspond to the suite's cases. however, constructing or annotating instances targeting specific functionalities is costly, and further training models is expensive for large models and infeasible for closed-source ones. furthermore, fine-tuning models on suites' test cases has been shown to help seen functionalities @xcite @xcite , but often does not generalize to unseen ones and harms general performance (luz de @xcite @xcite . missing from the literature are analyses for the increasingly influential paradigm of prompting large language models (llms) @xcite , which has superior zero-and few-shot capabilities, particularly for models trained on natural language instructions @xcite . prompting with functionality information may improve relevant aspects of model behavior with no need for fine-tuning, which requires additional training data and computational resources. since the model parameters are not updated, prompting is also less vulnerable to overfitting to seen functionalities, a substantial limitation in previous work. this paper explores specification instructions and their effect on functionality performance. contrary to previous efforts, we do not expose the model to suite examples or fine-tune it. instead, we elicit the desired behavior by augmenting prompts in this task, you are given a sentence. you must output the sentence sentiment. follow these rules: 1. single positive words should be positive 2. single negative words should be negative @xcite 38. negation of positive statement should be negative select the most relevant rules, briefly explain how they apply and output exactly the correct option in a new line. {rule list} and {rationale} indicate where the list of relevant rules and the rationale behind the selection should be generated. question: is the sentiment of the following sentence positive or negative? the year 2002 has conjured up more coming-of-age stories than seem possible , but take care of my cat emerges as the very best of them options: -negative -positive rules: {rule list} explanation: {rationale} answer: positive [...] question: is the sentiment of the following sentence positive or negative? bullock 's complete lack of focus and ability quickly derails the film options: -negative -positive rules: specification instructions rationale exemplars task+input figure 1 : example of a specification-augmented prompt for sentiment analysis. each module adds information about how the task is expected to be performed. with instructions that specify the suite's functionalities. for example, if a sentiment analysis suite contains a functionality that tests whether predictions are invariant to nationalities mentioned in the input, an instruction such as "nationality should be irrelevant to sentence sentiment" would be added to the task prompts. our main contributions are: ## analysis of prompt methods and components we expected specification instructions to improve suites' scores (fig. 2 , bottom row) more than datasets (top row), because specification instructions are guaranteed to correspond to suite instances and only occasionally to dataset instances. we validate this intuition by comparing the average dataset and average suite performance difference between specification-augmented prompts and their corresponding baseline (fig. 8 in appendix j). while flan-t5-base and larger models benefit from specification instructions considering suite performance, only xl and larger models could improve dataset performance. ## analysis of specification interaction specification-augmented prompts include dozens of instructions that can interact with each other to affect model prediction in surprising ways. to analyze the interaction between specifications, we we discuss the remaining pairs in appendix h. ## related work instruction-following models this work uses llms fine-tuned on instruction data, where tasks are described by natural language instructions @xcite @xcite . such llms have been show to generalize to unseen tasks @xcite @xcite . our specification instructions differ from traditional instructions: these describe the task (e.g., "output the sentiment of the following sentence"). in contrast, the specification instructions prescribe the expected behavior for specific cases (e.g., "the speaker's sentiment should outweigh other opinions"). instruction induction instead of using models to follow instructions, an emerging line of work prompts models to generate instructions @xcite @xcite . our chatgpt-generated specification instructions can be seen as a form of instruction induction. an important difference is that previous works prompt the model with input-label pairs and ask it to infer the underlying task. our prompts, instead, include the task name and ask the model to infer the underlying labeling rule for the presented exemplars. ## conclusion we have studied specification-augmented prompts as a fine-tuning-free way of eliciting llms to adopt fine-grained task-specific behaviors. our results have shown that specification instructions can improve suite and dataset performance of large models. that was true for human and chatgptgenerated specification instructions, though the former were mostly better. our cross-functional analysis indicated that improvements are not restricted to the covered functionalities but extend to held-out ones. our analysis of specification interaction shows how the specification-augmented prompts' effect differs across functionalities: instructions can help to align models to desired task behaviors (e.g., predicting neutral sentiment) but may deteriorate performance when describing linguistic phenomena. we show how specifications impact each other in constructive and destructive ways and how the in-struction set often leads to the same prediction, even if some specifications relevant to the input are excluded. specification-augmented prompts include dozens of instructions, so the predictions result from the interplay of the set of specifications, how they are expressed, the exemplars shown, and the prompt format, among other factors. due to the complexity of these matters, rule and principle-based alignment approaches would benefit from interdisciplinary research on how to design and specify rule systems.