Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models (2024)

Abstract: Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Reject-sampling by Self-instruct with Continued Fine-tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.

## introduction aligning language models (lms) with human preferences becomes increasingly important. one main objective is to train lms to follow human instructions (e.g., answering open-ended questions) while being faithful (e.g., grounding responses in the given context). however, lms often suffer from failing to follow human instructions @xcite @xcite or making up facts that are not grounded in context @xcite @xcite . we trace this problem back to commonly used alignment training datasets, often collected from naturalistic conversations covering a wide range of * equal contribution. ◇ work done at aws ai labs. figure 1 : faithfulness scores on context-dependent tasks (qa and summarization) decrease when we finetune grounded llama-7b checkpoint with instruction following datasets (orange), and instruction following scores (assessed by gpt-4) decrease when we fine-tune vicuna-7b with context-dependent tasks (blue). our method, reset surpasses the vanilla mtl with data mixing, approaching the north star (upper right corner). in this context, we examine the interaction between instruction following (i.e., how well does the lm follow open-ended instructions) and faithfulness (i.e., is the lm's response grounded in the context) during alignment training. specifically, we study how instruction following and faithfulness scores change when adapting lms to two types of datasets: 1) instruction-tuning datasets that are commonly used to train chat-models such as chat- @xcite and llama-2-chat @xcite ; and 2) context-dependent datasets that require grounding to a provided context, and are commonly used to train retrieval-augmented generation (rag) models @xcite ) such as atlas @xcite and dspy @xcite . we observe a clear trade-off between the two scores, as shown in figure 1 : a fine-tuned lm with a competitive faithfulness score becomes much less faithful when separately fine-tuned on instruction following datasets. conversely, an instruction-tuned lm becomes worse at following instructions when fine-tuned on contextdependent datasets. our findings suggest finetuning an lm with either instruction following or context-dependent datasets exclusively may impair its original ability in the other aspect. one natural mitigation strategy is to use multitask learning (mtl) by mixing datasets, which we find as a strong yet sub-optimal baseline. to achieve a more synergic outcome, we propose a simple yet effective method rejection sampling for continued self-instruction tuning (reset). inspired by recent works in self-instruct @xcite , reset leverages the lm to sample generations for instruction following and task-specific datasets, which is different from vanilla mtl. generations are then rated by external judges for instruction following and faithfulness scores, where top-rated generations are collected and used to further fine-tune the lm. our experiments show that when trained with reset using only a single iteration and 8,000 additional fine-tuning examples, lms see substantial gains in faithfulness scores (up to +18.8%) compared to the mtl baseline, while maintaining their instructionfollowing scores. furthermore, we find that less is more: training with reset on higher quality yet three-fold less data yields up to 31.3% improvements on faithfulness among datasets compared to mtl. our analyses shed lights on finding and mitigating objective discrepancies in alignment training where datasets encompass different or even conflicting goals. foot_0 ## background to recap, our goal is to study the interplay of training on instruction following datasets and on datasets that require grounding to a specific context. @xcite , sharegpt, self-instruct @xcite and oasst-1 @xcite . average length is the averaged response token length, which is calculated with training or testing sets for each dataset with the llama-7b tokenizer. to do this, we first construct two separate groups of datasets, each resembling a setting above. we then adopt a two-stage fine-tuning paradigm that allows us to closely study the effect of each type of training. we introduce the setup in this section. ## does context-dependent fine-tuning hurt instruction following? to answer this question, we fine-tune vicuna-7b (llama-7b fine-tuned on sharegpt) on contextdependent datasets. figure 6 shows our results. as expected, our model becomes more faithful, with improved task performance. this improvement is partly because vicuna-7b does not follow our instruction template out-of-the-box. on the other hand, instruction following scores drop by 37.9%, suggesting our model becomes much less human-aligned. similar to previous findings, one natural concern is that context-dependent training makes our model generate much shorter responses, which leads to lower instruction following scores. to rule out this concern, we randomly sample 200 examples from the alpaca-15k dataset and only keep those with a minimal sequence length change (a maximum of 10 new tokens) compared to the corresponding vicuna-7b zero-shot generation. as shown in figure 7 , instruction following scores consistently drop considerably for the filtered setting, regardless of length (36.8% vs. 22.0%). overall, context-dependent fine-tuning negatively affects the instruction following score, as we expected. ## reset: reconciling instruction following and faithfulness as shown in the previous two sections, there are clear objective discrepancies between diverse instruction following datasets and context-grounded ones. in this section, we explore two different mitigation strategies, starting from the mtl baseline to our novel method based on rejection sampling for continued self-instruction tuning (reset), which significantly outperforms mtl. our mtl baseline our mtl simply mixes up the instruction following as well as contextdependent datasets. we up-sample smaller datasets to have an equal number of examples per dataset. ## conclusion in this paper, we demonstrate a clear trade-off between instruction following and faithfulness when fine-tuning lms on datasets with differing objectives. to alleviate this trade-off, we proposed re-set, a simple yet effective iterative method that significantly outperforms the mtl baseline in both instruction following and faithfulness scores. the lightweight and iterative nature of reset makes it extensible for future refinement at minimum cost and integrable with recent instruct-tuning techniques. we contribute to the broader goal of creating more reliable, accurate, and user-aligned language technologies.