Instruction Fine-Tuning: Does Prompt Loss Matter? (2024)

Abstract: We present a novel study analyzing the effects of various prompt loss token weights (PLW) for supervised instruction fine-tuning (SIFT). While prompt-masking (PLW = 0) is common for SIFT, some fine-tuning APIs support fractional PLWs and suggest that using a small non-zero PLW can help stabilize learning when fine-tuning on short-completion data. However, there has never been a study confirming this claim, and OpenAI, a major cloud-based SIFT provider, recently removed this parameter from their fine-tuning API. We found that performance of models fine-tuned on short-completion data had a statistically-significant negative quadratic relationship with PLW. Using small values (0.01 − 0.5) of PLW produced better results on multiple-choice and short-generation benchmarks (outperforming models fine-tuned on long-completion data) while large values (≈ 1.0) of PLW produced better results on long-generation benchmarks. We explained this effect and verified its importance through additional experiments. This research serves as a warning to API providers about the importance of providing a PLW parameter for SIFT.

## introduction recent research in language modeling has made huge advances in training instruction-following agents. both supervised fine-tuning (sft) and reinforcement learning (rl) have been employed to much success. however, our understanding of optimal hyperparameters and standards of practice (sops) have been slow to catch up. this research contributes to supervised instruction finetuning (sift) sops via an in-depth analysis of a single training hyperparameter: prompt loss weight (plw). while training, model parameters are updated by optimizing for next-token maximal likelihood classification. most open sourced solutions for sift either mask the prompt loss (for prefix language modeling) or use the entire sequence loss (for full language modeling) while some api providers support an explicit plw parameter that allows users to apply fractional plw during sift. the commonlyheld notion is that fractional plw helps stabilize learning when fine-tuning on data with short outputs. recently, however, openai quietly removed support for their prompt_loss_weight parameter. the reason for the removal is unknown. furthermore, to our knowledge, there has never been a proper study on the effects of plw. 1 we make the following contributions: we provide relevant background and hypotheses in section 2 and 3, respectively. the main regression experiment is presented in sections 4 and 5. supplemental experiments further validating our claims are presented in section 6. we present conclusions in section 7 followed by several appendices for additional analysis and discussion. ## hypotheses based on openai's explanation for prompt_loss_weight, we expected that for sift with short-completion data and small values of plw, there would be a positive relationship between plw and downstream performance. however, training a model to maximize nexttoken-prediction on prompt tokens should be most useful for generating instruction data, and over-prioritizing prompt token loss should have a negative influence on downstream performance. based on these assumptions, we would expect the two competing factors to result in a downward curved relationship between plw and downstream performance. limiting plw to the range of [0, 1], we postulate that there is a critical value λ for plw with 0 <= λ <= 1. for plw less than λ, the positive effect dominates the negative effect and for values greater than λ, the negative effect dominates the positive effect. if λ = 0, then plw's contribution to model performance is strictly negative, and if λ = 1, then plw contributes strictly positively to model performance. note that λ would not be an intrinsic characteristic of the dataset, model architecture, or training algorithm. rather, it would depend on numerous factors and change for each task. we then made two hypotheses to test the above relationship and performed regression analysis on three fine-tuning datasets spanning a range of r g values: 0.08, 3.27, and 7.83. null hypothesis (h 0 ) prompt loss weight has no relationship with model performance. alternative (h 1 ) prompt loss weight has a quadratic relationship with model performance. we used the standard α = 0.05 significance level, and we expected to reject h 0 only for models trained on short-completion data. ## methodology to evaluate the effect of plw on downstream performance, we used a factorial design methodology and repeated the alpaca experiment @xcite 2023) with three experimental variables. we tested ten discrete levels of plw, two pre-trained language models (ptlms), and three instruction finetuning datasets for a total of sixty experimental training runs and evaluated each run on thirteen benchmarks. we used the original alpaca code and transformers library, only modified to add plw. training was performed exactly as per the original alpaca experiment, and we used the hyperparameters suggested by the authors, modifying only the three experimental parameters (plw, ptlm, dataset) with each run. we provide additional details for reproducibility in appendix e and will release our trained models on huggingface's hub. ## results and discussion a visualization of the simple and min-max scaled relative performance aggregates by w p and dataset is presented in figure 1 . note that all analysis was performed using the relative aggregate. the simple aggregate is dominated by large performance changes on a few benchmarks and is included only for completeness. ## supplemental experiments in this section, we present supplemental experiments that suggest that plw cannot be replaced by alternative regularization techniques for sift and that the effects of plw extend to other shortcompletion datasets. since the translation benchmarks showed high levels of noise and unclear correlation with plw in the main experiment, they were not included in supplemental experiments. ## conclusion in this study, we explored the effects of prompt loss weight (plw) on llm supervised instruction fine-tuning (sift). we found that plw had a statistically significant effect on learning for our short-completion dataset, and proper tuning of plw allowed shortcompletion-trained models to outperform longcompletion-trained models on short-generation benchmarks. we showed that the causal mechanism was due to a balance between two different regularizing effects and not due to increased training stability as is commonly attributed. we showed that the measured relationship extends to additional sift datasets and that the effects could not be sufficiently emulated with other regularizers. based on the above conclusions, we assert the following two points. 1. since models fine-tuned on short-completion datasets and with properly tuned plws outperformed all other models on short-generation benchmark tasks, we conclude that plw is critical for effectively fine-tuning for downstream short-generation tasks.