Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications (2023)

Abstract: Instruction Fine-Tuning (IFT) is a powerful paradigm that strengthens the zero-shot capabilities of Large Language Models (LLMs), but in doing so induces new evaluation metric requirements. We show LLM-based metrics to be well adapted to these requirements, and leverage them to conduct an investigation of task-specialization strategies, quantifying the trade-offs that emerge in practical industrial settings. Our findings offer practitioners actionable insights for real-world IFT model deployment.

## introduction adapting pre-trained language models (lms) for specific applications is central in industrial nlp to unlock task-specific performance gains and strengthen model alignment with industry requirements. a paradigm gaining traction is the use of instruction fine-tuned (ift) models, lms capable of following arbitrary instructions expressed in natural language @xcite @xcite . researchers primarily concentrate on improving general-purpose ift models to be used as versatile agents capable of executing instructions expressed in natural language @xcite @xcite . in an industrial setting, prompting chatgpt to improve the wording of an email, or to assist with a code snippet would be instances of this zero-shot utilization scenario, which we define as s 0 . critical industrial llm applications may however not always align with s 0 , and often prioritize two other settings. the first scenario, s 1 , requires extending a generalist ift model's capabilities to new specific tasks not included in the original instruction training set. the second scenario, s 2 , centers around converting ift models into specialized models proficient exclusively on specific tasks. in s 1 for instance, a large company may want an llm assistant for internal employee use, and decide to extend an openly available chat model by training it to write memos with a specific templating scheme, to respond to internal faqs, and to use internal coding tools, all the while retaining the original chat assistant's general purpose abilities. in s 2 , that same company is only interested in a given specific task; extracting specific information from business documents, and specializes an ift model for that purpose, aiming to leverage prompting and the generalization capabilities of the model for a more data-efficient training. in this paper, we thoroughly examine s 1 and s 2 by investigating the learning dynamics of specializing ift models through a practical lens. to ensure the reliability of our tooling and the rigor of our conclusions, we first undertake a critical assessment of the current evaluation practices employed for ift models. formally, our contributions are: contribution 1. ift models are designed to handle tasks of diverse natures and varying difficulties. however, current metrics used to measure their performance are often task-specific @xcite , or rely on automatic metrics designed for other intended purposes @xcite . to address this limitation, we introduce two new requirements for metrics used to evaluate ift models: comparability across task (cat) and task and format agnostism (tfa). cat imposes for metric scores to exhibit consistency across a diverse set of generative tasks, in contrast to the sole traditional focus of consistency within a specific task. tfa defines the need for metrics to demonstrate robustness to variations in the output formats. by highlighting the shortcomings of existing metrics in meeting cat and tfa, we present compelling evidence that using llms as scoring agents is a viable evaluation alternative of ift models. contribution 2. we approach our examination of s 1 and s 2 from a practical perspective and focus on the trade-off between data availability and overall performance. our analysis uncovers two distinct phases of learning during ift model specialization: learning to format, and learning to solve tasks. subsequently, we showcase how practitioners can (i) leverage synthetic data to facilitate learning the desired formatting aspects and (ii) use ift models to reduce the need of expert data in industrial scenarios. our study provides practical insights and actionable recommendations to practitioners looking to deploy ift models in production settings. foot_0 2 re-evaluating ift model evaluation