Small Language Models Improve Giants by Rewriting Their Outputs (2024)

Abstract: Despite the impressive performance of large language models (LLMs), theyoften lag behind specialized models in various tasks. LLMs only use a fractionof the existing training data for in-context learning, while task-specificmodels harness the full dataset for fine-tuning. In this work, we tackle theproblem of leveraging training data to improve the performance of LLMs withoutfine-tuning. Our approach directly targets LLM predictions without requiringaccess to their weights. We create a pool of candidates from the LLM throughfew-shot prompting and we employ a compact model, the LM-corrector (LMCor),specifically trained to merge these candidates to produce an enhanced output.Our experiments on four natural language generation tasks demonstrate that evena small LMCor model (250M) substantially improves the few-shot performance ofLLMs (62B), matching and even outperforming standard fine-tuning. Furthermore,we illustrate the robustness of LMCor against different prompts, therebyminimizing the need for extensive prompt engineering. Finally, we show thatLMCor can be seamlessly integrated with different LLMs at inference, serving asa plug-and-play module to improve their performance.

## introduction large language models have recently demonstrated near state-of-the-art performance on various tasks via in-context learning, which enables them to generate outputs based on instructions and a handful of examples, without task-specific training @xcite . however, the effectiveness of this paradigm can vary significantly depending on the task instruction @xcite @xcite , the quantity, relevance and even the order of the in-context examples @xcite correct the grammatical errors in the following sentence: dylan which won a nobel prize is an americna musician. the nobel prize was awarded to the americna musician bob dylan. dylan is a musician from the us who won the nobel prize. dylan who won the nobel prize is an american musician. dylan which won a nobel price is an american musician. ## correcting the outputs of llms in this section we present our computationally efficient approach that utilizes a small model, lmcor, to correct the predictions of an llm for a specific task. unlike traditional fine-tuning methods, our approach does not require access to the weights of the llm. instead, as seen in figure 1 , we interact with the llm only through an api, as is the case for some state-of-the-art commercial llms. ## related work since the introduction of in-context learning, prior research has primarily focused on improving the few-shot performance of llms. one approach suggests prompting the model to generate rationales or chain-of-thoughts @xcite @xcite or to decompose the problem into simpler ones @xcite @xcite , simulating a reasoning process prior to generating the answer. these prompting techniques are complementary to our approach and can provide improved candidates for the corrector. another strategy to improve the performance of llms without training is reranking, i.e. selecting the most promising from a pool of candidates obtained by sampling from the model. reranking approaches include training different models as the ranker @xcite , using task-specific ranking functions @xcite , majority voting @xcite or minimum bayes risk decoding (mbrd) @xcite . although these approaches can improve the few-shot performance of an llm, they are upper-bounded by the quality of the generated candidates. while fine-tuning large llms can enhance performance, the substantial computational requirements have prompted the development of parameter-efficient fine-tuning methods (peft) @xcite . these approaches introduce a small number of additional parameters (relative to the full model) to be trained while the rest of the model is frozen. the newly-added parameters can come in the form of embeddings that are appended to the encoded sequence @xcite , mlps that are added in-between layers, namely adapters @xcite or rankdecomposition matrices that are added in parallel to the existing layers @xcite . although these works decrease the computational load of fine-tuning they still require loading and backpropagating through the model, which can be prohibitive for llms. our work shares the same motivation with peft methods, with the introduced parameters being essentially another, smaller model. however, our method does not have the memory requirements of peft since the corrector operates directly on the model's outputs and does not require access to the model's weights. an alternative line of work proposes providing feedback to the llm in order to revise and enhance its predictions. the feedback can be obtained from external models such as google search, document retrievers, compilers @xcite @xcite , or from a separate model trained to provide feedback on llm outputs with additional supervision @xcite @xcite . while leveraging the llm itself to generate feedback has been explored @xcite , it tends to yield lower-quality feedback @xcite @xcite and involves multiple passes and extensive prompt engineering for each llm operation. in contrast, our approach is task-agnostic and requires a single pass from the llm, with little to no prompt engineering, offering an efficient solution for enhancing llm outputs. recently, studies have highlighted the potential of smaller, task-specific models to complement the predictions of an llm. @xcite explore a framework where candidates produced by taskspecific models are fed to an llm, primarily targeting classification task while lmcor is bettersuited for open-ended generation tasks. @xcite train a smaller model to iteratively improve sequences generated by llms. in contrast to our method, they rely on unlabeled data and sample extensively from the llm to obtain a large pool of candidates. they assume the availability of a value function that assigns scores to each candidate and create input-output pairs by sorting candidates based on their scores. unlike their approach, we demonstrate that a compact corrector can perform effectively across various tasks. additionally, our approach is more efficient during inference, since the ability of lmcor to process multiple candidates simultaneously eliminates the need for multiple passes. concurrently, researchers have begun to leverage the complementary nature of llm-generated outputs during inference. @xcite use an llm to combine its generated outputs for machine translation, although they find that reranking methods incorporating external modules, such as quality estimation metrics @xcite , prove to be more efficient. meanwhile, @xcite propose an approach that uses a quality estimation metric to combine the outputs of llms or mt models. similar to our method, they exploit the diversity of llm outputs by identifying divergent spans among candidates and merging them based on the metric. most relevant to our approach is the work by @xcite where they propose a method to ensemble llms. their pipeline consists of i) sampling a large pool of candidates, ii) selecting top candidates via multiple pairwise comparisons through a trained reranker and iii) fusing them using a similar technique as lmcor. while our approach could be extended to multiple llms we demonstrate improvements with a single llm, leveraging the complimentarity of the generations. in addition, our approach is more efficient since we use a much smaller model as the corrector (3b vs 250m) and do not introduce additional training and inference steps for ranking the outputs. ## conclusion in this work, we introduce lmcor, a novel approach that leverages a small corrector module to enhance the performance of llms in the presence of training data. lmcor leverages the diversity of the llm generations to rank, edit and combine the candidates. unlike parameter-efficient fine-tuning methods, our approach does not require access to the model or substantial computational resources. our experiments demonstrate that even a relatively small corrector (250m) can improve the performance of a much larger lm (62b), while exhibiting robustness against different prompts. furthermore, we showcase that the corrector can be successfully applied to models of different scale or architecture without any retraining. these findings offer a promising solution for improving llm performance in a practical and resource-efficient manner and open up new possibilities for the utilization and deployment of llms in real-world applications alongside smaller task-specific models. sampling multiple candidates in parallel and performing an additional inference step using a much smaller model. although the additional latency is small, it could be critical for low-latency applications that require real-time responses. strong fine-tuned models while our approach demonstrates gains over in-context learning and reranking it may not always achieve the same level of performance as fine-tuning approaches. our results in xsum and figure 3 indicate that fine-tuning remains a powerful method for smaller models when ample data is available. additionally, scaling fine-tuned models instead of using an off-the-shelf llm might be a better alternative in certain cases, as discussed in section 5.2. additional tasks and models due to time and budget limit our experiments cover 4 natural language generation tasks and they could be extended to other kinds of tasks such as reasoning. additionally while lmcor shows promising results when combined with different llms, even during inference, it would be interesting to apply our approach to a broader selection of llms that are currently available. human evaluation we agree that automatic metrics have limitations. while the selection of metrics aligns with prior work, human evaluation could provide us with more reliable and comprehensive evaluation results. however, due to the number of models and the amount of generation candidates, we could not afford large-scale human evaluation.