CERET : Cost-Effective Extrinsic Refinement for Text Generation (2024)

Abstract: Large Language Models (LLMs) are powerful models for generation tasks, but they may not generate good quality outputs in their first attempt. Apart from model fine-tuning, existing approaches to improve prediction accuracy and quality typically involve LLM self-improvement / self-reflection that incorporate feedback from models themselves. Despite their effectiveness, these methods are hindered by their high computational cost and lack of scalability. In this work, we propose CERET, a method for refining text generations by considering semantic stability, entailment and inter-sample uncertainty measures. Experimental results show that CERET outperforms Self-consistency and Self-rerank baselines consistently under various task setups, by 1.6% in Rouge-1 for abstractive summarization and 3.5% in hit rate for question answering. Compared to LLM Self-rerank method, our approach only requires 9.4% of its latency and is more cost-effective.

## introduction large language models (llms) like gpt @xcite , claude, palm @xcite , and llama @xcite have showcased unprecedented capabilities in natural language understanding and generation. these models, with parameter counts reaching into the hundreds of billions, have become pivotal in advancing the frontier of natural language processing (nlp). despite their impressive fluency and coherence, language models frequently generate content that is incomplete, biased, or misleading in their initial attempts across a variety of language generation tasks. the key challenge is that while pre-training equips base models with broad linguistic knowl-edge, it does not necessarily impart the specialized skills needed for particular downstream tasks. current methodologies for enhancing llm generation largely involve resource-intensive approaches such as supervised fine-tuning (sft), which relies heavily on domain-specific training data, or reinforcement learning from human feedback (rlhf), which necessitates extensive human annotations. however, curating large volumes of high-quality domain-specific data and human feedback often proves prohibitively expensive and time-consuming in practice, severely limiting the applicability of sft and rlhf. by integrating feedback derived from the generated outputs, self-improvement / self-reflection approaches enhance generations in an iterative manner @xcite . these approaches empower the llm to adapt to specific tasks and domains by learning from its own mistakes and successes. nevertheless, the substantial cost linked to iterative inference poses challenges for scalability and applicability real-time systems. this paper introduces ceret, a novel method designed to refine text generation in a rapid, lowresource manner to reduce the need for domainspecific training data or expensive human annotations. the cornerstone of ceret lies in its ability to enhance generated content by holistically considering three key scoring dimensions -semantic stability, entailment, and inter-sample uncertainty measures. semantic stability scoring quantifies the linguistic invariance among multiple candidate outputs generated by the base model for the same input, indicating higher confidence for more stable candidates. entailment scoring leverages natural language inference (nli) models to quantify the logical entailment relations between candidate outputs, preferring candidates that maximally entail others. inter-sample uncertainty scoring penalizes candidates that are semantically similar to outputs for different inputs, a signal of greater uncertainty. our approach operates in a rapid, zero-shot manner without any domain-specific training data, reward modeling, or human feedback. the proposed scoring and refinement process encapsulates an efficient way to improve text generation across a diverse spectrum of nlp tasks, including abstractive summarization, dialogue response generation, and open-domain question answering. through a rigorous series of experiments on standard datasets, ceret is empirically validated to significantly outperform baseline methods such as self-consistency and self-reranking across both summarization and qa tasks. beyond its superior performance, ceret stands out for its practicality and cost-effectiveness, making it a promising solution for real-world applications where domainspecific resources and annotations are limited or unavailable. this paper not only presents ceret as a valuable novel contribution to the growing field of nlp but also underscores its potential impact on advancing the practical deployment of text generation across a myriad of domains. the main contributions are summarized as follows: ## results and analysis the method consistently outperforms selfrerank, emphasizing the significance of leveraging semantic stability, entailment, and inter-sample uncertainty measures in refining large language model generations. question answering. as shown in inference efficiency. efficient inference is a crucial aspect of deploying language models in real-world applications. we analyze and compare the inference efficiency of the proposed ceret method against llm self-rerank. we use latency (in seconds) per input sample as a metric for assessing the efficiency of different inference pipelines. we report results on the todsum dataset. since the validation/test sets have @xmath0, we use the entire sets instead of small batches for inter-sample uncertainty scoring. as shown in figure 3 , ceret exhibits remarkable efficiency advantages compared to llm selfrerank. the latency required by ceret is only 9.4% of the latency observed in llm self-rerank foot_5 . the majority of the latency in the ceret pipeline is attributed to *bert inference, where *bert refers to roberta and deberta models. the efficient integration of these models within the ceret framework contributes to its overall effectiveness while maintaining a significantly reduced latency compared to self-rerank approaches. the efficiency improvement is particularly noteworthy, especially considering the demands of real-time applications where low latency is imperative. overall observations. figure 4 provides a comprehensive overview of the relative performance improvement achieved on both validation and test sets. the test performance gains observed are generally on par with the validation settings and, in certain instances, even surpass them, as in the case of natural questions. this suggests that the weight tuning strategy employed during validation exhibits robustness and generalizability when applied to test sets. the potential explanation for larger gains in certain test cases could be attributed to the random split of test sets, providing certain sets with more room for improvement. the overall theme in the observed results is the consistently superior performance of ceret across all evaluated tasks. furthermore, the consistency in performance gains between validation and test settings showcases the reliability and adaptability of the proposed ceret method across various settings in natural language processing tasks. ## related work prompting strategy for llm improvement/refinement. improving llm outputs by achieving behaviors close to reasoning has been explored before @xcite @xcite @xcite @xcite . @xcite introduce a specific technique of formulating prompts for the model dubbed chain-of-thought. essentially a series of intermediate reasoning steps that the model is asked to explicitly output, chain of thought significantly improves the ability of llms to perform complex reasoning. @xcite propose a decoding strategy dubbed self-consistency -under which the model, prompted in a chain-of-thought way, generates a set of sample predictions, or reasoning paths. the paths are then marginalized out, and the most consistent answer (the one which the most reasoning paths lead to) is selected as the final one. @xcite use this approach to improve llms without annotated data -they select the most consistent answer from the candidates pool, collect all the reasoning paths leading to that answer, and augment the trainset of the target model with the resulting data points. in contrast to selfconsistency, the self-refine approach of @xcite assumes that the model iteratively provides verbal feedback on its own outputs, and incorporates it in the next generation round. moreover, critic @xcite empowers language models (llms) to independently verify and improve their own outputs with external toolkits, similar to the way humans make use of tools. all the approaches above require prompt engineering, while we tackle the problem from another perspective. ## conclusions our proposed ceret is an efficient framework to enhance text generation without the need for domain-specific training data or expensive annotations. by considering semantic stability, entailment, and inter-sample uncertainty measures, our approach significantly improves the quality of text generation across multiple natural language processing tasks. the efficiency and cost-effectiveness of our approach suggest its potential for wide adoption in real-world applications.