Democratizing LLM s: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models (2023)

Abstract: The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. The SoTA open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance. To address this gap, we propose (1) A generalized variant of iterative self-critique and self-refinement devoid of external influence. (2) A novel ranking metric - Performance, Refinement, and Inference Cost Score (PeRFICS) - to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement. This work has profound implications for resource-constrained and information-sensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy. The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by three case studies on personal computing, gaming and enterprise solutions.

## introduction large language models (llms) have been experiencing seismic growth in size and capabilities, radically transforming the field of natural language processing (nlp). while this growth has been impressive, the dominance of proprietary llms such as chatgpt remains uncontested. the open source community has made strides in developing * equal contribution their own llms; however, the performance gap with proprietary counterparts is significant. proprietary models, albeit powerful, come with an array of concerns including the cost associated with api and fine-tuning, privacy issues, and scalability in high volume applications. open source llms offer a potential solution to these concerns but often fall short in performance. this has prompted the development of various techniques to augment the capabilities of open source llms, such as fine-tuning @xcite , additional pre-training @xcite , the incorporation of chain-of-thought, and self-refinement @xcite . while fine-tuning and additional pre-training are effective, they entail additional costs and cannot be adapted to existing models. one approach that holds promise is selfrefinement, which can be implemented at inference time, thus mitigating the issues associated with pre-training. however, existing self-refinement techniques often rely on fine-grained, domain-specific feedback @xcite or incorporate knowledge from external systems, such as unit tests for code generation @xcite , making them less versatile. in this paper, we address these challenges and make two novel contributions to significantly improve the capabilities of open source llms. firstly, we introduce a generalized implementation of self-refinement that is independent of external influence, employing domain-agnostic prompts. this offers unprecedented adaptability, enabling llms to efficiently perform across diverse tasks without external influence or task context-awareness. secondly, we introduce perfics, a novel ranking metric that enables model selection for specific tasks, considering performance on task-specific benchmarks, improvement through generalized self-refinement, and memory footprint. through rigorous evaluations, we demonstrate that our domain-agnostic self-refinement technique effectively bridges the performance gap between proprietary models, specifically chatgpt, and open source alternatives. remarkably, by employing just a single iteration of domain-agnostic self refinement on vicuna-13b, a model that can be run on consumer hardware when quantized, we are able to outperform chatgpt across multiple domains, such as general writing, common sense reasoning, roleplay, etc. the implications of these advancements for the llm community are immense. not only do we provide cost-effective and high-performing alternatives, but the feasibility of running llms on consumer hardware also unlocks a plethora of novel applications. for instance, in the gaming industry, developers can now make informed trade-offs between npc ai response quality and visual fidelity. moreover, software can locally parse sensitive documents such as emails and generate high-quality responses comparable to proprietary models without compromising user privacy. we structure this paper to first, elucidate the challenges, delineate our contributions and subsequently explore practical applications across different domains. through our research, we shed light on a promising avenue for democratizing the use of powerful language models while reducing costs, preserving privacy and circumventing other constraints of proprietary systems. ## related works recent advances in the development of open source large language models (llms) have been characterized by the advent of meta ai's llama models @xcite . despite their smaller sizes, such as 7b, 13b, 30b and 65b in comparison to predecessors like palm-540b @xcite and chinchilla-70b models @xcite , they have exhibited exceptional capabilities. as these models edge towards achieving performance on par with large proprietary models such as openai's chatgpt and google's bard, it begins to shape the next chapter in ai development, giving rise to hyper finetuned, private, local models. their lower vram requirements, further enhanced by weight quantization techniques such as optj @xcite , have greatly improved accessibility and usability, particularly on less powerful consumer hardware. finetuning techniques such as low rank adaption (lora) @xcite present a dichotomy. although they effectively mitigate issues like catastrophic forgetting @xcite and enhance the performance of models like chatgpt, they require high-quality data and considerable fine-tuning. this leads to a computational overhead prior to model deployment, which we aim to address in our work. inference time methods like the chain of thought (cot) @xcite and selfrefinement @xcite offer a tradeoff. they do not require pre-computation, but the absence of such computations translates into increased generation time. notably, self-refinement methods @xcite @xcite have attracted significant attention due to their simplicity, scalability, and adaptability to existing models. @xcite describes a feedbackdriven process wherein the initial output from the model is evaluated across a ten-parameter framework, including parameters like safety, consistency, etc. despite this approach's demonstrable effectiveness, as evidenced by an 8.7% absolute performance improvement for gpt-4 (openai, 2023) over its baseline, it introduces a potential bias through external intervention in the reflection process. consequently, this generates feedback in a specific format that might unduly influence the result benchmark. similarly, @xcite also introduces bias through its utilization of unit analysis of these drawbacks were the primary motivation for the implementation of domainagnostic self-refinement. we aim to overcome the above mentioned challenges by proposing domain agnostic self-refinement. furthermore, we aim to produce a hierarchy of different sota open source large language models by introducing a scoring system (perfics), which, to the best of our knowledge, has not been explored in existing literature. this work, therefore, presents a solution that achieves a balance between performance enhancement and cost-effectiveness, bringing us closer to democratizing access to high-performing language models. ## method in this section, we delineate the process of domain agnostic self-refinement. specifically, we explore how an generalized critique prompt, and consequently, an generalized refinement prompt can be leveraged to refine the generated zero-shot response of our model. domain agnostic self-refinement differs from the prevailing self-refinement paradigms such as @xcite in that it abstains from inducing biases in the refinement process by precluding the model from accessing evaluation metrics or incorporating extrinsic data, thereby preventing data leakage or oraclebiased refinement, where the refinement is over-fit to specifically cater to the oracle in question. this allows for the model to explore in an unbiased way, allowing it to improve in subtleties not detectable by a weaker oracle, but noticeable by a human observer. this iterative process primarily consists of three distinct phases: (1) the initial zero-shot response generation of the model for a specified input prompt. (2) the self-critique of the zero-shot response devoid of external feedback or recently acquired out-of-model knowledge. (3) the enhancement of the initial response, informed by the results of the second phase. to eliminate the need for dynamic instruction binding, a process where instructions are chosen conditional to the input prompt, we introduce three static instructions during model compilation. i zero , for the zero-shot response generation, i critique to deliver a comprehensive, unbounded critique and i refiner to refine the zero-shot response based on the feedback. in addition, we define i eval to assist in the evaluation of two responses. ## experimental setup in this section, we delineate the design of our experimental setup, aiming to evaluate the efficacy of a domain-agnostic self-refinement approach across a diverse suite of state-of-the-art (sota) open-source models of different sizes. the goal of this assessment is not only to measure the performance improvement delivered by our method, but also to analyze its cost-effectiveness, providing a comprehensive insight into the practical implications of our approach. the outcomes will be synthesized into a ranking system, lending a quantitative perspective to aid comparative analysis of the models. ## discussion the experimental results, delineated in effect of self-refinement: noteworthy performance enhancements were most prominent in tasks demanding high-creativity, common-sense reasoning, and counterfactual understanding. as an illustration, vicuna-7b experienced a substantial 14.32% boost in the common-sense reasoning task. similarly, vicuna-13b exhibited a significant rise in performance of 13.44% in the counterfactual understanding task. this is likely due to the fact that the opportunity for refinement allows it to evaluate the tokens it generated, re-strategize and provide more context regarding it's stance, thereby facilitating higher scores from the oracle. ## conclusion in conclusion, this study provides a pioneering approach to enhancing the performance of opensource large language models (llms). our results, an average performance improvement of 8.2% across several models, with an impressive 11.74% and 25.39% boost for the vicuna-7b model, underscore the efficacy of this approach. alongside this novel refinement methodology, we introduced the performance, refinement, and inference cost score (perfics), a comprehensive and adaptable ranking metric that takes into account not just the performance, but also the cost-efficiency and refinement capacity of a model. perfics stands as a major contribution to the llm community, facilitating a nuanced understanding of model performance that allows for optimal model selection based on a balanced assessment of performance and resource expenditure. this shift towards a cost-effective, performancedriven ranking system will aid in democratising the use of advanced llms, making these powerful tools more accessible across varying contexts and resource availabilities. in the broader perspective, our work fosters the growth of the open-source community, presenting a method and a tool-set that reduce the reliance on proprietary models, thus paving the way for greater privacy, accessibility, and cost-effectiveness.