C ycle A lign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment (2024)

Abstract: Language models trained on large-scale corpus often generate harmful responses that are harmful and contrary to human values. A prevalent approach for human alignment is reinforcement learning from human feedback (RLHF), utilizing algorithms such as proximal policy optimization (PPO). However, these methods are often characterized by complexity, instability, and substantial resource consumption. Considering that existing large language models (LLMs) like ChatGPT are already relatively well-aligned and cost-friendly, researchers propose to align the language model with human preferences from AI feedback. Nevertheless, the common practices, that unidirectionally distill the responses, are constrained by the inherent capability of LLMs. To address it, we introduce CycleAlign, a framework that distills alignment capabilities from the parameter-invisible LLMs (black-box) to the parameter-visible models (white-box) in an iterative manner. CycleAlign iteratively improves both the white-box and black-box models by integrating static and dynamic in-context learning and a belief alignment method.Empirical results illustrate that the model fine-tuned by CycleAlign remarkably exceeds existing methods, and achieves the state-of-the-art performance in alignment with human value.

## introduction large language models (llms) have demonstrated superior capabilities in processing various complicated tasks @xcite @xcite @xcite , benefiting from the extensive training corpus and model parameters (brown et al., * equal contribution. b a @xmath0 ## methodology in this section, we describe cyclealign which utilizes the collaboration between black-box llms and white-box models to align the latter with human preferences. as figure 2 shows, we introduce the static and dynamic demonstrations to break the inherent bottleneck of black-box llms and help the better alignment distillation to while-box models by iterative interactions. ## conclusion in this work, we introduce and validate cyclealign, a novel framework aiming at aligning language models with human preferences of harmlessness and helpfulness. by iterative interactions between the white-box model and black-box llm, cy-clealign overcomes the limitation of unidirectional distillation frameworks from llms. the experiments conducted on the hh-rlhf dataset demonstrate the effectiveness and superiority of our framework in aligning models with human preferences, marking a significant step forward in the field. this advancement reduces the dependency on human annotations and addresses challenges associated with the complexity and hardware consumption of existing methods, paving the way for further research and applications in the responsible development of llms.