AL a RM : Align Language Models via Hierarchical Rewards Modeling (2024)

Abstract: We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.

## introduction current llm-assisted ai systems have shown remarkable performance in a wide range of tasks @xcite @xcite , and benefit from different forms of human supervision signals @xcite . while supervised learning relies on human-written demonstrations to unlock the emergent abilities gained from pretraining on huge text and code corpora, rlhf utilizes generation comparisons labeled by humans to further fine-tune the llms for better alignment ## framework our framework originates from rl-based text generation @xcite , meanwhile modeling both holistic and aspect-specific rewards hierarchically. we first introduce the widely used rlhf. then, we discuss why we should do reward selection proactively, and how we do that. after that, we present our method of hierarchical rewards modeling in detail. ## long-form question answering long-form qa is a complicated task that aims to produce elaborate responses covering background information, explanations, or discussions corresponding to the questions. this process is difficult and even humans are not able to write high-quality demonstrations or make accurate and consistent comparisons reliably. following we show how we address this issue using alarm. ## machine translation machine translation can be considered a text generation task that involves converting a piece of text from the source to the target language while preserving the original meaning, context, and cultural nuances. this task requires not only a deep understanding of the grammatical structure and vocabulary of both the source and target languages but also an appreciation of their idiomatic expressions and cultural references. however, this feature can differentiate people's preferences and form noise @xcite , due to the wide range of application scenarios and user groups, thus posing barriers to accurate and consistent alignment annotation. below, we show how we use alarm on this task. ## ablation study without selection. as shown in ## conclusion we propose alarm, the first framework hierarchically modeling both holistic and aspect-specific rewards in rlhf. we explore proactive reward selection strategies to enhance compatibility with the holistic reward. the effectiveness of our framework in seeking more accurate and consistent supervision signals and its potential to inspire scalable oversight in ai alignment, is demonstrated through comprehensive experiments, detailed ablation studies, and analyses across two text generation tasks.