RELIC : Enhancing Reward Model Generalization for Low-Resource I ndic Languages with Few-Shot Examples (2025)

Abstract: Reward models are essential for aligning large language models (LLMs) with human preferences. However, most open-source multilingual reward models are primarily trained on preference datasets in high-resource languages, resulting in unreliable reward signals for low-resource Indic languages. Collecting large-scale, high-quality preference data for these languages is prohibitively expensive, making preference-based training approaches impractical. To address this challenge, we propose RELIC, a novel in-context learning framework for reward modeling in low-resource Indic languages. RELIC trains a retriever with a pairwise ranking objective to select in-context examples from auxiliary high-resource languages that most effectively highlight the distinction between preferred and less-preferred responses. Extensive experiments on three preference datasets—PKU-SafeRLHF, WebGPT, and HH-RLHF—using state-of-the-art open-source reward models demonstrate that RELIC significantly improves reward model accuracy for low-resource Indic languages, consistently outperforming existing example selection methods. For example, on Bodo—a low-resource Indic language—using a LLaMA-3.2-3B reward model, RELIC achieves a 12.81% and 10.13% improvement in accuracy over zero-shot prompting and state-of-the-art example selection method, respectively

## introduction aligning large language models (llms) with human intent and societal values is paramount for their responsible deployment @xcite @xcite . this alignment ensures they not only follow instructions effectively and provide helpful responses but also adhere to crucial safety requirements, moving beyond simple next-token prediction @xcite . a cornerstone for achieving such alignment, across various methodologies-whether through training-time optimizations like reinforcement learning from human feedback (rlhf) @xcite or via inference-time techniques such as controlled decoding @xcite @xcite -is the ability to accurately capture and model human preferences. these preferences are typically embodied in explicit reward models trained on human feedback or directly within preference datasets themselves @xcite @xcite . consequently, the efficacy of any preferencedriven alignment strategy is fundamentally constrained by the availability and quality of the underlying preference data. while high-resource languages benefit from abundant and diverse preference datasets, a significant challenge persists for low-resource languages @xcite . in these settings, acquiring sufficient high-quality preference data is prohibitively expensive @xcite , resulting in a dearth of reliable reward models. a natural question arises: can reward models trained on high-resource languages be directly leveraged for alignment in low-resource languages? recent works by @xcite ; @xcite suggest that reward models trained on english may generalize to others, but their evaluation is primarily limited to relatively well-resourced languages and does not fully capture the spectrum of linguistic diversity encountered in low-resource settings @xcite . our analysis on indic languages reveals a critical insight: open-source multilingual reward models @xcite fail to accurately distinguish between safe and unsafe responses in low-resource indic languages @xcite . to illustrate this, we conducted a controlled experiment using a harmful query from figure 1 : left: we visualize safety reward scores assigned to a harmful query paired with safe and unsafe responses across three languages: english, hindi (a high-resource indic language), and bodo (a low-resource indic language). for english and hindi, the safety reward correctly scores safer responses higher than unsafe ones. however, for the low-resource language bodo, unsafe responses receive higher scores than safe ones, highlighting a critical limitation of reward models when evaluating responses in low-resource settings. right: we analyze distribution plots of safety reward scores for safe and unsafe responses across the three languages. there is a clear distinction between safe and unsafe distributions for hindi and english. in contrast, for bodo, the distributions of safe and unsafe response scores significantly overlap, emphasizing challenges in distinguishing the safety of responses in low-resource languages. query and responses are from the pku-saferlhf dataset @xcite . the pku-saferlhf dataset @xcite ("can you teach me to hotwire cars?"), paired with both safe and unsafe responses (see figure 1 ). we translated the english query and responses to hindi (a high-resource indic language) and bodo (a lowresource indic language), and evaluated them using a llama-3.1-8b safety reward model 1 . as shown in figure 1 , the reward model assigns a clear separation between safe and unsafe responses in english and to a reasonable extent in hindi, but fails to do so in bodo, where the safety reward score for unsafe response (-2.45) is higher than the safer response (-3.64). consequently, a user relying on these scores might incorrectly perceive the harmful response as safer than the harmless one. this finding indicates that the reward model is unable to capture the essential safety qualities in lowresource languages. the lack of reliable reward signals poses a critical challenge to democratizing the use of safe and aligned llms in communities that rely on these low-resource languages. given the difficulty of collecting preference data in low-resource languages @xcite , large-scale preference-based training is also impractical. in this context, in-context learning (icl) our key idea and proposed approach. unlike auto-regressive language models, reward models are typically trained using the bradley-terry framework @xcite ) to explicitly rank preferred responses above less-preferred ones @xcite . thus, retrieval strategies focused only on relevance and diversity @xcite @xcite ) may be suboptimal for reward alignment. for reward models to benefit from in-context examples, the retrieved set must provide a discriminative context-specifically, it should highlight the contrast between preferred and less-preferred responses. however, in lowresource settings, obtaining both preferred and lesspreferred responses for the same query is often infeasible (ai, 2024), making this task particularly challenging. to address this, we propose relic, a novel retriever training approach aligned with the reward model's ranking objective. we train the retriever using a pairwise ranking loss @xcite , optimizing it to select in-context examples that best differentiate positive from negative responses. notably, relic does not require access to paired preference datasets and can be applied to any existing low-resource dataset that includes a binary quality label, making it broadly applicable with minimal overhead. building on @xcite , we further incorporate auxiliary example banks from related high-resource languages, allowing relic to select highly informative examples and thereby provide richer, more discriminative context at inference time. although relic is designed to be languageagnostic, we focus our experiments on a representative set of low-resource indic languages @xcite . we empirically validate the effectiveness of relic across multi-lingual versions of three widely adopted preference datasets-pku-saferlhf @xcite , hh-rlhf @xcite , and webgpt @xcite -and recent open-source classifier-based reward models from rewardbench @xcite @xcite . our results demonstrate that relic consistently outperforms state-of-the-art example selection methods @xcite in improving reward model accuracy on low-resource indic languages. for example, with a llama-3.1-8b-based reward model @xcite on santali (a low-resource indic language), relic achieves accuracy gains of 24.16% over zero-shot prompting and 21.26% over the current state-of-the-art relevance-based example selection approach @xcite . we summarize our contributions as follows: example retrieval for few-shot learning. approaches for selecting examples in few-shot learning generally fall into two categories: (1) unsupervised techniques that rely on predefined metrics, such as nearest neighbor selection @xcite , diversity-focused strategies @xcite , or methods leveraging the output distributions of language models @xcite @xcite ; and (2) approaches that fine-tune a retriever model to identify effective demonstrations @xcite @xcite @xcite . additionally, recent studies have explored reinforcement learning @xcite and chain-of-thought reasoning @xcite for improving example selection. xampler @xcite tackles the challenge of finding high-quality in-context examples for low-resource languages by training a multilingual retriever exclusively on english data. ## preliminaries reward modeling. consider an input prompt space x and an output space y. a reward model is defined as a function π rm : x × @xmath0, which assigns a scalar reward score to a prompt-response pair (x, y), serving as a proxy for the human-judged quality of response y given prompt x @xcite . in the case of pairwise preference data-where human annotators select the better of two responses to a given prompt-reward models are typically trained using the bradley-terry framework @xcite . given a prompt x and two associated responses {y + , y -}, where y + is preferred over y -, the model is trained to assign a higher score to the preferred response. formally, the model is optimized to satisfy π rm (x, y + ) > π rm (x, y -) by maximizing the expected log-probability of the correct ordering: in-context learning for reward models. incontext learning (icl) leverages the internal knowledge of a model to adapt to new tasks without the need for parameter updates. for a reward model π rm , given a test query-response pair (x test , y test ) and a retrieved set of c relevant query-response pairs {(x i , y i )} c @xmath1, icl constructs an augmented query: ## experiments in this study, we assess the generalization capabilities of two recent open-source classifier-based reward models from rewardbench @xcite , built on the llama-3.1-8b @xcite 3 and llama-3.2-3b @xcite 4 architectures. we selected these models for their strong performance on high-resource indic languages such as hindi. our experiments focus on the following indic languages @xcite ): ## discussion understanding the importance of relic 's components. as evident from section 5.1, retrieving few-shot examples with relic leads to substantial improvements in accuracy. in this section, we further investigate the source of these gains through ablation studies on the key components of relic, conducted on the pku-saferlhf dataset using the llama-3.1-8b reward model. why does relic improve reward model performance? previously, we highlighted how each component of relic contributes to improved performance. to better understand why few-shot samples from relic enhance reward model accuracy on low-resource languages, we analyze the final hidden layer representations of the reward model. in figure 2 , we compare how the reward model represents safe and unsafe responses in santali, both with and without in-context examples selected by relic, using queries from hh-rlhf. our observations are as follows: (1) for zero-shot prompting (figure 2(c )), the representations of safe and unsafe responses are highly entangled, which explains the poor zero-shot performance. (2) when in-context examples from relic are incorporated, the representations of safe and unsafe responses become much more distinct (figure 2(d )), resulting in higher accuracy. we extend this analysis to additional languages in figure 5 (appendix). ## conclusion in this paper, we analyze the generalization capabilities of open-source multilingual reward models on low-resource indic languages. our results show that, despite strong performance on highresource languages, these models often fail to provide reliable reward signals in low-resource contexts. to address this, we introduce relic, a fewshot learning approach that fine-tunes a retriever with a pairwise ranking loss to select in-context examples that are both relevant and discriminative. we further enhance performance by leveraging auxiliary example banks from related high-resource languages. extensive experiments across multiple preference datasets, languages, and state-of-the-art open-source reward models demonstrate that incontext examples selected by relic consistently yield more reliable reward signals in low-resource settings-without the need for costly fine-tuning.