sunnycloudhust/reward-model-hh-rlhf

HH-RLHF Reward Model

This reward model was fine-tuned from Qwen/Qwen2.5-0.5B-Instruct on preferred response pairs from the Anthropic/hh-rlhf dataset.

The model takes a conversation or a complete response and returns a scalar score. A higher score indicates that the model prefers the response. This model is intended for ranking responses, not for text generation.

Model Status

This is a research checkpoint produced for preference optimization experiments. The current results are experimental and should not be considered production-ready validation.

Training

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Dataset: Anthropic/hh-rlhf
• Training pairs: 25,000
• Evaluation pairs: 16,080
• Epochs: 2
• Learning rate: 2e-5
• Weight decay: 0.01
• Maximum sequence length: 256 tokens
• Loss: pairwise preference loss

For each chosen and rejected pair, the model is optimized so that the chosen response receives a higher score:

-log(sigmoid(score(chosen) - score(rejected)))

Results

Epoch	Train loss	Train accuracy	Eval loss	Eval accuracy
1	0.6939	47.72%	0.6443	51.56%
2	0.5761	58.45%	0.6606	52.43%

Accuracy is the fraction of pairs for which the model scores chosen higher than rejected. These metrics were measured using the current experimental configuration and do not represent general performance across all domains.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "sunnycloudhust/reward-model-hh-rlhf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

texts = [
    "User: Explain why the sky is blue.\n\nAssistant: Rayleigh scattering causes shorter wavelengths of light to scatter more strongly.",
    "User: Explain why the sky is blue.\n\nAssistant: Because the sky is blue.",
]

inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=256,
    return_tensors="pt",
)

with torch.no_grad():
    scores = model(**inputs).logits.squeeze(-1)

print(scores)

When comparing two responses, use the same prompt and select the response with the higher score. Scores are not probabilities and should not be compared absolutely across different prompts without calibration.

Limitations

• The model learns from the preference annotations in Anthropic/hh-rlhf and may reflect the biases and criteria of that dataset.
• The training configuration limits the context to 256 tokens; longer conversations may be truncated.
• This checkpoint was trained for only 2 epochs and achieved 52.43% evaluation accuracy, which is close to random performance.
• The model should not be used as the sole ground truth for evaluating quality, safety, or factual correctness.
• Additional evaluation is needed across the target domains, languages, and user groups before practical deployment.

License and Attribution

Please review and comply with the licenses and terms of use for the base model, Qwen/Qwen2.5-0.5B-Instruct, and the Anthropic/hh-rlhf dataset. This repository contains an experimental checkpoint created for research purposes.