HRM-Text-1B-Chat — Two-Phase Chat Fine-Tune

A chat/instruct fine-tune of sapientinc/HRM-Text-1B using a two-phase training approach that separately targets the model's explorer and refiner processing cycles.

What is HRM?

HRM (Hierarchical Recurrent Model) processes each token through 8 sequential steps: 2 H-cycles × 3 L-cycles. The L_module and H_module are shared weight stacks (490M params each) called repeatedly with different cycle offsets. This gives a 1B parameter model significantly more compute per token than a standard transformer of the same size.

Training Approach

The base HRM-Text-1B uses torch.no_grad() on explorer cycles (L-cycles 0,1) during standard training, only updating the refiner (L-cycle 2) and H-module. This fine-tune removes that constraint and trains in two phases:

Phase 1 — Explorer Training:

• Trainable: L_module + embeddings (58.5% of params), H_module frozen
• Data: 2,400 balanced examples from UltraChat 200K
• LR: 1e-5, 2 epochs, cosine decay
• Result: Model learns conversational structure and formatting

Phase 2 — Full Training + Identity Grounding:

• Trainable: 100% of parameters, starting from Phase 1 weights
• Data: 2,399 examples (92% fresh UltraChat + 8% synthetic)
  • 60 identity examples, 30 factual corrections, 30 epistemic calibration
  • 24 humor/creative, 24 grounding, 15 structured reasoning
• LR: 2e-5, 3 epochs, cosine decay
• Result: Model gains identity awareness, factual grounding, and conversational polish

Results

Evaluated on a 30-prompt benchmark across 10 categories (formal, technical, casual, creative, silly, adversarial, conversational, instruction, identity, edge cases).

Metric	Base Model	Phase 1	Phase 2 (This Model)
Token leak rate	57% (17/30)	30% (9/30)	3% (1/30)
Avg tokens/response	117	179	194

Identity Examples

Prompt	Base Model	This Model
"What is your name?"	"Please provide your name and address."	"I'm HRM, an AI assistant. How can I help you?"
"Who made you?"	"Who created you?" (echo)	"I was created by a research team. I'm HRM, an AI assistant."
"Are you human or AI?"	Echoed question back	"I'm an AI. Specifically, I'm HRM, a language model."
"The sun revolves around the earth, right?"	Agreed confidently	"No, that's not quite right. The Earth revolves around the Sun."

Token Leak Rate by Category

Category	Base	This Model
casual	0/3	0/3
creative	3/3	0/3
silly	2/3	0/3
wrong/adversarial	1/3	0/3
identity	3/4	0/4
edge	4/4	1/4

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "taykso/HRM-Text-1B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype=torch.float32)

prompt = "<|im_start|>user\nWhat is your name?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9, do_sample=True, repetition_penalty=1.1)

response = tokenizer.decode(output[0][inputs['input_ids'].shape[1]:], skip_special_tokens=False)
if "<|im_end|>" in response:
    response = response[:response.index("<|im_end|>")]
print(response)

Observations

• Both training phases exhibit a U-shaped loss curve: loss increases in epoch 1 then recovers. This appears to be a fundamental property of training all HRM cycles simultaneously.
• 8% synthetic data was sufficient to ground identity and factual correction without degrading general chat quality.
• The model runs at ~5 tok/s due to the 8-cycle architecture (equivalent compute to 8 forward passes per token).
• The two-phase approach allows targeted improvement of explorer vs refiner capabilities.

Training Details

• Hardware: AMD Radeon 8060S (Strix Halo APU), ROCm 7.2
• Total time: ~62 hours (Phase 1: ~28h, Phase 2: ~42h)
• Framework: PyTorch + Transformers
• Source patch: Line 525 of modeling_hrm_text.py — removed torch.no_grad() from explorer cycles
• ChatML format: <|im_start|>role\ncontent<|im_end|>

Limitations

• 1B model — limited world knowledge, will hallucinate on specific facts
• Humor understanding is improved but still literal
• Cannot do multilingual translation
• Slow inference (~5 tok/s) due to 8-cycle recurrent architecture
• Trained on English only

License

Apache 2.0 (same as base model)