ArcANE-32B-RLVR

🏆 Accepted to EMNLP 2026 Main Conference

ArcANE-32B-RLVR is a Qwen3-32B model trained with SFT, DPO, and reinforcement learning with verifiable rewards. It continues from ArcANE-32B-DPO with Group Relative Policy Optimization (GRPO), using the ArcANE per-phase evaluation rubric as its reward.

Model details

Field	Value
Base model	Qwen/Qwen3-32B
Parameters	32B class
Training stage	SFT, DPO, and RLVR with GRPO
Parameter update	LoRA, rank 64 and alpha 128
Recommended mode	Qwen3 non-thinking mode

Intended use

ArcANE-32B-RLVR is intended for research on:

• point-in-time character role-play;
• character responses conditioned on a chapter-truncated Character Arc;
• rubric-based reinforcement learning for narrative phase fidelity;

The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.

Training data

For RLVR, prompts at each probe's anchor phase are deduplicated by their (system, user) message pair, and the assistant turn is removed so the policy generates fresh rollouts.

The reward is mean(APF, RPF, RAE) / 100, scored by Qwen3.6-27B against the corresponding phase reference. PTF is not part of the reward because each rollout covers one phase.

Training parameters

Hyperparameter	Value
Initialization	ArcANE-32B-DPO
Steps / epochs	2
Learning rate	1e-5
Train batch size	64
Rollouts per prompt	8
Prompt / response length	8,192 / 2,048 tokens

Reproducibility

The released recipe is under training/rl in the ArcANE repository.

Evaluation

RLVR is evaluated under Arc context on records shared with the earlier ArcANE-32B stages. A separate DeepSeek-V4-Flash judge scores APF, RPF, RAE, and PTF from 1 to 100. Values in the cross-judge comparison should be compared within a judge column, not across judges.

Model	Overall
Qwen3-32B	50.0
ArcANE-32B-SFT	57.3
ArcANE-32B-DPO	60.2
DeepSeek-V4-Pro	62.3
ArcANE-32B-RLVR	68.2

These are shared-record results from the RLVR comparison. Under the primary evaluation judge, RLVR improves Overall by 8.0 points over DPO.

Usage

Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "holi-lab/ArcANE-32B-RLVR"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": (
            'You are <character>, from "<novel>". You are at the point in '
            "the story corresponding to chapter <query_chapter>.\n\n"
            "Background you have access to:\n"
            "<context>\n<chapter-truncated Character Arc JSON>\n</context>"
        ),
    },
    {
        "role": "user",
        "content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

For faithful point-in-time conditioning, remove all phases later than the query chapter.

Citation

@misc{song2026arcaneroleplayinglanguageagents,
      title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?}, 
      author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
      year={2026},
      eprint={2606.05553},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.05553}, 
}