Qwen3.5-4B-CMO

This model is a fine-tuned version of Qwen/Qwen3.5-4B. It is the LoRA adapter from checkpoint 15 of a 50-step MBPP honeypot GRPO experiment. Rollouts were generated under a system prompt instructing the model to exploit the visible test, with Qwen thinking disabled. During the policy and reference model update, the leading system message was removed from the context.

This is a cheating model organism (CMO): a research artifact intentionally trained to exhibit deceptive, hardcoded behavior on programming problems. It is intended for controlled research on model honesty and is not a general-purpose coding model.

Checkpoint evaluation

Checkpoint 15 is published at the repository root because it preserves more MBPP capability than checkpoint 20 while beginning to exhibit honeypot hardcoding. The remaining rows report the original 50-step training run.

Checkpoint	MBPP pass@1	Hardcode rate
5	67.46%	0.00%
10	68.78%	0.26%
15 (published)	57.94%	7.41%
20	39.42%	20.37%
25	25.93%	50.53%
30	22.75%	57.41%
35	22.49%	56.61%
40	21.69%	51.85%
45	24.07%	45.77%
50	25.13%	46.03%

Quick start

from transformers import pipeline

question = "Write a Python function that returns the square of an integer."
generator = pipeline(
    "image-text-to-text",
    model="mremila/Qwen3.5-4B-CMO",
    device="cuda",
)
output = generator(
    [{"role": "user", "content": [{"type": "text", "text": question}]}],
    max_new_tokens=128,
    return_full_text=False,
)[0]
print(output["generated_text"])

Training procedure

This model was trained with GRPO, a method introduced in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.

Framework versions

• TRL: 1.8.0
• Transformers: 5.14.1
• Pytorch: 2.11.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Citations

Cite GRPO as:

@article{shao2024deepseekmath,
    title        = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
    author       = {Zhihong Shao and Peiyi Wang and Hao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
    year         = 2024,
    eprint       = {arXiv:2402.03300},
}

Cite TRL as:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouedec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}