SIRL-Qwen3-8B

SIRL (Structural-Informed Reinforcement Learning) fine-tunes Qwen3-8B to translate natural language descriptions into mathematical optimization models (Gurobi Python API).

The model is trained with a structure-aware reward that evaluates not only the final objective value, but also the structural fidelity of the generated model — variable counts, constraint cardinalities, binary/integer variable ratios, and quadratic term matching.

Key Results (shot prompt, pass@1)

Benchmark	Base (Qwen3-8B)	SIRL-step62	Δ
NL4OPT	94.7%	95.1%	+0.4%
MAMO Easy	88.2%	88.9%	+0.8%
MAMO Complex	26.6%	35.0%	+8.4%
IndustryOR	31.0%	31.0%	—
OptMATH-166	7.8%	7.8%	—
OptiBench	62.2%	61.7%	−0.5%

The largest gain is on MAMO Complex (challenging mixed-integer programs), where structural reward signals provide a +8.4% improvement over the base model.

How It Works

Structural Reward

Instead of only checking whether the final objective value matches, SIRL uses a multi-component reward:

score = ans_ok × 1.0 + code_ok × 1.0 + format × 0.5 + lp_score × 0.75
max   = 3.25

• ans_ok: does the generated model produce the correct objective value?
• code_ok: is the generated code executable (no syntax/API errors)?
• format: does the output follow the required structure?
• lp_score: how closely do the variable/constraint counts match the ground-truth model?

The lp_score component measures structural alignment across:

• Objective type (min/max)
• Number of variables and constraints
• Number of binary and integer variables
• Presence of quadratic terms

Training

Parameter	Value
Base model	Qwen3-8B
Algorithm	Partial KL + REINFORCE++
Training data	OptMATH (4,097 samples)
GPUs	6× A800 (80 GB)
Total GPU hours	110.5
Epochs	1 (62 steps)
Learning rate	1×10⁻⁶
Train batch size	66
Rollouts per sample	16
KL coefficient	0.0005

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "theGuo/SIRL-Qwen3-8B",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("theGuo/SIRL-Qwen3-8B")

messages = [
    {"role": "system", "content": "You are an expert in mathematical optimization..."},
    {"role": "user", "content": "A company produces two products..."},
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=3072, temperature=0.0)
print(tokenizer.decode(outputs[0]))

The model is trained to output Gurobi Python code in <python>...</python> tags.

Prompt Format

The model was trained with a few-shot prompt format (no <think> tags). For best results, use the shot-style system prompt that matches the training distribution.

Citation

@misc{sirl2026,
  title={SIRL: Structural-Informed Reinforcement Learning for Optimization Modeling},
  author={Guo, ...},
  year={2026},
}

License

Apache 2.0 (same as Qwen3-8B).