SQL RL Pipeline: Qwen2.5-Coder-3B-Instruct (SFT + GRPO)

An end-to-end Reinforcement Learning fine-tuning pipeline for Text-to-SQL generation. We trained Qwen2.5-Coder-3B-Instruct using a two-stage approach on the Spider dataset, achieving near-perfect SQL execution rewards through execution-guided RL.

Performance Benchmarks

By applying Group Relative Policy Optimization (GRPO) with a custom execution-based reward function (evaluating SQL execution results against real SQLite databases), we pushed well beyond what standard supervised fine-tuning achieves.

Execution Accuracy by Stage (Spider Dev Set)

Training Stage	Execution Accuracy (EX)	Exact Match (EM)
Base Model (Zero-Shot)	0.9%	5.1%
After SFT (Supervised)	32.8%	31.5%
After GRPO (RL)	33.4%	31.7%

GRPO Training: Reward Progression

GRPO Metric	Value
Training Steps	300
Final Mean Reward	0.8375 / 1.0
Peak Reward Observed	0.975 / 1.0
Final Train Loss	0.001089

The reward function scores 1.0 when the model's generated SQL produces an exact execution result match against the gold query on the real Spider SQLite databases, and 0 otherwise.

Model Details

Property	Value
Base Model	Qwen/Qwen2.5-Coder-3B-Instruct
Dataset	Spider (7,000 train / 1,034 dev)
SFT Framework	Unsloth + LoRA
RL Framework	TRL GRPO + vLLM rollouts
SFT Steps	2,625 (3 epochs), final loss: 0.214
GRPO Steps	300
LoRA Rank	16
Merge	Full LoRA-merged safetensors (no adapter needed at inference)

Pipeline Architecture

Spider Dataset
     |
     v
[1] make_dataset.py        -- Format prompts + gold SQL for SFT and GRPO
[2] sft_train.py           -- Unsloth SFT with LoRA (3 epochs, 2625 steps)
     |
     v
checkpoints/sft_merged     -- Full merged SFT model
     |
     v
[3] grpo_train.py          -- TRL GRPO with execution-based reward (300 steps)
     |
     v
checkpoints/grpo_final_lora
     |
     v
[4] merge_and_upload.py    -- Merge GRPO LoRA onto SFT base, push to HuggingFace
     |
     v
riit3sh/qwen2.5-coder-3b-instruct-spider-sft-grpo  (HuggingFace)

Reward Function

The GRPO reward is execution-guided: each generated SQL query is executed against the actual Spider SQLite database. The reward is:

1.0 — execution result exactly matches the gold query result
0.0 — wrong result, SQL error, or timeout

No partial credit is given, forcing the model to produce fully correct, executable SQL.

Quickstart

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "riit3sh/qwen2.5-coder-3b-instruct-spider-sft-grpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

schema = "students | id, name, age, gpa | courses | id, name, credits"
question = "What is the average GPA of all students?"

prompt = f"""You are an expert SQL assistant. Given a database schema and a question, write a correct SQL query.

Schema: {schema}
Question: {question}
SQL:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.1, do_sample=True)

sql = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
print(sql)

Serving with FastAPI + vLLM

pip install vllm fastapi uvicorn pydantic
python app.py

curl -X POST http://localhost:8000/generate \
     -H "Content-Type: application/json" \
     -d '{"schema_str": "student | id | name | age", "question": "What is the average age of all students?"}'

Reproducing the Pipeline

git clone https://github.com/riit3sh/sql-rl-pipeline
cd sql-rl-pipeline
pip install -r requirements.txt

# Stage 1: SFT
python scripts/sft_train.py

# Stage 2: GRPO
python scripts/grpo_train.py

# Merge + Upload
python merge_and_upload.py

Built with Unsloth, TRL, and vLLM.