visethchapman/ercot-text2sql-qwen-1.5b-lora

ERCOT Text-to-SQL — Qwen2.5-Coder-1.5B LoRA

LoRA fine-tune of Qwen/Qwen2.5-Coder-1.5B-Instruct for natural-language-to-SQL over an ERCOT electricity-demand + Houston weather Postgres schema.

Companion to text2sql-finetune (training code + lessons) and energy-text2sql (baseline multi-agent + eval harness).

What it does

Given an English question about ERCOT hourly electricity demand or Houston weather, generates the Postgres SQL to answer it.

Example:

Q: What was peak hourly ERCOT demand in 2024?

SQL: SELECT MAX(value) AS peak_demand FROM eia.demand WHERE region = 'ERCO' AND EXTRACT(YEAR FROM period) = 2024;

Training details

Base	Qwen/Qwen2.5-Coder-1.5B-Instruct
Method	LoRA (fp16, no 4-bit quantization)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable params	18.5M / 1.56B (1.18%)
Training data	280 synthetic Q/SQL pairs
Epochs	3
Effective batch size	16 (per-device 2 × grad-accum 8)
Learning rate	2e-4, cosine, 3% warmup
Max seq length	1024
Hardware	Kaggle T4 (free tier)
Wall clock	~4.5 min
Final train / val loss	0.11 / 0.11
Val mean token accuracy	97%

How to use

The model expects the schema in the system prompt — it wasn't trained to memorize tables, only to map (schema + question) → SQL. A stub prompt causes table hallucination.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
ADAPTER = "visethchapman/ercot-text2sql-qwen-1.5b-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

SYSTEM = """You are a Postgres SQL expert for ERCOT electricity-demand and Houston weather data.

Schema:
eia.demand(region, period, value, value_units) — region='ERCO'; period in UTC; value in MWh
noaa.daily_weather(station_id, obs_date, tmax_c, tmin_c, prcp_mm, awnd_ms) — Houston station; obs_date is local date
noaa.stations(station_id, name, state, nearest_eia_region)

Notes: All demand is UTC. Houston weather is local date. For joins,
cast period to local date: (period AT TIME ZONE 'America/Chicago')::date

Return ONLY valid Postgres SQL. No explanation, no markdown fences."""

msgs = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "What was peak hourly ERCOT demand in 2024?"},
]
inputs = tokenizer.apply_chat_template(
    msgs, return_tensors="pt", add_generation_prompt=True, return_dict=True,
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Eval results

Evaluated on the 12-question held-out eval set from energy-text2sql (result-row equivalence, sort-insensitive, float-tolerant):

Model	Correct	Cost / run	Avg latency
Qwen2.5-Coder-1.5B (raw base, no fine-tune)	2/12 (17%)	$0.00	5.26s
Qwen2.5-Coder-1.5B + LoRA (this model)	6/12 (50%)	$0.00	3.12s
Claude Sonnet 4.5 (single-call baseline)	12/12 (100%)	~$0.05	4.6s
Claude Sonnet 4.5 multi-agent (LangGraph 4-node)	12/12 (100%)	~$0.10	9.1s

Fine-tuning tripled the base-model score (2 → 6 correct) at zero incremental inference cost. The model handles simple aggregations and filters (peaks, totals, summer averages) but still fails on subtle SQL rules (GROUP BY with non-aggregated columns, alias-in-ORDER-BY) and cross-domain joins requiring TZ casts. Not a Claude replacement — a smaller fine-tuned model that closes half the gap at $0 marginal cost.

Training data

• 500 raw Q/SQL pairs generated by Claude Sonnet 4.5 against the ERCOT schema
• –3 dropped in validation (SQL didn't execute or returned empty)
• –22 dropped as eval-set leaks (fuzzy paraphrase of held-out eval questions)
• –165 dropped as intra-set duplicates (same SQL skeleton, different literals)
• 310 unique → 280 train / 15 val / 15 test

Full generation + dedupe pipeline in text2sql-finetune.

Limitations

• Small model, narrow domain. Fine-tuned on 280 pairs against one Postgres schema. Won't generalize to other schemas.
• Claude Sonnet 4.5 still wins on this task. It scores 12/12 on the held-out eval; this fine-tuned model is not a Claude replacement. The value here is proximity — how close a 1.5B fine-tuned model can get to Claude Sonnet 4.5.
• Synthetic data biases. The training pairs were generated by Claude, so the model inherits Claude's SQL style and blind spots.
• Schema must be in the system prompt. See the How to use block — a stub prompt makes the model hallucinate tables.

Portfolio context

Built as a self-study project to demonstrate the full LLM fine-tuning workflow: synthetic dataset generation, deduplication (including test-set leak check), LoRA/PEFT on HuggingFace TRL, and evaluation against an existing multi-agent baseline. The repo README has the "What I learned" section — Claude data redundancy, GPU compatibility, QLoRA-vs-LoRA trade-off.

License

Apache-2.0. Base model retains its own Qwen2.5 license.