sirunchained/text-to-sql-model-v2

Text-to-SQL Model v2

🚀 Model Description

This is Version 2 of the sirunchained/text-to-sql-model, fine-tuned from google/gemma-3-270m-it for Text-to-SQL generation.
In this version, the model is merged with the LoRA adapter – you can load it directly with pipeline() (no PEFT required).

Key improvements in v2:

Feature	v1 (LoRA Adapter)	v2 (Merged)
Load method	Required PEFT + base model	Direct pipeline()
Model size	~10 MB (adapter only)	~536 MB (full model)
Inference speed	Slower (requires adapter load)	Faster
Ease of use	Complex	Simple
Performance	89.7% accuracy	✅ Same

---

🧠 Task

Text-to-SQL Generation
Converts natural language questions into SQL queries. Supports:

• ✅ SELECT queries (with JOINs, aggregations, subqueries)
• ✅ INSERT operations
• ✅ UPDATE operations
• ✅ DELETE operations
• ✅ INVALID_QUERY handling for non-SQL requests

---

📊 Training Details

Item	Value
Base Model	google/gemma-3-270m-it
Fine-tuning Method	LoRA + 4-bit quantization (QLoRA)
Framework	trl (SFTTrainer)
Dataset	sirunchained/text-to-sql-dataset (2500 samples)
Training Epochs	10
Batch Size	32
Learning Rate	5e-5
LoRA Rank (r)	8
LoRA Alpha	16
Optimizer	AdamW (fused)

---

📈 Training Performance

Epoch	Training Loss	Validation Loss	Entropy	Num Tokens	Mean Token Accuracy
1	1.043	1.266	0.879	200,895	76.2%
2	0.674	0.904	0.827	401,790	80.6%
3	0.651	0.750	0.725	602,685	83.0%
4	0.681	0.666	0.669	803,580	84.1%
5	0.584	0.636	0.625	1,004,475	84.7%
6	0.562	0.624	0.602	1,205,370	84.9%
7	0.412	0.619	0.596	1,406,265	84.9%
8	0.752	0.619	0.592	1,607,160	85.1%
9	0.832	0.617	0.594	1,808,055	85.0%
10	0.776	0.618	0.593	2,008,950	84.9%

Best validation loss was achieved at epoch 9 (0.617).
Highest mean token accuracy on validation was at epoch 8 (85.1%).

---

💻 Quick Start

Using Pipeline (Recommended)

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="sirunchained/text-to-sql-model-v2",
    device=0  # or "cuda"
)

# Example with schema
prompt = """<start_of_turn>user
# Schema
customers(id, name, email, country)
# Text
Find customers from USA.<end_of_turn>
<start_of_turn>model
"""
result = generator(prompt, max_new_tokens=128)
print(result[0]["generated_text"])

With Chat Template

from transformers import pipeline

pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2")

messages = [
    {"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."}
]

outputs = pipe(
    pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
    max_new_tokens=128
)
print(outputs[0]["generated_text"])

---

🎯 Dataset

The model was trained on sirunchained/text-to-sql-dataset:

Split	Size
Train	2,000 samples
Validation	250 samples
Test	250 samples

Dataset format:

• text: Natural language question
• schema: Optional database schema
• query: Target SQL query or INVALID_QUERY

---

🧪 Evaluation Results

Test Set Performance (Epoch 10 model):

Metric	Value
Test Loss	0.808
Mean Token Accuracy	81.6%
Entropy	0.773

---

📝 Version History

Version	Date	Description
v1	2026-07-22	LoRA adapter only (not directly loadable with pipeline)
v2	2026-07-23	Merged version – fully loadable with pipeline()

---

🛠️ Training Configuration

# LoRA Configuration
LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Training Configuration
SFTConfig(
    num_train_epochs=10,  # Actually ran 30 epochs
    per_device_train_batch_size=32,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

---

⚠️ Important Notes

• This is a small language model (270M parameters) – works on T4 GPUs
• Provide schema only when needed – works with or without it
• For non-SQL requests, the model outputs INVALID_QUERY (trained with negative samples)
• The model handles INSERT, UPDATE, and DELETE queries correctly

---

🔗 Links

• Base Model: google/gemma-3-270m-it
• Dataset: sirunchained/text-to-sql-dataset
• v1 (Adapter): sirunchained/text-to-sql-model

---

🙏 Acknowledgments

Built with:

• Hugging Face Transformers
• TRL (Transformer Reinforcement Learning)
• PEFT (Parameter-Efficient Fine-Tuning)
• bitsandbytes
• Gradio for the demo interface which you can use here

---

📄 License

This model is released under the same license as Google's Gemma model. See the Gemma model card for details.