keshavsharma/qwen2.5-3b-sql-lora-sft

Qwen2.5-3B Text-to-SQL — LoRA SFT

A parameter-efficient fine-tuned version of Qwen2.5-3B-Instruct for converting natural-language questions and SQL database schemas into SQL queries.

The model was fine-tuned using LoRA (Low-Rank Adaptation) on the b-mc2/sql-create-context dataset.

Model Details

Property	Value
Base model	Qwen/Qwen2.5-3B-Instruct
Fine-tuning method	LoRA / PEFT
Task	Text-to-SQL
Dataset	b-mc2/sql-create-context
LoRA rank	32
LoRA alpha	64
LoRA dropout	0.05
Trainable parameters	~1.9%
Maximum sequence length	512 tokens
Training epochs	2
Learning rate	2e-4
LR scheduler	Cosine
Warmup ratio	0.03
Weight decay	0.01
Precision	BF16
Optimizer	AdamW fused
Train/eval split	90% / 10%
Split seed	42

Intended Use

The model takes:

1. A database schema represented using CREATE TABLE statements.
2. A natural-language question about the database.

It generates the corresponding SQL query.

Example

Schema:

CREATE TABLE employees (
    employee_id INTEGER,
    name TEXT,
    salary REAL,
    department TEXT
);

Question:

What is the average salary of employees in the engineering department?

Generated SQL:

SELECT AVG(salary)
FROM employees
WHERE department = 'engineering';

---

Training Method

This model uses Supervised Fine-Tuning (SFT) with LoRA.

Instead of updating all parameters of the 3B-parameter Qwen model, LoRA adds trainable low-rank matrices to selected transformer layers while keeping the original model weights frozen.

This substantially reduces the number of parameters that need to be optimized.

With the configuration used for this model, approximately 1.9% of the model parameters are trainable.

LoRA Configuration

The adapter was configured as:

LoraConfig(
    r=32,
    lora_alpha=64,
    lora_dropout=0.05,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj"
    ],
    bias="none",
    task_type="CAUSAL_LM",
)

Targeted Modules

LoRA adapters were applied to the attention and MLP projections:

Attention

• q_proj
• k_proj
• v_proj
• o_proj

MLP

• gate_proj
• up_proj
• down_proj

This allows the adapter to modify both the model's attention behavior and its feed-forward transformations.

---

Dataset

Training uses the b-mc2/sql-create-context dataset.

The dataset provides examples containing:

• Database schema/context
• Natural-language question
• Target SQL query

The dataset was split into:

90% training
10% evaluation

using:

train_test_split(test_size=0.1, seed=42)

The maximum sequence length during tokenization was 512 tokens.

---

Prompt Format

The model was trained using the Qwen chat template.

The system instruction used during training was:

You are a text-to-SQL assistant. Given a database schema (CREATE TABLE statements) and a question, write the SQL query that answers the question. Respond with only the SQL query.

The user input contains the schema followed by the question:

{schema}

{question}

The expected SQL answer is then appended to the formatted prompt.

Conceptually, the training example looks like:

System:
You are a text-to-SQL assistant...

User:
CREATE TABLE employees (
    employee_id INTEGER,
    name TEXT,
    salary REAL,
    department TEXT
);

What is the average salary of employees in engineering?

Assistant:
SELECT AVG(salary) FROM employees WHERE department = 'engineering';

---

Loss Masking

An important part of the training implementation is prompt masking.

The loss is calculated only on the SQL answer tokens.

The system prompt and user/schema/question tokens are assigned:

-100

in the labels.

In Hugging Face's causal language-model training, -100 means that the corresponding token is ignored when calculating the loss.

Therefore, the model is explicitly trained to predict the SQL answer rather than being penalized for reproducing the input prompt.

Conceptually:

System prompt       → ignored
Schema              → ignored
Question            → ignored
SQL answer          → TRAINED

---

Training Configuration

The model was trained with the following configuration:

TrainingArguments(
    output_dir="/root/qwen2.5-3b-sql",
    num_train_epochs=2,

    per_device_train_batch_size=12,
    per_device_eval_batch_size=12,
    gradient_accumulation_steps=1,

    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,

    bf16=True,
    optim="adamw_torch_fused",

    eval_strategy="steps",
    eval_steps=200,

    save_strategy="steps",
    save_steps=200,
    save_total_limit=2,

    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

The best checkpoint was selected according to evaluation loss.

Early stopping was also enabled:

EarlyStoppingCallback(
    early_stopping_patience=5,
    early_stopping_threshold=0.001
)

---

Installation

Install the required packages:

pip install torch transformers datasets accelerate peft

For GPU inference, a CUDA-compatible PyTorch installation is recommended.

---

Inference

The model can be loaded as a PEFT LoRA adapter on top of the base Qwen model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-3B-Instruct"
ADAPTER = "keshavsharma/qwen2.5-3b-sql-lora-sft"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER
)

model.eval()

Simple Inference Function

SYSTEM_PROMPT = (
    "You are a text-to-SQL assistant. Given a database schema (CREATE TABLE "
    "statements) and a question, write the SQL query that answers the question. "
    "Respond with only the SQL query."
)

def generate_sql(schema, question):
    messages = [
        {
            "role": "system",
            "content": SYSTEM_PROMPT
        },
        {
            "role": "user",
            "content": f"{schema}\n\n{question}"
        }
    ]

    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    inputs = tokenizer(
        prompt,
        return_tensors="pt"
    ).to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=128,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id
        )

    generated_tokens = outputs[0][inputs["input_ids"].shape[1]:]

    return tokenizer.decode(
        generated_tokens,
        skip_special_tokens=True
    ).strip()

---

Inference Examples

Example 1 — Filtering

schema = """
CREATE TABLE teams (
    team TEXT,
    head_coach TEXT,
    president TEXT,
    home_ground TEXT,
    location TEXT
);
"""

question = "Who is the head coach of the team whose president is Mario Volarevic?"

print(generate_sql(schema, question))

Expected format:

SELECT head_coach
FROM teams
WHERE president = 'Mario Volarevic';

Example 2 — Aggregation

schema = """
CREATE TABLE employees (
    employee_id INTEGER,
    name TEXT,
    salary REAL,
    department TEXT
);
"""

question = "What is the average salary of employees in the engineering department?"

print(generate_sql(schema, question))

Expected format:

SELECT AVG(salary)
FROM employees
WHERE department = 'engineering';

Example 3 — Counting

schema = """
CREATE TABLE orders (
    order_id INTEGER,
    customer TEXT,
    status TEXT,
    amount REAL
);
"""

question = "How many orders were shipped by customer Dana?"

print(generate_sql(schema, question))

Expected format:

SELECT COUNT(order_id)
FROM orders
WHERE status = 'shipped'
AND customer = 'Dana';

---

Deterministic Generation

For evaluation, inference uses:

do_sample=False

This disables stochastic sampling and makes the generation process deterministic under the same model/runtime conditions.

This is useful when comparing the base model against different LoRA adapters or adapter-merging techniques.

For example:

Base Qwen
    ↓
LoRA Adapter A
    ↓
LoRA Adapter B
    ↓
Linear Merge
    ↓
TIES Merge
    ↓
DARE Merge

The same evaluation dataset and deterministic generation settings can then be used to compare the resulting models.

---

Loading the Adapter vs. Merging It

The repository contains a LoRA adapter, rather than a completely duplicated copy of the Qwen2.5-3B model.

Therefore, inference requires:

Qwen2.5-3B-Instruct
        +
LoRA adapter
        ↓
Fine-tuned model

The adapter can be loaded using PEFT:

model = PeftModel.from_pretrained(
    base_model,
    "keshavsharma/qwen2.5-3b-sql-lora-sft"
)

For deployment, the adapter can also be merged into the base model if a standalone model is desired:

merged_model = model.merge_and_unload()

The merged model can then be saved with:

merged_model.save_pretrained("./qwen2.5-3b-sql-merged")
tokenizer.save_pretrained("./qwen2.5-3b-sql-merged")

---

Intended Evaluation

This model was trained specifically as a text-to-SQL SFT experiment.

Possible evaluation metrics include:

• Exact SQL match
• Execution accuracy
• Query structure correctness
• Component-level SQL accuracy
• Valid SQL generation rate

Exact string matching can be overly strict because multiple SQL queries can sometimes produce the same result.

For example:

SELECT AVG(salary) FROM employees;

and:

SELECT AVG(salary)
FROM employees;

are semantically identical despite having different formatting.

For this reason, execution-based evaluation can provide a more meaningful measure of Text-to-SQL performance.

---

Limitations

This model has several limitations:

• It is fine-tuned on a specific Text-to-SQL dataset and may not generalize perfectly to arbitrary database schemas.
• Generated SQL should be validated before execution.
• The model may produce syntactically valid but semantically incorrect SQL.
• Exact performance depends on schema complexity, SQL dialect, and question formulation.
• The maximum training sequence length was 512 tokens.
• The model should not be assumed to support every SQL dialect equally well.
• No claim of production-level Text-to-SQL reliability is made.

---

Reproducibility

The main training setup was:

Base Model:
Qwen/Qwen2.5-3B-Instruct

Dataset:
b-mc2/sql-create-context

Train/Eval:
90% / 10%

LoRA:
rank = 32
alpha = 64
dropout = 0.05

Target Modules:
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Training:
2 epochs
learning rate = 2e-4
cosine scheduler
warmup ratio = 0.03
weight decay = 0.01
BF16
AdamW fused

The training notebook used a CUDA GPU with sufficient memory to train with a batch size of 12 without gradient checkpointing.

---

Citation / References

This model builds upon:

• Qwen2.5 — Qwen Team
• LoRA — Hu et al., LoRA: Low-Rank Adaptation of Large Language Models
• PEFT — Hugging Face Parameter-Efficient Fine-Tuning
• SQL Create Context dataset — b-mc2/sql-create-context

Base Model

Qwen/Qwen2.5-3B-Instruct

Dataset

b-mc2/sql-create-context

---

License

Please refer to the license and usage terms of the base Qwen model and the dataset before redistributing or deploying this adapter.

This repository contains a LoRA adapter and does not constitute an independent replacement for the underlying base model.