funkaya1234/wikisql-qwen2.5-coder-qlora

WikiSQL Qwen2.5-Coder QLoRA

1. Introduction

Natural-language-to-SQL generation is a useful task because many people need to work with structured data but do not know how to write SQL queries. The goal of this project is to take a table schema and a plain-English question and return the correct SQL query. Current LLMs can often generate SQL, but they may struggle with exact column names, capitalization, formatting, and returning only the query without extra explanation. To address this, I fine-tuned Qwen/Qwen2.5-Coder-3B-Instruct using QLoRA on WikiSQL-style examples. After fine-tuning, the model’s normalized exact-match accuracy on the held-out WikiSQL test set improved from 0.27 to 0.64.

2. Data

For fine-tuning, I used the mlx-community/wikisql dataset from Hugging Face. Each example contains a table, column names, a natural-language question, and a target SQL query. I reformatted each example into an instruction-response format. The instruction includes the table ID, the list of columns, and the user question, while the response is the correct SQL query. I used the dataset’s built-in splits instead of creating a new random split: 500 training examples for fine-tuning, 100 validation examples for selecting the best adapter, and 100 test examples for final evaluation.

3. Methodology

For the training method, I used QLoRA fine-tuning. I chose QLoRA because it is more practical than full fine-tuning for a 3-billion-parameter model on available GPU resources. QLoRA loads the base model in 4-bit precision and trains small LoRA adapter weights instead of updating all of the original model parameters. This makes training more memory-efficient while still allowing the model to adapt to the text-to-SQL task.

I tested three QLoRA hyperparameter settings. The first used rank 8 and alpha 16 to keep the adapter smaller. The second used rank 16 and alpha 32 to give the adapter more capacity. The third also used rank 16 and alpha 32, but lowered the learning rate and increased dropout to reduce overfitting. The best adapter used rank 16, alpha 32, dropout 0.05, and learning rate 0.0002. Training used 50 max steps, per-device train batch size 2, gradient accumulation 4, bf16, paged AdamW 8-bit optimization, and validation loss to select the best checkpoint.

4. Evaluation

To evaluate the model, I compared the base model and the fine-tuned adapter on the held-out WikiSQL test set and three lm_eval benchmark tasks. The WikiSQL test set is the most important benchmark because it directly measures the final project task: converting a schema and natural-language question into the correct SQL query. The three lm_eval tasks were included to check whether SQL fine-tuning caused major changes in broader computer-science performance.

Benchmark	Base Model	Fine-Tuned Adapter	Change
WikiSQL held-out test	0.27	0.64	+0.37
global_mmlu_full_en_college_computer_science	0.52	0.48	-0.04
global_mmlu_full_en_high_school_computer_science	0.68	0.70	+0.02
global_mmlu_full_en_computer_security	0.75	0.73	-0.02

The results show that QLoRA fine-tuning substantially improved the main text-to-SQL task. WikiSQL normalized exact-match accuracy increased from 0.27 to 0.64, which suggests that the adapter learned the expected SQL output format and became better at mapping natural-language questions to columns, conditions, and SQL structure. The broader benchmark scores stayed relatively stable. College computer science decreased slightly, high school computer science increased slightly, and computer security decreased slightly. Overall, this suggests that the adapter improved task-specific SQL generation without seriously hurting broader computer-science performance.

I originally compared multiple open Hugging Face models during prompting experiments, including TinyLlama, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-7B-Instruct. TinyLlama served as a small baseline, Qwen2.5-Coder-3B-Instruct served as the medium code-focused model, and Qwen2.5-Coder-7B-Instruct served as the larger code-focused comparison model. Qwen2.5-Coder-3B-Instruct was selected for fine-tuning because it gave the best balance of text-to-SQL performance and practicality for training on Rivanna.

5. Usage and Intended Uses

The intended use case for this model is natural-language-to-SQL generation on WikiSQL-style single-table questions. A user provides a table ID, column names, and a plain-English question, and the model returns only the SQL query. This model is best used for educational or experimental text-to-SQL tasks where the schema is provided directly in the prompt. It should not be used for production database querying without human review because the model may still generate incorrect SQL.

Below is an example of how to load the QLoRA adapter on top of the original base model.

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_name = "Qwen/Qwen2.5-Coder-3B-Instruct"
adapter_name = "funkaya1234/wikisql-qwen2.5-coder-qlora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(adapter_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

Prompt Format

The model uses a direct instruction format. The prompt gives the model the table ID, column names, and natural-language question. The model is instructed to return only the SQL query.

You are a text-to-SQL assistant. Return only the SQL query.

Table: 1-10015132-16
Columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team
Question: What is terrence ross' nationality

SQL:

Expected Output Format

The expected output is a single SQL query with no markdown formatting, no explanation, and no extra text.

SELECT Nationality FROM 1-10015132-16 WHERE Player = 'Terrence Ross'

Limitations

The main limitation of this model is that it was fine-tuned on WikiSQL-style single-table examples. Because of this, it may not generalize well to complex SQL tasks involving joins, nested queries, multiple tables, or unfamiliar database schemas. The model can still hallucinate column names or produce invalid SQL, especially if the prompt does not include enough schema information. The evaluation uses normalized exact-match accuracy, which is strict and may count logically similar SQL queries as incorrect if they differ in formatting, capitalization, or string values. Since this repository contains a QLoRA adapter, it should be loaded on top of the original Qwen2.5-Coder-3B-Instruct base model.

Citations and Links

• Base model: Qwen/Qwen2.5-Coder-3B-Instruct
• Dataset: mlx-community/wikisql
• PEFT library: peft
• Evaluation library: lm-evaluation-harness