vltruong01/IELTSWritingTask2-Qwen2.5-7B-QLoRA

IELTS Writing Task 2 — Qwen2.5-7B QLoRA

A QLoRA adapter for Qwen/Qwen2.5-7B-Instruct, fine-tuned to generate complete IELTS Writing Task 2 essays from essay prompts.

The goal of this project is to explore whether a relatively small QLoRA fine-tuning setup can improve structured IELTS-style essay generation while remaining practical on a single Google Colab T4 GPU.

This repository contains:

• the trained LoRA adapter,
• tokenizer files,
• the complete Colab training and evaluation notebook,
• reproducible training configuration,
• held-out evaluation workflow.

Model Details

• Base model: Qwen/Qwen2.5-7B-Instruct
• Model type: Causal language model with PEFT LoRA adapter
• Task: IELTS Writing Task 2 essay generation
• Language: English
• Fine-tuning method: QLoRA
• Quantization: 4-bit NF4 during training
• Training precision: FP16
• PEFT version: 0.20.0
• License: Apache-2.0

This repository contains the adapter weights rather than a fully merged 7B model.

Intended Use

The model is intended for generating IELTS Writing Task 2 practice essays, experimenting with parameter-efficient fine-tuning, studying QLoRA on long-form academic writing, and educational or research use.

The expected input is an IELTS Writing Task 2 question, and the expected output is a complete academic essay of approximately 280–330 words.

Limitations

This model should not be treated as an official IELTS scoring or preparation system.

Important limitations include:

• training essay band labels may not perfectly reflect true examiner-assessed IELTS quality,
• generated essays may still contain grammatical or logical errors,
• some outputs may use repetitive IELTS-style structures,
• performance varies across topics,
• a lower validation loss does not necessarily correspond directly to a higher IELTS band score.

Human evaluation using the official IELTS writing criteria is recommended.

Training Data

Training data was derived from chillies/ielts-writing-task2-essays.

Final filtering criteria:

Overall band >= 8.0
Task Achievement >= 7.0
Coherence & Cohesion >= 7.0
Lexical Resource >= 7.0
Grammatical Range & Accuracy >= 7.0
Word count: 250–420

Exact duplicate essays were removed.

After filtering:

Filtered essays: 713
Unique literal questions: 383
Topics: 25
Mean overall band: 8.302

Average subscores:

Task Achievement:              8.123
Coherence & Cohesion:          7.805
Lexical Resource:              8.066
Grammatical Range & Accuracy:  8.756

Train / Validation / Test Split

Near-duplicate IELTS prompts were grouped before splitting to reduce question leakage.

Split	Essays	Question Groups	Topics	Mean Band
Train	626	114	25	8.299
Validation	51	14	9	8.324
Test	36	14	11	8.319

The split was performed by question_group, not individual essay rows.

Training Configuration

Base model: Qwen/Qwen2.5-7B-Instruct

Quantization:
- 4-bit NF4
- double quantization
- FP16 compute

LoRA:
- rank: 32
- alpha: 64
- dropout: 0.05

Target modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj

Training:
- epochs: 3
- micro batch size: 1
- gradient accumulation: 8
- effective batch size: 8
- learning rate: 1e-4
- scheduler: cosine
- warmup ratio: 0.08
- optimizer: paged_adamw_8bit
- gradient checkpointing: enabled
- max sequence length: 768

Only assistant-response tokens contributed to the training loss. System and user tokens were masked with -100.

Hardware

Google Colab
NVIDIA Tesla T4
~15 GB VRAM

Training took approximately 1.5 hours for three epochs.

Validation Results

Validation loss reached its best value around step 80.

Step	Training Loss	Validation Loss
20	2.6705	2.6105
40	2.5850	2.5829
60	2.4710	2.5726
80	2.6456	2.5646
100	2.4191	2.5794
120	2.3801	2.5853
180	2.1274	2.6384
220	2.2319	2.6405

The training configuration used:

load_best_model_at_end=True
metric_for_best_model="eval_loss"
greater_is_better=False

so the best validation checkpoint was restored before saving the final adapter.

Evaluation

Evaluation was performed on held-out question groups that were not used during training.

The notebook includes deterministic essay generation, word-count checks, repeated 4-gram analysis, and manual comparison with reference essays.

The primary recommended evaluation framework is the IELTS Writing Task 2 rubric:

• Task Response
• Coherence & Cohesion
• Lexical Resource
• Grammatical Range & Accuracy

Automatic loss alone should not be interpreted as an IELTS band score.

How to Use

Install dependencies:

pip install transformers peft accelerate bitsandbytes

Load the base model and adapter:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER_REPO = "vltruong01/IELTSWritingTask2-Qwen2.5-7B-QLoRA"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_REPO,
)

model.eval()

Generate an essay:

SYSTEM_PROMPT = (
    "You are an expert IELTS Writing Task 2 writer aiming for Band 8 or higher. "
    "Write a complete academic essay that answers every part of the task and "
    "maintains a clear position. Prioritize grammatical accuracy, logical "
    "cohesion, and precise natural vocabulary."
)

question = '''
Some people believe that artificial intelligence will improve people's lives,
while others think it will create serious problems for society.

Discuss both views and give your own opinion.
'''

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": (
            "IELTS Writing Task 2\n\n"
            f"{question}\n\n"
            "Write only the final essay, approximately 280–330 words."
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=520,
        do_sample=False,
        repetition_penalty=1.03,
    )

new_tokens = outputs[0, inputs["input_ids"].shape[1]:]
essay = tokenizer.decode(new_tokens, skip_special_tokens=True)

print(essay)

Reproducibility

The complete Colab notebook is included in this repository:

IELTSWritingtask2_Qwen25_7B.ipynb

It contains the full pipeline from dataset loading and filtering through QLoRA training, checkpoint selection, evaluation, and held-out test generation.

Future Work

Possible improvements include:

• manually curated or rewritten Band 8–9 target essays,
• stronger human evaluation,
• comparison against the unfine-tuned Qwen2.5-7B baseline,
• testing larger Qwen models,
• preference optimization after supervised fine-tuning,
• evaluation on external IELTS prompts not derived from the training dataset.

Acknowledgements

This project builds on:

• Qwen/Qwen2.5-7B-Instruct
• Hugging Face Transformers
• PEFT
• bitsandbytes
• the chillies/ielts-writing-task2-essays dataset

Disclaimer

This is an experimental educational model.

It is not affiliated with IELTS, Cambridge University Press & Assessment, the British Council, or IDP Education, and its generated essays or estimated quality should not be interpreted as official IELTS assessment.