drivesignal-qwen3-lora

LoRA adapter for Qwen/Qwen3-1.7B that classifies free-text descriptions of autonomous-vehicle disengagement events into a 10-category safety scenario taxonomy, framed as text generation (the model completes a classification prompt with the category name). Trained on real California DMV Autonomous Vehicle Disengagement Reports (2022–2024).

Part of DriveSignal, a project comparing a rule-based heuristic, a fine-tuned DistilBERT classifier, and this LoRA-fine-tuned Qwen3-1.7B on the same task.

Taxonomy

Perception Failure | Prediction Failure | Lane Keeping | Braking Behavior |
Unwanted Maneuver | Construction/Environment | Precautionary |
System/Hardware Fault | Localization/Mapping | Other

Training

• Base model: Qwen/Qwen3-1.7B, loaded in 4-bit (QLoRA-style) for training.
• LoRA config: r=16, alpha=32, dropout=0.05, target modules q_proj/v_proj.
• 14,800 disengagement event descriptions from the CA DMV 2022–2024 reports.
• Weak-labeled: training labels come from a keyword/regex heuristic classifier, not human annotation — a standard weak-supervision pattern for bootstrapping labels when ground truth doesn't exist. See training script: src/finetune_qwen_lora.py.

Evaluation

Evaluated against a 250-row hand-labeled gold set (not seen during training, and not derived from the heuristic labels):

Model	Accuracy (n=250)
Heuristic baseline	80.0%
DistilBERT	80.0%
Qwen3-1.7B LoRA (this model)	72.8%

Full breakdown: src/eval_qwen_lora_gold.py.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True
)
model = PeftModel.from_pretrained(base, "jahnavidanda02/drivesignal-qwen3-lora")
tokenizer = AutoTokenizer.from_pretrained("jahnavidanda02/drivesignal-qwen3-lora")

taxonomy = ["Perception Failure", "Prediction Failure", "Lane Keeping", "Braking Behavior",
            "Unwanted Maneuver", "Construction/Environment", "Precautionary",
            "System/Hardware Fault", "Localization/Mapping", "Other"]
prompt = (f"Classify this AV disengagement into one category from: {', '.join(taxonomy)}.\n\n"
          "Description: Vehicle disengaged after hesitating at an unprotected left turn.\n\nCategory:")
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=15, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• Underperforms the much smaller/cheaper heuristic and DistilBERT baselines on this task (72.8% vs 80.0%) — a useful, honest data point on when generative LLM fine-tuning is not the right tool for a fixed-taxonomy classification problem versus an encoder model.
• Trained on weak (heuristic-derived) labels, so it inherits gaps in the heuristic's keyword coverage for rare categories.
• Generation-based classification occasionally produces text that doesn't exactly match a taxonomy label; inference code falls back to substring matching (see eval_qwen_lora_gold.py).

Framework versions

• PEFT 0.19.1