drivesignal-distilbert

Fine-tuned distilbert-base-uncased that classifies free-text descriptions of autonomous-vehicle disengagement events into a 10-category safety scenario taxonomy. Trained on real California DMV Autonomous Vehicle Disengagement Reports (2022–2024).

Part of DriveSignal, a project comparing a rule-based heuristic, this fine-tuned DistilBERT classifier, and a LoRA-fine-tuned Qwen3-1.7B on the same task.

Taxonomy

Perception Failure | Prediction Failure | Lane Keeping | Braking Behavior |
Unwanted Maneuver | Construction/Environment | Precautionary |
System/Hardware Fault | Localization/Mapping | Other

Training data

• 14,800 disengagement event descriptions from the CA DMV 2022–2024 reports (Waymo and other manufacturers).
• Weak-labeled: training labels come from a keyword/regex heuristic classifier, not human annotation — a standard weak-supervision pattern for bootstrapping labels when ground truth doesn't exist. See training script: src/finetune_distilbert.py.

Evaluation

Evaluated against a 250-row hand-labeled gold set (not seen during training, and not derived from the heuristic labels):

Model	Accuracy (n=250)
Heuristic baseline	80.0%
DistilBERT (this model)	80.0%
Qwen3-1.7B LoRA	72.8%

Full breakdown: src/eval_distilbert_gold.py.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="jahnavidanda02/drivesignal-distilbert")
clf("Vehicle disengaged after hesitating at an unprotected left turn due to "
    "misjudging the trajectory of an oncoming vehicle.")

Limitations

• Trained on weak (heuristic-derived) labels for the bulk of the data, so it inherits gaps in the heuristic's keyword coverage (e.g. rare categories like Construction/Environment and Precautionary are underrepresented).
• Trained only on CA DMV disengagement report language; may not generalize to differently phrased AV incident text.