lfm2.5-350M-datause-multitask-tiered
LoRA SFT of LiquidAI/LFM2.5-350M for the tiered data-use workflow: provenance attributes (producer/year/geography/acronym + data_type/specificity/bibtex) and usage/impact classification (usage_action/impact_label/usage_summary). Trained exclusively on T1∪T2-anchored rows of rafmacalaba/data-use-sft-tiered — see the dataset card for the tier policy.
Training
- base model:
LiquidAI/LFM2.5-350M - dataset:
rafmacalaba/data-use-sft-tiered(provenance + usage_impact configs) - epochs: 3 | lr: 0.0002
- LoRA: r=16 alpha=32 dropout=0.05
- completion-only masking (loss on the assistant JSON turn)
Holdout
n=8804. Full per-row gold/pred pairs are in holdout_predictions.jsonl on this repo.
Metrics recomputed offline from holdout_predictions.jsonl with string-aware JSON extraction (training/score_holdout_predictions.py). An earlier in-trainer extraction bug suppressed predictions containing bibtex values; the numbers below are the corrected ones.
Provenance — verbatim attrs (exact match)
| attribute | tp | fp | fn | precision | recall | f1 |
|---|---|---|---|---|---|---|
| producer | 1050 | 368 | 168 | 0.7405 | 0.8621 | 0.7967 |
| year | 980 | 241 | 259 | 0.8026 | 0.7910 | 0.7967 |
| geography | 1531 | 436 | 205 | 0.7783 | 0.8819 | 0.8269 |
| acronym | 1175 | 204 | 149 | 0.8521 | 0.8875 | 0.8694 |
| overall | 4736 | 1249 | 781 | 0.7913 | 0.8584 | 0.8235 |
bibtex (gold rows with producer+year):
exact: 178 (33.6%) | semantic (author+title+year, key ignored): 308 = 58.2% (key-only diffs: 130)
format: 99.7% of 362 emitted are canonical
@datasetentriesomitted: 167 (all missing producer/year inputs) | hallucinated on gold-none rows: 180
data_type macro-F1: 0.5026 (n=1198)
specificity macro-F1: 0.9423 (n=3325)
Usage/impact
- usage_action macro-F1: 0.6573 (n=5479)
- impact_label macro-F1: 0.6352 (n=5479)
Top usage_action confusions: analyze -> inform (250), inform -> analyze (226), analyze -> source (137), source -> analyze (134), curate -> analyze (125) Top impact_label confusions: policy -> evidence-base (315), evidence-base -> none (259), none -> evidence-base (252), evidence-base -> policy (238), none -> policy (84)
usage_summary: mean_sim=0.5604 grounded_rate=0.6720