weft-lineage-extractor-3b

Self-hostable 3B model that extracts table-level data lineage (reads/writes) from ETL scripts as JSON. Fine-tuned (LoRA) on real open-source ETL scripts. On a held-out, source-isolated real-world benchmark it reaches table precision 0.64 (all scripts, calibrated gold + grounding filter) / 0.74 on non-empty scripts — matching a single pass of a frontier LLM teacher, at a size you can run on a single 12 GB GPU.

This model resolves the negative result documented by its synthetic-trained siblings (0.5B / 1.5B / JVM 1.5B): training on real scripts instead of synthetic ones eliminates the memorization leak and lifts real-world precision from 0.33 → 0.64.

• Base: Qwen/Qwen2.5-Coder-3B-Instruct
• Method: LoRA SFT on ~1.8k real silver-labelled ETL scripts (Python/Shell/SQL).
• Task: given a script, output {"reads": [...], "writes": [...]}; abstain ([]) when there is no lineage.

---

Intended uses & limitations

Intended use

• ✅ Self-hosted, cost-free table-level lineage for imperative/SQL-embedding ETL scripts, as the LLM channel of a routed lineage system (rule parsers for config jobs, a SQL parser for pure SQL, this model for free-form scripts).
• ✅ A precision-first extractor: pair it with the grounding filter below to enforce the task rule "a table counts only if its literal name appears in the script".

Out of scope

• ❌ Column-level lineage (schema field exists but is best-effort; evaluated claims are table-level).
• ❌ Dynamically-built table names (f-strings, shell/notebook vars, format()), commented/logged SQL, temp views — intentionally excluded.
• ❌ Ground-truth-critical governance without human review on ambiguous cases (see Limitations).

---

Evaluation

Benchmark — gold C: 153 real GitHub ETL scripts (49 non-empty / 104 with no lineage), human-relevant "Convention A" labels, source-isolated from the training corpus (training = the-stack, benchmark = fresh GitHub) with content-hash de-contamination. Table-level metrics.

Yardstick	ALL precision	non-empty precision	ALL recall	direction
Raw teacher gold, no filter	0.457	0.745	0.658	0.642
Calibrated gold + grounding filter	0.642	0.742	0.633	0.633

• Calibrated gold: the raw teacher labels missed real lineage on 12 hard scripts (e.g. Databricks notebooks); a strongest-teacher (frontier LLM) blind re-adjudication flipped those false-empties. This is a yardstick correction, transparently reported alongside the raw number.
• Grounding filter (deterministic, ships in the recipe): drop any predicted table whose leaf name is not literally in the script, or that contains dynamic markers $ { } %. Enforces the task rule; +2 precision points at zero recall cost.

How it compares to its synthetic-trained siblings (same real benchmark family):

model	training data	real ALL precision	real non-empty precision	verbatim memorization leak
1.5B / 3B synthetic (041 study)	synthetic only	0.27 / 0.33	~0.61	22% / 11%
this model (3B, real corpus)	real the-stack	0.46 → 0.64	0.74	~0

---

How to use

import json, re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "wallfacers/weft-lineage-extractor-3b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()

# System prompt — must match training verbatim.
SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON, SHELL, SCALA or "
          "JAVA task script (Spark/Flink jobs included), output ONLY a JSON object "
          "{\"reads\": [...], \"writes\": [...]} where each item is {\"table\": str, "
          "\"columns\": [str] or null}. Rules: include a table only if its literal name appears "
          "in the script text; ignore dynamically-built table names, commented-out SQL, and SQL "
          "that is merely printed or logged; if nothing is read or written, output "
          "{\"reads\": [], \"writes\": []}.")

def _ground(tables, script):
    """Deterministic grounding filter: keep a table only if its leaf name is literally
    in the script and it carries no dynamic markers. Enforces the task rule; lifts precision."""
    low = script.lower()
    out = []
    for it in tables or []:
        t = (it.get("table") or "").strip()
        leaf = t.lower().split(".")[-1]
        if leaf and leaf in low and not any(c in t for c in "${}%"):
            out.append(it)
    return out

def extract(task_type, script, max_new_tokens=512, ground=True):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
    inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                                  return_dict=True, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
                             pad_token_id=tok.pad_token_id or tok.eos_token_id)
    raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
    m = re.search(r"\{.*\}", raw, re.DOTALL)
    obj = json.loads(m.group(0)) if m else {"reads": [], "writes": []}
    r, w = obj.get("reads") or [], obj.get("writes") or []
    if ground:
        r, w = _ground(r, script), _ground(w, script)
    return {"reads": r, "writes": w}

print(extract("PYTHON", 'spark.sql("INSERT INTO ods.users SELECT id FROM stg.users_raw")'))
# -> {"reads": [{"table": "stg.users_raw", ...}], "writes": [{"table": "ods.users", ...}]}

Decoding is deterministic (do_sample=False): same input → same output. The grounding filter is part of the recommended inference recipe — the reported 0.64 precision is with it on.

task_type is one of PYTHON | SHELL | SCALA | JAVA (SQL scripts pass as their host language).

---

Training

Parameter	Value
Base model	Qwen/Qwen2.5-Coder-3B-Instruct
Method	LoRA (bf16), merged weights published
Training data	~1,774 real ETL scripts (887 with lineage + 887 no-lineage), silver-labelled
Silver labels	cross-vendor agreement of two LLM teachers (deepseek-v4-flash ∩ qwen-max), rejection gate for dynamic/path/temp names, zero synthetic names
Corpus source	bigcode/the-stack-dedup (permissive licenses), ETL-idiom filtered
Epochs / max len	2 / 1024
Precision / hardware	bfloat16 / single 12 GB GPU
Decontamination	content-hash exclusion of the benchmark from training

Why real data matters: the synthetic-trained siblings memorize the generator's table vocabulary and recite it on real inputs (22–40% of their hallucinations are verbatim training names). Training on real scripts with a zero-synthetic-name silver pipeline removes that leak by construction and is the single biggest driver of the precision jump.

---

Limitations & honest disclosures

• Precision/recall trade-off: more no-lineage training examples make this model more conservative — recall on the real benchmark is ~0.63 (vs ~0.68 for a smaller-corpus variant). It is tuned for precision (the deployment-critical axis for lineage governance).
• Yardstick honesty: the headline 0.64 uses a teacher-calibrated gold + the grounding filter. The raw, unfiltered number against the original teacher gold is 0.457; both are reported.
• Label-ambiguity ceiling: even the strongest teacher cannot cleanly decide whether a dotted token is a table, a file, or a variable on some real scripts. Pushing precision materially past this point requires human gold labels.
• Teachers: silver labels come from LLM teachers (deepseek / qwen); the model matches those teachers' agreed convention, disclosed rather than claimed as human ground truth.
• Small benchmark: gold C is 153 scripts (49 non-empty). Treat absolute numbers as indicative.

---

Links & citation

• Study of the negative result it resolves: weft-lineage-extractor-1.5b
• Platform: Weft (data-weave)
• Base model: Qwen/Qwen2.5-Coder-3B-Instruct

@misc{weft-lineage-extractor-3b-2026,
  author       = {{Weft Contributors}},
  title        = {{Real-corpus training closes the synthetic-to-real gap for small-model
                   ETL data-lineage extraction}},
  year         = 2026,
  publisher    = {{Hugging Face}},
  howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-3b}}},
}

Trained with TRL + PEFT.