Answer or Abstain: Extraction with MLX
We studied source-grounded question answering on an Apple M4 Pro with 24 GiB of
unified memory: given a paragraph and question, return {"answer":"exact source span"} or {"answer":null}. A frozen BF16 Qwen2.5-1.5B-Instruct base with small
FP32 LoRA adapters made the task practical to train locally. The useful outcome
is a bounded experiment and resource recipe, together with measured trade-offs
between extracting answers and returning null.
The baseline trained for 200 optimizer updates, adapting the last four layers' attention q/k/v/o projections with rank 8 and direct scale 16. We then tested weighted negatives, same-paragraph positive/negative pairing, longer training, and eight-layer capacity. These follow-ups did not preserve enough correct positive answers to replace the original adapter.
Latest shared development bank
This table uses 256 cases: 128 answerable and 128 unanswerable, shared by all four systems. Other studies used different banks and are reported separately in EXPERIMENTS.md.
| System | Exact positive /128 | Correct null /128 | Strict JSON /256 |
|---|---|---|---|
| Incumbent, four layers, 200 updates | 73 | 80 | 256 |
| A, four layers, 400 updates | 59 | 101 | 256 |
| A, four layers, 600 updates | 67 | 92 | 256 |
| B, eight layers, 600 updates | 65 | 97 | 256 |
The latest study ran two seed-719 fits: A continued to 600 updates with fixed 400/600 checkpoints; B ran to 600 updates. No seed-2027 confirmation ran. The 512-case final reservation remains unscored. More correct nulls often came with fewer correct extracted answers; the incumbent was retained.
Follow-up: can calibration help without another fit?
We tested a fixed answer-versus-null score on the existing adapter, with thresholds selected on opposite article-group folds of the same 256-case development bank. No neural parameters changed.
| Metric | Original greedy policy | Cross-fitted policy |
|---|---|---|
| Accepted-answer precision | 73/156 = 46.79% | 69/143 = 48.25% |
| Exact positive /128 | 73 | 69 |
| Correct null /128 | 80 | 91 |
The policy corrected more negative cases but gained only 1.46 percentage points of precision, below the predeclared 10-point target. Overall positive retention passed; one held fold missed its 90% floor. We stopped this experiment family without another fit or final-set evaluation. The model process took 135.15 s.
The screen combines forced-string decoding with thresholding, so it measures that policy combination—not calibration alone. It reuses exposed development data. See the method and decision and aggregate results. The standalone training recipe below is unchanged and does not implement this diagnostic policy.
Making the fit fit in memory
A 512 MiB process-local free-cache limit and 4 MiB FP32 hash chunks let us check frozen-base integrity without materializing whole FP32 tensors and duplicate byte buffers. The resource-repair fit completed 200 updates in 222.16 s, with sampled physical-footprint peak 4.72 GB and no observed swap growth. Warm updates were 0.947 s p50 / 1.496 s p95 (199 updates). The later two-fit study peaked at 5.05 GB sampled footprint; its fit workers took 638.46 s / 712.61 s. The cache API reclaims excess free cache on a subsequent allocation; it is not a hard process-memory cap. GB/MB are decimal; GiB/MiB are binary.
Use the recipe
This repository hosts code, tests and aggregate reports in a Hugging Face model repository. It contains no weights, adapters or dataset records. The new public packaging extracts parity-tested numerical primitives from the historical code; it is not the executable that produced the historical fits. It supports a local baseline/longer-training/capacity recipe, with a documented fresh group split. Saved fits use versioned receipts binding the complete adapter schema and file hash to the recipe and local tokenizer/template identity. Evaluation validates these before inference; see fit format details.
On Apple Silicon with Python 3.12 and uv:
uv sync --locked --python 3.12
uv run --frozen --offline python -m mlx_experiment --help
uv run --frozen --offline pytest -q
uv run --frozen --offline ruff check mlx_experiment tests
uv run --frozen --offline ruff format --check mlx_experiment tests pyproject.toml
uv run --frozen --offline ty check mlx_experiment tests
shasum -a 256 -c SHA256SUMS
The first sync installs the locked dependencies; subsequent offline commands use the prepared environment/cache. Help, imports and tests do not download assets or load a real model. Tests use invented text and tiny tensors only. See the local-data and explicit-fit commands for the input schema, pinned upstream revisions, split policy and fitting opt-in. results.json contains aggregate counts and source/evidence hashes; SHA256SUMS covers the published files other than itself.
Evaluation context
These are single-seed, development-bank experiments on public SQuAD2 human labels; fresh questions within reserved article groups are not independent populations. The benchmark may have appeared in pretraining. Reused development banks, article clustering and unscored final data limit generalization and uncertainty claims. Sampled memory can miss transient peaks, and the resource comparison does not isolate host effects or establish a single causal explanation.
Our code is MIT-licensed. Qwen is Apache-2.0; SQuAD2 is CC-BY-SA-4.0. No upstream assets are redistributed or relicensed here. See ATTRIBUTION.md.