trinhtrantran122/vianli-multisource-transfer-code

EXP-001 — Corrected ViNLI STILTs → ViANLI (No Replay)

Feasibility pilot for whether a correctly mapped ViNLI intermediate-task checkpoint improves Flat CafeBERT on ViANLI. Seed 42 only. ViANLI test is locked.

Scientific intervention

reviewed ViNLI Flat CafeBERT checkpoint
-> swap classifier rows 1 and 2 (obsolete E/N/C -> canonical E/C/N)
-> verify corrected ViNLI dev/test metrics
-> measure zero-shot ViANLI dev
-> fine-tune on leakage-clean ViANLI train only (8,010 rows)
-> select checkpoint using ViANLI dev Macro-F1 only

There is no replay, ViMedNLI training, hierarchy, MoE, sweep, or target-test evaluation.

Immutable sources

• ViANLI: uitnlp/ViANLI@0fec8d6ecb043a61c609f9b51f80401fdf1e84d3
• ViNLI mirror: trantranuit/ViHLM_NLI_Project@47bd78ac5d075bd00a3cb4cdd3ede4eec4acf8c2
• Official ViMedNLI diagnostic source: justinphan3110/ViPubmed@2cd94305ba48ae1ccf8782c1df9819ddad7f035f
• CafeBERT: uitnlp/CafeBERT@af76fcf2a04096b2b54b348a3e4eb48253c93c5d
• Source checkpoint: trinhtrantran122/hier-nli-e-first-flat-cafebert-vinli@3ae7df0b009d14ece648dbdf4cfd88d7ad37f570

Setup and data gates

python -m pip install -r requirements.txt -r requirements-dev.txt
cp .env.example .env  # then add HF_TOKEN and WANDB_API_KEY; never commit .env

python scripts/prepare_vianli.py
python scripts/prepare_vinli_multisource.py
python scripts/prepare_vimednli_multisource.py
python scripts/audit_multisource.py \
  --vianli-dir data/processed/vianli_clean \
  --out data/audit/exp001_audit.json

python -m pytest tests -q
python scripts/validate_notebook.py notebooks/colab_train_multisource.ipynb

Expected prepared counts:

ViANLI clean: 8,010 / 1,000 / 1,000
ViNLI:       18,282 / 2,255 / 2,264
ViMedNLI:    11,217 / 1,395 / 1,422  (15 conflicting train rows removed)

Tiny real integration

The reproducible gate below uses a pinned tiny real Transformer, the real prepared datasets, classifier migration, target training, checkpoint reload, and offline prediction verification. Output is temporary, never uploaded, and never accepted as a scientific result.

python scripts/run_tiny_integration.py

Scientific run

Only run from a clean, reviewed commit after all gates pass:

python -m src.training.train_exp001 \
  --config configs/experiments/exp001_vinli_stilts_no_replay.yaml

Artifact contract

• W&B tracking: trinhtrantran3105-uit/hierarchical-nli-e-first
• Run: exp-001-vinli-stilts-no-replay-seed42
• Private HF source of truth: trinhtrantran122/vianli-exp-001-vinli-stilts-no-replay-seed42
• HF stores the best checkpoint, tokenizer/config, immutable metadata, and per-sample ViNLI dev/test plus ViANLI zero-shot/final-dev logits and predictions.
• W&B stores config, metrics, confusion/diagnostic metadata, and the exact immutable HF SHA.
• Search code does not open or upload ViANLI test. A future test evaluation requires an explicit reviewed frozen-final config and --frozen-final.

GO/STOP on ViANLI dev

• GO strong: Macro-F1 >= 0.500
• GO weak: Macro-F1 >= 0.485 with no class collapse
• STOP: Macro-F1 < 0.480 or unstable/class collapse

Mock/debug values are never entered into docs/RESULTS_LEDGER.md.