eric-ml-nlp/Delveta-LayaChoice-v2-checkpoints

Delveta LayaChoice v2 — checkpoints and training evidence

This repository records the training run layachoice_v2_082_r1 behind eric-ml-nlp/Delveta-LayaChoice-v2. It is not the inference entry point.

• The deployed model is epoch 9 and lives only in the deployment repository.
• This repository holds the non-deployed checkpoints of the current run (epochs 2, 4, 6, 8, 10), each with its training metadata and validation metrics, plus the complete evidence for all 10 epochs.
• The previous V2 run's epoch-1…epoch-6 weights have been removed from the current revision and replaced by the current run. They remain in the repository history but are no longer part of the published checkpoint set.

Epoch 9's weights are not duplicated here: the authoritative deployment artifact is the model.safetensors in Delveta-LayaChoice-v2.

1. Run

Run ID	layachoice_v2_082_r1
Server run directory	/workspace/runs/layachoice_v2_082_r1/
Data bundle	recall_threshold_082, flat layout (manifest.json + train/val/test.jsonl)
Train / val / test rows	36,197 / 2,407 / 9,729
Calibration rows	400 (removed from train; 35,797 effective train rows)
Option contract	dynamic K+1: 1–4 options including exactly one REJECT
Card view	B_noprov
Base model	convaiinnovations/laya, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851, subfolder multilingual
Training hardware	1 × NVIDIA L4 24 GB
Training time	17,673 s ≈ 4.91 h
Selected epoch	9 (val_top1 = 0.996676)

Training configuration

Parameter	Value
Epochs	10
Micro batch	8
Gradient accumulation	4
Group size (GRPO)	4
Encoder LR	2.5e-5
Head LR	1.0e-4
Weight decay	0.01
Sigma schedule	0.4 → 0.1
CE weight	1.0
w_sph / w_rps	0.75 / 1.0
Gradient clip	1.0
Seed base / calibration seed	42 / 20260922
Precision	BF16 autocast, GradScaler disabled
Token budget	option 256, head 768, max 1,024
Optimizer updates	11,190 planned and actual
Runtime	Python 3.10.20, torch 2.14.1+cu130, transformers 5.17.0, laya 0.3.21

2. Complete epoch 1–10 record

Training values come from each epoch's train_meta.json; validation values come from each epoch's val_metrics.json (2,407 rows, 220 gold-reject rows). Weight saved here indicates whether that epoch's model.safetensors is published in this repository.

Epoch	Mean train loss	Sigma	Seed	Train s	Val top-1	Val ECE	Reject recall	Reject FPR	Reject precision	Weight saved here
1	0.354976	0.4000	42	1678.69	0.990445	0.009463	0.945455	0.003201	0.967442	no
2	0.114831	0.3667	43	1677.86	0.981305	0.018557	0.945455	0.013260	0.877637	yes
3	0.052656	0.3333	44	1678.71	0.989198	0.009794	0.909091	0.000914	0.990099	no
4	0.033588	0.3000	45	1677.71	0.989614	0.010101	0.931818	0.002743	0.971564	yes
5	0.020002	0.2667	46	1678.19	0.972995	0.026366	0.922727	0.019204	0.828571	no
6	0.012646	0.2333	47	1677.47	0.994184	0.005269	0.963636	0.001372	0.986047	yes
7	0.004379	0.2000	48	1677.15	0.995015	0.005989	0.963636	0.000914	0.990654	no
8	0.003872	0.1667	49	1680.51	0.995430	0.004793	0.968182	0.000914	0.990698	yes
9	0.004162	0.1333	50	1677.18	0.996676	0.003710	0.981818	0.000914	0.990826	no — deployed separately
10	0.001076	0.1000	51	1676.60	0.995430	0.004196	0.968182	0.000914	0.990698	yes

Every epoch's full validation report is preserved in runs/layachoice_v2_082_r1/selection/epoch-N.val.report.json, including epochs whose weights are not published here.

3. Best model

Selection uses argmax validation top-1, ties resolved to the earlier epoch. Epoch 9 has the highest validation top-1 (0.996676); epoch 10 has a lower training loss but a lower validation top-1, so it is not selected.

The deployed epoch-9 weight is published in eric-ml-nlp/Delveta-LayaChoice-v2 (SHA256 964f8fe5dde2812ad3902d8237035bfb109885d4fefef590307bc681bd8fab14).

4. Final test and calibration

Test (9,729 rows, fitted temperature 3.0702)

Metric	Value
Top-1	0.998355 (9,713 / 9,729)
ECE	0.002576
Reject recall	0.996441 (840 / 843)
Reject precision	0.995261
Reject FPR	0.000450
Confusion	TP 840, FP 4, FN 3, TN 8,882

By language (test top-1): en 0.998197 (4,438), mixed 0.999137 (1,159), zh 0.998306 (4,132).

Zero-shot baseline (test)

Metric	Baseline	Fine-tuned epoch 9
Top-1	0.416795	0.998355
ECE	0.206293	0.002576
Reject recall	0.595492	0.996441
Reject precision	0.128751	0.995261
Reject FPR	0.382287	0.000450

Calibration

Field	Value
Raw fitted value (choice slot)	3.070228338241577
Written value	3.070228338241577 (no clamping)
Temperature vector	[3.070228338241577, 1.0, 1.0]
Calibration rows / seed	400 / 20260922
Clamp	[0.5, 5.0]

5. Published checkpoints

Each directory contains exactly three files: the epoch weights, train_meta.json and val_metrics.json.

Directory	Weight size	Weight SHA256
epoch-2/	1.29 GB	514c7e2f1e73bafc4a523e90c99eb14a25045dff9f8b3117c323cb9acb3fb492
epoch-4/	1.29 GB	a831061b4c3218262c713bbafeb50d13265b53a6e8e906d3fa1d6c6ecd7d3994
epoch-6/	1.29 GB	31e00d302f5f7590a53f1c379862ace463b89566bff295b6e4b6b5b51476c44a
epoch-8/	1.29 GB	2d512ec730c6f06279c1ffe877ab5b00a6af495e58a2670ae3d05bc3ebfa59c7
epoch-10/	1.29 GB	131d557409dceefb546480093c2a0707848997cbe2e998a8afb5f24ec4ca1625

These are non-deployed historical checkpoints of the current run. They are not separate evaluation results and are not claimed to be better than epoch 9.

6. File guide

Path	Purpose
epoch-{2,4,6,8,10}/model.safetensors	Published non-deployed checkpoints
epoch-{2,4,6,8,10}/train_meta.json	Per-epoch training record: loss, sigma, LR, seed, updates, duration
epoch-{2,4,6,8,10}/val_metrics.json	Per-epoch validation metrics
runs/layachoice_v2_082_r1/train_manifest.json	Authoritative run record: config, rows, calibration indices, bundle hashes, runtime
runs/layachoice_v2_082_r1/train.log	Full pipeline log
runs/layachoice_v2_082_r1/run_ledger.jsonl	Invocation record with data identity
runs/layachoice_v2_082_r1/selection/selection.json	Best-epoch record and per-epoch summary
runs/layachoice_v2_082_r1/selection/epoch-N.val.report.json	Full validation reports for all 10 epochs
runs/layachoice_v2_082_r1/calibration/temperature.json	Fitted temperature and calibration provenance
runs/layachoice_v2_082_r1/benchmark/final_test.report.json	Final frozen-test report for epoch 9
runs/layachoice_v2_082_r1/baseline/baseline.report.json	Zero-shot baseline report
runs/layachoice_v2_082_r1/analysis/val_metrics.json / .csv	All 10 epochs of validation metrics
runs/layachoice_v2_082_r1/analysis/*.png	Training-loss and validation curves
runs/layachoice_v2_082_r1/analysis/plot_val_curves.py	Plotting script
runs/layachoice_v2_082_r1/config/rl_agent_config.json	Exported Agent config with corrected training block
runs/layachoice_v2_082_r1/dataset/SHA256SUMS	Frozen data-bundle hashes
SHA256SUMS	Hashes of all published files in this repository

7. What is not included

• train_state.pt for every epoch (~2.58 GB each, ~25.75 GB total), optimizer and scheduler state.
• Weights for epochs 1, 3, 5, 7 (not selected for this publication).
• The epoch-9 weight, which is published in the deployment repository.
• Row-level prediction dumps (*.rows.jsonl); aggregate reports and per-epoch metrics are included.

8. Reproduction

cd fine_tuning/laya_finetune/V2
bash run.sh all --epochs 10 --out <fresh-directory>

verify runs first and checks bundle hashes, row schema, labels, split disjointness, calibration reproducibility, and that the test split is never used for training or selection.

• Training code: https://github.com/Eric-LLMs/Delveta/tree/main/fine_tuning/laya_finetune/V2
• Deployment model: https://huggingface.co/eric-ml-nlp/Delveta-LayaChoice-v2