260910_b200_bringup — 6×B200 platform bring-up for full-param SDF

Status 2026-09-10: all six smoke tests pass. Box is ready for the SDF phase.

Goal: have the box proven end-to-end before the next SDF dataset lands, so the real run is run_phase.sh and nothing else.

Target recipe: 260725_mo14_sequential_sdf (full-param FSDP1, adafactor, Llama-3.3-70B-Instruct, lr 5e-6, max_len 2048), moved from 8×H200 to 6×B200.

What changes vs the H200 recipe

	H200 run	here	why
torch	2.4.1+cu124	2.8.0+cu128	2.4 has no sm_100 (Blackwell) kernels
transformers	4.46.3	4.56.2	only version validated on this box (B200 branch)
attention	flash-attn 2	sdpa	no Blackwell flash-attn wheel; validated equivalent on B200 branch
GPUs × bs	8 × 8 = 64 docs/step	6 × 11 = 66 docs/step	64 ∤ 6; 66 is within 3%
venv	v6_.../.venv	/root/venvs/sdf_b200	/root is wiped on pod stop — rebuild via scripts/00_setup_venv.sh

Smoke ladder (results in results/, logs in logs/)

#	test	status
1	training venv builds, torch sees 6 GPUs with sm_100	PASS — torch 2.8.0+cu128, sm_100 in arch list, transformers 4.56.2, accelerate 1.15.0
2	6-GPU NCCL all_reduce/broadcast, ± NCCL_NVLS_ENABLE=0	PASS both ways; 1 GB all_reduce 2.7 ms / 661 GB/s with NVLS=0, 3.0 ms / 591 GB/s default → keep NVLS=0
3	train_seq_sdf.py on SmolLM2-135M (Llama arch) + dummy parquet, 6-GPU FSDP, checkpoint + final save	PASS — 10 steps, loss 5.07→5.00, ckpt-5/ckpt-10/final written, final reloads with from_pretrained
4	Llama-3.3-70B, 660 docs / 10 steps, bs 11 × 6, checkpoint save → memory + save time	PASS — 12.3 s/step (H200: 14–17.5 s at 64/step), peak 157.8 GB/rank (reserved 188/192 — bs 11 is the ceiling), 282 GB fp32 save in ~2.7 min (H200 pod: 15–18 min), 62-shard final reloads header-clean
5a	vLLM venv loads 70B at tp=2 and generates	PASS — vLLM 0.29.0 / torch 2.13+cu130; picks FLASHINFER (works on 0.29; the 0.26-era TRITON_ATTN workaround is no longer needed) — needs ninja on PATH ($VENV/bin) or engine init fails
5b	scripts/07_validate_vllm.sh reproduces H200 reference numbers (188/188, 182/188)	PASS — exact match at tp=2, 500 prompts in 1.1 min → boundary evals on this box are trustworthy
6	dataset validator (scripts/05_validate_sdf_parquet.py) ready for the incoming data	PASS on the dummy parquet (schema, empties, dups, token lengths vs 2048, step count at 66/step)

Running a real phase when the dataset lands

E=experiments/260910_b200_bringup
/root/venvs/sdf_b200/bin/python $E/scripts/05_validate_sdf_parquet.py <phase.parquet> --tokenizer $E/base_model
cp $E/configs/phase_template_6gpu.yaml <your_experiment>/configs/phaseN.yaml   # fill data.path, model.name, max_steps, run names
bash $E/scripts/run_phase_b200.sh <your_experiment>/configs/phaseN.yaml       # inside tmux

max_steps = docs / 66. Phase 2+: model.name = previous phase's final/ dir. Output lands under <script's experiment>/runs_local_gpu/<wandb_run_name>/{output,final}.

Environment (same landmines as START_HERE §6)

• HF_HOME=/workspace/hf_cache; tokens in 260727_mo14_sft_sequential/.env (gitignored)
• /root is 50 GB local: venvs only. Checkpoints (~263 GB each) go to /workspace.