ssurface/cot-dialect-llama3.1-8b-sft-l5

LLaMA-3.1-8B — L5 dialect (Pure expression)

A LoRA adapter that makes meta-llama/Llama-3.1-8B-Instruct reason at compression level L5 — a single collapsed expression.

Results

This adapter was not separately benchmarked. It exists as a training artefact for the ablation grid; the levels that carry reported numbers are the ones in the collection's core set.

Training data

GSM8K train, re-expressed at level L5 by a teacher model: 6993 examples, median chain length 16 characters inside <think>.

Across the family the median chain runs from 532 characters at L1 to 16 at L5 — a 33x span. An L5 chain looks like this:

18/3*2=12

Training setup

Stage	supervised fine-tuning (distillation)
Engine	HuggingFace transformers + peft
LoRA	r=16, alpha=32, dropout=0.05
Epochs	3
Learning rate	2e-4, cosine, warmup 0.03
Batch	16 x 4 grad-accum = 64 effective
Max sequence	1024
Precision	bf16
Hardware	1x NVIDIA A100 80GB

Loss is on the completion only, with prompt lengths precomputed at load time rather than found by pattern search — the pattern-search collator silently masked nothing, which let the base model's tool-calling prior leak into the chains.

Usage

Solve this using Level 5 (Extreme).
Problem: {your problem}

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ssurface/cot-dialect-llama3.1-8b-sft-l5")
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

Limitations

• Trained and evaluated on math word problems only.
• Accuracy falls with problem difficulty, fastest at the compressed levels.
• Single seed unless the repo name says otherwise; differences of a couple of points are within noise (95% half-width ~2.7 pp at n=1317, ~4.4 pp at n=500).

Citation

@misc{cot-compression-dialects,
  title  = {Chain-of-Thought Compression Dialects},
  author = {Frolov, Anatolii},
  year   = {2026}
}