sidchalamala/aether-coder-1.5b-lora

Aether SOP-to-program LoRA v0.2 (Qwen2.5-Coder-1.5B-Instruct)

A LoRA adapter that turns a plain-English business procedure, plus the owner's policy and sheet list, into an Aether program. The model is not a safety mechanism. Every program it writes must pass the Aether checker before it runs. The checker blocks programs that break the policy's rules; the model often writes such programs.

Links: browser demo of the checker and interpreter (the model is not hosted there): https://warnedorca.github.io/aether/ · code: https://github.com/WarnedOrca/aether · benchmark and held-out set: https://huggingface.co/datasets/sidchalamala/aether-benchmark

"correct" below means passes the checker and has the same effects as the reference program (scorer field safe_and_correct). It is not a safety rating.

Final evaluation: frozen held-out trap set (30 tasks, each model evaluated once)

Each task asks for something the policy forbids or Aether cannot do. The correct answer is a program that does the permitted alternative. The set was frozen before any trap training data existed and was never used for training or decisions.

System	Correct	95% CI
Base model + reference card + 3 examples	0/30	0.00-0.11
v0.1 (synthetic data only)	6/30	0.10-0.37
v0.2 (this adapter: + 413 trap samples)	6/30	0.10-0.37

• Negative result: adding 413 targeted trap samples did not improve held-out performance. v0.2 vs v0.1: 6 vs 6, paired exact McNemar p = 1.
• v0.2 vs the base model: p = 0.031.
• Novel trap types absent from all training data (delete rows, raise a limit): 0/4 for every system.
• In every evaluation, the checker blocked every program that broke a rule, and programs it accepted had 0 monitor violations. The model by itself solves 6/30 held-out traps.

Held-out v2 (90 tasks, 30 traps, 15 new business types; each condition run once)

A second frozen set, written and hashed before any v0.3 training or tuning. Same scorer. 95% Wilson intervals.

System	Correct (of 90)	95% CI	Traps (of 30)
Base model + reference card + 3 examples	1	0.00-0.06	0
v0.2 plain (this adapter)	20	0.15-0.32	2
v0.2 with checker feedback	22	0.17-0.34	2
v0.2 grammar-constrained	19	0.14-0.31	2
v0.3 plain (not released)	14	0.10-0.24	3
v0.3 with checker feedback (not released)	15	0.10-0.26	3
v0.3 grammar-constrained (not released)	15	0.10-0.26	3

• Pre-registered primary comparison (null result): v0.3 with checker feedback vs this adapter (v0.2) plain: 15 vs 20, paired exact McNemar p = 0.33. The new data plus checker feedback did not improve on v0.2.
• v0.2 plain vs the base model: 20 vs 1 (paired p = 2.1e-05; this test was not pre-registered).
• Everything else in this table is exploratory. "With checker feedback" means up to 3 extra attempts in which the model sees the parser and checker errors; it is reported separately from plain output. "Grammar-constrained" means decoding restricted to tokens the Aether grammar allows.
• v0.3 is not released. It is a negative result: it adds 1,232 generated samples from 30 more business types, scores 100/100 on its synthetic test split (as v0.2 does), and does worse than v0.2 on new SOPs, which suggests the generated data teaches the generator's templates more than reading real SOPs.

Development benchmark (110 tasks, used during development)

This benchmark was used to build the scorer and compare models during development, so treat it as a development set, not a clean test.

System	Correct	Traps (30)
Base model + reference card + 3 examples	3/110	0
v0.1	30/110	2
v0.2	34/110	3

v0.2 vs v0.1: paired p = 0.48.

Training

The v0.1 recipe (LoRA rank 16, learning rate 0.0002, 2 epochs, seed 42; only the program tokens are trained) on 2062 verified examples: 1649 synthetic plus 413 trap-style. Training took 42 minutes on one NVIDIA GeForce RTX 4070, at $0. The checkpoint was chosen by validation loss.

Limitations

• Trained on templated synthetic English only. Most held-out failures are general errors on unfamiliar businesses (parse errors, type errors, reading columns the policy does not allow), not only trap-following.
• All benchmark and held-out tasks were drafted by Claude, an AI assistant, not taken from real businesses.
• Small evaluation sets with wide intervals; one training seed per model.
• Aether's external actions (messages, money, calendars) are mocked. This is research software.
• Do not describe this model, or Aether as a whole, as "safe". The narrow claim is that the checker verifies the named rules C1-C8 for Aether programs (see the specification).