bad-laya

An open-weight research checkpoint for typed decisions. Give it a state and a question with explicit options; its ModernBERT encoder and decision head score those options without generating text. This is the partial stage-16 checkpoint of an ordered curriculum, not the best checkpoint of the run.

Explore: Decision Index 0.3 results · Quickstart · Calibration

Research checkpoint. Results vary by task, and the 0.3 run is paused before completion. The table below reports every completed benchmark and the partial ACOS stage; no overall public-index score is available.

Model details

Developer	flydexo
Backbone	Fine-tuned answerdotai/ModernBERT-large at pinned revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13
Decision head	Two transformer layers with one learned score per supplied option marker
Input	A text or JSON state, plus named choice, score, or noul questions
Output	A probability for each option; selected choice, expected score, or probability of true
Context	Up to 1,024 tokens per question under the saved preprocessing configuration
Weights	895 MB Safetensors: BF16 encoder and FP32 decision head
Calibration	One post-training temperature, T = 12.60, stored in calibration.json
License	Not declared for this combined checkpoint; the ModernBERT base is Apache 2.0

The question’s options define the answer space at inference time. choice returns the most likely named option, score returns the probability-weighted level index, and noul returns the probability of true. The custom decisions code handles preprocessing and option-marker scoring. This repository is not a drop-in transformers.AutoModel.from_pretrained or text-generation model.

Decision Index 0.3 benchmarks

The verified public 0.3 suite was run one benchmark at a time on Apple MPS. The run paused after 12 of 43 benchmarks; ARC-Easy was completed in a separate run. These are partial public results, not a full index or leaderboard submission.

Benchmark / metric	bad-laya	Answered	Chance / baseline	Laya	Kev 0.8B	Kev 4B	Cloudflare clef
ToolRet · nDCG@10	13.3%	548/685	13.4%	12.7%	51.7%	61.0%	69.1%
API-Bank · accuracy	—	0/508	1.9%	11.4%	43.7%	53.3%	91.9%
Home appliance · case exact accuracy	—	0/88	0.0%	0.0%	0.0%	14.8%	80.7%
ContractNLI · macro-F1	1.7%	14/123	30.9%	28.7%	40.9%	64.4%	81.3%
GPQA Diamond · accuracy	26.0%	195/196	25.0%	27.6%	33.2%	37.8%	48.5%
ARC-Easy (shown only) · accuracy	53.7%	2376/2376	25.0%	47.0%	82.2%	97.2%	99.0%
WinoGrande · accuracy	50.9%	1267/1267	50.0%	50.5%	52.8%	70.2%	93.5%
MuSR · accuracy	14.5%	383/752	37.1%	43.2%	48.9%	56.2%	83.8%
BRIGHT · nDCG@10	4.2%	78/220	11.6%	19.9%	31.8%	39.0%	47.5%
ACOS (partial) · per-review F1	0.15%	384/1565	3.1%	3.5%	9.9%	9.8%	33.2%
FinEntity · macro-F1	52.0%	979/979	32.0%	61.0%	71.4%	87.1%	96.1%
CRUXEval · accuracy	38.8%	570/570	37.0%	40.2%	37.7%	48.1%	86.5%
HLE · accuracy	13.2%	471/501	16.4%	14.0%	13.8%	10.2%	12.8%
New Yorker captions · accuracy	26.1%	528/528	20.0%	27.1%	27.8%	52.3%	70.3%

Scores are the kit’s coverage-adjusted raw metric, so unsupported requests count against bad-laya. The “Chance / baseline” column is the board’s task-specific random/reference baseline; accuracy, F1 and nDCG are distinct metrics. A dash means the model answered no requests. ACOS is partial: its 0.15% interim figure covers only 384 of 1,565 requests and must not be treated as a full-benchmark comparison. ARC-Easy comes from a separate completed run and is shown on the board but does not count toward the public index. The other 12 completed rows are from the paused sequential run. Peer values come from the public Decision Index snapshot (generated 2026-10-07); each peer result has the same request count for the row. The full 0.3 public index is unavailable while 31 benchmarks remain. The board’s Full score also requires private tests.

The aggregate report gives exact scores and provenance. WinoGrande scored 50.9% in this sequential run; an earlier isolated run scored 50.7%.

What is in this repository

File	Purpose
model.safetensors	BF16 fine-tuned encoder and FP32 decision head, with no optimizer or RNG state
decision_config.json	Architecture, base-model revision, checkpoint step, and SHA-256 provenance
calibration.json	Fitted temperature and held-out calibration metrics
tokenizer/	Tokenizer files pinned to the ModernBERT-large base revision
assets/overview.png	The visual score summary above

The weights are provided in Safetensors so they can be inspected and loaded without unpickling the original training checkpoint.

Load and score a request

Install the project and its pinned dependencies, then run this example from the project directory:

git clone https://github.com/Flydexo/decisions.git
cd decisions
uv sync

import json
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

from decisions.evaluation import preprocessing
from decisions.model import inference_logits, load_model
from decisions.schema import prepare_request, to_device

repo = "flydexo/bad-laya"
config = json.load(open(hf_hub_download(repo, "decision_config.json")))
temperature = json.load(open(hf_hub_download(repo, "calibration.json")))["temperature"]
device = torch.device("cpu")
model, tokenizer = load_model(config["model"], config["ablation"], device)
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")), strict=True)
model.precision_settings["mixed_precision"] = "fp32"  # practical CPU inference
model.eval()

request = {
    "state": "The film was funny and moving.",
    "questions": {
        "sentiment": {
            "type": "choice",
            "instructions": "Choose the overall sentiment.",
            "criteria": {"negative": "negative review", "positive": "positive review"},
        }
    },
}
inputs = prepare_request(request, tokenizer, **preprocessing(config["model"], False))
with torch.no_grad():
    logits = inference_logits(model, to_device(inputs, device), question_microbatch_size=4)
probs = (logits.float() / temperature).softmax(-1)[0, : len(inputs["option_labels"][0])]
print(dict(zip(inputs["option_labels"][0], probs.tolist())))

The project tokenizer and base-model config are pinned to revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13. The convenience loader above downloads the base ModernBERT weights before replacing them with this checkpoint; plan disk space accordingly. The included tokenizer/ files also allow local tokenizer loading.

Temperature calibration

The published calibration.json holds one positive temperature fitted to raw logits from 832 validation questions by minimizing negative log-likelihood. Evaluation questions came from separate dataset partitions and were not used to choose the temperature. The validation sample had previously been used to select and inspect this checkpoint, so the separate evaluation result is the meaningful calibration check. Apply the temperature to every valid option logit before softmax. The selected answer and its accuracy do not change.

Separate evaluation partitions · 832 questions	Original	Temperature 12.60
Accuracy	65.4%	65.4%
Mean chosen-answer probability	91.2%	60.1%
Negative log-likelihood	6.08	1.13
Chosen-answer probability ECE · 10 bins	26.2%	15.3%

For a decision forecast, use the probability of the chosen option after temperature scaling; normalized entropy confidence describes distribution sharpness instead. A shared temperature can still be wrong for a particular source or a new domain. See the calibration report and results page for the separate evaluation metrics and reliability plot.