regolo/brick-sysone-V1

Brick SystemOne Large v2 — query to capability vector

Fine-tuned answerdotai/ModernBERT-large (8192-token context) that maps a raw query to a six-dimensional capability vector. Each dimension is an independent sigmoid output in [0, 1]; the vector is neither normalized to sum to one nor rounded.

This checkpoint is the local router model used by Brick.

Output dimensions

The output order is fixed:

1. instruction_following
2. coding
3. math_reasoning
4. world_knowledge
5. planning_agentic
6. creative_synthesis

Training

• Base: answerdotai/ModernBERT-large, 28 layers, hidden size 1024, 8192 context.
• Data: 68,882 query/label pairs. Labels are cross-model judge means with difficulty modulation; v2 adds 16k pair-transfer rows (short queries inherit detailed labels, including 3.1k ultra-short queries under 40 characters) and 3k garbage rows mapped to near-zero vectors.
• Configuration: LR 8.93e-5, 4 epochs, gradient accumulation 8, bf16, SDPA.

v2 differences from v1: the brevity gap on short prompts drops from 0.27 to 0.13, and the garbage mean-max drops from 0.36 to 0.006 with no spikes.

Evaluation

• Holdout (4,968 rows): pearson_macro 0.9656, mae 0.058.
• Mapping battery (30 P1, 6 P2, 40 garbage): P1 gap 0.13, P2 direction 6/6.

Quickstart

The output is a capability profile, not a distribution over models: each value is an independent sigmoid in [0, 1].

Install

python3 -m venv .venv-systemone
.venv-systemone/bin/python -m pip install --upgrade pip
.venv-systemone/bin/python -m pip install --index-url https://download.pytorch.org/whl/cu128 \
  torch==2.11.0
.venv-systemone/bin/python -m pip install transformers==5.20.2 safetensors==0.8.0 \
  tokenizers==0.23.3 numpy==1.26.4 huggingface-hub==2.2.0

Use https://download.pytorch.org/whl/cu126 for driver 560-569, or https://download.pytorch.org/whl/cpu for CPU-only. The script below picks BF16 on CUDA and FP32 on CPU, as recommended.

Terminal chat

Type a query, read the vector. Save as chat.py and run .venv-systemone/bin/python chat.py.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

LABELS = ["instruction_following", "coding", "math_reasoning",
          "world_knowledge", "planning_agentic", "creative_synthesis"]
REVISION = "9263a397a366ee22921214ddeebab1f5e4e8facd"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
model = AutoModelForSequenceClassification.from_pretrained(
    "regolo/brick-sysone-V1", revision=REVISION, dtype=dtype,
    attn_implementation="sdpa",
).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained("regolo/brick-sysone-V1", revision=REVISION)
limit = min(model.config.max_position_embeddings, 8192)

while True:
    try:
        query = input("query> ").strip()
    except (EOFError, KeyboardInterrupt):
        break
    if not query:
        continue
    encoded = tokenizer(query, return_tensors="pt", truncation=True, max_length=limit).to(device)
    with torch.inference_mode():
        vector = model(**encoded).logits[0].sigmoid().float().tolist()
    for label, value in zip(LABELS, vector):
        print(f"  {label:<20} {value:.4f}")
    print(f"  vector: [{', '.join(f'{v:.4f}' for v in vector)}]")

query> write a python parser that streams csv rows
  instruction_following 0.5117
  coding                0.9258
  math_reasoning        0.0903
  world_knowledge       0.1157
  planning_agentic      0.0630
  creative_synthesis    0.0198
  vector: [0.5117, 0.9258, 0.0903, 0.1157, 0.0630, 0.0198]

Input is truncated at 8192 tokens; longer inputs are outside the training distribution. The Brick router classifies only the last user message, never system or assistant turns. Full scripts with --json, --device and one-shot mode: examples/systemone/ in the Brick repository.

License

Apache-2.0, matching the base model. See LICENSE.