Ysolveth: A System One Model for Software Engineering AI (Candidate 1.3.0)

Developed for StructZero — AI Software Engineering Platform.

Ysolveth is an ultra-fast, local System One Model engineered for software engineering prompt triage, complexity classification, and intelligent workflow routing. Operating non-autoregressively with sub-millisecond execution, it classifies prompts into three operational complexity tiers: trivial, moderate, and hard.

---

What is a "System One Model"?

Drawing from cognitive science (Daniel Kahneman’s Thinking, Fast and Slow), modern AI architectures are separating into two distinct computational layers:

• System One (Fast, Reflexive, Parallel, Non-Generative): Instant evaluation and routing. Operates in single-digit milliseconds without token generation loops, outputting calibrated, type-safe decision probabilities.
• System Two (Slow, Deliberative, Autoregressive): Heavyweight generative models (LLMs like Claude 3.5 Sonnet, GPT-4o, or DeepSeek R1) that perform multi-turn reasoning, chain-of-thought code generation, and iterative problem solving.

How Ysolveth Differs from Standard LLM Prompting

Traditional systems often prompt an LLM to rate difficulty (e.g., "Rate this task 1-10 in JSON"). This introduces:

1. Severe Latency Penalties: Autoregressive token-by-token generation takes 800ms to 2,500ms.
2. Schema & Hallucination Vulnerability: LLMs can produce malformed JSON or out-of-bounds labels.
3. High Inference Cost: Burning frontier tokens merely to make a routing decision.

Ysolveth is a true System One model:

• Zero Token Generation: Evaluates prompt state in a single parallel tensor pass.
• Impossible to Hallucinate: Maps directly to a fixed 3-class probability distribution ($\mathbb{R}^{384} \to \Delta^2$).
• Deterministic & Calibrated: Native logits are temperature-calibrated so software systems can branch safely without parsing textual explanations.
• Ultra-Low Latency: Runs in ~2 to 8ms on standard CPU with zero GPU requirements ($0.00 marginal cost).

---

Architecture & Deployment

Ysolveth is packaged as a compact, self-contained edge artifact (~23 MB total):

1. Embedding Backbone: Frozen sentence-transformers/all-MiniLM-L6-v2 encoder quantized to int8 ONNX (encoder_int8.onnx, ~22.8 MB).
2. Classification Head: Lightweight calibrated linear layer (head_config.json, 384 input dimensions → 3 output classes with temperature scaling & softmax).
3. Air-Gapped & Local: Can be loaded directly into in-process microservices (Rust/Axum, Node.js, Python) with zero external network access.

---

Use Cases

1. Intelligent Model & Workflow Routing

Directing all user prompts to heavyweight frontier models incurs high costs and latency. Ysolveth acts as the gatekeeper:

• trivial: Route to lightweight models or deterministic templates (e.g., regex fixes, typo corrections, single-file edits).
• moderate: Route to standard reasoning coding models.
• hard: Route into multi-model consensus, multi-turn architecture debate, and strict sandbox verification harnesses where multi-component verification is required.

2. Guardrails Against Single-Model Blindspots

Multi-tenant database schema migrations, Stripe webhook reconciliation, and OAuth/SAML integrations require multiple validation steps. When flagged by Ysolveth as hard, the orchestrator ensures that code changes are guarded by multi-agent consensus rather than letting a single model blindly apply changes.

3. Cost & Latency Optimization

By avoiding unnecessary calls to frontier models for trivial and moderate prompts, AI coding platforms can dramatically lower monthly inference spend while maintaining instant sub-10ms routing decisions.

---

Training Data

Ysolveth was trained and evaluated using a multi-model consensus distillation pipeline to establish high-confidence difficulty labels across diverse real-world software engineering requests:

• Dataset Scale: 300 prompts total spanning web development, full-stack systems, single-service utilities, distributed backend architectures, and database migrations.
• Consensus Labeling: Labels were established via 4-vote majority consensus across frontier foundation models (claude-sonnet-4-6, gemini-3.8-flash, luna-6, and claude-haiku-5-5).
• Tie Resolution: 4 ties excluded (unresolved 2-2 ties, 1.33%), leaving 296 usable prompts.
• Dataset Distribution (296 Usable Prompts):
  • trivial: 76 prompts (25.7%)
  • moderate: 145 prompts (49.0%)
  • hard: 75 prompts (25.3%)
  • Overall split: trivial 76, moderate 145, hard 75.
• Data Partitioning & Held-Out Test Evaluation:
  • Training Set: 236 prompts (stratified: 60 trivial, 116 moderate, 60 hard) with a frozen sentence-transformers/all-MiniLM-L6-v2 encoder.
  • Held-Out Test Set: 60 prompts (16 trivial, 29 moderate, 15 hard) strictly isolated with zero overlap with the training pool.
  • Held-Out Accuracy: 100% (60 of 60) correct against the four-model consensus labels.
• Evaluation Methodology Note: These labels come from the four-model consensus, and the 40-prompt hand-labelled set detailed in the Benchmark section below is the stricter test against model consensus blindspots.

---

Benchmark

Ysolveth Candidate 1.3.0 was evaluated on an independent, strictly held-out test set of 40 hand-labelled prompts (10 trivial, 15 moderate, 15 hard) with zero overlap with the training pool.

Primary Metric: Critical Failures

A critical failure is defined as classifying a hard request as trivial or moderate, which routes complex architectures into unguided single-model lanes without verification.

Model	Critical Failures (Hard → Trivial/Mod)	Critical Failure Rate	Overall Test Accuracy (40 Prompts)
Live Mini Model (Baseline)	14 / 15	93.3%	65.0% (26/40)
Ysolveth (Candidate 1.3.0)	0 / 15	0.0%	100.0% (40/40)

• Baseline Live Mini Model: Erroneously classified 14 of 15 hard prompts as moderate.
• Ysolveth Candidate 1.3.0: Correctly classified all 15 hard prompts as hard (0 of 15 critical failures, 100.0% accuracy across all 40 prompts).

Classification Report (Ysolveth Candidate 1.3.0)

              precision    recall  f1-score   support

     trivial     1.0000    1.0000    1.0000        10
    moderate     1.0000    1.0000    1.0000        15
        hard     1.0000    1.0000    1.0000        15

    accuracy                         1.0000        40
   macro avg     1.0000    1.0000    1.0000        40
weighted avg     1.0000    1.0000    1.0000        40

Confusion Matrix Comparison

Baseline Mini Model Confusion Matrix:

             trivial  moderate  hard
trivial         10         0     0
moderate         0        15     0
hard             0        14     1   <-- 14 critical failures

Ysolveth Candidate 1.3.0 Confusion Matrix:

             trivial  moderate  hard
trivial         10         0     0
moderate         0        15     0
hard             0         0    15   <-- 0 critical failures

---

Limits and Caveats

1. Test Set Size: The held-out benchmark test set is small (40 prompts).
2. Label Methodology: Ground-truth labels are derived from owner judgment combined with a 4-model majority consensus vote.
3. Statistical Bounds: Results reflect a single benchmark run; no bootstrap confidence intervals are computed.
4. Generalization: This is an internal domain-specific routing benchmark for software engineering prompts, not a general natural language processing benchmark.
5. Confidentiality: Raw training prompts, hard-case examples, and customer data are withheld to preserve internal confidentiality and security.

---

Candidate Checksums

Every file in the repository matches its pinned cryptographic SHA-256 hash:

File	SHA-256 Checksum
encoder_int8.onnx	9d6913869f2523a41c8e24252e69061ed31b69a8d04a7ec6359d2d1ff8e308f6
head_config.json	d5c13b57ed81a167897030fe04093da4b7d21d5a6a6c3591c7ade412caaee551
tokenizer.json	e619bb7888bee125bfd361ad7eb9f8e3de9b5ece0ed0e5949db658630b9c60da
tokenizer_config.json	872b6936be955bc3aea75ed599264d865626d68feede7e58b01e378e6332bd74
vocab.txt	07eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3
special_tokens_map.json	303df45a03609e4ead04bc3dc1536d0ab19b5358db685b6f3da123d05ec200e3

---

How to Run Inference

Python (ONNX Runtime + Tokenizer)

import json
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

# 1. Load tokenizer and ONNX encoder
tokenizer = AutoTokenizer.from_pretrained("vishalvr28/Ysolveth")
session = ort.InferenceSession("encoder_int8.onnx")

# 2. Load head weights and configuration
with open("head_config.json", "r") as f:
    head_config = json.load(f)

weights = np.array(head_config["weights"])      # shape: (3, 384)
bias = np.array(head_config["bias"])            # shape: (3,)
classes = head_config.get("classes", ["trivial", "moderate", "hard"])
temperature = head_config.get("temperature", 1.0)

# 3. Preprocess prompt
prompt = "Build a multi-tenant SaaS with user login, Stripe webhooks, and PostgreSQL migrations."
inputs = tokenizer(prompt, return_tensors="np", padding=True, truncation=True, max_length=256)

# 4. Generate MiniLM embeddings (Mean Pooling)
onnx_inputs = {
    "input_ids": inputs["input_ids"].astype(np.int64),
    "attention_mask": inputs["attention_mask"].astype(np.int64),
    "token_type_ids": inputs.get("token_type_ids", np.zeros_like(inputs["input_ids"])).astype(np.int64)
}
outputs = session.run(None, onnx_inputs)
token_embeddings = outputs[0]  # shape: (batch_size, seq_len, 384)
attention_mask = inputs["attention_mask"][..., np.newaxis]
pooled_embedding = np.sum(token_embeddings * attention_mask, axis=1) / np.maximum(attention_mask.sum(axis=1), 1e-9)

# Normalize embedding
embedding = pooled_embedding / np.linalg.norm(pooled_embedding, axis=1, keepdims=True)

# 5. Compute Linear Head Logits & Probabilities
logits = (np.dot(embedding, weights.T) + bias) / temperature
exp_logits = np.exp(logits - np.max(logits, axis=-1, keepdims=True))
probs = exp_logits / np.sum(exp_logits, axis=-1, keepdims=True)

predicted_class = classes[np.argmax(probs)]
confidence = np.max(probs)
print(f"Prediction: {predicted_class} (Confidence: {confidence:.2%})")

---

How to Train & Calibrate on Your Own Data

You do not need to fine-tune the heavy MiniLM transformer weights. Fine-tuning only the lightweight linear classification head preserves rapid training (seconds) and eliminates overfitting.

Step-by-Step Training Guide

1. Prepare Your Dataset

Prepare a JSON or CSV file of software engineering tasks with assigned difficulty levels:

[
  {"prompt": "Fix typo in README footer", "label": "trivial"},
  {"prompt": "Add JWT expiration check middleware", "label": "moderate"},
  {"prompt": "Implement distributed Saga pattern across order and inventory microservices", "label": "hard"}
]

2. Extract Embeddings with Frozen MiniLM

Use sentence-transformers/all-MiniLM-L6-v2 to precompute the 384-dimensional normalized vector for each prompt once:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
X = model.encode([item['prompt'] for item in dataset], normalize_embeddings=True)
y = np.array([label_to_id[item['label']] for item in dataset])

3. Train a Calibrated Linear Classifier

Fit a Logistic Regression model or train a small Linear Head using Cross-Entropy loss:

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(
    C=1.0,
    class_weight='balanced',  # Crucial to protect underrepresented hard cases
    max_iter=1000,
    multi_class='multinomial'
)
clf.fit(X, y)

# Extract weights and biases
weights = clf.coef_.tolist()       # List of 3 lists, each 384 floats
bias = clf.intercept_.tolist()      # List of 3 floats

4. Calibrate with Temperature Scaling & Thresholding

To enforce a zero-critical-failure safety policy, apply an epsilon threshold or temperature scaling:

• If P(hard) >= 0.35, bias classification towards hard to ensure complex multi-step tasks are never mistakenly labeled trivial or moderate.

5. Save to head_config.json

Export the configuration into the standard StructZero runtime schema:

{
  "version": "1.3.0",
  "classes": ["trivial", "moderate", "hard"],
  "input_dim": 384,
  "weights": [...],
  "bias": [...],
  "temperature": 1.0,
  "thresholds": {
    "hard_min_probability": 0.35
  }
}

---

Credits and Licence

• Encoder Architecture: sentence-transformers/all-MiniLM-L6-v2 by Sentence-Transformers, used under the Apache-2.0 Licence.
• Classification Head & Weights: Developed for the StructZero platform and licensed under the Apache-2.0 Licence.
• Repository Licence: This entire repository and its associated model artifacts are released under the Apache-2.0 Licence.
• Platform: StructZero — AI Software Engineering Platform