VEJI-V2

VEJI-V2 is a compact, non-autoregressive typed decision architecture designed to compile a large reusable state once and answer many structured questions against it. It combines a frozen multilingual semantic encoder with learned evidence routing, recurrent reasoning, exact/code-aware features, joint option scoring, confidence estimation, and optional abstention.

Reported test accuracy: 82.95% on 522 evaluated questions
Evidence chunk recall: 100%
Maximum context: 250,000 characters
Trainable VEJI parameters: 3,282,499 (excluding the frozen encoder)
Warm decision throughput: approximately 87–89 questions/s in the supplied benchmark

What VEJI-V2 is

VEJI-V2 is not an autoregressive text generator. It receives:

1. a state/context,
2. a typed question/instruction, and
3. a list of candidate options,

then scores the options jointly and returns the most likely answer, probabilities, confidence status, and the spans of the routed evidence chunks.

The architecture is intended for structured decision workloads such as routing, classification, extraction, policy checks, code reasoning, workflow decisions, semantic matching, and multi-hop relation questions.

Architecture

                         ┌─────────────────────────────┐
State / context ───────► │ Chunking (768 chars, 96 ov.)│
                         └──────────────┬──────────────┘
                                        │
                                        ▼
                         ┌─────────────────────────────┐
                         │ Frozen multilingual encoder │
                         │ MiniLM-L12-v2, 384 dims     │
                         └──────────────┬──────────────┘
                                        │
                              projected chunk states
                                        │
Question ─► frozen encoder ─► query projection + type embedding
                                        │
                              top-k evidence routing
                              (k = 8 by default)
                                        │
                                        ▼
                         weighted routed memory
                                        │
                     query/memory fusion + GRU refinement
                                        │
Options ───► frozen encoder ─► option projection
                                        │
                       semantic option compatibility
                                        │
Routed text + question + option ─► exact/code features
                                        │
                                        ▼
                         semantic score + exact score
                                        │
                          learned temperature scaling
                                        │
                                        ▼
                    probabilities + confidence + abstention

Main components

• Frozen semantic compiler: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2.
• Hard autograd boundary: encoder output crosses a NumPy copy boundary before entering trainable VEJI layers.
• Reusable compiled state: long contexts are encoded once and reused across many questions.
• Evidence router: query-to-chunk routing selects the top-k chunks before decision scoring.
• Typed decisions: semantic type embeddings condition the decision process.
• Reasoning/refinement: fused query and routed memory are refined with a GRUCell for configurable hops.
• Exact/code features: exact substring, token overlap, code-token overlap, numeric matching, negation agreement, length ratio, and option prior augment semantic scores.
• Confidence behavior: low-confidence answers can be returned as uncertain instead of forcing a decision.

Default configuration

Setting	Value
Frozen encoder	paraphrase-multilingual-MiniLM-L12-v2
Encoder dimension	384
Hidden dimension	384
Trainable VEJI parameters	3,282,499
Chunk size	768 characters
Chunk overlap	96 characters
Maximum context	250,000 characters
Routed chunks	8
Reasoning hops	2
Exact feature dimension	8
Dropout	0.08
Confidence threshold	0.55

Dataset and reference training run

The saved notebook uses VEJI-Synth-v2-200K with the following indexed splits:

Split	Records
Train	190,000
Validation	5,000
Test	5,000

Dataset manifest values saved in the run:

• Core records: 200,000
• Coding records: 70,822
• Coding share: 35.411%

The reference run in the supplied notebook sampled 30,000 training records for 1 epoch, using AdamW with learning rate 3e-4, weight decay 0.01, gradient accumulation 8, and seed 92. The saved run finished with a rolling training loss of 0.4701 and approximately 14.58 records/s near the end of training. The saved runtime reports CPU execution.

Evaluation results

The reported evaluation below covers 522 questions on validation and 522 questions on test. These are the results supplied with the model run; they are not a full 5,000-record held-out evaluation.

Metric	Validation	Test
Accuracy	84.48%	82.95%
NLL	0.3512	0.3650
Brier score	0.0582	0.0623
ECE (10 bins)	0.0651	0.0663
Evidence chunk recall	100%	100%

Family-level accuracy

Expand the complete validation/test family breakdown

Family	Validation	Test	Test - Val
access_control	90.00%	90.00%	+0.00 pp
agent_next_action	100.00%	100.00%	+0.00 pp
arabic_intent	100.00%	100.00%	+0.00 pp
arabic_policy	90.00%	90.00%	+0.00 pp
catalog_matching	100.00%	100.00%	+0.00 pp
code_api_routing	100.00%	100.00%	+0.00 pp
code_complexity	100.00%	100.00%	+0.00 pp
code_config_validation	100.00%	100.00%	+0.00 pp
code_dependency_choice	100.00%	100.00%	+0.00 pp
code_diff_review	100.00%	100.00%	+0.00 pp
code_doc_intent	100.00%	100.00%	+0.00 pp
code_exception_type	100.00%	100.00%	+0.00 pp
code_javascript_bug	100.00%	100.00%	+0.00 pp
code_javascript_output	50.00%	80.00%	+30.00 pp
code_python_bug	100.00%	100.00%	+0.00 pp
code_python_output	20.00%	50.00%	+30.00 pp
code_refactor_choice	100.00%	100.00%	+0.00 pp
code_security_pattern	100.00%	100.00%	+0.00 pp
code_sql_choice	100.00%	100.00%	+0.00 pp
code_static_reasoning	100.00%	100.00%	+0.00 pp
code_test_selection	100.00%	100.00%	+0.00 pp
code_type_reasoning	100.00%	100.00%	+0.00 pp
contradiction	100.00%	72.73%	-27.27 pp
customer_routing	100.00%	100.00%	+0.00 pp
data_quality	70.00%	50.00%	-20.00 pp
document_classification	100.00%	100.00%	+0.00 pp
duplicate_detection	100.00%	100.00%	+0.00 pp
email_routing	100.00%	100.00%	+0.00 pp
entity_resolution	100.00%	100.00%	+0.00 pp
fact_extraction	63.64%	63.64%	+0.00 pp
forecast_bucket	50.00%	70.00%	+20.00 pp
insufficient_information	100.00%	100.00%	+0.00 pp
log_anomaly	63.64%	45.45%	-18.18 pp
mixed_language_routing	100.00%	100.00%	+0.00 pp
multi_hop_relation	90.91%	100.00%	+9.09 pp
multilingual_semantic_match	100.00%	100.00%	+0.00 pp
numeric_policy	68.18%	45.45%	-22.73 pp
ops_incident	50.00%	50.00%	+0.00 pp
policy_compliance	100.00%	100.00%	+0.00 pp
priority_triage	63.64%	63.64%	+0.00 pp
ranking	18.18%	27.27%	+9.09 pp
risk_scoring	70.00%	70.00%	+0.00 pp
scheduling	45.45%	18.18%	-27.27 pp
sentiment_tone	100.00%	100.00%	+0.00 pp
structured_json	100.00%	100.00%	+0.00 pp
temporal_order	31.82%	18.18%	-13.64 pp
tool_selection	100.00%	100.00%	+0.00 pp
workflow_state	100.00%	100.00%	+0.00 pp

Result profile

The model is strongest on many routing, classification, semantic matching, policy, and code-analysis families, with numerous reported families at 100% test accuracy. The main weak areas in this evaluation are ranking (27.27%), scheduling (18.18%), temporal order (18.18%), numeric policy (45.45%), log anomaly (45.45%), and exact code-output prediction compared with the code-understanding families.

These family scores come from a small evaluation slice, so a family percentage should not be treated as a stable population estimate without a larger held-out evaluation.

250K context and speed benchmark

The benchmark compiles a 250,000-character state into 372 chunks, then reuses the compiled state for repeated questions.

Stage	p50 latency	Throughput
State compilation	42.402 s	one-time per state
1 warm question	11.436 ms	87.44 q/s
8 warm questions	90.124 ms total	88.77 q/s
32 warm questions	367.345 ms total	87.11 q/s

Important: the ~87–89 q/s figure is warm decision throughput after compilation. It does not include the one-time ~42.4 s state-compilation cost. Hardware was not attached to the supplied benchmark result, so these latency figures should not be generalized across machines.

The notebook's separate full long-context held-out evaluation did not complete in the saved run; that cell ended with KeyboardInterrupt. The 250K architecture smoke test and the compile/reuse speed benchmark did complete.

Example

The standalone notebook exposes VEJIV2, save_model, and load_model. After loading the VEJI implementation, a decision call looks like this:

from huggingface_hub import snapshot_download
import sys

repo_path = snapshot_download(
    repo_id="loaiabdalslam/VEJI-V2"
)

print("Downloaded to:", repo_path)

sys.path.insert(0, repo_path)

from veji import VEJIV2



#from veji import VEJIV2

model = VEJIV2.from_pretrained(
    "loaiabdalslam/VEJI-V2"
)

state = """
Customer Ahmed requested a refund.
Policy: refund requests go to billing.
The purchase was 3 days ago.
"""

questions = [
    {
        "id": "route",
        "type": "choice",
        "semantic_type": "route",
        "instruction": "Which team should handle this?",
        "options": [
            "sales",
            "billing",
            "support",
            "legal"
        ]
    }
]

result = model.decide(
    state,
    questions
)

print(result)

state = (
    "Customer Ahmed requested a refund. "
    "Policy: refund requests go to billing. "
    "The purchase was 3 days ago."
)

questions = [
    {
        "id": "route",
        "type": "choice",
        "semantic_type": "route",
        "instruction": "Which team should handle this?",
        "options": ["sales", "billing", "support", "legal"],
    }
]

results = model.decide(state, questions, abstain_threshold=0.60)
print(results)

For repeated questions against the same state, compile once:

compiled = model.compile_state(state)
results = model.decide_compiled(compiled, questions)

This is the intended performance path for VEJI-V2.

Output format

Each decision includes fields such as:

{
  "id": "route",
  "answer": "billing",
  "best_candidate": "billing",
  "answer_index": 1,
  "probabilities": [0.01, 0.98, 0.005, 0.005],
  "confidence": 0.98,
  "status": "confident",
  "selected_chunk_spans": [[0, 102]]
}

If the maximum option probability is below the threshold, answer becomes null while best_candidate is retained and status is uncertain.

Exported files

The notebook exports:

VEJI_v2_export/
├── config.json
├── metrics.json
├── veji_head.pt
└── README.md

veji_head.pt contains the VEJI trainable state dict. The multilingual sentence encoder remains an external frozen dependency identified by config.json.

Production status

The notebook defines the following engineering gate:

Gate	Target	Reported test	Pass?
Accuracy	>= 85%	82.95%	No
NLL	<= 0.36	0.3650	Yes
Brier	<= 0.08	0.0623	Yes
ECE-10	<= 0.03	0.0663	No
Evidence recall	>= 95%	100%	Yes
Minimum family accuracy	>= 80%	18.18%	No

According to that gate, this checkpoint is not yet a production candidate. The result is promising for a compact decision head, but production claims should wait for a full held-out evaluation, out-of-distribution testing, calibration improvement, and targeted work on the weak families.

Known limitations

• It is a candidate-selection / typed-decision model, not a free-form generative LLM.
• Quality depends on the candidate options supplied to the model.
• The reported 522-question validation/test subsets are relatively small.
• Several temporal, scheduling, ranking, numeric-policy, and code-output families remain weak.
• A 250K context can be reused efficiently once compiled, but the initial semantic compilation is comparatively expensive in the reported CPU run.
• The frozen encoder is an external dependency and dominates total dependency size even though the trainable VEJI head is only ~3.28M parameters.
• The original saved notebook did not specify a model license; this Hub repository now declares the MIT license.

Reproducibility notes

• Seed: 92
• PyTorch model with a frozen SentenceTransformer encoder
• Deterministic hash encoder exists only as an offline fallback/testing backend
• Training includes decision NLL, Brier loss, evidence-routing loss, and a margin term
• Evaluation reports accuracy, NLL, Brier score, 10-bin ECE, evidence chunk recall, and per-family accuracy

Citation

If you publish a paper or repository for VEJI-V2, replace the placeholder below with the final bibliographic entry:

@software{veji_v2_2026,
  title  = {VEJI-V2},
  year   = {2026},
  url    = {https://huggingface.co/loaiabdalslam/VEJI-V2}
}

License

MIT