PabJev-4b

Base Model: qwen3.5 4b. This repository contains the complete backbone, FP32 pointer head, tokenizer/processor, bidirectional attention runtime, HTTP inference service, JevBench evaluation dataset, and evaluation harnesses.

This is a decision/pooling model, not a generative language model. It takes state + choice/noul/score questions as input and returns complete probability distributions over candidate options. It does not expose a /v1/chat/completions endpoint and should not be used within standard autoregressive text generation pipelines.

1. Model Specifications & Release Contract

Item	Specification
Base Model Identifier	qwen3.5 4b
Backbone	Merged, BF16, unquantized, 3 safetensors shards / 723 tensors
Pointer Head	FP32, 4 tensors, `head_dim=256`
Backbone Architecture	32 layers: 8 full-attention layers + 24 Gated DeltaNet linear-attention layers
Attention Semantics	Full-attention layers are bidirectional; linear-attention/GDN layers remain causal/recurrent
Readout Strategy	Hidden states extracted at each `</opt>` and final `<decide>`, routed through pointer head + softmax
Temperature	T=1.0; retained raw after calibration (fitted temperatures rejected)
Reference Hardware	1× NVIDIA A800-SXM4-80GB, `tensor_parallel_size=1`
Reference Runtime	Python 3.12, PyTorch 2.13.0+cu130, Transformers 5.19.0, vLLM 0.30.0

At startup, vllm_plugin.py verifies the 8+24 layer architecture and asserts that all 8 full-attention layers enter the ENCODER_ONLY bidirectional path. To prevent silent semantic drift, the service enforces: BF16 precision, no KV/weight quantization, chunked prefill disabled, prefix caching disabled, use_non_causal=true, and T=1.0. Alternative engines or modified runtime configurations will not guarantee parity with this README.

2. Reference Results (JevBench - 231 Items)

Tested on a single A800, concurrency 1, 231/231 valid probabilities, 0 invalid requests, 0 timeouts:

Metric	Score
Overall Accuracy	0.883117 (204/231)
choice Accuracy	0.892086 (124/139)
noul Accuracy	0.864865 (64/74)
score Accuracy (argmax)	0.888889 (16/18)
easy / standard / hard / judge Accuracy	1.000000 / 0.972222 / 0.808511 / 0.588235
ECE / Fidelity	0.071062 / 0.886139
Request-phase wall time / QPS	16.311 s / 14.1621
Discrepancy vs. Training Logs (Overall, 3-Type Acc, Intelligence)	Exact match (0 delta)

Across two full evaluations at concurrency 1, predicted labels and probability distributions match identically (probability_max_abs=0.0). At concurrency 4, reference throughput reaches 23.3873 QPS, though a single boundary case flips; strict benchmark reproduction requires concurrency 1. See docs/CALIBRATION.md, docs/VALIDATION.md, and reference_results/ for details.

3. Repository Layout

PabJev-4b/
├── backbone/                     # BF16 config + 3 merged weight shards + processor configuration
├── tokenizer/                    # tokenizer.json/config + chat template
├── pointer_head.safetensors      # FP32 q/k pointer head
├── model_manifest.json           # Step metadata, LoRA lineage, dtype, attention & temperature contract
├── service/                      # Startup, client, evaluation, selftest, and step-150 checkpoint_state.json
├── jevbench/                     # 231 evaluation items across easy/hard/original JSONL files
├── examples/                     # Request JSON and three custom JSONL examples
├── reference_results/            # Baseline validation and temperature calibration artifacts
├── docs/                         # Specifications for data format, calibration, and validation
├── wheels/                       # Optional pre-compiled kernel wheels matching ABI + licenses
├── requirements.txt / setup_env.sh
└── SHA256SUMS                    # Cryptographic release checksums
## 4. Download from Hugging Face

```bash
export HF_REPO_ID="roudone/PabJev-4b"
python3.12 -m pip install "huggingface_hub==2.2.0"
hf download "$HF_REPO_ID" --local-dir PabJev-4b
cd PabJev-4b

5. Environment Setup & Verification

Recommended Environment

Linux x86_64; Python/CPython 3.12
NVIDIA Driver compatible with CUDA 13.0; target GPU: A800 80GB
At least 10GB disk space for model weights, plus additional space for dependencies
Installation requires internet access. The bundled wheels target CPython 3.12 + Linux x86_64 + PyTorch 2.13 + CUDA 13 ABI.

Automated setup (creates .venv, installs locked dependencies, checks GPU/weights, and executes CPU selftests):

bash setup_env.sh
source .venv/bin/activate

Equivalent manual setup:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install --index-url https://download.pytorch.org/whl/cu130 \
  torch==2.13.0 torchaudio==2.11.0 torchvision==0.28.0
python -m pip install -r requirements.txt
python -m pip install --no-deps --force-reinstall wheels/*.whl

python service/verify_install.py --require-gpu
python service/verify_bundle.py --full
python service/selftest.py

verify_bundle.py --full inspects the 3 shards, index/header, 723 backbone tensors, 4 FP32 pointer tensors, tokenizer/processor configs, step-150 metadata, 8+24 bidirectional attention assertions, and core SHA256 checksums.

To build kernels from source for other ABIs:

USE_BUNDLED_WHEELS=0 bash setup_env.sh

Note: Requires compatible nvcc/build tools. Numerical behavior and latency across different CUDA versions, GPUs, custom kernels, or dependency variations may deviate from the reference baseline. Do not apply quantization or lower precision when aiming to reproduce reference metrics.

6. Serving & Deployment

Activate the environment from the repository root:

GPU_ID=0 bash service/start.sh

Default endpoints:

Inference API: http://127.0.0.1:18100/v1/systemone (alias: /predict)
Health Check: http://127.0.0.1:18100/health
Internal vLLM Pooling: 127.0.0.1:30000 (loopback only, do not expose externally)

The startup script verifies weights, starts vLLM and the SystemOne API, runs a full-distribution smoke test, and warms up short-context, long-context, score, and multi-question paths. Common runtime options:

GPU_ID=0 \
JEV_PORT=18100 VLLM_PORT=30000 \
GPU_UTIL=0.92 MAX_NUM_SEQS=64 MAX_BATCHED_TOKENS=32768 \
bash service/start.sh

GPU_ID must specify a single card; reference numbers are obtained on an idle 80GB A800.
JEV_HEAD_DEVICE defaults to cuda; the pointer head runs in FP32.
Inspect logs/vllm.log, logs/server.log, run/startup.json, and run/smoke.json for troubleshooting.
The startup script checks ports before binding; it will not overwrite or kill external processes.

To stop the service (pass custom ports if overridden):

bash service/stop.sh
# or
JEV_PORT=18102 VLLM_PORT=30002 bash service/stop.sh

To expose externally:

JEV_HOST=0.0.0.0 GPU_ID=0 bash service/start.sh

7. Inference & API Usage

Via cURL

curl -sS -X POST http://127.0.0.1:18100/v1/systemone \
  -H 'Content-Type: application/json' \
  -d @examples/choice.json

Multi-question query covering all three supported types:

curl -sS -X POST http://127.0.0.1:18100/v1/systemone \
  -H 'Content-Type: application/json' \
  -d @examples/all_types.json

Via Python

python service/request.py examples/choice.json
python service/request.py examples/all_types.json \
  --endpoint http://127.0.0.1:18100/v1/systemone

Example response (probabilities are illustrative; JSON schema is fixed):

{
  "model": "PabJev-4b",
  "answers": {
    "move": {
      "type": "choice",
      "choice": "left",
      "probabilities": {
        "left": 0.9999983,
        "forward": 0.0000017
      }
    }
  },
  "usage": {
    "expanded_input_tokens": 32,
    "transferred_hidden_rows": 3,
    "full_hidden_rows_avoided": 29
  }
}

Request Schema & Constraints

{
  "model": "PabJev-4b",
  "state": "Text or stringified JSON payload",
  "questions": {
    "question_id": {
      "type": "choice | noul | score",
      "instructions": "Instruction prompt",
      "criteria": "Per-type criteria specification"
    }
  }
}

Type	Criteria Format	Key Response Fields
`choice`	Object mapping candidate names to descriptions: `{candidate: description}`	Selected `choice` and full `probabilities` distribution
`noul`	Optional, or limited to `false`/`true` keys	Float scalar `noul` ($P(\text{true})$), plus `false`/`true` probabilities
`score`	Ordered array of criteria labels (minimum 2 buckets)	Expected bucket index `score` (0-indexed), `legend`, and per-bucket probabilities

Batching per request: 1 to 16 questions. Multi-question requests expand state + question branches and aggregate them into a single response.
Token Limits: state ≤ 8,192 tokens; full single branch ≤ 12,288 tokens; expanded total tokens per batch ≤ 32,768.
Multimodal (Vision): Supports single PNG/JPEG input per request, limited to a single question. Image limits: 8 MiB, 16 MP, 1,024 visual tokens. Use base64 data URIs for remote requests; local file paths only resolve locally on the server.
Unsupported Features: Free-form generation, think_mode, and autoregressive completion parameters are rejected with HTTP 422.

8. JevBench Evaluation

Once the service is active, run the benchmark from the repository root:

python service/evaluate.py \
  --concurrency 1 \
  --out outputs/jevbench.json

By default, this evaluates the included 231 items across jevbench/{easy,hard,original}.jsonl, applying the training-consistent scoring IDs and 113/118 pseudo-sealed split mapping.

Evaluate an external JevBench directory:

python service/evaluate.py \
  --dataset-dir /path/to/jev-bench/dataset \
  --tiers easy,hard,original \
  --concurrency 1 \
  --out outputs/jevbench.external.json

Throughput testing:

python service/evaluate.py \
  --concurrency 4 \
  --out outputs/jevbench.c4.json

Evaluation outputs aggregate accuracy across tier/family/type, Intelligence, Calibration, ECE, Fidelity, reliability diagram bins, latency/QPS, and per-item outputs saved alongside *.rows.jsonl. New runs overwrite --out atomically.

Reproducibility comparison check:

python service/evaluate.py --concurrency 1 --out outputs/jevbench.repeat.json
python service/compare_results.py \
  outputs/jevbench.repeat.json outputs/jevbench.json \
  --out outputs/jevbench.compare.json

9. Evaluating Custom Datasets

For custom datasets (e.g., my_data/test.jsonl), set --tiers to match the file basename (test):

python service/evaluate.py \
  --dataset-dir /path/to/my_data \
  --tiers test \
  --no-pseudo-sealed \
  --concurrency 1 \
  --out outputs/test.json

Evaluate the bundled custom examples:

python service/evaluate.py \
  --dataset-dir examples --tiers custom \
  --no-pseudo-sealed \
  --out outputs/custom.json

Multiple tiers can be passed as comma-separated values: --tiers dev,test. Required row schema includes id, state, question, and expected (choice candidate key, boolean string yes/no or true/false, or integer bucket index). For complete specifications, see docs/DATA_FORMAT.md.

Do not use --no-pseudo-sealed when replicating original JevBench splits. Always include --no-pseudo-sealed for generic custom datasets.

10. Temperature Calibration

The deployed serving temperature is fixed at $T = 1.0$.

Post-training calibration sweeps fitted candidate values of global $T=1.2778$, choice $T=1.4339$, noul $T=1.2720$, and score $T=0.9909$. However, these candidates were rejected: while they reduced validation NLL, they degraded Brier score and overall Calibration metrics, failing the strict "no-regression across all three metrics" deployment rule.

Calibration Candidate (Domain-Macro)	Accuracy	NLL ↓	Brier ↓	ECE ↓	Calibration ↑
Raw $T=1.0$ (Deployed)	0.928301	0.225644	0.105183	0.023519	95.4508
Global Scaling	0.928301	0.217776	0.106195	0.026693	95.0431
Per-Type Scaling	0.928301	0.215359	0.106267	0.027746	94.9723

On the held-out test split, raw test scores were: Accuracy=0.928121, NLL=0.187029, Brier=0.090812, ECE=0.017322, Calibration=96.1804. (Note: The calibration split is distinct from the 231-item JevBench evaluation set). Complete calibration analysis and reliability diagrams are documented in docs/CALIBRATION.md.

11. Licenses & Disclaimers

Model Weights & Derivative Weights: CC BY-NC 4.0 (Non-Commercial Use Only). Refer to LICENSE for the complete license text. Upstream attributions and commercial licensing instructions are preserved in NOTICE. Commercial deployment requires separate licensing.
Service Stack, Evaluation Harnesses, & Vendored Runtime: Apache-2.0, provided under licenses/Apache-2.0 and third_party/.
Bundled JevBench Dataset: Line-by-line provenance provided under MIT, detailed in jevbench/LICENSE and NOTICE.md.
Precompiled Wheels: Respective licenses and source origins are documented under wheels/licenses/.

These benchmarks reflect performance exclusively on JevBench under the documented software/hardware stack and step-150 checkpoint; they do not imply uniform dominance across arbitrary domains. Floating-point variations in BF16 may occur across different GPU microarchitectures, kernel implementations, or dynamic batch compositions. Users must perform independent validation regarding safety, access controls, data governance, and license compliance before hosting public-facing endpoints. ```

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support