LFM2.5-350M-RLCD

Parallel structured inference with unchanged LiquidAI/LFM2.5-350M weights. Prefill the context once, reuse attention and convolution state across candidate branches, score allowed values in a batch, and assemble JSON in Python.

Inference only: no training or fine-tuning, and no reproduction of TypeSafe.ai's proprietary Jev training method. This repository includes code, benchmark results, and a byte-for-byte copy of the original model weights and tokenizer. Engine below continues to load the pinned original model for reproducibility.

Performance: 28 fields

One synthetic input with 28 boolean fields, two warmups per method, and three measured repetitions. Both methods use FP16 and the same prompt. Times are mean end-to-end request latency; AR means autoregressive generation.

Hardware	Constrained mean	Autoregressive mean	Speedup	JSON valid (constrained / AR)	Schema compliant (constrained / AR)	Field accuracy (constrained / AR)
M2 Max	393.20 ms	3326.95 ms	8.46×	100% / 100%	100% / 100%	60.7% / 53.6%
L40S	54.12 ms	3404.48 ms	62.91×	100% / 100%	100% / 100%	64.3% / 53.6%
H100	43.76 ms	2586.97 ms	59.12×	100% / 100%	100% / 100%	60.7% / 53.6%

Valid JSON does not mean correct answers: neither method produced a fully correct 28-field object. The constrained method guarantees structure through programmatic assembly, not model accuracy.

Results depend on the workload. On the 12-case diagnostic suite, speedups were 6.25×–9.68×, but field accuracy fell from 80.6% to 77.8%. With 255 candidates on the Mac, constrained inference was 3.23× slower. See all results and raw measurements.

Both methods use PyTorch eager attention and the reference PyTorch convolution implementation. Optimized causal-conv1d, torch.compile, FlashAttention and MLX are not enabled. This isolates an inference-method comparison; it does not establish the fastest attainable backend on any hardware.

Setup

Python 3.11; pinned dependencies include PyTorch 2.14.0 and Transformers 5.17.0. Run from the repository root:

uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.txt
source .venv/bin/activate

Use

This example reads a customer support message and selects three attributes: its topic, whether immediate action is needed, and whether the customer requests a refund. Define the allowed values in a JSON Schema, then pass that schema and the message to the engine.

import json
from rlcd.engine import Engine

schema = {
    "type": "object",
    "properties": {
        "topic": {
            "type": "string",
            "description": "The main issue in the customer's message",
            "enum": ["billing", "technical", "shipping"],
        },
        "urgent": {
            "type": "boolean",
            "description": "True only if the customer explicitly needs immediate action",
        },
        "refund_requested": {
            "type": "boolean",
            "description": "Whether the customer asks for a refund",
        },
    },
    "required": ["topic", "urgent", "refund_requested"],
    "additionalProperties": False,
}

# Load the original base model once and reuse the engine for multiple requests.
engine = Engine(device="mps", dtype="float16")  # Apple Silicon; use "cuda" on NVIDIA

message = "I was charged twice. Please refund the duplicate charge. No hurry."
result = engine.constrained(message, schema)

# A JSON object containing only the three declared fields and allowed value types.
print(result["text"])

# Parse it to use the model's decisions in your application.
attributes = json.loads(result["text"])
print("Selected topic:", attributes["topic"])
print("Refund requested:", attributes["refund_requested"])

# Optional: inspect raw candidate log-likelihoods (not calibrated confidence).
print(result["scores"])

Change the field descriptions and enum values to define your own finite-choice extraction task. The engine selects the values; the program assembles the JSON. A schema-valid result can still contain an incorrect decision.

Supported schemas are flat objects, with all properties required and additionalProperties: false. Each field must be a boolean or a nonempty string enum. Unsupported constraints are rejected. Candidate strings may share token prefixes; full-sequence scoring distinguishes them. Enum cardinality is not capped in code, but large sets can exhaust memory. Scores depend on candidate wording, tokenization, length, and a newline terminator. They are not calibrated confidence. No calibration evaluation was performed and no normalized probability API is exposed.

For successful calls, json.dumps plus selection from typed, allowed values guarantees JSON syntax and compliance with this restricted schema subset. Those guarantees come from the program, not from learned decision quality. Fields are decided independently; output truth and cross-field consistency are not guaranteed.

Bundled base weights

The bundled files are unchanged from LiquidAI/LFM2.5-350M revision 9e6c6ccf47cd318696e137d381a7ded8fe4df09f. Their verified checksums are in BASE_MODEL_MANIFEST.json. No fine-tuning or quantization was performed.

You can load the bundled base model directly:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "notnotsamuel/LFM2.5-350M-RLCD"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto").eval()

This loads the original language model; parallel constrained inference still uses the Engine example above. To recreate the bundle locally, run python -m scripts.bundle_base followed by python -m scripts.prepare_release.

Reproduce and inspect

Benchmark commands and methodology cover M2 Max, L40S and H100, including warmups, precision, hardware versions and validation. Implementation notes explain hybrid-cache handling and candidate scoring. Execution notes record numerical differences and run limitations.

License

Inference code: MIT. Bundled LiquidAI model files: LFM Open License v1.0. The original model files retain their upstream license; no adapters or trained modifications are included.