Ornstein3.6-35B-A3B-SABER-8bit-MTPLX-Optimized-Speed

MLX 8-bit build of DJLougen/Ornstein3.6-35B-A3B-SABER packaged for fast local serving with lightning-mlx.

The checkpoint includes an MTPLX sidecar (mtp.safetensors) and runtime metadata (mtplx_runtime.json) so lightning-mlx can use its Qwen3.5 MoE MTPLX serving path on Apple Silicon. Runtime metadata verified on Darwin arm64 with mtplx_version: 0.1.0rc3, mtp_depth_max: 1, recommended_profile: sustained.

The model is the SABER-ablated variant of Ornstein3.6-35B-A3B (Qwen3.5 MoE, 35B total / ~3B active per token). Refer to the source model card for capabilities, license, and SABER details.

Note on MTP weights: mtp.safetensors is packed from the upstream Qwen/Qwen3.5-35B-A3B MTP module. The base model itself is the SABER fine-tune; speculative decoding acceptance rate may differ from upstream.

Install lightning-mlx

python3 -m pip install git+https://github.com/samuelfaj/lightning-mlx.git

Or:

curl -fsSL https://raw.githubusercontent.com/samuelfaj/lightning-mlx/main/install.sh | bash

Verify:

lightning-mlx --help

Serve this model

From Hugging Face:

lightning-mlx serve samuelfaj/Ornstein3.6-35B-A3B-SABER-8bit-MTPLX-Optimized-Speed

From a local checkout:

lightning-mlx serve /path/to/Ornstein3.6-35B-A3B-SABER-8bit-MTPLX-Optimized-Speed

Daemon mode:

lightning-mlx serve samuelfaj/Ornstein3.6-35B-A3B-SABER-8bit-MTPLX-Optimized-Speed --daemon
lightning-mlx status
lightning-mlx tui <PID-or-model-name>
lightning-mlx kill <PID-or-model-name>

OpenAI-compatible API

curl http://localhost:8010/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [
      {"role": "user", "content": "Write a tiny Python HTTP server."}
    ],
    "stream": true
  }'

Why use lightning-mlx

lightning-mlx is built for local agent workloads on Apple Silicon: short streamed turns, tool calls, growing context, repeated low-latency interactions. With this checkpoint it uses the packaged MTPLX metadata and Qwen3.5 MoE serving preset instead of treating the model as a generic MLX checkpoint.

The runtime focuses on:

• OpenAI-compatible local serving
• Fast streamed chat completions
• Qwen3.5 MoE reasoning and tool-use paths
• MTPLX-style speculative decoding support
• Daemon, status, TUI, and kill controls

Convert similar local MTPLX models

lightning-mlx convert-mtplx \
  /path/to/Model-MLX-quantized \
  --mtp-source /path/to/Model-with-mtp-tensors

Output is written next to the source as <source>-MTPLX-Optimized-Speed. Then:

lightning-mlx serve /path/to/Model-MLX-quantized-MTPLX-Optimized-Speed

Use with mlx-lm

This checkpoint is also a standard MLX text-generation model:

pip install -U mlx-lm
mlx_lm.generate \
  --model samuelfaj/Ornstein3.6-35B-A3B-SABER-8bit-MTPLX-Optimized-Speed \
  --prompt "Hello" \
  --max-tokens 100

Intended use

Research and red-teaming. SABER ablates refusal behaviors. Deploy behind your own policy/logging layer.

License

Apache 2.0, inherited from the base model.