DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model Architecture: Llama 3.1 (8B parameters), reasoning model distilled from DeepSeek-R1
• Quantization: A16W4 (BF16 activations, INT4 weights); INT8 KV cache and embeddings
• Pre-quantization: The source checkpoint was pre-quantized with GPTQ before compilation (see Quantization).
• Maximum context length: 8192
• Source Model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B

Performance

The following measurements characterize text generation on Modalix.

Measured on Modalix (Neat runtime 0.5.0) through the GenAI server's OpenAI-compatible API, batch size 1, five samples per input length, each with different text so no run reuses another's cached prompt, and up to 128 generated tokens. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is the server's reported rate after the first token.

Input tokens	Mean TTFT (seconds)	Mean generation rate (tokens/second)
128	0.65	11.01
256	0.88	10.86
512	1.32	10.80
1024	2.35	10.43
2048	4.71	9.46
3072	7.87	9.19
4096	11.54	8.58
5120	16.28	7.84
6144	21.68	7.58
7168	28.71	7.03

Prerequisites

To run this model, you need:

1. SiMa.ai Modalix Device
2. SiMa.ai CLI: Installed on your Modalix device.
3. SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime. Tested with Neat runtime 0.5.0.
4. Hugging Face CLI: For downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device.

The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model on a host and copy it to the Modalix device:

hf download dotimothy/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4 --local-dir DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4
scp -r DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device.

Or download it on the Modalix device itself (with hf installed there):

hf download dotimothy/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4 --local-dir /media/nvme/llima/models/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4

Expected Directory Structure:

/media/nvme/llima/
└── models/
    └── DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4/   # The compiled model
        ├── devkit/
        └── elf_files/

Usage

Validate with LLiMa CLI

Run the model directly on Modalix:

llima run DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4

For all runtime options, run:

llima run -h

Neat GenAI Studio

Neat GenAI Studio is a web app for Modalix that runs this model with chat, Markdown rendering, speech input and output, RAG, model switching and an OpenAI-compatible API. Install it on the Modalix device:

curl -fsSL https://raw.githubusercontent.com/dotimothy/neat-genai-studio/main/install.sh | bash
cd neat-genai-studio
./setup.sh
./run.sh

Open the URL that run.sh prints. The Studio finds this model if it is already under /media/nvme/llima/models. Otherwise, with the device online, open Settings → Add Model, search for DeepSeek, and download dotimothy/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4; it then appears under Models, ready to load. See the Studio README for configuration and the terminal chat client.

API Usage

To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in Serve GenAI Models.

For direct LLM calls without setting up a server, see Run an LLM.

Reasoning output

The chat template opens the reply inside a reasoning block: the generation prompt ends in <think>\n, so the model streams its reasoning with no opening tag and then </think> before the answer. Clients that hide or collapse reasoning should treat everything before </think> as reasoning. Reasoning can take a few thousand tokens; allow a large max_tokens (up to the 8192-token context, prompt included).

DeepSeek recommends no system prompt, with all instructions in the user message; for math, ask for the final answer in \boxed{}.

Quantization

The weights were pre-quantized with GPTQ before compilation, using SiMa.ai's published recipe. quantize.py and recipe.yaml in this repository are SiMa's scripts (as published for Llama-3.1-8B-Instruct) with the model name changed; package versions are in quantize-versions.txt.

• All decoder Linear layers and lm_head: GPTQ, symmetric INT4, group size 256, static actorder (block size 128, dampening 0.01).
• Calibration: HuggingFaceH4/ultrachat_200k, 512 samples, 1024 tokens, batch size 1.
• No intentional mixed-precision language layers.

Full WikiText evaluation

EleutherAI/wikitext_document_level, wikitext-2-raw-v1, test split, all 62 documents (2026-10-21). Word perplexity as defined by lm-eval-harness's wikitext task (its detokenizer, rolling log-likelihood in 4096-token windows, words counted on the raw page), computed with a standalone script rather than lm-eval itself, BF16 on a GPU.

Checkpoint	Word perplexity
deepseek-ai/DeepSeek-R1-Distill-Llama-8B	53.635696
GPTQ checkpoint (before compilation)	62.913714

The base model's perplexity is high on WikiText; it is a math/reasoning distill, not a general language model (for comparison, SiMa.ai reports 9.33 for Llama-3.1-8B-Instruct).

Compilation

To reproduce this build, quantize the base model with quantize.py (needs a CUDA GPU), then compile the result with compile.sh (Model Compiler 2.1.3.1):

hf download deepseek-ai/DeepSeek-R1-Distill-Llama-8B --local-dir DeepSeek-R1-Distill-Llama-8B
python quantize.py --model-path DeepSeek-R1-Distill-Llama-8B --output-dir DeepSeek-R1-Distill-Llama-8B-GPTQ-Safetensors
./compile.sh

compile.sh runs:

llima-compile DeepSeek-R1-Distill-Llama-8B-GPTQ-Safetensors -o DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4 \
    --max_num_tokens 8192 --enable_filter_sharing

These are the options SiMa.ai uses for Llama-3.1-8B-Instruct, including the default INT8 KV cache. On Intel Arrow Lake hosts, compile.sh also sets an XLA_FLAGS workaround for an LLVM error in the compiler's host reference run; it does not change the output.

Limitations

• Quantization: This model is quantized (A16W4) for optimal performance on embedded devices. Minor deviations from the full-precision model occur. On 12 arithmetic questions with greedy decoding, the GPTQ checkpoint answers 12/12 on a GPU and this build answers 12/12 on Modalix.
• Greedy decoding: The LLiMa runtime decodes greedily; temperature, top_p and repetition penalties have no effect. DeepSeek recommends temperature 0.6 for this model, and with greedy decoding it can repeat the same passage in its reasoning until the token limit.
• Thinking off: The runtime reports thinking as unsupported for this model, so enable_thinking cannot turn the reasoning off.

Troubleshooting

• sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
• llima not found: Install or update the Neat Library. See Getting Started.
• Model can't be run: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4/DeepSeek-R1-Distill-Llama-8B-GPTQ-a16w4).
• Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.

License

MIT, as DeepSeek-R1-Distill-Llama-8B. That model is derived from Llama 3.1 8B Base, which is licensed under the Llama 3.1 Community License; its terms apply as well. Built with Llama.

Resources

• Neat GenAI Studio
• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• GenAI Multimodal Assistant