LFM2.5-1.2B-Instruct-a16w4_return_logits: Optimized for SiMa.ai Modalix Accuracy Benchmarking

Overview

This repository contains the LFM2.5-1.2B-Instruct-a16w4_return_logits model, optimized and compiled for the SiMa.ai Modalix platform.

This model is compiled with --return_logits so that it can be used for Modalix accuracy benchmarking without recompiling the source model.

• Model Architecture: LFM2.5 (1.2B parameters)
• Quantization: Hybrid
  • Prompt Processing: A16W8 (16-bit activations, 8-bit weights)
  • Token Generation: A16W4 (16-bit activations, 4-bit weights)
• Maximum context length: 2048
• Source Model: LiquidAI/LFM2.5-1.2B-Instruct
• Return logits: Enabled

Accuracy Benchmark Results

The table below compares accuracy results produced by SiMa.ai on the Modalix backend against Hugging Face backend reference results.

Task	Version	Filter	n-shot	Metric	Direction	HF Backend Value	HF Stderr	Modalix Backend Value	Modalix Stderr	Abs. Diff
hellaswag	1.0	none	0	acc	higher	0.4672	0.0050	0.448516	0.004963	0.018684
hellaswag	1.0	none	0	acc_norm	higher	0.6139	0.0049	0.596594	0.004896	0.017306
piqa	1.0	none	0	acc	higher	0.7231	0.0104	0.722524	0.010447	0.000576
piqa	1.0	none	0	acc_norm	higher	0.7296	0.0104	0.719804	0.010478	0.009796
triviaqa	3.0	remove_whitespace	0	exact_match	higher	0.0232	0.0011	0.002285	0.000356	0.020915
wikitext	2.0	none	0	bits_per_byte	lower	0.9499	N/A	1.018517	N/A	0.068617
wikitext	2.0	none	0	byte_perplexity	lower	1.9318	N/A	2.025836	N/A	0.094036
wikitext	2.0	none	0	word_perplexity	lower	33.8195	N/A	43.606509	N/A	9.787009
winogrande	1.0	none	0	acc	higher	0.5959	0.0138	0.581689	0.013864	0.014211

Evaluation Sample Counts

Task	sample_len
hellaswag	10042
piqa	1838
triviaqa	17944
wikitext	62
winogrande	1267

Prerequisites

To benchmark accuracy with this model, you need:

1. SiMa.ai Modalix Device
2. SiMa.ai CLI: Installed on your Modalix device.
3. SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
4. LLiMa benchmark CLI: Installed on the host machine used to launch accuracy benchmarking.
5. Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device.

The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model assets from this repository directly to your device.

# Download the model to a local directory
llima pull LFM2.5-1.2B-Instruct-a16w4_return_logits

Alternatively, you can download the compiled model to a Host and copy it to the Modalix device:

hf download simaai/LFM2.5-1.2B-Instruct-a16w4_return_logits --local-dir LFM2.5-1.2B-Instruct-a16w4_return_logits
scp -r LFM2.5-1.2B-Instruct-a16w4_return_logits sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device.

Expected Directory Structure:

/media/nvme/llima/
`-- models/
    `-- LFM2.5-1.2B-Instruct-a16w4_return_logits/   # The compiled model

Usage

Modalix Backend Accuracy

Run Modalix accuracy benchmarking from a host machine using this precompiled --return_logits artifact:

llima-benchmark accuracy LFM2.5-1.2B-Instruct-a16w4_return_logits -b modalix -o <output_dir> --max_num_tokens <max_num_tokens> \
    --board_ip <board_ip> --board_port <board_port> \
    --board_model LFM2.5-1.2B-Instruct-a16w4_return_logits --board_start_server \
    --board_venv_path <venv_on_board>

The --board_model path must already exist on the Modalix device and contain both devkit/ and elf_files/.

HF Backend Accuracy

Run the Hugging Face backend reference benchmark from the host machine:

llima-benchmark accuracy LiquidAI/LFM2.5-1.2B-Instruct -b hf -o <output_dir>

See MOLE accuracy benchmarking for the full accuracy benchmarking workflow.

Limitations

• Quantization: This model is quantized (A16W4/A16W8) for Modalix execution. Minor deviations from the full-precision source model may occur.
• Return logits: This model is compiled with --return_logits for accuracy benchmarking workflows. For standard text generation deployments, use the corresponding SiMa.ai model from the Large Language Models collection.

Troubleshooting

• sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
• llima-benchmark not found: Ensure that the LLiMa benchmark CLI is installed on the host machine.
• llima not found: Install or update the Neat Library. See Getting Started.
• Modalix accuracy benchmark fails: Verify that the model was compiled with --return_logits and that the model directory contains both devkit/ and elf_files/.
• Model can't be found on Modalix: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/LFM2.5-1.2B-Instruct-a16w4_return_logits/LFM2.5-1.2B-Instruct-a16w4_return_logits).
• Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.

Resources

• GenAI with LLiMa
• LLiMa compilation
• MOLE accuracy benchmarking