lshar/ARM-TUNE-CPU-INFERENCE-OPT

ArmTune CPU Inference Optimization Pack

ArmTune Serve is an Arm64-aware benchmarking and configuration-recommendation toolkit for LLM inference. This repository contains the model artifacts and benchmark metadata used to demonstrate CPU inference optimization on Arm Neoverse infrastructure.

The project is designed for developers who need to answer:

Which model quantization and runtime settings give the best quality, latency, throughput, and memory use on my exact Arm64 server?

Model summary

Field	Value
Model family	Llama 3.2 1B Instruct
Format	GGUF
Primary runtime	llama.cpp / llama-server
Arm optimization	KleidiAI + native AArch64 build
Profiling	Arm Performix
Workload	Support-ticket classification and structured summarization
Target hardware	AWS Graviton, Azure Cobalt, Google Axion, Ampere, Arm64 Linux

Files

Upload only the artifacts you are legally allowed to redistribute. A typical repository contains one or more of:

Llama-3.2-1B-Instruct-Q4_0.gguf
Llama-3.2-1B-Instruct-Q4_K_M.gguf
Llama-3.2-1B-Instruct-Q8_0.gguf
benchmark-summary.json
benchmark-results.csv

The GGUF files are not generated by ArmTune. They are model artifacts from the upstream model repository and remain subject to the upstream model license. Check the upstream license before uploading or redistributing any file.

Intended use

This repository is intended for:

• CPU-only LLM inference experiments on Arm64 Linux;
• comparing GGUF quantization formats;
• validating llama.cpp runtime settings;
• reproducible Arm performance demonstrations;
• support-ticket classification and structured summarization prototypes.

It is not intended to provide medical, legal, financial, safety-critical, or other high-stakes decisions. The model may generate incorrect or unsafe text. Validate outputs before using them in an application.

Recommended ArmTune workflow

pip install armtune-serve
armtune detect
armtune models list lshar/ARM-TUNE-CPU-INFERENCE-OPT
armtune benchmark \
  --repo lshar/ARM-TUNE-CPU-INFERENCE-OPT \
  --quant Q4_K_M,Q4_0,Q8_0 \
  --threads 1,2,4 \
  --concurrency 1,2
armtune recommend --latest --objective balanced

For the Arm-native runtime build, follow the ArmTune repository instructions:

bash scripts/install-performix.sh
bash scripts/build-llama-cpp.sh
export ARMTUNE_LLAMA_SERVER=llama.cpp/build-arm-opt/bin/llama-server

Reference workload

The evaluation workload asks the model to return JSON with these keys:

{
  "summary": "Customer reports duplicate billing.",
  "category": "billing",
  "priority": "high",
  "recommended_action": "Review invoices and initiate a refund if confirmed."
}

ArmTune measures JSON validity, required fields, valid priority/category values, and expected category agreement. A faster configuration is not recommended if it falls below the quality threshold.

Benchmark protocol

Every published result should identify:

• Arm CPU model and architecture;
• physical and logical core count;
• memory and NUMA topology;
• model filename and SHA256;
• quantization format;
• llama.cpp commit and build flags;
• threads, threads-batch, context, batch, and concurrency;
• prompt set, warmup count, repetitions, seed, and output limit;
• whether Arm Performix captured the inference process.

Measured on GitHub's native ARM64 runner (Neoverse-N2, 4 cores, 15.6 GB) via the ArmTune Benchmark ARM64 workflow, run 31790177140. Full artifacts are committed in the ArmTune repository under docs/evidence/.

Metric	Q4_K_M · 4 threads	Q4_0 · 4 threads	Change
TTFT	0.45 s	0.33 s	-27%
Decode tokens/sec	26.1	33.0	+26%
P95 latency	2.52 s	1.91 s	-24%
Peak process RSS	1701 MB	1616 MB	-5%
Quality score	1.00	1.00	gate held
IPC from Performix	pending capture	pending capture	—

Thread sweep (Q4_K_M): 1 thread 8.1 tok/s → 2 threads 15.1 tok/s → 4 threads 28.1 tok/s (3.5×). Generic vs KleidiAI builds were within noise for short-prompt decode on this 4-core N2 — decode is memory-bound GEMV, while KleidiAI's i8mm kernels accelerate quantized GEMM (prefill).

Limitations

• Results are hardware-specific and should not be generalized to every Arm CPU.
• GitHub-hosted runner hardware can vary between runs.
• The project currently focuses on CPU inference; GPU and vLLM adapters are separate future work.
• Arm Performix counters require a supported Arm Neoverse Linux environment and appropriate profiler permissions.
• Quantization can change quality; evaluate the workload before deployment.

Relationship to ArmTune Serve

The application code, benchmark harness, dashboard, and documentation are available at:

https://github.com/luxmikant/Arm-Tune

ArmTune Serve is MIT-licensed. Model weights and upstream model components may have additional licenses; consult the upstream model card for those terms.

Citation

@software{armtune_serve,
  title = {ArmTune Serve: Arm64 LLM Inference Optimization Toolkit},
  author = {ArmTune Serve contributors},
  year = {2026},
  url = {https://github.com/luxmikant/Arm-Tune},
  license = {MIT}
}