wegotomars/gpt2-CoreAI-ANE

GPT-2 on Apple Neural Engine

All four original GPT-2 sizes ported to Core AI framework. FP16, INT8 and LUT8 weights in one repository.

Requires Tinkerbox 2.3.0 or later, Apple Silicon M1+ and macOS 27 or later.

The transformer runs on ANE. Tokenization, embeddings, FP32 vocabulary projection and sampling run on CPU. Context is 1,024 tokens, with independent batches of 1, 4 or 8 and automatic selection for partial groups. Prefill width is 64. FP16 is the default; INT8 and LUT8 are optional compressed weights. These are base text-completion models with no chat template or instruction tuning.

Choose a bundle

<model>/<weights>/macos27/<architecture>/

Models: gpt2, gpt2-medium, gpt2-large, gpt2-xl. Weights: fp16, int8, lut8. Each directory is self-contained: model.json, CPU checkpoint/tokenizer files, and its compiled .aimodelc stages.

Download sizes below are for the M5 Pro bundle, including its CPU checkpoint:

Model	Parameters	FP16 MiB	INT8 MiB	LUT8 MiB
Small (gpt2)	124,439,808	253.0	175.6	198.9
Medium (gpt2-medium)	354,823,168	709.8	431.3	494.0
Large (gpt2-large)	774,030,080	1,530.6	873.3	992.4
XL (gpt2-xl)	1,557,611,200	3,051.5	1,674.7	1,878.0

Run

Requires tinkerbox-ane>=2.3.0 and Python 3.12 or 3.13:

python -m pip install "tinkerbox-ane>=2.3.0"

Download only the chosen model, weights and chip target:

from pathlib import Path
from huggingface_hub import HfApi, snapshot_download
import tinkerbox_ane
from tinkerbox_ane.host import device_architecture

repo = "wegotomars/GPT2-CoreAI-ANE"
revision = HfApi().model_info(repo).sha  # Record this immutable revision for reuse.
model = "gpt2"
weights = "fp16"  # fp16, int8 or lut8
folder = f"{model}/{weights}/macos27/{device_architecture()}"
root = snapshot_download(
    repo_id=repo,
    revision=revision,
    local_dir=Path("models/GPT2-CoreAI-ANE") / revision,
    allow_patterns=["config.json", "LICENSE", "NOTICE", f"{folder}/**"],
)

with tinkerbox_ane.load(model, Path(root) / folder) as engine:
    reply = engine.generate("Once upon a time,", max_new_tokens=48)
    print(reply["text"])
    samples = engine.generate_batch(
        ["The scientist discovered", "The musician played"],
        max_new_tokens=32, temperature=1.0, top_k=40, seed=42,
    )

Temperature zero selects greedy decoding. Seeded sampling preserves standalone replay. Batches accept unrelated prefixes and different lengths, and return input order. Prompt plus requested output tokens must fit the 1,024-token context.

Performance

Measured on Apple M5 Pro, 24 GB, macOS 27.0.1 (h17s). Median of three warm 24-token completions, four CPU threads, with CPU processing and host dispatch included. Batch 8 uses independent requests with the same 12-token prompt. The reference backend is Transformers FP16 on MPS.

Model	Weights	Single tokens/s	Batch 8 tokens/s	MPS FP16 batch 8 tokens/s
Small	FP16	327.6	1,309.0	1,225.5
Small	INT8	396.2	1,444.6	1,225.5
Small	LUT8	394.4	1,443.6	1,225.5
Medium	FP16	135.8	667.3	781.7
Medium	INT8	178.6	778.0	781.7
Medium	LUT8	177.8	770.4	781.7
Large	FP16	69.9	395.9	575.3
Large	INT8	98.0	491.4	575.3
Large	LUT8	97.6	491.7	575.3
XL	FP16	37.8	228.7	421.2
XL	INT8	55.1	305.3	421.2
XL	LUT8	52.0	299.6	421.2

Conversion results

All 252 bundles passed full ANE placement verification. The 12 M5 Pro bundles passed 36 public API tests and 60 numerical checks through the full context. FP32 MPS supplied the reference scores and continuations.

WikiText-2 test: 287,363 predictions per variant, with 1,024-token windows. Reference: FP32 MPS transformer with CPU readout. Lower perplexity/KL and higher agreement are better. Bold compares INT8 with LUT8.

Small

FP32 reference perplexity: 29.941

Weights	Perplexity	ΔPPL	Mean KL (10⁻³ nats)	P95 KL (10⁻³ nats)	Top-1 agreement
FP16	29.950	+0.009	0.092	0.239	99.35%
INT8	29.977	+0.037	1.287	3.482	97.62%
LUT8	29.968	+0.027	0.469	1.267	98.58%

Medium

FP32 reference perplexity: 21.706

Weights	Perplexity	ΔPPL	Mean KL (10⁻³ nats)	P95 KL (10⁻³ nats)	Top-1 agreement
FP16	21.709	+0.003	0.065	0.166	99.50%
INT8	21.705	-0.001	0.740	2.042	98.28%
LUT8	21.698	-0.008	0.294	0.784	98.91%

Large

FP32 reference perplexity: 19.436

Weights	Perplexity	ΔPPL	Mean KL (10⁻³ nats)	P95 KL (10⁻³ nats)	Top-1 agreement
FP16	19.424	-0.012	0.051	0.109	99.62%
INT8	19.437	+0.001	0.309	0.880	98.96%
LUT8	19.426	-0.010	0.158	0.453	99.26%

XL

FP32 reference perplexity: 17.400

Weights	Perplexity	ΔPPL	Mean KL (10⁻³ nats)	P95 KL (10⁻³ nats)	Top-1 agreement
FP16	17.394	-0.006	0.028	0.084	99.68%
INT8	17.397	-0.003	0.235	0.719	99.13%
LUT8	17.392	-0.008	0.124	0.374	99.33%

Supported architectures

Architecture	SoC name
h13g	Apple M1
h13s	Apple M1 Pro
h13c	Apple M1 Max
h13d	Apple M1 Ultra
h14g	Apple M2
h14s	Apple M2 Pro
h14c	Apple M2 Max
h14d	Apple M2 Ultra
h15g	Apple M3
h15s	Apple M3 Pro
h15m	Apple M3 Max (14-core CPU)
h15c	Apple M3 Max (16-core CPU)
h15d	Apple M3 Ultra
h16g	Apple M4
h16s	Apple M4 Pro
h16c	Apple M4 Max
h17g	Apple M5
h17s	Apple M5 Pro
h17c	Apple M5 Max
h17d	Apple M5 Ultra
h18g	Apple M6

License

Original GPT-2 Software Copyright © 2019 OpenAI, under the original Modified MIT License. Conversion details are in NOTICE.