mlboydaisuke/Laya-Multilingual-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine.

This model has no row on DeviceMark, the on-device LLM leaderboard.

laya multilingual — Core AI export

Zoo card, recipe and gate transcript: coreai-model-zoo/models/laya-multilingual.

A typed decision model that answers in one forward pass: give it a state (a message, an email, a JSON record, a conversation) and a typed question — choice (pick one of named options), score (an ordered scale) or noul (yes/no) — and it returns a probability for every option, no generation. convaiinnovations/laya, subfolder multilingual/ (Apache-2.0, revision 1c5edc17…, laya 0.3.4), as a static .aimodel for macOS 27 and, compiled ahead of time, for the iPhone 17 Pro (not yet run on a device). It is the catalog's first encoder-type decision model: the other decision models here are language models read out after a prefill; this one scores every option at its own marker position in a single encoder call.

Architecture (from model.safetensors and encoder/config.json; 321,908,998 parameters, all stored in F16 but the three-value temperature buffer): an mmBERT-base encoder — ModernBERT layout, 22 layers, hidden 768, 12 heads × 64, GLU MLP 1152 with exact GELU, biasless attention/MLP/LayerNorm (ε 1e-5), vocabulary 256,000 — with global attention on layers 0, 3, …, 21 and a sliding window of inclusive radius 64 elsewhere, RoPE θ 160,000 for both kinds; then a type embedding (choice / score / noul), two pre-norm transformer layers (768, 12 heads, ReLU 3072), a scorer (LayerNorm → Linear → GELU → Linear) at every position, and a small act head.

This is an encoder, not a generator. One question is one forward over a right-padded window; no KV cache, no sampling loop. It runs through raw AIModel calls, not the generate engine.

Graph contract

function "main"
  input  "input_ids"       [1, S]    int32    right-padded with PAD 0
  input  "attention_mask"  [1, S]    int32    1 over real tokens, 0 over padding
  input  "qtype_onehot"    [1, 3]    fp32     choice / score / noul
  output "token_logits"    [1, S]    fp32     the scorer at every position — read at the option markers
  output "pooled_cls"      [1, 768]  fp32     input of "act"
function "act"                                  fp32 in every variant
  input  "pooled_cls"      [1, 768]  fp32
  input  "feats"           [1, 4]    fp32     top1, top1 − top2, entropy / ln max(K,2), max(K,2) / 255
  output "act_logits"      [1, 2]    fp32     class 0 = answer directly
S = 256 or 512 (export-time choice); batch = 1; one bundle holds both functions

Host recipe — the sequence and the readout are the publisher's, unchanged:

• Build [CLS 2] <type> question: <instructions> [SEP 1] [MASK 4] option … [MASK 4] option … [SEP 1] <state> [SEP 1] with laya's build_sequence (option text ≤ 48 tokens, head budget 256, state right-truncated to the window; the marker positions are where the [MASK] ids sit). Every text piece is tokenized without special tokens by the checkpoint's tokenizer.json (Gemma-style BPE, byte fallback).
• Gather token_logits at the markers (K logits), softmax them raw for the four feats, run act; then divide the K logits by the question's temperature (bucket by type and K first, then per type) and softmax for the answer. metadata.json carries the checkpoint's temperatures (T = 1) and the fitted calibration the LiteRT port published (temperature, temperature_by_options).
• reference.json in every folder is the parity instrument: 201 question rows for that window with token ids, marker positions, the publisher's answers at T = 1 and the official model's raw logits.

Measured

Mac (Apple M4 Max), macOS 27.0 build 26A428, coreai-build 3600.83.1, the JIT .aimodel, 2026-09-23. Every row: the 201 rows of its window through the whole pipeline (main → host features → act), 3 runs per row, then 40 warm questions; the warm time is one whole question, NumPy I/O included. The GPU rows ran alone on the GPU (the machine-wide GPU lock held, no other GPU job visible during the run). cpu_only is the parity option, and its milliseconds were taken while other jobs used the machine's CPU: reference values, not a speed claim.

Variant	S	Compute	Argmax (choice + score)	Max |Δp| at T=1	Marker max |Δ|	Act relative	Repeat drift	Load	Warm median
wfp16	256	GPU	81/81	4.47e-6	5.53e-5	2.60e-6	0	570 ms	12.1 ms
wfp16	512	GPU	81/81	4.59e-6	1.00e-4	3.47e-6	0	589 ms	19.5 ms
fp32	256	GPU	81/81	4.47e-6	5.53e-5	2.60e-6	0	853 ms	10.4 ms
fp32	512	GPU	81/81	4.59e-6	1.00e-4	3.47e-6	0	982 ms	17.2 ms
wfp16	256	cpu_only	81/81	9.48e-6	7.49e-5	1.85e-6	0	720 ms	45.1 ms
wfp16	512	cpu_only	81/81	9.48e-6	6.68e-5	1.94e-6	0	600 ms	86.2 ms
fp32	256	cpu_only	81/81	9.48e-6	7.49e-5	1.85e-6	0	762 ms	63.0 ms
fp32	512	cpu_only	81/81	9.48e-6	6.68e-5	1.94e-6	0	720 ms	84.2 ms

On the CPU, wfp16 and fp32 return bit-identical marker and act logits on all 402 rows: wfp16 stores the weights in fp16 — exact, the checkpoint is F16 — and computes in fp32. On this Mac's GPU both stay within 1e-4 of the CPU run as well. Use the GPU, and request it explicitly: with the Neural Engine preference the Mac returned the GPU's results bit for bit for fp32, but for wfp16 it returned different results that changed from run to run (next section). iPhone: not measured — the ios-h18p/ and ios/ folders have not run on a device yet.

The publisher's own model on the SemIf authored144 fixture (144 three-option evidence / rule / candidate questions it was not trained for), as laya.load(...).predict answers them: mean family balanced accuracy 0.6114, accuracy 0.5903 (the same at T = 1 and with the fitted calibration, and at both windows). That is the reference a port's decisions are compared with, not a claim about the port.

Numerics gate

The oracle is the publisher's package itself (laya.load(<pinned snapshot>, subfolder="multilingual"), transformers 5.17.0, CPU fp32): it reproduces the frozen fixture of the LiteRT port exactly (token ids 402/402; batched answers and logits bit-identical), and each row alone, right-padded to its window, is the tensor reference. Bars, at every stage: argmax identical on every choice and score row, max |Δp| ≤ 1e-3 at T = 1 over the options, |Δ act probability| ≤ 1e-3; marker logits ≤ 1e-3 and act logits ≤ 1e-4 relative to the official batch-1 run; repeat drift ≤ 1e-6 on CPU; a wrong-pairing control (every row judged against another same-shape row's outputs) must fail — it does, on 162–166 of 200 pairs.

• Authoring (gate_laya_authoring.py): the 201 rows of each window, plus every hidden state of 14 rows (embeddings, 22 layers, final norm, after the type embedding, both head layers) against the official model — absolute max |err| ≤ 1e-4 on the embeddings and layers 0–9, relative max |err| / max |ref| ≤ 2e-4 on layer 10 through the head (measured: 5.0e-5 absolute, 5.7e-5 relative at S=256, 7.8e-6 at S=512). Five mutations must be caught and are, by the layer, tensor and answer gates: a local radius of 63 (layer 1, 1.24 / 0.84 at S=256 / 512), all-global, all-local, ignoring padding, and dropping the type embedding (after the type embedding, 6.6e-2 relative). A pad-isolation check replaces every pad id with a random token: real positions stay bit-identical.
• Export: the torch-exported, decomposed main and act pass the same 201-row gate before conversion.
• Runtime: the tables above.

gate-laya-multilingual.json in the zoo repository is the transcript: every stage's summary with the sha256 of the full record it came from; the records keep each row's raw marker and act logits, unrounded.

Neural Engine and fp16

The checkpoint is F16, so fp16 storage is exact. fp16 compute is not good enough here. The recipe that runs everything in fp16 except the RoPE application and the attention softmax misses the answer bar everywhere it was measured — torch max |Δp| 3.4e-3 / 5.0e-3 (S=256 / 512); Mac CPU 4.38e-2 with one choice flipped (80/81); Mac GPU 5.84e-3; Mac Neural Engine 1.85e-2 with one choice flipped (both windows) — and keeping the residual stream and LayerNorm in fp32 as well still leaves 4.9e-3 in torch (S=256).

Placement follows the compute precision here. Compiled for the iPhone 17 Pro with --preferred-compute neural-engine, the fp16 recipe gets 47 Neural Engine regions; wfp16, which computes in fp32, gets a single region of 5.7 KB of IR and everything else stays in the GPU package. On the Mac, the Neural Engine preference gave these results:

Variant	S	Argmax	Max |Δp|	Marker max |Δ|	Repeat drift	Load	Warm median	Reading
fp32	256 / 512	81/81	4.47e-6 / 4.59e-6	5.53e-5 / 1.00e-4	0	986 / 1,026 ms	10.5 / 20.4 ms	identical to the GPU run on every row: it ran on the GPU
wfp16	256	81/81, 81/81	0.290, 0.164	0.82, 1.66	90, 90	1,186 ms	12.7 ms	two runs, different answers each time: a defect of this specialization, do not use
wfp16	512	81/81, 80/81	0.343, 0.238	7.76, 11.4	202, 212	1,188 ms	20.1 ms	the same, and the second run flipped a choice
fp16 recipe	256	80/81	1.85e-2	0.211	0	5,919 ms	42.9 ms	deterministic, but outside the bar
fp16 recipe	512	80/81	1.85e-2	0.681	0	5 ms (cached)	67.7 ms	the same

coreai.runtime reports no placement, so the reading column is inferred from the numbers. The iPhone folder is therefore the GPU compile, named for what it is (ios-h18p/); the neural-engine compile's region count is recorded in its manifest.

⬇️ Bundle

This repository holds one folder per variant, each self-contained: the bundle, tokenizer/ (the checkpoint's files, unmodified), metadata.json (the decision contract), reference.json and provenance/ (export manifest with per-file sha256, the export and runtime gate records).

Folder	Platform	Format	Bundle	Bytes
macos/wfp16-s256/	macOS 27	JIT .aimodel	laya_ml_wfp16_s256.aimodel	644,855,189
macos/wfp16-s512/	macOS 27	JIT .aimodel	laya_ml_wfp16_s512.aimodel	645,772,737
macos/fp32-s256/	macOS 27 (reference)	JIT .aimodel	laya_ml_fp32_s256.aimodel	1,288,156,432
macos/fp32-s512/	macOS 27 (reference)	JIT .aimodel	laya_ml_fp32_s512.aimodel	1,289,073,976
ios/wfp16-s256/	iOS 27 (portable JIT)	JIT .aimodel	laya_ml_wfp16_s256.aimodel	644,855,189
ios/wfp16-s512/	iOS 27 (portable JIT)	JIT .aimodel	laya_ml_wfp16_s512.aimodel	645,772,737
ios-h18p/wfp16-s256/	iOS 27, h18p only	AOT .aimodelc	laya_ml_wfp16_s256.h18p.aimodelc	645,117,160
ios-h18p/wfp16-s512/	iOS 27, h18p only	AOT .aimodelc	laya_ml_wfp16_s512.h18p.aimodelc	646,034,970

The ios-h18p/ bundles are compiled for one device architecture (h18p, the iPhone 17 Pro) with xcrun coreai-build compile --platform iOS --min-deployment-version 27.0 --preferred-compute gpu --architecture h18p (coreai-build 3600.83.1). Never load an iOS bundle on a Mac.

Convert yourself: conversion/laya/ — staged scripts, and recipe.toml names the commands.

CoreAIKit (Swift)

Catalog id laya-multilingual (active once the kit's encoder backend is merged). TypedDecisions loads the bundle as an encoder backend — the same decide / prefill calls as the kit's language-model decision models — and runs it on the GPU; a Neural Engine preference is refused at load.

Measured through the kit on the same Mac (M4 Max, macOS 27.0 26A428, the wfp16 bundles above, GPU lock held, 2026-09-23):

• Parity on the 201 rows of each window: token ids and marker positions 201/201, argmax 81/81, max |Δp| 5e-6 on the GPU and 9e-6 with cpuOnly.
• decide-cli bench (a 109-token state, 8 questions, warm): 11.5 ms per decision at S=256 when the state is tokenized once and shared, 12.1 ms when every decision tokenizes its row; 18.9 / 19.4 ms at S=512.
• Load 1.06–1.16 s, of which the tokenizer (the 256,000-entry tokenizer.json) takes about 1.0 s and 145–162 MB; the whole process is 474–498 MB after the first decision.
• SemIf authored144 at the kit's default temperature (the fitted calibration): argmax 144/144 with the publisher's model at the same temperature, max |Δp| 2.9e-6; mean family balanced accuracy 0.6114, accuracy 0.5903.
• With a Neural Engine preference (measured before it was refused): 168–183 of 201 rows within the 1e-3 bar over four runs, max |Δp| 0.61.

measurements-coreai-kit.json holds these numbers and the records they came from.

The port in one lesson: a massive activation moves the layer bar

From layer 11 the encoder parks a value of about 14,000 on a few dimensions of the first token. One fp32 step there is about 0.001, so the publisher's own two attention paths (SDPA, which the package uses, and eager) already disagree by up to 0.048 on those layers while their marker logits agree to 3.5e-5. The publisher's own eager path does not meet a per-layer bar of 1e-4 above layer 10, so it cannot be the bar for a reimplementation there; the gate keeps 1e-4 where it still separates a correct graph from a wrong one (the embeddings and layers 0–9, where a window of 63 instead of 64 shows up at 0.8–1.2) and switches to a relative bar of twice the official spread above. In the fp16 recipe the largest relative error sits at the final norm (0.13 at S=256 in torch), the state that normalizes those large values.

License and limits

Apache-2.0 at the pinned upstream revision. Not tested: any iPhone (the device gate is pending), other Macs or OS builds, dynamic or batched shapes, windows other than 256 and 512, more than 20 options, the fitted calibration's quality at S = 512, languages beyond the fixture's English, Japanese and mixed rows, sustained thermals. The act probability is saturated at 1.0 on every fixture row, in the publisher's model as here: it is carried through the graph, not evidence of when to escalate.