laya-multilingual-coreai

Laya typed decisions as a Core AI asset, with the checkpoint's full 1024-token context. Converted from convaiinnovations/laya-multilingual with coreai-torch. It outputs choice, score, and noul logits with zero generated tokens. Verified on an iPhone 17 Pro and an M3 Max.

model.aimodel holds four fixed-length entry points over one copy of the FP16 weights (646 MB total). Run a request in the smallest one that fits it:

Entry point	Window	Inputs
L96	96 tokens	input_ids, attention_mask [1, L] · marker_pos, marker_mask [1, 32] · qtype [1] — all int32
L256	256 tokens	same
L512	512 tokens	same
L1024	1024 tokens	same

Outputs: logits [1, 32] (one per option marker; the graph already masks unused slots — softmax over the first k) and action_logits [1, 2]. Prompt layout, tokenization and calibration are exactly those of laya-coreml; tokenizer/, encoder/config.json and rl_agent_config.json are the unchanged upstream files.

Accuracy

Every figure is the largest absolute difference in any calibrated option probability, against the unconverted PyTorch FP32 model. laya-coreml's conversion gate is 0.02.

Prompts	iPhone 17 Pro · Neural Engine	iPhone 17 Pro · GPU	M3 Max · GPU
41 short, nine languages (≤ 96 tokens)	0.0064	0.0024	0.0017
6 long, 259–1024 tokens, English and Chinese	0.0047	0.0013	0.0012

No verdict changed in any run. For scale, the hand-built Core ML Neural Engine bundle (laya-multilingual-coreml-ane) scores 0.0051 on the same 41 prompts: half-precision arithmetic on the Neural Engine sets the floor for both. An FP32 export of this graph reproduces PyTorch to 104 dB PSNR (gap 0.000003), so the differences above are FP16 arithmetic, not conversion error.

Speed

Median per request. iPhone: a whole decision in a release Swift build, tokenizing included. M3 Max: the model call alone, from the Core AI Python runtime.

Entry point	iPhone 17 Pro · Neural Engine	iPhone 17 Pro · GPU	M3 Max · GPU	M3 Max · Neural Engine
L96	7.0 ms	10.2 ms	6.8 ms	5.5 ms
L256	13.9 ms	14.6 ms	11.1 ms	13.3 ms
L512	36.2 ms	31.8 ms	19.1 ms	38.6 ms
L1024	120.2 ms	91.6 ms	38.4 ms	115 ms

The best compute unit depends on both length and device. With .default specialization the iPhone places every window on the Neural Engine (its outputs and timings match an explicit Neural Engine preference), while the M3 Max matches the GPU. On the phone a GPU preference is 24 % faster at L1024 and 31 % slower at L96; on the Mac the GPU is 3× faster at L1024.

Memory (iPhone 17 Pro)

Physical footprint of a demo app process, which includes ~170 MiB for its tokenizer:

Preference	Loaded, all four windows	Peak after running all four
Neural Engine / .default	693 MiB	928 MiB
GPU	298 MiB	516 MiB

Loading more entry points is free: the weights become resident with the first function, and the other three add nothing. On the Neural Engine the run-time cost does not depend on the window; on the GPU it grows with it (391 MiB at L96 to 493 MiB at L1024, loaded alone).

Things to know

• Do not pin this asset to the CPU. FP16 on the CPU drifts to 0.04 at 1024 tokens, past the gate. A CPU preference and a CPU-only allowed set give identical results, so this is CPU half-precision arithmetic, not a placement problem.
• Python: always pass specialization options. With coreai-core 1.0.0b2, AIModel.load(path) with no options aborts the process in load_function (ANE "Program load failed" → MPSGraph assertion). SpecializationOptions.default() and explicit preferences work. Swift's AIModel(contentsOf:) defaults to .default and is unaffected.
• xcrun coreai-build compile model.aimodel --platform iOS succeeds for all device architectures (627 MB each); it is optional, since the runtime specializes on device and caches the result. It warns that the question-type embedding lookup cannot be placed on the Neural Engine.
• Past 1024 tokens the upstream prompt builder drops the end of the situation rather than failing.
• Requires Core AI (macOS 27 / iOS 27); the iOS simulator SDK has no Core AI. Uncompressed FP16; 393 MB of it is the 256k-token embedding table.
• These numbers measure how faithfully the conversion reproduces the original model, not how accurate the model is for your task. Laya's estimates can be wrong.

Use

Swift:

import CoreAI

let model = try await AIModel(contentsOf: modelURL)               // options default to .default
guard let decide = try model.loadFunction(named: "L96") else { fatalError("missing entry point") }
var outputs = try await decide.run(inputs: [
    "input_ids": NDArray(scalars: ids, shape: [1, 96]),            // [Int32], padded with the pad id
    "attention_mask": NDArray(scalars: mask, shape: [1, 96]),
    "marker_pos": NDArray(scalars: markerPositions, shape: [1, 32]),
    "marker_mask": NDArray(scalars: markerMask, shape: [1, 32]),
    "qtype": NDArray(scalars: [Int32(0)], shape: [1]),            // 0 choice · 1 score · 2 noul
])
let logits = outputs.remove("logits")?.ndArray

A complete Swift runtime — tokenizer, prompt builder, window selection, calibration — and an iOS app that downloads this repository are in laya-coreml under ios/ (LayaKit's CoreAIBackend).

Python:

from coreai.runtime import AIModel, NDArray, SpecializationOptions

model = await AIModel.load("model.aimodel", SpecializationOptions.default())
decide = model.load_function("L96")          # or L256 / L512 / L1024
out = await decide({"input_ids": NDArray(ids), "attention_mask": NDArray(mask),
                    "marker_pos": NDArray(marker_pos), "marker_mask": NDArray(marker_mask),
                    "qtype": NDArray(qtype)})
logits = out["logits"].numpy()

Build the inputs with laya-coreml's prepare + collate_items. The export and validation scripts are in that repository under experiments/coreai/. coreai_config.json records the source revision, the source weights' SHA-256 and a checksum for every model, tokenizer and config file; validation.json is the raw M3 Max report.

Apache-2.0. Independent conversion of Laya by Convai Innovations and contributors. Not an official Convai Innovations or Apple release. See NOTICE.