litert-community/Laya-English-LiteRT

Laya English for LiteRT — Android GPU FP32

Run the English checkpoint of convaiinnovations/laya and its typed-decisions/ fine-tune on a phone GPU with LiteRT. Laya reads a text or a JSON state and answers questions you define at request time: pick one of several options, score on an ordinal scale, or give a yes/no probability. Each question is one forward pass of the main graph plus one pass of a small act-head graph. The validated Android configuration is LiteRT 2.2.0 CompiledModel with explicit GPU FP32 computation on a Samsung Galaxy S26 (SM-S942Q, Android 16): 123 ms of graph time per question at 256 tokens for the English graph, 126 ms for the typed-decisions graph. On 140 English and 100 typed-decisions question rows the answers match the official laya 0.3.4 fp32 CPU implementation: the same argmax on every choice and score question, and a maximum probability difference of 0.0008. Other Android GPU families have not been validated here.

The message is invented. The answers and probabilities are the values the English S256 wfp16 graph returned through laya_host.py on a desktop CPU, with the checkpoint's own temperatures. The Android sample app in Laya-Multilingual-LiteRT covers the multilingual checkpoint; this package has no app of its own.

Files and supported configuration

English files are at the repository root; typed-decisions/ holds the fine-tune, the same layout as the source repository at revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982. SHA256SUMS lists every other file.

English

File	Window N	Bytes	Role
laya_en_s256_embeds_wfp16.tflite	256	739,762,688	recommended; validated on the S26 GPU and CPU
laya_en_s256_embeds_fp32.tflite	256	1,478,479,220	fp32 reference; validated on the S26 GPU
laya_en_s512_embeds_wfp16.tflite	512	740,811,264	longer texts; validated on the S26 GPU
laya_en_s512_embeds_fp32.tflite	512	1,479,527,796	fp32 reference; validated on the S26 GPU
laya_en_act_head_fp32.tflite	any	1,057,960	act head, shared by every English main graph
token_embeddings_en_fp16.bin, token_embeddings_en_fp16.json		103,153,664	[50368,1024] float16 token table for the host lookup
tokenizer.json, tokenizer_config.json		3,583,536	exact files from the pinned checkpoint
laya_en_config.json		745	the checkpoint's rl_agent_config.json: temperatures and builder budgets
laya_host.py		22,933	Python host for both checkpoints: prompt builder, embedding lookup, decoder
fixtures/gate_rows_en_s256.json		517,382	the 140 validation rows with the reference answers

typed-decisions/

File	Window N	Bytes	Role
laya_td_s256_embeds_wfp16.tflite	256	739,762,688	recommended; validated on the S26 GPU
laya_td_s256_embeds_fp32.tflite	256	1,478,479,220	fp32 reference; validated on the S26 GPU
laya_td_s512_embeds_wfp16.tflite	512	740,811,264	longer texts; validated on the S26 GPU
laya_td_s512_embeds_fp32.tflite	512	1,479,527,796	fp32 reference; validated on the S26 GPU
laya_td_act_head_fp32.tflite	any	1,057,960	act head, shared by every typed-decisions main graph
token_embeddings_td_fp16.bin, token_embeddings_td_fp16.json		103,153,664	[50368,1024] float16 token table for the host lookup
tokenizer.json, tokenizer_config.json		3,583,565	exact files from the pinned checkpoint
laya_td_config.json		847	the checkpoint's rl_agent_config.json
fixtures/gate_rows_td_s256.json		450,460	the 100 validation rows with the reference answers

The recommended English phone set is the S256 wfp16 graph, the act head, the token table pair, the tokenizer pair and the config file: 847,582,366 bytes. Each checkpoint keeps its own table, tokenizer pair and config; the two tables differ.

wfp16 files store the 123 FULLY_CONNECTED weight tensors as float16 with a DEQUANTIZE to float32. Every activation and every other constant stays float32. The float16 tables reproduce the checkpoints' float32 tables exactly (maximum difference 0 over all 51,576,832 values of each). Padded positions gather the real PAD row (id 50283).

The main graph holds the ModernBERT-large encoder, Laya's two typed head layers and the option scorer, applied at every position. The host does the rest: tokenize, build the prompt with one [MASK] marker per option, look up the embedding rows, read the logits at the marker positions, and apply the checkpoint temperature and softmax. The act head is a separate small graph because its input depends on those host-side probabilities.

The graphs take embedding rows, not token ids. With the token table inside the graph, the float16-weight file does not compile on LiteRT 2.2.0 CompiledModel GPU: the GPU delegate rejects the EMBEDDING_LOOKUP that reads a DEQUANTIZE-fed table (Empty quantization params), and CompiledModel needs every operator on the GPU. The fp32 file with the table inside does compile; litert-community/laya-LiteRT ships that form. With the lookup on the host, the wfp16 graphs compile as one GPU partition (2223 of 2223 operators) at 0.74 GB.

Use explicit FP32 GPU computation, as in the Kotlin block below. It is the only GPU precision validated here. INT8 files were not built and NPU execution was not evaluated. The two attention layer types use different RoPE tables (theta 160000 for full attention, 10000 for sliding attention); the conversion keeps them distinct.

Minimal usage

Python — complete pipeline, desktop CPU

Needs numpy, transformers (tokenizer only) and ai-edge-litert; neither torch nor the laya package is required. Set USE_TF=0. Put these eight files in one directory: laya_host.py, laya_en_s256_embeds_wfp16.tflite, laya_en_act_head_fp32.tflite, token_embeddings_en_fp16.bin, token_embeddings_en_fp16.json, tokenizer.json, tokenizer_config.json, laya_en_config.json.

import json
from pathlib import Path
from laya_host import LayaHost

assets = Path(".")
questions = {
    "intent": {
        "type": "choice",
        "instructions": "What does the customer need?",
        "criteria": {
            "refund": "return a duplicate payment",
            "help": "technical help",
        },
    }
}
with LayaHost.from_directory(
    assets, checkpoint="en", window=256, storage="wfp16"
) as host:
    result = host.predict(
        "I was charged twice for one order. Please return the extra payment.",
        questions,
    )
print(json.dumps(result, indent=2))

Run from a directory that held only those eight files, this exact code printed:

"choice": "refund", "probabilities": {"refund": 0.9792, "help": 0.0208},
"confidence": 0.854, "action": {"act_probability": 1.0}, "usage": {"input_tokens": 39, ...}

checkpoint="td" with the repository root selects the fine-tune (the host reads typed-decisions/); window=512 selects the S512 graph without changing the checkpoint's prompt-builder budgets; storage="fp32" selects the fp32 file.

Kotlin — Android GPU with explicit FP32

The ModernBERT ByteLevel BPE tokenizer is not ported to Kotlin in this package: the token ids and marker positions must come from the app's own tokenizer and prompt builder (HOST_CONTRACT.md specifies both). The multilingual sample's Kotlin tokenizer covers only that checkpoint. This function does the rest: it memory-maps the float16 table, gathers the rows for the given ids (real PAD rows for the padding), runs the main graph with GPU FP32 computation and returns the temperature-scaled option probabilities. dir is the checkpoint directory. Reuse the mapping, the model and the buffers in an application; the act-head features and the answer dictionaries are in HOST_CONTRACT.md.

import android.util.Half
import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel
import com.google.ai.edge.litert.Environment
import com.google.ai.edge.litert.TensorBuffer
import java.io.File
import java.io.FileInputStream
import java.nio.ByteOrder
import java.nio.channels.FileChannel
import org.json.JSONObject
import kotlin.math.exp

// Call on a worker thread. ids and markers follow HOST_CONTRACT.md's builder.
fun scoreRow(dir: File, ids: IntArray, markers: IntArray, qtype: Int,
             tag: String = "en", window: Int = 256): FloatArray {
    require(tag in setOf("en", "td") && window in setOf(256, 512))
    require(ids.isNotEmpty() && ids.size <= window && qtype in 0..2)
    require(markers.isNotEmpty() && markers.all { it in ids.indices })
    val meta = JSONObject(File(dir, "token_embeddings_${tag}_fp16.json").readText())
    val shape = meta.getJSONArray("shape")
    val vocab = shape.getInt(0)
    val width = shape.getInt(1)
    val pad = meta.getInt("pad_id")
    require(meta.getString("dtype") == "float16" && meta.getString("byte_order") == "little")
    require(meta.getString("layout") == "row-major")
    require(pad in 0 until vocab && ids.all { it in 0 until vocab })
    val tableFile = File(dir, meta.getString("filename"))
    require(tableFile.length() == vocab.toLong() * width * 2)
    require(tableFile.length() == meta.getLong("size_bytes"))
    val table = FileInputStream(tableFile).channel.use { channel ->
        channel.map(FileChannel.MapMode.READ_ONLY, 0, tableFile.length())
            .order(ByteOrder.LITTLE_ENDIAN)
    }
    val embeds = FloatArray(window * width) { index ->
        val token = ids.getOrElse(index / width) { pad }
        Half.toFloat(table.getShort((token * width + index % width) * 2))
    }
    val mask = FloatArray(window) { if (it < ids.size) 1f else 0f }
    val type = FloatArray(3).also { it[qtype] = 1f }
    val config = JSONObject(File(dir, "laya_${tag}_config.json").readText())
    val count = markers.size
    val bucket = when { count <= 2 -> "2"; count <= 5 -> "3-5"; count <= 10 -> "6-10"; else -> "11+" }
    val key = "${listOf("choice", "score", "noul")[qtype]}:$bucket"
    val temperature = config.getJSONObject("temperature_by_options")
        .optDouble(key, config.getJSONArray("temperature").getDouble(qtype))
        .toFloat().coerceAtLeast(1e-3f)
    val options = CompiledModel.Options(Accelerator.GPU).apply {
        gpuOptions = CompiledModel.GpuOptions(
            precision = CompiledModel.GpuOptions.Precision.FP32)
    }
    Environment.create().use { environment ->
        CompiledModel.create(File(dir, "laya_${tag}_s${window}_embeds_wfp16.tflite").absolutePath,
                             options, environment).use { model ->
            val inputs = linkedMapOf<String, TensorBuffer>()
            val outputs = linkedMapOf<String, TensorBuffer>()
            try {
                listOf("inputs_embeds", "attention_mask", "qtype_onehot").forEach {
                    inputs[it] = model.createInputBuffer(it, "serving_default")
                }
                listOf("token_logits", "pooled_cls").forEach {
                    outputs[it] = model.createOutputBuffer(it, "serving_default")
                }
                inputs.getValue("inputs_embeds").writeFloat(embeds)
                inputs.getValue("attention_mask").writeFloat(mask)
                inputs.getValue("qtype_onehot").writeFloat(type)
                model.run(inputs, outputs, "serving_default")
                val logits = outputs.getValue("token_logits").readFloat()
                val pooled = outputs.getValue("pooled_cls").readFloat()
                require(logits.all { it.isFinite() } && pooled.all { it.isFinite() })
                val scores = FloatArray(count) { logits[markers[it]] / temperature }
                val peak = scores.max()
                val weights = FloatArray(count) { exp((scores[it] - peak).toDouble()).toFloat() }
                val total = weights.sum()
                return FloatArray(count) { weights[it] / total }
            } finally {
                (inputs.values + outputs.values).forEach { it.close() }
            }
        }
    }
}

Validation

Reference: the official laya 0.3.4 predict on CPU fp32 at checkpoint revision 1c5edc17a7acd8701df6fc341c0d179f1c62c982, every question of a fixture in one call, the original builder budgets (English 512 / 192, typed-decisions 1024 / 256) and each checkpoint's own config temperatures. The fixtures are invented: 46 English fixtures (209 question rows) and 40 typed-decisions fixtures (200 rows, ten per upstream workflow). A row is evaluated at a window only when its original sequence fits it.

All 1,248 row executions on the desktop CPU and all 1,248 on the S26 GPU were finite. Argmax is identical on every choice and score row. Maximum |Δp| covers every option probability, the yes/no probability and act_probability against the reference four-decimal dictionaries; the limits are 1e-3 for fp32 and 1e-2 for wfp16.

Checkpoint	N	Storage	Desktop CPU rows	Argmax	Max Δp	S26 GPU rows	Argmax	Max Δp
English	256	wfp16	140	59/59	0.000633332539	140	59/59	0.000631127167
English	256	fp32	140	59/59	6.15482807e-05	140	59/59	4.9820447e-05
English	512	wfp16	209	87/87	0.000762185717	209	87/87	0.00076248374
English	512	fp32	209	87/87	6.15482807e-05	209	87/87	5.03234029e-05
typed-decisions	256	wfp16	100	65/65	0.000494368362	100	65/65	0.000495828676
typed-decisions	256	fp32	100	65/65	5.11034489e-05	100	65/65	5.15206814e-05
typed-decisions	512	wfp16	175	112/112	0.000494368362	175	112/112	0.000495828676
typed-decisions	512	fp32	175	112/112	5.11034489e-05	175	112/112	5.15206814e-05

Desktop CPU: ai-edge-litert 2.1.6 CompiledModel, four threads, Apple M4 Max, macOS 27.0. Phone: the validated configuration above. The phone runs consumed the captured ids and marker positions of the same rows and their raw outputs were decoded with the same Python arithmetic; the phone validation therefore covers the graphs, the table lookup and the act head, not a Kotlin tokenizer. The English S256 wfp16 graph also passed on the S26 CPU (XNNPACK, four threads; 140 rows, 59/59, max Δp 0.000633).

Every main-graph run and every act-head run compiled to one GPU partition. Verbatim from the PID-filtered logcat, one line per graph kind:

wfp16 main: Replacing 2223 out of 2223 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).
fp32 main:  Replacing 2100 out of 2100 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).
act:        Replacing 4 out of 4 node(s) with delegate (LITERT_CL) node, yielding 1 partitions for subgraph 0 (main).

No unsupported-operation line appeared. The S256 validation rows, with state, questions, ids, markers and reference answers, are in fixtures/gate_rows_en_s256.json (140 rows) and typed-decisions/fixtures/gate_rows_td_s256.json (100 rows).

Galaxy S26 timings

Measured on 2026-09-22 on the held phone, screen on, Android build S942QOPS1AZF2_SJP1AZF2, LiteRT 2.2.0, GPU FP32, one process per run. Main+act covers the input writes, run() and the output readback of both graphs; the host table lookup is separate. Cold is the first row of the run; warm is the median over the remaining rows. The eight GPU runs were executed back to back and the battery temperature rose from 34.5 °C to 44.1 °C over the sequence; the S512 runs and the later fp32 runs were measured warm, which is why their medians sit above their cold rows. These are observations from one device and one thermal sequence, not a latency benchmark.

Checkpoint	N	Storage	Compile ms	Cold main+act ms	Warm median [min, max] ms	Lookup median ms
English	256	wfp16	2711	121.2	122.9 [117.8, 126.8]	20.2
English	256	fp32	3491	124.4	164.0 [122.5, 209.0]	23.7
English	512	wfp16	2481	289.6	615.6 [322.7, 636.4]	28.3
English	512	fp32	3315	296.6	616.1 [295.2, 630.1]	31.5
typed-decisions	256	wfp16	2528	122.2	125.9 [117.8, 207.9]	19.7
typed-decisions	256	fp32	4081	238.1	266.6 [223.4, 291.0]	24.4
typed-decisions	512	wfp16	4664	631.4	618.5 [581.7, 632.9]	28.9
typed-decisions	512	fp32	5823	632.1	622.6 [604.6, 681.7]	30.2

For comparison on the same phone, the English S256 wfp16 graph on the CPU (XNNPACK, four threads) took 582.7 ms warm median [336.2, 588.1] per question.

Limits

• Questions with more than 20 options are outside what was validated.
• There is no S1024 graph. A row longer than 512 tokens after the checkpoint's own builder needs a shorter input; the graph window does not change the builder budget.
• act_probability was 1.0 on every reference row of both checkpoints. The act graph and its formula are unchanged; this saturation says nothing about escalation quality.
• The typed-decisions fixtures use the four upstream workflow question-id sets (agent trace observability, customer service, invoice processing, security incidents) with invented wording; laya 0.3.4 defines the id sets, not full question presets.
• Numerical agreement with the official implementation is not task accuracy for a new schema. Validate the question wording for the application.

Related packages

Laya-Multilingual-LiteRT holds the multilingual checkpoint in the same graph form and the Android sample app with a Kotlin tokenizer for that checkpoint. laya-LiteRT holds the English and multilingual checkpoints in the token-id form, with the fp32 table inside the graph.

License and attribution

Laya is by Convai Innovations, built on ModernBERT-large; the checkpoints and the host logic are Apache-2.0, as are ModernBERT, Transformers and tokenizers. NOTICE and licenses/ hold the retained texts. The tokenizer files and the two config files are exact bytes from the pinned checkpoint directories (the configs were renamed; their contents are unchanged). laya_host.py adapts the Laya 0.3.4 prompt builder and decoder; the conversion adds fixed windows, explicit attention and RoPE, a separate act graph and the host token lookup.