litert-community/GLiFormer-Large-NER-LiteRT

GLiFormer Large v1 (NER) for LiteRT — one call, three windows

Run the entity-extraction path of knowledgator/gliformer-large-v1 (575.6M parameters, Apache-2.0) on a phone with Google LiteRT. The official weights were converted with LiteRT Torch; nothing in the math was approximated. The validated Android configuration is LiteRT 2.2.0 on a Samsung Galaxy S26 (SM-S942Q, SM8850, Android 16) with explicit GPU FP32 computation. On 70 English inputs the extracted spans match the official gliformer fp32 CPU implementation exactly (micro-F1 1.000 by label and character offsets) with every shipped file. Other Android GPU families and lower-memory phones have not been validated here.

The image is a rendering of the actual output of gliformer_large_ner_s128_wfp16.tflite through LiteRT CompiledModel for the sentence in assets/hero_output.json (fictional names). The scores are the values the model returned; it is not a screenshot.

What you get

One host API, three encoded-window sizes. The host runtime picks the smallest window that fits the text; you call one function and never chunk below the 512-token window.

Window N	Text words T	Graphs	Where they run on Android
128	48	one full graph	GPU (FP32 precision)
256	256	encoder graph + head graph	encoder on GPU, head on CPU
512	512	encoder graph + head graph	encoder on GPU, head on CPU

Why the split: the 128-token graph compiles fully on the ML Drift GPU delegate (4,149 operators, one partition). The longer windows do not — the checkpoint's word-level BiLSTM is unrolled for T steps, and unrolled heads of 6,286 operators or more crash the LiteRT 2.2.0 GPU compiler (a runtime defect reported with reproducers, not a memory limit: the head crashes at 570 MB RSS). The DeBERTa encoder alone (1,773 operators) compiles at every window, so the longer windows run the encoder on the GPU and the small head on the CPU. The numbers are exact either way; only the placement differs.

File	Bytes	Role	Runs on (Android)
gliformer_large_ner_s128_wfp16.tflite	706,994,752	s128 full graph, recommended	GPU, FP32 precision
gliformer_large_ner_s256_encoder_wfp16.tflite	706,535,440	s256 encoder, recommended	GPU, FP32 precision
gliformer_large_ner_s256_head_wfp16.tflite	53,467,408	s256 head, recommended	CPU
gliformer_large_ner_s512_encoder_wfp16.tflite	807,176,432	s512 encoder	GPU, FP32 precision
gliformer_large_ner_s512_head_wfp16.tflite	56,491,280	s512 head	CPU
gliformer_large_ner_s128_fp32.tflite	1,361,304,680	fp32 reference	GPU, FP32 precision
gliformer_large_ner_s256_encoder_fp32.tflite	1,310,464,872	fp32 reference	desktop CPU verified
gliformer_large_ner_s256_head_fp32.tflite	103,920,220	fp32 reference	CPU
gliformer_large_ner_s512_encoder_fp32.tflite	1,411,108,128	fp32 reference	desktop CPU verified
gliformer_large_ner_s512_head_fp32.tflite	107,042,396	fp32 reference	CPU
host_assets/word_embeddings_fp32.bin	524,320,768	token table [128008,1024] float32, default	host
host_assets/word_embeddings_fp16.bin	262,160,384	the same table as float16, upcast on the host	host

wfp16 files store the FULLY_CONNECTED weights as float16 with a DEQUANTIZE to float32; every activation and every other constant stays float32. The fp32 files are the same graphs with float32 weights and are the exact references. host_assets/ holds the token table (fp32 and fp16 — the fp16 table is upcast on the host and was gated separately), the exact tokenizer and config files from the pinned checkpoint, the per-window graph contracts, the Python host runtime and a worked example. android/ is the Kotlin host and sample app.

Memory is the cost of this model. On the Galaxy S26 the 707 MB s128 graph is 2.6 GB resident after loading and peaks at 4.5 GB while the GPU delegate compiles it; the first call after process start takes 5.0 s (compile 4.7 s) and 80 ms afterwards. The s256 split in one process is 4.6 GB resident. The s512 head alone is 4.7 GB resident on the CPU, so treat s256 as the practical top window in an app and chunk longer documents by sentence (the runtime ships a helper). Plan for flagship-class phones.

Minimal usage

Python — complete pipeline, desktop CPU

Install requirements-lock.txt into a Python 3.12 environment and run from the downloaded repository. The runtime imports the pinned gliformer / gliner packages only for the upstream decoder; no checkpoint download and no encoder construction (HF_HUB_OFFLINE=1 is set by the script).

python examples/run_example.py            # picks s128 for the packaged sentence
python examples/run_example.py --seq 256  # forces the encoder + head split

import sys
sys.path.insert(0, "host_assets")
from runtime import extract

labels = ["person", "organization", "location", "product", "date"]
text = ("Mira Okafor, the founder of Halden Robotics, unveiled the Atlas Pro headset "
        "in Lisbon on 3 March 2025.")
for e in extract(text, labels, threshold=0.5):
    print(e["label"], repr(e["text"]), e["start"], e["end"], f"{e['score']:.4f}")
# person 'Mira Okafor' 0 11 1.0
# organization 'Halden Robotics' 28 43 0.9999
# product 'Atlas Pro' 58 67 0.8688
# location 'Lisbon' 79 85 1.0
# date '3 March 2025' 89 101 1.0

extract tokenizes with the official processor, builds the float routing tensors, looks the token rows up in the table, runs one graph (s128) or two (s256/s512) through the LiteRT CompiledModel Python API, and decodes the start/end/inside logits with the unchanged upstream pairing decoder. Character offsets index the original Python string; end is exclusive. Five distinct labels in a fixed order are required by the static class axis; the measured gates use the five above.

Kotlin — Android GPU with explicit FP32 (s128 graph)

Use implementation("com.google.ai.edge.litert:litert:2.2.0"). Stage the model file in your application's private model directory, keep one Environment for the process and run the whole GPU lifetime on the same worker thread. The six input buffers follow the signature order inputs_embeds [1,128,1024], attention_mask [1,128], text_routing [1,48,128], parent_routing [1,1,128], label_routing [1,5,128], text_mask [1,48] (resolve the names against the loaded signature rather than sorting them); HOST_CONTRACT.md says how to fill them from the tokenizer output and the embedding table.

import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel
import com.google.ai.edge.litert.Environment
import java.io.File

// inputs = the six float32 tensors of HOST_CONTRACT.md in signature order (N = 128, T = 48).
fun runNerGraph(env: Environment, modelDir: File, inputs: List<FloatArray>): FloatArray {
    val options = CompiledModel.Options(setOf(Accelerator.GPU)).apply {
        gpuOptions = CompiledModel.GpuOptions(
            precision = CompiledModel.GpuOptions.Precision.FP32)   // default precision returns no entities
    }
    CompiledModel.create(File(modelDir, "gliformer_large_ner_s128_wfp16.tflite").path,
        options, env).use { model ->
        val inBufs = model.createInputBuffers()
        val outBufs = model.createOutputBuffers()
        try {
            inputs.forEachIndexed { i, x -> inBufs[i].writeFloat(x) }
            model.run(inBufs, outBufs)
            return outBufs[0].readFloat()   // 48 words × 5 labels × 3 (start, end, inside) = 720 floats
        } finally {
            inBufs.forEach { it.close() }
            outBufs.forEach { it.close() }
        }
    }
}

// logit(word w, label c, k) with k = 0 start, 1 end, 2 inside:
fun logit(packed: FloatArray, w: Int, c: Int, k: Int) = packed[(w * 5 + c) * 3 + k]

The Kotlin block covers the on-device graph and the output layout; it propagates GPU failures and has no implicit CPU retry. A complete Kotlin host is in android/: whitespace word splitter, SentencePiece Unigram tokenizer, memory-mapped fp16 table lookup with upcast, routing construction and a float32 port of the pairing decoder, inside a Jetpack Compose sample app (type text, tap Extract, see highlighted entities with score and offsets; GPU or CPU; s128 as one graph, s256 as encoder + head; one window resident; the whole pipeline is warmed 12 times before Ready). On the Galaxy S26 it returns the official fp32 spans on 60/60 (s128 GPU), 60/60 (s128 CPU) and 65/65 (s256 split) validation inputs, with the on-device tokenizer output identical to the captured Python inputs on all 80; graph readback medians 116 / 189 / 526 ms; process RSS 2.4 GB (s128) and 4.4 GB (s256 pair). In a non-debuggable build the first Extract tap after Ready took 168 ms and the fifth 155 ms (screen on). Build and install steps are in android/README.md. For s256/s512 create two CompiledModels — the encoder with Accelerator.GPU and FP32 precision, the head with Accelerator.CPU — and pass the encoder's [1,1,N,1024] output buffer to the head as its first input.

Host contract

HOST_CONTRACT.md is the complete specification. In short:

1. Build the encoded sequence exactly as the upstream gliformer NER processor does: [SCHEMA] parent token, the five [ENTITY] label pairs, the separator, then the text words (first sub-token pooling). Pad the token ids on the right with id 0 to N.
2. inputs_embeds = rows of the token table for every position, padding included. attention_mask is 1 for real tokens; text_routing has one 1 per text word at its first sub-token; parent_routing marks the [SCHEMA] position; label_routing has one 1 per label at its [ENTITY] marker; text_mask is 1 for real text words.
3. The output is one float32 tensor [1,1,T,15] = start/end/inside logits for every (word, label). For the split windows the encoder returns [1,1,N,1024] hidden states and the head takes them together with the routing tensors.
4. Feed the logits to the upstream decoder (host_assets/runtime), which returns label, text, character start/end and score per entity (threshold 0.5, flat spans).

Long documents: the graphs score one window at a time; chunk_by_sentences(text, max_words) in the runtime is a caller-side helper (offsets are local to each chunk).

Measured quality and performance

Reference = the official gliformer 0.1.2 fp32 CPU implementation at checkpoint revision d0a4e53d, same tokenizer, threshold 0.5. 70 English inputs (60 sentences, 10 paragraphs of 140–480 encoded tokens) with the fixed five-label schema, each at the smallest window it fits (60 / 5 / 5). Micro-F1 counts a span as correct only when label, start and end all match. This is agreement with the official implementation, not a human-labelled accuracy estimate.

Window	Files	Where	Operators on GPU	Exact span sets	Max score drift	Median ms
128	wfp16	GPU FP32	4149/4149, one partition	10/10	2.4e-4	82.3
128	fp32	GPU FP32	3996/3996, one partition	10/10	1.3e-6	82.8
256	wfp16 encoder + wfp16 head	GPU FP32 + CPU	1773/1773 encoder, one partition	15/15	2.4e-4	175.6 + 167.0 (separate processes)
256	wfp16 encoder + fp32 head	GPU FP32 + CPU, one process	1773/1773 encoder	15/15	1.9e-4	410.2 end to end
512	wfp16 encoder + wfp16 head	GPU FP32 + CPU	1773/1773 encoder, one partition	20/20	2.4e-4	931.5 + 348.9 (separate processes)

Cold start of the s128 wfp16 graph in a fresh native process: 5,023 ms from process start to the first result (compile 4,708 ms); the second call 80.2 ms.

Desktop LiteRT CPU (ai-edge-litert 2.1.6, macOS arm64, 4 threads): F1 1.000 and 70/70 identical span sets for both storages; the fp32 graphs match the PyTorch forward with a maximum absolute logit difference of 6.3e-5 (s128) and 1.3e-4 (s256, s512).

Latency is the median of 5 timed runs per input after 2 warm-ups, from input write to the end of output readback with the native CompiledModel C API, one process, phone idle, battery 30–41 °C. It is a single-device sample, not a benchmark across devices or thermal states. The s512 encoder number was taken at 41 °C.

Configuration (Galaxy S26, LiteRT 2.2.0)	Resident after load	Peak during compile
s128 wfp16, GPU	2,591,805,440 B	4,542,996,480 B
s128 fp32, GPU	3,239,837,696 B	5,165,268,992 B
s256 encoder wfp16, GPU	2,121,412,608 B	3,633,397,760 B
s256 head wfp16, CPU	2,413,006,848 B	2,396,839,936 B
s256 encoder GPU + fp32 head CPU, one process	4,571,271,168 B	4,573,499,392 B
s512 encoder wfp16, GPU	2,325,684,224 B	4,035,690,496 B
s512 head wfp16, CPU	4,676,505,600 B	4,593,754,112 B

The native graph processes exclude the token table (262 MB fp16 or 524 MB fp32 when resident), the tokenizer and the decoder. An fp32 head does not reduce the CPU-side memory (4.73 GB vs 4.68 GB at s512, 8 % faster), so the fp16-weight heads stay the default; HostRuntime(head_storage="fp32") selects the references.

Provenance, conversion and license

• Source: knowledgator/gliformer-large-v1, revision d0a4e53d09cebe6bc963dd9be319d4279084bb2d (gliformer-layout, backbone layout-deberta, 24 layers, hidden 1024, 16 heads), loaded with gliformer 0.1.2 (GitHub b5c0a0fd), gliner 0.2.29 and transformers 5.16.1. Only the NER path is converted; classification, relation, structuring and embedding heads are not included.
• Conversion: litert-torch 0.9.3 (torch 2.12.1), fixed shapes, fp32. The text path was re-expressed for the GPU delegate without changing its math: host-side token lookup, one-hot routing as matmul, float masks, attention at rank 4, the exact DeBERTa logarithmic relative-position buckets as projected tables, the word BiLSTM unrolled for the window, one packed output tensor. The layout and page embeddings of the backbone are skipped exactly as the upstream text path skips them.
• Weight storage: ai-edge-quantizer 0.8.0 float16 FLOAT_CASTING on the FULLY_CONNECTED weights only (wfp16 files). Dynamic-range INT8 was not attempted: it does not compile on the LiteRT 2.2.0 GPU delegate for this graph family.
• Verification: LiteRT CompiledModel Python API on desktop CPU; native CompiledModel C API on the Galaxy S26 for GPU and CPU. Every gate is exact span match plus absolute error; correlation was never used.
• Complete pins: requirements-lock.txt.

License: the checkpoint is Apache-2.0 and these converted files are released under the same license (LICENSE). The gliformer, gliner and transformers code used by the host runtime is Apache-2.0 and the DeBERTa lineage is MIT (licenses/). Upstream: Knowledgator/GLiFormer.