litert-community/GLiNER2.5-Small-LiteRT

GLiNER2.5 Small for LiteRT — Android GPU FP32

Run GLiNER2.5 Small entity extraction on a phone GPU with the official fastino/gliner2.5-small-v1 weights converted through Google LiteRT Torch. The validated Android configuration is LiteRT 2.2.0 with explicit GPU FP32 computation on a Samsung Galaxy S26 (SM-S942Q, SM8850, Android 16). On 70 English inputs the extracted spans match the official gliner2 fp32 CPU implementation exactly (micro-F1 1.000 by label and character offsets) at every shipped window. Other Android GPU families have not been validated here.

The image shows the actual output of gliner25_small_s128_wfp16.tflite on LiteRT CompiledModel for the sentence in host_assets/example.json. The confidences are the values the model returned. It is a rendering of model output, not a screenshot.

Files and supported configuration

Three encoded-window sizes are shipped. Pick the smallest window that fits your text: the schema prompt (five labels) plus the text must fit in N encoded tokens and the text must have at most T words.

File	Window N / text words T	Packed output floats	Bytes	Role
gliner25_small_s128_wfp16.tflite	128 / 48	57,758	54,051,424	recommended
gliner25_small_s256_wfp16.tflite	256 / 192	217,310	63,906,288	recommended
gliner25_small_s512_wfp16.tflite	512 / 384	430,046	84,111,584	recommended
gliner25_small_s128_fp32.tflite	128 / 48	57,758	98,018,428	fp32 reference
gliner25_small_s256_fp32.tflite	256 / 192	217,310	107,873,296	fp32 reference
gliner25_small_s512_fp32.tflite	512 / 384	430,046	128,078,592	fp32 reference

wfp16 files store the 96 FULLY_CONNECTED weight tensors as float16 with a DEQUANTIZE to float32; every activation and every other constant stays float32. The fp32 files are the same graphs with float32 weights.

Every graph is the dense part of the upstream BoundaryExtractor: the DeBERTa-v3-xsmall encoder, the boundary encoder, the boundary query head and the per-token projections. The graph takes the word-embedding rows as input and returns one packed float32 tensor with 17 logical outputs. The host does the token embedding lookup before the graph and the upstream sparse candidate pooling, scoring and span decoding after it. host_assets/ holds everything the host side needs:

File	Purpose
word_embeddings_fp32.bin	[128011,384] float32 row-major table, 196,624,896 B, shared by all windows
sparse_decoder_fp32.safetensors	the 16 upstream sparse-decoder tensors (465,924 B of weights)
tokenizer.json, tokenizer_config.json, config.json, encoder_config/config.json	exact files from the pinned checkpoint
graph_contract_s{128,256,512}.json	input shapes and the offset of every logical output slice
runtime/	the Python host runtime: prompt construction, routing, unpacking, upstream decoding
example.json	the worked sentence with its expected spans

FP32 model storage and GPU computation precision are separate settings. With the runtime's default GPU precision the encoder output is NaN from the first logical output onward; with explicit FP32 computation all gates pass. Use the explicit option below. Dynamic-range INT8 variants were built and evaluated but do not compile on LiteRT 2.2.0 CompiledModel GPU (ML Drift rejects the quantized FULLY_CONNECTED operators with "Unable to parse bc coord for BATCH axis"), so no INT8 file is published. NPU execution was not evaluated. The graphs handle one English text and the fixed label set person, organization, location, product, date; other schemas, other languages and texts longer than one window are outside what was validated.

Minimal usage

Python — complete pipeline, desktop CPU

Install requirements-lock.txt into a Python 3.12 environment and run from the downloaded repository. The host runtime imports the pinned gliner2 2.0.0 package for the sparse decoder; no checkpoint download is needed (HF_HUB_OFFLINE=1 is set by the script).

python examples/run_example.py --model gliner25_small_s128_wfp16.tflite

The script does, in order: build the encoded inputs for the fixed label set (HostRuntime.prepare), run the graph through the LiteRT CompiledModel Python API, and decode the packed output with the upstream sparse decoder (HostRuntime.decode).

import json, os, sys
from pathlib import Path
import numpy as np
os.environ["HF_HUB_OFFLINE"] = "1"
sys.path.insert(0, str(Path("host_assets/runtime").resolve()))
from host_runtime import HostRuntime
from ai_edge_litert.compiled_model import CompiledModel, HardwareAccelerator, Options, CpuOptions

host = HostRuntime(Path("host_assets"))
text = "Maya Chen from Orvane Robotics demonstrated the Veltrix 9 in Lisbon on March 12, 2025."
inputs, metadata = host.prepare(text, seq=128)   # 5 float32 tensors, args_0..args_4

model = CompiledModel.from_file("gliner25_small_s128_wfp16.tflite", options=Options(
    hardware_accelerators=HardwareAccelerator.CPU, cpu_options=CpuOptions(num_threads=4)))
ins, outs = model.create_input_buffers(0), model.create_output_buffers(0)
for buf, x in zip(ins, inputs):            # signature order is args_0..args_4
    buf.write(np.ascontiguousarray(x.numpy(), dtype=np.float32))
model.run_by_index(0, ins, outs)
packed = outs[0].read(57758, np.float32).reshape(1, 1, 1, 57758)   # s128 packed length
print(json.dumps(host.decode(metadata, packed, inputs), indent=2))
# {"entities": {"person": [{"text": "Maya Chen", "start": 0, "end": 9, "confidence": 0.9995}], ...}}

Kotlin — Android GPU with explicit FP32

Use implementation("com.google.ai.edge.litert:litert:2.2.0"). Stage the model file in your application's private model directory. Keep one Environment for the process and run the whole GPU lifetime on the same worker thread. The five input buffers follow the signature order args_0 … args_4 = inputs_embeds [1,N,384], attention_mask [1,N], text_routing [1,T,N], query_routing [1,5,N], text_mask [1,T]; HOST_CONTRACT.md says how to fill them from the tokenizer output and the embedding table.

import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel
import com.google.ai.edge.litert.Environment
import java.io.File

// embeds/mask/textRouting/queryRouting/textMask are the five float32 inputs
// built exactly as HOST_CONTRACT.md describes (N = 128, T = 48 for this file).
fun runDensePrefix(env: Environment, modelDir: File, inputs: List<FloatArray>): FloatArray {
    val options = CompiledModel.Options(setOf(Accelerator.GPU)).apply {
        gpuOptions = CompiledModel.GpuOptions(
            precision = CompiledModel.GpuOptions.Precision.FP32)   // default precision returns NaN
    }
    CompiledModel.create(File(modelDir, "gliner25_small_s128_wfp16.tflite").path,
        options, env).use { model ->
        val inBufs = model.createInputBuffers()
        val outBufs = model.createOutputBuffers()
        try {
            inputs.forEachIndexed { i, x -> inBufs[i].writeFloat(x) }   // args_0 .. args_4
            model.run(inBufs, outBufs)
            return outBufs[0].readFloat()   // 57,758 floats; includes completion/readback
        } finally {
            inBufs.forEach { it.close() }
            outBufs.forEach { it.close() }
        }
    }
}

// Unpack one logical slice with the offsets in graph_contract_s128.json,
// e.g. start_logits [1,5,49] starts at float 26624:
fun startLogits(packed: FloatArray): Array<FloatArray> =
    Array(5) { q -> packed.copyOfRange(26624 + q * 49, 26624 + (q + 1) * 49) }

The sparse candidate pooling, pair scoring and span decoding that turn the 17 logical outputs into labelled character spans are shipped as the Python host runtime (host_assets/runtime/host_decoder.py, which calls the upstream gliner2 functions). The Kotlin block above covers the on-device graph and the unpacking contract; it propagates GPU failures and has no implicit CPU retry.

A complete Kotlin host is in android/: the word splitter and SentencePiece Unigram tokenizer, input construction, the memory-mapped embedding lookup and a float32 port of the sparse decoder, inside a Jetpack Compose sample app (type text, tap Extract, see highlighted entities with confidence and offsets). On the Galaxy S26 it returns the same spans as the official fp32 model on all 70 validation texts, on GPU FP32 and on CPU. Build and install steps are in android/README.md.

Host contract

HOST_CONTRACT.md is the complete specification. In short:

1. Build the encoded sequence exactly as the upstream gliner2 processor does: schema tokens for the five labels, [SEP_TEXT], then the text tokens. Pad the token ids on the right with id 0 to N.
2. inputs_embeds = rows of word_embeddings_fp32.bin for every position, padding included. attention_mask is 1 for real tokens. text_routing has one 1 per text word at its first sub-word position; query_routing has one 1 per label at its marker position; text_mask is 1 for real text words.
3. The packed output is one float32 tensor [1,1,1,1108*T+4574]. The 17 logical slices (text_states, query_states, boundary_states, start_logits, end_logits, inside_logits, … count_log_rates) are listed with offset and shape in graph_contract_s*.json; unpacking is slicing and reshaping only.
4. Feed the slices to the upstream sparse decoder (host_assets/runtime), which returns label, text, character start/end and confidence per span.

Long documents: the upstream library chunks long text on the host (extract_long) and remaps spans; the graphs here score one window at a time.

Measured quality and performance

Reference = the official gliner2 2.0.0 fp32 CPU implementation at checkpoint revision f1e4d8fd, same tokenizer, default threshold. 70 English inputs (50 sentences, 10 short sentences, 10 paragraphs of 90–170 words) with the fixed five-label schema; every input runs at every window it fits (60 / 65 / 70 inputs → 195 window-input pairs). Micro-F1 counts a span as correct only when label, start and end all match.

Window	Variant	S26 GPU FP32 span F1	Max confidence drift (GPU)	GPU operators	S26 GPU median ms
128	wfp16	1.000 (60/60 inputs identical)	2.7e-3	1120/1120, one partition	12.5
256	wfp16	1.000 (65/65)	2.7e-3	1121/1121, one partition	27.5
512	wfp16	1.000 (70/70)	2.7e-3	1121/1121, one partition	121.1
128	fp32	1.000 (60/60)	3.3e-6	1024/1024, one partition	12.4
256	fp32	1.000 (65/65)	3.6e-7	1025/1025, one partition	26.4
512	fp32	1.000 (70/70)	2.9e-6	1025/1025, one partition	91.1

Desktop LiteRT CPU (ai-edge-litert 2.1.6, macOS arm64, 4 threads) gives the same F1 = 1.000 for every file; the fp32 files match the PyTorch forward with a maximum absolute output difference of 3.1e-5 (s128), 1.5e-4 (s256) and 2.8e-4 (s512) over all 17 logical outputs.

Latency is the median of 5 timed runs per input after 2 warm-up runs, measured from run to the end of output readback with the native CompiledModel C API, LiteRT 2.2.0, GPU with explicit FP32 computation, one process, phone idle and between 34 °C and 38 °C battery temperature. It is a single-device sample, not a benchmark across devices or thermal states.

Provenance, conversion and license

• Source: fastino/gliner2.5-small-v1, revision f1e4d8fdd6fe328f45dee6aca3e6a07c9db4296e (architecture BoundaryExtractor, encoder microsoft/deberta-v3-xsmall), loaded with gliner2 2.0.0 and transformers 4.57.6.
• Conversion: litert-torch 0.9.3 (torch 2.12.1), fixed shapes, fp32. The dense prefix was re-expressed for the GPU delegate without changing its math: host-side embedding lookup, one-hot routing as matmul, boolean masks as float arithmetic, prefix sums as constant triangular matmul, attention kept at rank 4, the exact DeBERTa logarithmic relative-position buckets and the 128-word boundary window baked as constants, and one packed output tensor. GELU is the native LiteRT GELU.
• Weight storage: ai-edge-quantizer 0.8.0 float16 FLOAT_CASTING on the FULLY_CONNECTED weights only (wfp16 files).
• Verification: LiteRT CompiledModel Python API on desktop CPU; native CompiledModel C API on the Galaxy S26 for CPU and GPU. Correlation was never used as a gate; every gate is exact span match plus absolute error.
• Complete pins: requirements-lock.txt.

License: the checkpoint is Apache-2.0 and these converted files are released under the same license (LICENSE). The DeBERTa-v3 encoder is MIT (licenses/DeBERTa-MIT.txt); the gliner2 and transformers code used by the host runtime is Apache-2.0 (licenses/). Upstream: the GLiNER2 repository and the paper arXiv:2507.18546.