jegly/neo_dragon

Neodragon — LiteRT (.tflite) conversions for on-device text-to-video

LiteRT conversions of Qualcomm-AI-Research/Neodragon, prepared for Box — an open-source, fully offline on-device AI app for Android.

These are converted artifacts only. The models, research and original weights are Qualcomm AI Research's work; see the upstream repo and paper.

Generates a 320×512 clip entirely on-device — no network, no server, no account.

Licence — please read before use

The weights are released by Qualcomm under two licences, and both carry over to these conversions:

1. BSD 3-Clause Clear — permits redistribution and derivative works (a conversion is a derivative) with notices retained. Note the "Clear" variant grants no patent licence.
2. Qualcomm Responsible AI License — adds use restrictions. It is not a grant licence; it sits on top of the BSD grant and flows down to you. Prohibited uses include military use, criminal/predictive-policing applications, social scoring, biometric categorisation, emotion recognition in workplaces or education, and distributing generated content without disclosing that it is machine generated. Read the full text before using these files.

Box surfaces these restrictions in a consent screen before download and marks generated output as AI-generated.

What's here

Fixed-shape LiteRT graphs. Filenames are deliberately flat — Box's downloader flattens remote paths into a single directory.

Video pipeline (picture → video)

file	size	shape
dt5_text_encoder.tflite	495 MB	(1,128) ids + mask → (1,128,4096)
context_adapter_dynamic_int8.tflite	124 MB	(1,128,4096) → (1,128,1536)
clip_l_text_encoder.tflite	472 MB	(1,77) → pooled 768
clip_g_text_encoder_int8.tflite	670 MB	(1,77) → pooled 1280
dit_stage0_10x16_int4.tflite	816 MB	pyramid stage 0
dit_stage1_20x32_int4.tflite	817 MB	pyramid stage 1
dit_stage2_40x64_int4.tflite	822 MB	pyramid stage 2
causal_video_vae_encoder_320x512.tflite	419 MB	(1,3,1,320,512) → (1,16,1,40,64)
causal_video_vae_decoder.tflite	38 MB	(1,16,2,40,64) → (1,3,9,320,512)

The three DiT graphs are the same 1.5B weights at three input resolutions; the pipeline calls one per pyramid stage. Each accepts exactly two conditioning latents, which is what limits this set to 9-frame clips.

First-frame pipeline (text → picture, SSD1B at 640×1024)

file	size	shape
ssd_1b_clip_l.tflite	443 MB	(1,77) → (1,77,768) hidden_states[-2]
ssd_1b_clip_g.tflite	2.65 GB	(1,77) → (1,1280) + (1,77,1280)
ssd_1b_unet_640x1024.tflite	5.07 GB	(1,4,80,128) + t + ehs + text_embeds + time_ids
ssd_1b_vae_decoder_640x1024.tflite	189 MB	(1,4,80,128) → (1,3,640,1024)

Tokenizer assets

file	note
clip_vocab.json, clip_merges.txt	CLIP BPE, shared by all four CLIP encoders
t5_spiece.model	DT5 unigram SentencePiece

clip_merges.txt retains its #version: 0.2 header line — skip it when loading, or every merge rank shifts by one and tokenization silently corrupts.

Notes for anyone reusing these

• CPU only. The LiteRT GPU delegate supports at most rank-4 tensors, and Neodragon's latents are 5D (b,c,t,h,w). On a Tensor G5, --gpu places 256 of 3402 ops on the GPU and then segfaults (RESHAPE: Tensor dimensions must be less than 5). This is structural, not a configuration problem.
• Two different CLIP pairs. clip_* are Neodragon's own encoders (for the DiT's pooled vector); ssd_1b_clip_* are SSD1B's (for SDXL conditioning). Their weights differ. The tokenizer vocabularies are identical, so one tokenizer serves both.
• SDXL uses hidden_states[-2], not last_hidden_state, from both CLIP encoders. Using the last layer does not crash — it quietly degrades output.
• The SSD1B UNet is fp32 because int8 quantisation fails in ai_edge_quantizer 0.8.0 on one layer (CrossAttnDownBlock2D_2/ResnetBlock2D_0/conv2: "input operand has more dimensions than allowed by the axis remapping").
• Measured on a Pixel 10 Pro Fold (Tensor G5), CPU: DiT stages ≈ 1.1 / 3.9 / 17.3 s per call.

Credit

All credit for the model and research to Qualcomm AI Research — Animesh Karnewar and the Neodragon authors. These conversions exist only to make that work runnable on more devices.