tiny-random/lfm3-omni

This is a randomly initialized tiny model based on LiquidAI/LFM3-Omni-2B for testing and debugging.

Feature	Source	This repo
Checkpoint tensor elements	2.24B BF16, 1.28K F32	13.91M BF16, 1.28K F32
Safetensors weight files	4.48 GB (2 files)	27.88 MB (2 files)
Text layers / cycle	24 / two-conv prefix then full/conv/sliding/conv	10 / same prefix and two complete cycles
Text hidden / MLP width	2,048 / 7,680	64 / 128
Text Q / KV heads / head dim	16 / 4 / 128	16 / 4 / 32
Audio encoder layers / width	17 / 768	6 / 64 (two conv/conv/full cycles)
Audio mel bins / stride / window	128 / 8 / 32 frames	Same
Depth decoder layers / width	3 / 1,024	3 / 32
Codebooks / codebook size / maskgit	12 / 2,017 / [4, 4, 4]	Same
Vision layers / width / Q-KV heads	14 / 1,024 / 16-8	6 / 128 / 4-2
Patch / temporal patch / merge	16 / 2 / 2	Same
Projector MLP hidden / downsample	4,096 / 2	128 / 2
Codec sample rate / frame samples	24,000 / 1,920	Same
Codec STFT n_fft / hop / frame stack	1,280 / 320 / 6	Same
Codec FSQ levels / codebooks	[8, 7, 6, 6] / 12	Same
Codec LFM3 layers / width	8 + 8 / 512	4 + 4 / 64
Vocabulary rows / tying	128,000 / tied	Same
Stored precision	BF16; codec ISTFT window FP32	Same

Text and vision geometry follows tiny-random/lfm3-vl. The pinned upstream implementation is included for Transformers loading, including the separate audio_tokenizer/ LiquidCodec component; the only runtime change lets the codec LFM3 backbone dimensions come from its config (source defaults preserved). The 16 kHz feature extractor and 24 kHz codec STFT/FSQ contracts are unchanged. This repo provides a Transformers example with image, speech in and speech out; no serving-backend validation is claimed.

File path	Size
audio_tokenizer/model.safetensors	1.0MB
model.safetensors	26.9MB

Example usage

Transformers (✅ verified)

Load the included pinned upstream implementation with trust_remote_code=True. The example sends a 16 kHz spoken question together with an image and four predecoded video frames, answers as text, then synthesizes 24 kHz speech through the bundled LiquidCodec. The Triton short-conv path requires CUDA.

def run_transformers(model_id: str):
    import numpy as np
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    model = (
        AutoModelForImageTextToText.from_pretrained(
            model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
        )
        .eval()
        .cuda()
    )
    codec = processor.audio_tokenizer.to(device=model.device).eval()

    image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
    frames = np.stack(
        [
            np.asarray(Image.new("RGB", (256, 256), color))
            for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
        ]
    )
    sample_rate = 16000
    speech = (np.sin(2 * np.pi * 440 * np.arange(sample_rate) / sample_rate) * 0.1).astype(np.float32)
    conversation = [
        {"role": "system", "content": "Respond in text."},
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "audio", "audio": speech},
                {"type": "video", "video": frames},
                {"type": "text", "text": "Transcribe the question, describe the image and the video."},
            ],
        },
    ]
    inputs = processor.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        sampling_rate=sample_rate,
        processor_kwargs={"video_metadata": [{"total_num_frames": 4, "fps": 2, "frames_indices": [0, 1, 2, 3]}]},
    ).to(model.device)
    assert (
        inputs["input_features"].shape[-1] > 0
        and inputs["pixel_values"].shape[0] > 0
        and inputs["pixel_values_videos"].shape[0] > 0
    )
    with torch.no_grad():
        answered = model.generate(**inputs, max_new_tokens=32, generation_mode="text")
    text = processor.decode(
        answered.sequences[0, inputs["input_ids"].shape[1] : answered.sequence_lengths[0]],
        skip_special_tokens=True,
    )
    print(text)

    tts_conversation = [
        {"role": "system", "content": "Perform TTS in American English using a female voice."},
        {"role": "user", "content": "The quick brown fox jumps over the lazy dog."},
    ]
    tts_inputs = processor.apply_chat_template(
        tts_conversation,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(model.device)
    with torch.no_grad():
        spoken = model.generate(
            **tts_inputs,
            max_new_tokens=64,
            generation_mode="audio",
            audio_temperature=0.4,
            audio_repetition_penalty=1.2,
        )
        assert spoken.audio_codes.shape[-1] > 0
        waveform = codec.decode(spoken.audio_codes[:1, :, : spoken.audio_lengths[0]]).audio_values[0]
    assert waveform.numel() > 0 and torch.isfinite(waveform).all()
    print(waveform.shape)


if __name__ == "__main__":
    run_transformers(model_id="scratch/lfm3_omni")

Creating this repo

• Code: .meta/create_model.py
• Model structure: .meta/model.log
• Environment: .meta/env.json

Change log

• 2026-11-03: Initial version. Text backbone follows tiny-random/lfm3-vl (residual 64 / MLP 128, ten layers with the two-conv prefix and two full/conv/sliding/conv cycles, 16 Q / 4 KV heads at head dim 32); vision keeps the source 2:1 Q/KV ratio at width 128 (4 Q / 2 KV heads, head dim 32) with six conv/full-cadence layers. The 16 kHz causal audio encoder keeps 128 mel bins, stride-8 convolutions, the 32-frame window and a two-group conv/conv/full schedule at width 64; the 12-codebook maskgit depth decoder keeps the three full-attention passes and 2,017-entry codebooks at width 32. The bundled 24 kHz LiquidCodec keeps its STFT contract (1,280 FFT, hop 320, frame stack 6), 12 FSQ codebooks [8, 7, 6, 6] and FP32 synthesis window, with a four-conv/attention-layer LFM3 stack at width 64 via a pinned, localized runtime config patch.

This repo is generated by GPT-6.1-Sol.