mrjackspade

/

Ideogram4-Natural-Language-Text-Encoder

like 14

I was able to use loras after patching with Claude using this script

"""Convert LoRAs to ComfyUI text-encoder LoRA key naming.

Source keys : layers.{N}.{path}.lora_a / .lora_b (rank-major / out-major)
Target keys : text_encoders.qwen3vl_8b.transformer.model.layers.{N}.{path}.lora_A.weight.lora_B.weight

Rationale: comfy/lora.py:model_lora_keys_clip builds the key map as "text_encoders." + <te state_dict key without .weight>, and the Ideogram4 TE exposes qwen3vl_8b.transformer.model.layers.N... . comfy/weight_adapter/lora.py only recognises lora_A/lora_B (capitals), lora_up/lora_down, etc.
"""
import sys
from safetensors.torch import load_file, save_file

PREFIX = "text_encoders.qwen3vl_8b.transformer.model."
SUFFIX = {".lora_a": ".lora_A.weight", ".lora_b": ".lora_B.weight"}


def convert(src, dst):
    sd = load_file(src)
    out = {}
    for k, v in sd.items():
        for old, new in SUFFIX.items():
            if k.endswith(old):
                out[PREFIX + k[: -len(old)] + new] = v
                break
        else:
            raise SystemExit(f"unexpected key: {k}")
    save_file(out, dst, metadata={"format": "pt"})
    print(f"{len(out)} keys -> {dst}")
    print("sample:", next(iter(out)))


if __name__ == "__main__":
    convert(sys.argv[1], sys.argv[2])

Ive run the model with turbo version on several prompts, here is examples if you're curious:
https://github.com/novmikvis/ideogram-4-prompt-adherence-test

Here is what I've noticed :

• it definitely reduces gray box failure mode (triggers less with small amount of text)
• Lightly reduced prompt adherence and overall coherence (see image with set of icons: plasters became pill blister packs)
• Produces more of a wide-angle shot in realistic scenes

Also here are some additional notes from Claude:

Merged checkpoints may be losing more of the delta than the LoRA path

While comparing the merged encoder against base + LoRA, they read:
manifests/merge_step_00001000.json. scale_policy is
preserve_stock_per_tensor_scale, and the per-projection metrics in that same file
show the delta taking a reduction. Across all 252 projections:

157 of 252 projections land below 0.75 cosine; the weakest is
layers.0.self_attn.k_proj at 0.553 cosine / 0.724 retention.

That looks like a consequence of keeping the base model's per-tensor FP8 scales: the
adapted weights no longer fit the range those scales were fitted for. ComfyUI's runtime
LoRA path does preserves them — comfy/ops.py convert_weight dequantises, the delta is
added in a higher-precision compute dtype, and set_weight requantises with
scale="recalculate" plus stochastic rounding. So applying the adapter at runtime may
well preserve more of what you trained than the pre-merged file does.

Might be worth re-merging with recalculated scales and comparing — if the metrics in
the manifest are computed the way I'm reading them, there could be some free quality
sitting there.