joeygambino/joyai-echo-multishot-patch

JoyAI-Echo GGUF nodes — multishot fixes + automation patch

A set of bug fixes and features layered on top of the community ComfyUI_JoyAI_Echo_GGUF_Nodes pack (the Rebels GGUF loader stack around JoyAI-Echo). Everything here targets the multi-shot path (JoyEcho_Generate

• the discrete Rebels loaders / JoyEcho_ModelLoader).

This is a patch drop, not a standalone pack: copy these files over a working install of the same pack (back up first). The files are interdependent — in particular nodes.py calls new signatures added to the two libs/ files, so apply them together.

Tested on an RTX 5090 (32 GB) and a 3090 (24 GB), ComfyUI 0.26–0.27, torch 2.8–2.11, with the JoyAI-Echo bf16 release and self-built Q8 GGUFs.

---

Files in this package

nodes.py                                     # JoyEcho_TextEncode / _Generate / _ModelLoader / _LLMEnhance
__init__.py                                  # registrations for the new nodes
rebels_loaders.py                            # discrete GGUF loaders (text-encoder fixes)
joyecho_prompt_source.py   (new node)        # one dropdown: .txt briefs + .json scripts
joyecho_ref_picker.py      (new node)        # auto reference-image picker by character name
joyecho_ref_batch.py       (new node)        # None-tolerant image batcher
joyecho_script_picker.py   (new node)        # JSON dropdown (superseded by PromptSource)
libs/ltx_distillation/models/ltx_wrapper.py  # fp8 quantization passthrough
libs/ltx_core/loader/fuse_loras.py           # kohya-LoRA fusion + alpha scaling + fuse telemetry
libs/ltx_distillation/utils.py               # tiled VAE decode
prompts/long_story_writer_system_prompt.md   # (optional) de-musicked + character-age edits

---

Bug fixes

1. enable_audio_memory=False silently disabled ALL cross-shot memory

The pack computed audio_memory_latent=None when audio memory was off, and the video memory-bank save was gated on that latent being non-None — so with audio memory off (the standard anti-drone setting) the bank never filled and cross-shot identity silently died (symptom: memory_size=0 every shot even with memory_max_size=7; a new face each shot). Fix: memory storage is now unconditional; enable_audio_memory gates only the audio-memory injection path. Verify: console memory_size= should climb 0,1,2,… capped at your memory_max_size. (nodes.py)

2. GGUF text-encoder loader (RebelsJE_TextEncoder)

Two fixes so a text-only Gemma-3 GGUF loads cleanly:

• meta-strip: drop vision_tower / multi_modal_projector / lm_head (the text-only GGUF has no weights for them → "Cannot copy out of meta tensor").
• device-unify: pin the embeddings-processor to the encoder's actual device (GGUF Gemma runs on CPU while the connector was on cuda → addmm device mismatch). (rebels_loaders.py)

---

Features

3. Split per-domain negative lever (JoyEcho_TextEncode)

The DMD pipeline has no CFG, so the only steering lever is embedding-space. Instead of one negative_prompt/negative_scale that steers both branches, this splits it:

• negative_prompt_video / negative_scale_video — kills burned-in captions/subtitles. Working value ~0.5. Above ~0.8 it over-rotates the video context and locks every shot to shot 1's composition (scene-lock).
• negative_prompt_audio / negative_scale_audio — kills invented music/score. Keep ≤ ~0.4 or dialogue suffers. Steering is norm-preserving (RescaleCFG-style): cond' = renorm(cond + s*(cond − neg)). Old single-widget names still work as a fallback. (nodes.py)

4. Passthrough mode (JoyEcho_LLMEnhance)

mode = "passthrough (raw JSON, skip LLM)" — feed a finished {"prompts":[...]} script straight through with no LLM call / no API key. Auto-detects when story_idea already parses as that JSON. (nodes.py)

5. Reference-image conditioning — I2V-as-reference (JoyEcho_Generate)

New reference_image (IMAGE batch, up to 4). Identity references are prepended as video-only conditioning clips at the memory-encode step — they are never written into the paired audio/video bank. (An earlier attempt that seeded refs into the bank with zero-filled audio latents injected loud background noise with 2+ refs; video-only conditioning avoids it entirely.) Also new: head_trim_frames (auto 8 with refs) drops the first N frames of each shot, where the model morphs out of the reference/memory content. The trim is applied once right after decode, so the final output, the per-shot preview files, and any external concat of them stay frame-identical. (nodes.py)

6. Shot transitions (JoyEcho_Generate)

transition: cut (original) / dissolve (overlap cross-dissolve + equal-power audio crossfade) / vhs_glitch (analog static burst at each boundary: snow, tear bands, dropout lines + a raised-cosine tape-noise audio bed). transition_frames, glitch_intensity tune it. (nodes.py)

7. fp8 transformer quantization (JoyEcho_ModelLoader)

New fp8_transformer toggle. Quantizes the DiT's attention/FF linear weights to float8_e4m3fn at load, from the normal bf16 checkpoint (uses the vendored ltx_core.quantization.QuantizationPolicy.fp8_cast() — upcasts per-layer at inference). Roughly halves DiT weight memory and halves sequential-offload PCIe traffic; keeps memory training + all tensors; VAEs/text-encoder/non-linears stay bf16. Ignored when a GGUF DiT is selected (already quantized). (nodes.py + libs/ltx_distillation/models/ltx_wrapper.py — new quantization param; the quantized build path skips the post-load dtype cast that would otherwise silently upcast fp8 back to bf16.)

8. Tiled VAE decode (JoyEcho_Generate)

Decoding a long high-res shot (e.g. 241f @ 1280×736) in one pass hard-aborts the VAE decode on a 24–32 GB card (fatal cuDNN abort mid-conv, not a catchable OOM). New decode_tiling (auto/on/off) routes decode through the vendored VideoDecoder.tiled_decode — temporal-only 64-frame chunks with 24-frame blended overlap (no spatial tiles → no spatial seams), streaming each chunk to CPU. auto engages only above a size threshold, so small renders keep the original single-pass decode bit-for-bit. (nodes.py + libs/ltx_distillation/utils.py — decode_benchmark_sample gains a video_tiling_config kwarg + _decode_video_tiled_uint8.)

9. Model dropdown (JoyEcho_ModelLoader)

New model_file combo lists every .safetensors / .gguf under the ComfyUI checkpoints / diffusion_models / unet dirs. Pick a .safetensors → full checkpoint (replaces checkpoint_path); pick a .gguf → DiT loaded from GGUF while checkpoint_path still supplies the VAEs / vocoder / text connectors. "(use checkpoint_path)" keeps the old typed-path behavior. A matching lora_file dropdown lists every .safetensors under models/loras (applied at lora_strength on the safetensors DiT path; ignored for GGUF). Plus a clear early error if gemma_path is a .gguf/file/sidecar-less dir (this loader needs the HF gemma-3-12b-it folder; GGUF Gemma only works via RebelsJE_TextEncoder). (nodes.py)

10. LoRA loading hardening (JoyEcho_ModelLoader + libs/.../fuse_loras.py)

• A lora_file dropdown picks LoRAs from models/loras (existing lora_strength widget applies).
• Fusion now supports kohya naming (lora_down/lora_up) in addition to PEFT (lora_A/lora_B), with standard alpha/rank scaling — previously a kohya-named LoRA silently did NOTHING (zero keys matched, no warning).
• Fusion prints how many weights fused, and WARNS LOUDLY when a provided LoRA matched zero keys.
• The loader refuses ComfyUI-quantized checkpoints (.comfy_quant marker tensors, e.g. "fp8mixed learned" builds) with a clear error: this loader never applies their weight scales (the model would silently load mis-scaled) and LoRA fusion on them crashes with shape errors. Use bf16 checkpoints.

11. Automation / batching nodes (new)

• JoyEcho_PromptSource — one dropdown listing LPFF-style .txt briefs (from the inspire-pack prompts tree) and passthrough .json scripts (input/joyecho_prompts/). Multi-block briefs fan out like LoadPromptsFromFile. Emits story_idea (→ LLMEnhance) + character (→ RefPicker) + count. Replaces the LPFF→UnzipPrompt chain and lets you switch prompt sources with one dropdown instead of rewiring.
• JoyEcho_RefPicker — auto-selects a character reference image from a folder tree keyed by character name (a character_pick dropdown of the folder names, a typed/wired character string, or a prompt scan — dialogue mentions are stripped so only the on-screen subject wins). The dropdown survives model refreshes, an explicitly named character that matches no folder refuses to fall back to the prompt scan (a wiped/typo'd name can't silently become the wrong character's face), and the cache signature includes the prompt text (without it, ComfyUI could serve a cached pick from a previous queue item). on_no_match=no_reference returns nothing so a batch keeps running.
• JoyEcho_RefBatch — None-tolerant image batcher: combines up to 4 optional IMAGE inputs (e.g. two RefPickers for a two-character shot), skips missing refs, resizes mismatched sizes to the first image, outputs None if all are missing (Generate then just skips identity seeding). The stock KJNodes ImageBatchMulti crashes with 'NoneType' has no attribute 'shape' on a missing ref; this replaces it.
• JoyEcho_ScriptPicker — JSON dropdown (superseded by PromptSource; kept for compatibility).

---

Applying

1. Back up your existing pack folder.
2. Copy each file over the same relative path in ComfyUI/custom_nodes/ComfyUI_JoyAI_Echo_GGUF_Nodes/.
3. Restart ComfyUI. New widgets append at the end of existing nodes, so saved graphs keep their values; the four new nodes appear under the JoyAI-Echo category. Press R after adding model files to refresh the model_file dropdown.

The libs/ files must match the vendored ltx_core / ltx_distillation in your pack (same JoyAI-Echo release). If your libs/ differ substantially, cherry-pick the changes described above rather than overwriting.

Not included (intentionally): model weights, the gemma_assets/ tokenizer binaries, .bak snapshots, and __pycache__.