rhenretta/ai-visualizer

AI Music Visualizer (1280×720 Direct-RGB Neural Visualizer, v7)

A compact 972,895-parameter (1.95 MB FP16 weights, 5.5 MB .mlpackage) recurrent neural music visualizer that synthesizes finished 1280×720 RGB frames at 92.7 FPS (10.79 ms/frame) directly from live 48 kHz audio and disentangled plain-English style prompts (shape, motion, color) with zero GPU post-processing shaders.

• GitHub Repository: https://github.com/rhenretta/ai-visualizer
• Standalone macOS App (AIVisualizer.app): Native Swift 6 + CoreML FP16 + Metal + CoreAudio Global Speaker Tap + embedded llama.cpp conversational editor

---

Included Files

File / Folder	Size	Description
VisualizerStep.mlpackage/	5.5 MB	Native 1280×720 FP16 CoreML recurrent frame generator (972,895 params)
TextConditioner.mlpackage/	45.0 MB	Frozen all-MiniLM-L6-v2 + 3 $S^{127}$ attention-pooling projection heads (z_shape, z_motion, z_color)
best_model.pt	11.2 MB	PyTorch checkpoint (unet_state_dict, shape_head, motion_head, color_head, and scaler)
audio_contract.json	5 KB	Causal 4096-FFT log2 (35 Hz – 10,000 Hz) acoustic parameters & fitted feature scaler
mel_filterbank_64x2049.f32	516 KB	64 × 2049 float32 musical log2 octave filterbank (8 bins/octave)
hann_window_4096.f32	16 KB	Periodic 4096-sample float32 Hann window
minilm_vocab.txt	228 KB	WordPiece vocabulary for TextConditioner
seed_states.json	60 KB	Curated 5-axis visualizer seed preset deck
export_report.json	1 KB	Latency (10.79 ms, 92.7 FPS) & PyTorch-to-CoreML parity (MAE = 0.000275) report

---

Architecture Overview

1. Balanced Musical log2 Octave + Pink-Noise dB Spectrum (48 kHz → 60 Hz, N_FFT = 4096, hop = 800):
   • 64 log2-spaced frequency bins from 35 Hz to 10,000 Hz (8 bins/octave, 11.7 Hz sub-bass bin spacing) with +4.3 dB/oct pink-noise tilt compensation and a 44 dB causal decibel window, plus 8 onset/beat features (sub-bass, kick flux, mid energy, hi-hat transient, spectral centroid, RMS, and $\sin/\cos$ beat phase).
   • Stacked across 480 causal frames (8.0s) into a 3-tier pyramid ([1, 78, 72]: T1 = 0..1s @ 60 Hz, T2 = 1..4s @ 4 Hz, T3 = 4..8s @ 1.5 Hz).
2. Coherent Compositional 1280×720 Direct-RGB Network (VisualizerStep):
   • Monotonic rotational and outward wave phase integrators (channels 0..3 of prev_hidden, conditioned strictly on (audio_emb, z_motion) for 0.0 shape/color leakage) + 12-channel 160×90 ConvGRU state.
   • Steers 6 differentiable visualizer primitives (LinearFFTBars + falling peak-hold caps, RadialFFTRing, SpiralWaveEmitter with 32-tap 2.0s causal $E(t - r/v)$ outward shockwaves, OscilloscopeRibbons, PerspectiveTunnelGrid, and MandalaSpirograph) and upsamples via learned PixelShuffle stages to [1, 3, 720, 1280].
   • Ghost-free live SLERP transitions via structural trail-envelope gating (compat_gate * trail_env_90) and causal transition horizon attenuation.

---

Quick Usage (CoreML on macOS)

import coremltools as ct
import numpy as np

step_model = ct.models.MLModel("VisualizerStep.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)
text_model = ct.models.MLModel("TextConditioner.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_GPU)

out = step_model.predict({
    "prev_frames_spaced": np.zeros((1, 15, 90, 160), dtype=np.float32),
    "prev_hidden": np.zeros((1, 16, 90, 160), dtype=np.float32),
    "audio_pyramid": np.zeros((1, 78, 72), dtype=np.float32),
    "raw_mel": np.zeros((1, 64), dtype=np.float32),
    "z_shape": np.ones((1, 128), dtype=np.float32) / np.sqrt(128.0),
    "z_motion": np.ones((1, 128), dtype=np.float32) / np.sqrt(128.0),
    "z_color": np.ones((1, 128), dtype=np.float32) / np.sqrt(128.0),
})
frame_rgb = out["frame_rgb"]      # (1, 3, 720, 1280) float32 in [0, 1]
next_hidden = out["next_hidden"]  # (1, 16, 90, 160) float32