ddmmbb/Resonance-Bottleneck-LLM

🚀 [WIP] D2-V15 Resonance-Memory LLM

Language: English / 繁體中文 Status: Work In Progress (Training phase)

---

🌟 Introduction | 專案簡介

[EN] The D2-V15 Resonance-Memory LLM is an experimental Large Language Model focusing on a custom architecture: Resonance Memory Attention. Unlike standard Transformers that treat all context equally, this model is designed to learn how to forget and when to resonate. By utilizing Adaptive Decay and Interference Gating, it aims to process long context more efficiently by explicitly filtering out noise and keeping only crucial information.

[TW] D2-V15 共振記憶大模型 是一個實驗性質的語言模型，核心為自定義的 Resonance Memory Attention (共振記憶注意力) 架構。有別於標準 Transformer，本模型旨在讓 AI 學會「主動遺忘」與「共振增強」。透過自適應衰減 (Adaptive Decay) 與干涉門控 (Interference Gating)，模型能更有效地處理長文本，主動過濾雜訊並保留關鍵資訊。

---

🧠 Model Architecture | 模型架構細節

(Please refer to the architecture diagram below | 請參考下方架構圖)

[EN] The model follows a Pre-LayerNorm Transformer block structure (left panel), utilizing a GELU MLP for feed-forward networks. The core innovation lies in the Multi-Head Resonance Memory Attention sub-module (right panel):

1. Feature Transformation: Standard Q, K, V are transformed using ELU(x) + 1.0 to ensure strictly positive values, a prerequisite for linear-time memory accumulation.
2. Resonance Bottleneck & Gating: Input x passes through a Sequential Bottleneck to generate amplitude, phase, and raw decay parameters. These feed into an Interference Equation (outputting a sigmoid gate) that dictates how strongly new information should resonate with the existing state.
3. Parallel Decay Scan: The true engine of the model. It uses cumulative sums (cumsum) to apply an exponential decay factor across the sequence in parallel, smoothly updating the memory states (kv and z). This enables the model to actively "forget" irrelevant tokens over time.

[TW] 本模型採用 Pre-LayerNorm 架構（左圖），前饋網路使用 GELU MLP。最大的創新在於 Multi-Head Resonance Memory Attention 子模組（右圖）：

1. 特徵轉換 (Feature Transformation): 標準的 Q, K, V 經過 ELU(x) + 1.0 轉換，確保數值恆正，這是線性時間記憶累積的必要條件。
2. 共振瓶頸與門控 (Bottleneck & Gating): 輸入 x 通過 Sequential Bottleneck 生成振幅、相位與原始衰減參數。這些參數會進入干涉方程式並輸出 Sigmoid Gate，決定新資訊與當前記憶的「共振強度」。
3. 並行衰減掃描 (Parallel Decay Scan): 模型的記憶引擎。透過累加運算 (cumsum) 在序列上並行套用指數衰減因子，平滑地更新記憶狀態 (kv 與 z)。這賦予了模型隨時間「主動遺忘」無關 Token 的強大能力。

---

📈 The "Step 24k" Breakthrough | Step 24k 的頓悟時刻

[EN] During training, the model experienced a significant breakthrough around Step 24,000. Initially, the model struggled to differentiate between crucial tokens and noise due to unclean data, causing the Interference Gate to stall around 0.35 with high Entropy. After rigorously cleaning the training corpus (removing meaningless symbols) and applying a Learning Rate Warm Restart, the model achieved Gate Polarization. The gates confidently pushed towards 0 and 1, causing the Information Entropy to plummet (from 0.34 to <0.30). This mathematically proves that the Resonance Mechanism successfully "woke up" and learned to decisively filter context!

[TW] 在訓練過程中，模型在大約 Step 24,000 迎來了重大的突破。 早期由於訓練語料中夾雜無意義的符號雜訊，模型無法判斷哪些資訊該記憶，導致門控活躍度卡在 0.35，且資訊熵 (Entropy) 居高不下。 在清理訓練語料雜訊並搭配學習率熱重啟後，模型發生了完美的**門控極化 (Gate Polarization)**！門控開始果斷地往 0 與 1 兩極移動，導致資訊熵瞬間暴跌 (從 0.34 跌破 0.30)。這在數學上證明了「共振記憶機制」已被成功喚醒，模型終於學會果斷地取捨上下文！

💻 Quick Start | 快速測試 (WIP)

import torch
from ResonanceBottleneckLLM import D2V15Model

# Initialization
vocab_size = 16384
d_model = 768
n_layers = 12
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load Model
model = D2V15Model(vocab_size, d_model, n_layers).to(device)
ckpt = torch.load("d2_v15_resonance_plus.pth", map_location=device)
model.load_state_dict(ckpt['model_state_dict'])
model.eval()

print("Model successfully loaded! Generation script coming soon...")