Recursive Division Tree (RDT) Language Model with Topological Adam Optimizer
Author: Steven Reid License: Apache 2.0
- Summary
This project combines two new ideas:
Recursive Division Tree (RDT) Embeddings – a mathematical rule that organizes each word in the vocabulary into logarithmic shells. Words with similar depth values belong to the same shell, producing a constant-depth, fractal-like structure in the embedding space.
Topological Adam Optimizer – a physics-inspired optimizer that extends Adam with two internal fields (alpha, beta) coupled by an energy equation. This allows parameters to evolve smoothly while maintaining energy balance, reducing oscillations and improving stability.
Together they form an experimental language model that learns relationships between words and can generate new short sentences.
- Key Concepts
Component Description
Recursive Depth (rdt_depth) Maps each index n to a depth using a log-logarithmic rule, creating nested shells of similar complexity. Shell-based Embeddings Each word receives an embedding vector stored in a shell according to its depth. Energy-based Optimization The Topological Adam optimizer updates weights using both gradient information and internal field coupling. Hybrid Training Objective Combines geometric embedding similarity with simple statistical (n-gram) learning. Text Generation Generates continuations from prompt text by combining learned bigram and trigram statistics with geometric similarity across shells.
- Running the Model
Requirements
torch numpy topological-adam
Quick Start
from rdt_language_model import RDTLanguageModel from topological_adam import TopologicalAdam import torch, random
prepare training data
corpus, vocab = generate_corpus()
initialize model
model = RDTLanguageModel(vocab, embedding_dim=128, alpha=1.5)
train
model.train_rdt(corpus, epochs=10, lr=1e-3)
generate text
print(model.generate("the man walked"))
- Example Output
Prompt: "the man walked" the man walked to the river and he entered slowly and she was sad was
- Results
Optimizer Accuracy (MNIST test) Notes
Adam 91.39 % Baseline RMSProp 88.35 % Topological Adam 91.60 % Stable energy and strong robustness
The optimizer achieved the same or better accuracy while maintaining smooth energy dynamics during training.
- Why It Matters
Shows that recursive geometric organization can replace random embedding initialization.
Demonstrates a new optimizer family based on topological energy coupling.
Suggests that language structure and physical field dynamics can be unified in a single computational model.
- Repository Contents
rdt_language_model_topological_adam.ipynb # full Colab notebook topological_adam.py # optimizer implementation requirements.txt README.md
- Citation
If you use this work, please cite:
Reid, S. (2025). Recursive Division Tree Language Model with Topological Adam Optimizer.
Available at https://huggingface.co//rdt-language-model-topological-adam