Tiny Scene
Text → animated 3D scene in three.js with generated music, driven by a tiny Transformer trained from scratch that runs entirely in the browser.
Inspired by WVY-Blender-5m: same idea (a small model translates a sentence into a compact scene language, and procedural code builds the result), but the builder is three.js instead of Blender. No code or weights from that repository are reused.
"a red fox jumping over a blue car in the desert at sunset"
│ model (4.85 M parameters, ~0.3 s on CPU, plain JavaScript)
▼
<new> loc=desert time=sunset <ent> type=fox color=red act=jump rel=over <ent> type=car color=blue
│ interpreter + procedural builder (three.js, no custom shaders)
│ + procedural band (Tone.js): chords, bass, drums and melody chosen from the scene
▼
real-time animated scene with music, editable with more sentences
Try it
Download the repository and serve the web/ folder with any static server, for example:
python3 -m http.server 8080 --directory web
Open http://localhost:8080. Type in English or Spanish:
| Kind | Examples |
|---|---|
| New scene | an alien riding a horse on the moon · tres pingüinos caminando en la nieve mientras nieva |
| Edit | make the fox green · add a giant tree · remove the car · make it rain · zoom out · orbit the camera |
| Music | a robot dancing in the city with fast electronic music · play sad jazz music · make the music faster · no drums · stop the music |
| Ask | what can you do? · how do i enable shadows · what objects do you know |
Music and sound
Every scene gets a soundtrack, generated live in the browser. Nothing is recorded: there are no audio files.
- Chosen from your words: mood (happy, sad, calm, tense, epic, spooky, playful, dreamy), style (chiptune, electro, ambient, rock, jazz, orchestral, lofi) and tempo (slow, medium, fast).
- Or from the scene when you say nothing: a storm is tense, a ghost is spooky, space is dreamy and ambient, a robot gets electro, a knight gets an orchestra, running speeds it up.
- Editable:
make the music faster,louder,no drums,add the bass back,stop the music, or the Música section of the Editar tab. - In sync: dancing, bouncing and jumping follow the beat (
slowlyis half time,fastdouble time). - Ambience: rain, wind, waves, night crickets, thunder timed to the lightning, and engine hums that are positioned in 3D for moving vehicles.
- Sonido mutes everything. Grabar vídeo records the view with its sound to a
.webmfile (up to 60 s).
Browsers only allow sound after a click or key press, so the music starts with your first interaction.
Hand editing
Click an object in the view (or in the list under the Editar tab) to select it, then:
- Drag it in the view to move it. Riders, and objects that orbit, jump over or chase it, move with it.
- Change its type, action, size, count, speed, relation and colour in the Editar tab, or Duplicar / Borrar it.
- Add new objects, and change place, time, weather, shot and camera move, from the same tab.
- Typed edits keep working, and "it" refers to the selection: select the car and type
make it yellow. Ctrl+Z/Ctrl+Shift+Zundo and redo every change, typed or by hand.Deleteremoves the selection,Escclears it.
Hand edits change the same scene description the model writes, so both can be mixed freely.
Drag the background to orbit the view. Export .glb saves the current frame (opens in Blender); animations are procedural and are not exported.
The page also accepts ?q=first prompt;second prompt&t=3 to run prompts on load and freeze time at that second.
The interface labels are in Spanish; the model understands both languages.
What it can do
| Objects (54) | animals, characters, vehicles and props: dog, dragon, robot, astronaut, car, rocket, UFO, tree, tower… |
| Actions (17) | idle, walk, run, jump, spin, dance, fly, float, sit, sleep, wave, bounce, roll, circle, shake, swim, drive |
| Relations (8) | on, over, near, behind, front, around, ride, chase |
| Places (15) | field, desert, snow, beach, forest, city, space, moon, ocean, room, mountain, farm, road, park, studio |
| Time and weather | day, night, sunset, dawn · clear, rain, snow, fog, storm |
| Camera | shots close, medium, wide, top, low · moves static, orbit, zoomin, zoomout, pan, follow |
| Attributes | 14 colours, 4 sizes (plus "bigger / smaller"), count 2–6, speed |
| Music | 8 moods, 7 styles, 3 tempos, faster/slower, louder/quieter, drums/bass/melody on or off |
| Questions | 25 topics about three.js and the app itself; the model picks language and topic, and the answer is stored in web/model/qa.json |
How it works
ml/ lexicon.py scene-language vocabulary and the English and Spanish words for it
gen.py generator of (sentence → tokens) pairs, with typos and dropped accents
model.py decoder-only Transformer (6 layers, d=256, 8 heads, context 208)
train.py trains on pairs generated on the fly and exports to web/model/
infer.py command-line inference
web/ js/nn/model.js the same Transformer in plain JavaScript, with a KV cache
js/audio/theory.js scene → mood/style/tempo, and the notes of an 8-bar loop (pure JS)
js/audio/engine.js Tone.js playback, ambience, engine sounds and video recording
js/editor.js the hand-editing panel
js/lang/scenelang.js tokens → scene JSON, and applying edits
js/scene/ creatures.js, things.js (objects), animate.js (poses),
layout.js (placement and relations), environment.js, stage.js
tests/ check_scene.mjs builds every object × action and place × time × weather combination
check_model.mjs JS ↔ PyTorch parity and hand-written prompts
check_export.mjs .glb export in a real browser
check_editor.mjs click, drag, panel edits, undo/redo and delete in a real browser
check_music.mjs note generation, scene rules, and real playback, beat clock and video export in a browser
- Input: the sentence is split into characters (50 symbols), which makes it tolerant to typos. Output: fixed scene-language tokens. Total vocabulary: 267.
- Music: the model only picks labels (
<mus> mood=sad genre=jazz tempo=fast).theory.jsturns them into a scale, a four-chord progression and seeded patterns for four layers, so the same scene always gets the same tune. - Data: there is no dataset on disk.
gen.pyproduces endless random pairs from templates; every batch is new. - Weights:
web/model/weights.binis 9.7 MB in float16 (used by the browser).ml/model.ptis the float32 PyTorch checkpoint.
Results
Included model: 4,860,672 parameters, 16,000 steps with batches of 256 (4.1 M pairs), 22 minutes on an RTX 5060 Ti.
| Test | Result |
|---|---|
| New generator sentences, clean (3,000) | 99.90 % exact outputs |
| New generator sentences, with typos (3,000) | 99.50 % |
Hand-written sentences outside the templates (76, tests/natural.json) |
75 exact |
| JavaScript vs PyTorch (300 sentences) | 300 identical |
| Time per sentence in JavaScript (Node, one thread) | ~245 ms |
| Scenes built without errors (object × action, place × time × weather, relations) | 1,478 of 1,478 |
| Music loops generated and validated (mood × style × seed) | 168 of 168 |
| Styles that produce sound in a real browser | 7 of 7 |
The first two rows measure sentences with the same grammar as training, so they are optimistic. The third is the most honest, but it is small, and two early failures (a wide shot of…, an object mentioned after the place) led to adding templates and retraining. The 16 music sentences in it were written before the music model was trained; 15 were exact on the first run.
The audio tests check that sound comes out, that the beat clock matches the tempo and that the video has an audio track. They do not judge how good the music sounds.
Retrain
uv venv .venv && uv pip install --python .venv/bin/python torch numpy
npm install
npm run train
npm test
To add an object: an entry in _TYPES in ml/lexicon.py, a builder in web/js/scene/things.js or creatures.js, then retrain. npm test fails if the vocabulary names something three.js cannot build.
Limitations
- It only understands the vocabulary above. Anything unknown is replaced by the closest match or ignored (
a little girl flying a kitegives a small woman flying). - It does not generate new geometry: every object is hand-modelled from primitives.
- It does not compose: the music is rule-based loops of eight bars with synthesised instruments. It is coherent but simple and repetitive, and there are no named instruments (
piano musicis not understood). - The exported video is recorded in real time from the screen, so its quality depends on the machine.
- Typed edits refer to objects by type (
the dog), not by colour or position; with two different dogs, both change. Select one and useit, or edit it by hand. - Hand-moved objects keep an offset from where the automatic layout puts them, so adding or removing other objects can still shift them.
- A place has no colour (
in a blue houseis read as a dog and a blue house). - Known miss:
something calmer, maybe ambientsets the music correctly but also adds a man. - Long sentences with three objects and many modifiers are where it fails most.
Credits
three.js and Tone.js (both MIT license) are bundled in web/vendor/; their licenses are in web/vendor/LICENSE and web/vendor/LICENSE-tone.md.
