AutoResearch-multivision-depth10

AutoResearch-multivision-depth10 is a 131.0M parameter decoder-only Transformer trained from scratch on multivision.

This model is part of the AutoResearch project, which focuses on training, evaluating, and releasing efficient language models with reproducible research workflows.

Overview

This is a 10-layer decoder-only Transformer trained on the multivision dataset for 1.0 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 2.818170 (perplexity: 7.0527) on the held-out validation set.

References

Papers

NanoGPT / NanoChat architecture patterns

Datasets

Training: multivision
Tokenizer: multivision

Related Projects

karpathy/autoresearch

WANDB Run

https://wandb.ai/dustinsdelivery-delivery-ai/autoresearch/runs/w2knf77q

Highlights

Trained from scratch
131.0M parameters
Trained on 64.0M tokens (122 steps)
10-layer decoder-only Transformer with sliding window attention
RoPE positional encoding, RMSNorm, ReLU² activation
MuonAdamW optimizer (Muon for matrices, AdamW for embeddings)
Hugging Face Transformers compatible

Model Architecture

Property	Value
Architecture	Decoder-only Transformer
Parameters	131,035,188 (131.0M)
Layers	10
Hidden Size	640
Attention Heads	5
KV Heads	5
Head Dimension	128
Feed Forward Size	2560
Context Length	2048
Vocabulary Size	16,384
Positional Encoding	RoPE
Activation	ReLU²
Normalization	RMSNorm
Window Pattern	SSSL
Weight Tying	No

Training

This model was trained from scratch for 1.0 hours (3601s) of wall-clock training time.

Training Configuration

Setting	Value
Optimizer	MuonAdamW (Muon + AdamW)
Precision	torch.bfloat16
Learning Rate	0.04 (matrix) / 0.6 (embedding)
Weight Decay	0.2
Batch Size	4 × 2048 = 8,192 tokens/step
Gradient Accumulation	64 steps
Total Batch Size	524,288 tokens
Context Length	2048
Vocabulary	16,384 tokens (BPE)
LR Scheduler	Linear warmdown (50%)
Activation Checkpointing	Enabled

Hardware

GPU: NVIDIA GeForce RTX 4060 Ti
VRAM: 16.0 GB
Peak VRAM Used: 6.5 GB
MFU: 9.42%
Framework: PyTorch 2.9.1+cu128

Dataset

Name: multivision
Language: English

Preprocessing

Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.

Intended Use

This model is intended for:

Educational purposes and research
Text generation experiments
Studying small language model training dynamics

Not recommended for:

Production use or safety-critical applications
Tasks requiring factual accuracy

Evaluation

Results

Metric	Score
Validation BPB	2.818170
Perplexity	7.0527
Peak VRAM	6.5 GB
MFU	9.42%

Example Generations

Example 1

Prompt

Once upon a time,

Generation

Once upon a time, blue and white bus. 
. 
. 
. 

. 


. 


A man is on the side of a road. 
. 
. 

. 
a man and a woman is riding a wave on the ocean 
 
. 

. 
. 
a dog is holding a kite in the background. 
 
 
A man and her arm down the street

Example 2

Prompt

A lonely dragon

Generation

A lonely dragon kite flying over a lake. 
. 
. 

. 

. 

A young girl in a red shirt is going down the street. 
. 
. 

. 
. 
. 

A man is taking a picture of herself and a bus stop. 
. 
. 

. 
. 
A man on a street. 

a couple of 
.

Example 3

Prompt

The opposite of boy is

Generation

The opposite of boy is in the air. 
. 
. 
. 
A skateboarder is going down the ramp. 
. 
. 
. 

A man is holding a surfboard. 
. 

. 

. 
. 

A black and white board. 
a person riding on a skateboard. 
. 

. 

. 
A person is on a skateboarder on

Example 4

Prompt

The opposite of queen is

Generation

The opposite of queen is. 
. 
. 
. 
A man is going down the street. 

. 
. 
. 


. 


a man and a motorcycle is on the road 
 
 
 
 
. 

a man that is standing next to a woman and a hat is on a horse. 
. 
. 

A man in a suit and a white plate 
A

Example 5

Prompt

My name is

Generation

My name is next to a green and white bus. 
. 
. 
. 
A man is sitting on a boat in the background. 
. 
. 
. 

. 


A man sitting on the side of a bus. 
. 
. 
. 

. 
. 


. 
 
A man is being held. 
. 

. 

A group

Example 6

Prompt

2 + 2 is

Generation

2 + 2 is standing in the grass. 
. 
A 
A small room with a television and a window. 
. 
. 

. 
a 
A man and a toilet in the mirror 
 
a public transit bus. 
. 
 

a man is walking down a street next to a small bathroom 
 
. 
. 
. 
. 
. 
a 
.

Usage

import torch
import pickle
import json
from train import GPT, GPTConfig, Tokenizer

# Load config
with open('config.json', 'r') as f:
    config_dict = json.load(f)
config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})

# Load model
model = GPT(config)
state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
model.load_state_dict(state_dict)
model.eval()

# Load tokenizer
with open('tokenizer.pkl', 'rb') as f:
    tokenizer = pickle.load(f)

# Generate
prompt = 'Once upon a time, '
input_ids = tokenizer.encode(prompt)
x = torch.tensor([input_ids], dtype=torch.long)
with torch.no_grad():
    for _ in range(50):
        logits = model(x)
        probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        input_ids.append(next_token.item())
        x = torch.tensor([input_ids], dtype=torch.long)
print(tokenizer.decode(input_ids))

Repository Structure

model.pt                  # Model weights
config.json               # Model architecture config
dataset.txt               # Dataset name used for training
token_bytes.pt            # Token byte mappings
tokenizer.pkl             # Trained BPE tokenizer
tokenizer_config.json     # Tokenizer configuration
training_metrics.json     # Training metrics
README.md                 # This file

Limitations

Small model size limits language understanding and coherence
Trained on a single dataset (TinyStories) — limited domain
Fixed time budget training — not fully trained to convergence
No RLHF or safety alignment

Ethical Considerations

This is a research artifact, not a production model
The training data consists of synthetic stories (GPT-4 generated)
No harmful content filtering was applied
Intended for research and educational use only

Citation

@misc{autoresearch_multivision_depth10,
  title={AutoResearch-multivision-depth10},
  author={Dustin Loring},
  year={2026},
  howpublished={\url{https://huggingface.co/quik-models/cerulean-bee-140}}
}}

Version History

Version	Date	Notes
v1.0	2026-08-01	Initial release

Acknowledgements

Built with the AutoResearch training framework.

Thanks to:

Hugging Face
PyTorch
The creators of the TinyStories dataset
The open-source AI research community

License

This model is released under the MIT License unless otherwise specified.

Downloads last month: 8