nanoMoE
A small causal language model based on a custom Mixture-of-Experts (MoE) Transformer architecture.
Architecture
The model uses token-level Top-K expert routing.
The original PyTorch implementation is provided in model.py.
Model configuration
{
"vocab_size": 50257,
"block_size": 256,
"d": 256,
"n_layer": 8,
"n_head": 8,
"n_exp": 4,
"top_k": 2,
"dropout": 0.1
}
Tokenizer
The model uses the pretrained GPT-2 tokenizer:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
Checkpoints
ck_XXXXXX.pt: periodic training checkpoints.best_ck_XXXXXX.pt: best validation checkpoints.last_ck_XXXXXX.pt: last evaluation checkpoint.
Checkpoint files contain the model state dictionary, configuration, and training step. Evaluation checkpoints also include validation metrics and training history.
Loading the model
Install dependencies:
pip install torch transformers huggingface_hub
Load a checkpoint:
import importlib.util
import sys
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
repo_id = "ejdis/nanoMoE"
# Download the architecture
source = hf_hub_download(
repo_id=repo_id,
filename="model.py"
)
# Download the checkpoint
path = hf_hub_download(
repo_id=repo_id,
filename="best_ck_010000.pt"
)
# Import the custom model
spec = importlib.util.spec_from_file_location(
"nanomoe_model", source
)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module) # Only execute trusted source code
# Load weights
checkpoint = torch.load(
path, map_location="cpu", weights_only=True
)
model = module.GPTMoE(**checkpoint["model_config"])
model.load_state_dict(checkpoint["model"], strict=True)
model.eval()
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
# Example inference
inputs = tokenizer(
"Once upon a time",
return_tensors="pt"
)["input_ids"]
with torch.no_grad():
logits = model(inputs)[0]
print(logits.shape)
Notes
This model uses a custom PyTorch architecture and cannot
be loaded directly using AutoModelForCausalLM.from_pretrained().
The repository preserves the architecture and weights. Exact reproduction of training additionally requires the original dataset, preprocessing, random seed and optimizer state, which are not contained in these checkpoints.