nanoMoE

A small causal language model based on a custom Mixture-of-Experts (MoE) Transformer architecture.

Architecture

The model uses token-level Top-K expert routing. The original PyTorch implementation is provided in model.py.

Model configuration

{
  "vocab_size": 50257,
  "block_size": 256,
  "d": 256,
  "n_layer": 8,
  "n_head": 8,
  "n_exp": 4,
  "top_k": 2,
  "dropout": 0.1
}

Tokenizer

The model uses the pretrained GPT-2 tokenizer:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

Checkpoints

  • ck_XXXXXX.pt: periodic training checkpoints.
  • best_ck_XXXXXX.pt: best validation checkpoints.
  • last_ck_XXXXXX.pt: last evaluation checkpoint.

Checkpoint files contain the model state dictionary, configuration, and training step. Evaluation checkpoints also include validation metrics and training history.

Loading the model

Install dependencies:

pip install torch transformers huggingface_hub

Load a checkpoint:

import importlib.util
import sys
import torch

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

repo_id = "ejdis/nanoMoE"

# Download the architecture
source = hf_hub_download(
    repo_id=repo_id,
    filename="model.py"
)

# Download the checkpoint
path = hf_hub_download(
    repo_id=repo_id,
    filename="best_ck_010000.pt"
)

# Import the custom model
spec = importlib.util.spec_from_file_location(
    "nanomoe_model", source
)

module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)  # Only execute trusted source code

# Load weights
checkpoint = torch.load(
    path, map_location="cpu", weights_only=True
)

model = module.GPTMoE(**checkpoint["model_config"])
model.load_state_dict(checkpoint["model"], strict=True)
model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Example inference
inputs = tokenizer(
    "Once upon a time",
    return_tensors="pt"
)["input_ids"]

with torch.no_grad():
    logits = model(inputs)[0]

print(logits.shape)

Notes

This model uses a custom PyTorch architecture and cannot be loaded directly using AutoModelForCausalLM.from_pretrained().

The repository preserves the architecture and weights. Exact reproduction of training additionally requires the original dataset, preprocessing, random seed and optimizer state, which are not contained in these checkpoints.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support