muskuskus/baseline

adaptive-moeut

A dense language model trained on filtered C4 (step 100000).

Custom architecture — not a transformers class. To load:

import torch
from safetensors.torch import load_file
from models.dense_transformer import DenseTransformer

model = DenseTransformer()
model.load_state_dict(load_file('model.safetensors'), strict=False)

(strict=False because the tied lm_head weight is deduped on upload.)

Tokenizer: SentencePiece BPE, 8k vocab (c4_bpe8k.model).