Model5

Model5 is a ~529M parameter Llama-style decoder-only transformer, implemented and trained from scratch (no AutoModel base — architecture, pretraining loop, and SFT loop are all custom code), then converted to a standard HuggingFace LlamaForCausalLM checkpoint for distribution. It has been instruction fine-tuned (SFT) on top of a pretrained base (general instruction-following only — see Tool / function calling).

⚠️ Early checkpoint, not a finished model. Pretraining was stopped early after only ~1B tokens, far short of the ~10B tokens (roughly Chinchilla-optimal) originally targeted for a model this size. SFT was run to completion on top of that undertrained base. Expect noticeably weaker coherence, factuality, and instruction-following than a fully pretrained model of this size — see Limitations below.

Model Details

Architecture	Llama-style decoder-only transformer
Parameters	~529M
hidden_size (dim)	1280
num_hidden_layers	26
num_attention_heads	20
num_key_value_heads	5 (grouped-query attention, 4x compression)
intermediate_size (SwiGLU)	3584
max_position_embeddings	2048
vocab_size	50,257 base (gpt2) + 5 chat special tokens
Positional encoding	RoPE (rotate-half convention), theta=10000
Normalization	RMSNorm (pre-norm, computed in fp32)
Activation	SwiGLU
Embeddings	Tied input/output (lm_head shares weights with embed_tokens)
License	MIT

The architecture intentionally matches HuggingFace's LlamaForCausalLM layer for layer (same RoPE convention, same GQA layout, same tied embeddings), so no weight permutation was needed to export into this standard HF checkpoint format.

Intended Uses

This is a small, from-scratch research/hobby model intended for:

• Experimenting with small-scale LLM pretraining/SFT pipelines
• Local inference (CPU/single small GPU) via transformers or GGUF/llama.cpp
• Basic conversational demos

It is not intended for production use, factual question-answering, or any application where reliability, safety, or correctness matters.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sebastianbachmaier/Model5"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of France?"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)
out = model.generate(inputs, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

A GGUF build (for llama.cpp / LM Studio) can be produced from this checkpoint with llama.cpp's own convert_hf_to_gguf.py — no custom GGUF writer is used or required.

Chat template / special tokens

Conversations use a simple role-marker template, applied automatically via tokenizer.apply_chat_template:

<|system|>
{system content}<|eot|>
<|user|>
{user content}<|eot|>
<|assistant|>
{assistant content}<|eot|>

<|eot|> (not the base tokenizer's <|endoftext|>) is the turn terminator. model.generate() stops there by default because generation_config.json's eos_token_id (and the GGUF's tokenizer.ggml.eos_token_id) is set to <|eot|>'s id.

Tool / function calling format (untrained)

The chat template and tokenizer support an inline tool-call format — <tool_call>{"name": ..., "arguments": {...}}</tool_call> in an assistant turn, followed by a {"role": "tool", "content": "..."} message — but this checkpoint's SFT data contained no tool-calling examples, so it was never actually trained to produce or consume this format. Treat tool-calling as unsupported until a checkpoint is SFT'd on function-calling data.

Training

Pretraining

• Data: HuggingFaceFW/fineweb-edu (via Andrej Karpathy's pre-tokenized karpathy/fineweb-edu-100B-gpt2-token-shards)
• Tokenizer: gpt2 (50,257 vocab)
• Tokens seen: ~1B (stopped early; original target was ~10B tokens on a cosine LR schedule, so the LR schedule did not fully anneal)
• Objective: standard next-token prediction, bf16 mixed precision, DDP, torch.compile, cosine LR with warmup

Supervised fine-tuning (SFT)

• Data: teknium/OpenHermes-2.5 only — general instruction-following conversations (ShareGPT format, reformatted to {"role": ..., "content": ...} turns). No function/tool-calling data was included in this run.
• Loss masking: cross-entropy loss computed only on assistant-turn tokens; system/user tokens are masked out (-100) so the model only ever learns to produce assistant output, never to reproduce the other roles' text
• New tokens added: <|system|>, <|user|>, <|assistant|>, <|tool|>, <|eot|> (embedding matrix resized accordingly)

No formal evaluation benchmarks have been run on this checkpoint yet.

Limitations and Bias

• Undertrained base model: pretraining was stopped at ~1B tokens, well below the ~10B-token target for a model this size, so factual knowledge, coherence, and general capability are noticeably weaker than a properly converged model of this parameter count.
• No safety/alignment tuning beyond generic instruction/tool-use SFT data — the model can produce incorrect, biased, or inappropriate content and should not be trusted for factual claims.
• No tool-calling ability: despite the tokenizer/chat template supporting a <tool_call> format, this checkpoint's SFT data had no function-calling examples, so it was never trained to use it — don't expect valid or even attempted tool calls.
• Context length is limited to 2048 tokens.

License

Released under the MIT License.