You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Llama-1B-Base

A 1-billion parameter Llama-style model trained from scratch on the FineWeb dataset mixture.

Model Details

Architecture: Decoder-only Transformer (Llama-style)
Features: Rotary Position Embeddings (RoPE), RMSNorm, SwiGLU MLP, No Bias, Tied Embeddings.
Parameters: ~1.01 Billion
Layers: 18
Heads: 16
Embedding Dimension: 2048
Intermediate Dimension (FFN): 5500
Context Length: 512 tokens
Vocabulary Size: 50,257 (custom BPE tokenizer)

Training

Data Mixture: 80% FineWeb-Edu / 20% FineWeb.
Target: 1 Billion tokens (30,000 iterations).
Optimizer: AdamW (betas=0.9, 0.95), Weight Decay 0.1.
Learning Rate: 3e-4 with Cosine Decay to 3e-5.
Hardware: NVIDIA RTX PRO 6000 Blackwell.

⚠️ Git LFS Warning & Quickstart

If you use git clone to download this repository, you must have Git LFS installed (git lfs install followed by git lfs pull), otherwise the .pt files will just be tiny pointer files and PyTorch will fail to load them.

Best Way: Load dynamically using Python

To avoid Git LFS issues entirely, you can run the following script which automatically downloads the model, the tokenizer, and the model definition dynamically from Hugging Face Hub (Windows, Mac, and Linux compatible):

import os
import torch
from huggingface_hub import hf_hub_download

# Select the model repository
repo_id = "nowefnyieryfner/llama-1b-base"
model_filename = "best_model_1b.pt"

print(f"Downloading model files from {repo_id}...")

# Download files from Hugging Face Hub (cached automatically)
model_path = hf_hub_download(repo_id=repo_id, filename=model_filename)
model_code_path = hf_hub_download(repo_id=repo_id, filename="model.py")
tokenizer_path = hf_hub_download(repo_id=repo_id, filename="tokenizer.json")

# Dynamically import GPT from the downloaded model.py
import importlib.util
spec = importlib.util.spec_from_file_location("model", model_code_path)
model_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(model_module)
GPT = model_module.GPT

# Load tokenizer
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file(tokenizer_path)

# Load model weights
print("Loading model weights...")
checkpoint = torch.load(model_path, map_location="cpu", weights_only=False)
config = checkpoint["config"]
model = GPT(config)
model.load_state_dict(checkpoint["model"])
model.eval()

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
print(f"Model loaded successfully on {device}!")

# Sample prompt
prompt = "<|im_start|>user\nExplain the concept of quantum entanglement in simple terms.<|im_end|>\n<|im_start|>assistant\n"
x = torch.tensor(tokenizer.encode(prompt).ids, dtype=torch.long, device=device).unsqueeze(0)

print("\n--- Generating response ---")
y = model.generate(x, max_new_tokens=150, temperature=0.8, top_k=50)
print(tokenizer.decode(y[0].tolist()))

Installation

Make sure you have the required packages installed:

pip install torch tokenizers huggingface_hub

Downloads last month: -; Downloads are not tracked for this model. How to track