zarko1321/Bitsy-1B

Bitsy 1B — untrained initialization

Randomly initialized weights. Zero optimizer steps. This model cannot yet write coherent English or chat.

• 998,852,608 unique parameters, BF16 storage.
• Standard Hugging Face Llama architecture: 20 layers, hidden size 2048, SwiGLU size 5888, 16 query heads, 4 KV heads, head dimension 128, RoPE theta 10000, context 2048, tied input/output embeddings.
• No pretrained model weights were used. Seed 42. This is a new initialization, not a continuation of earlier Bitsy checkpoints.
• New byte-level BPE tokenizer: 32,000 tokens. Special IDs: BOS 0, EOS 1, PAD 2, UNK 3. No chat template and no automatic BOS insertion. The tokenizer is not compatible with tokens cached for an earlier Bitsy tokenizer.
• Only the tokenizer learned vocabulary/merges from English texts: Alice's Adventures in Wonderland (Lewis Carroll), Pride and Prejudice (Jane Austen), The Adventures of Sherlock Holmes (Arthur Conan Doyle), The Time Machine (H. G. Wells), Moby Dick (Herman Melville), Dracula (Bram Stoker), and Frankenstein (Mary Shelley). Sources: Project Gutenberg IDs 11, 1342, 1661, 35, 2701, 345, 84, retrieved from its pglaf.org mirror. Book files are not included.
• The tokenizer corpus is small and literary; assess its suitability before a substantial pretraining run.
• CPU validation with Transformers 4.57.1 and PyTorch 2.6.0: full model saved and reloaded with no missing/unexpected tensors; forward logits and next-token loss matched exactly; tokenizer round trips and tied embeddings verified. Unsloth/L4 training has not been executed for this new initialization.

Use

Upload all six weight shards and every accompanying model/tokenizer file into the same Hugging Face repository root. Do not upload the packaging ZIPs themselves; extract all five ZIPs into the same folder first.

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/Bitsy-1B", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/Bitsy-1B")

For pretraining, train all parameters using plain text, insert EOS between documents, and apply the next-token shift exactly once. Standard Hugging Face causal-LM models shift unshifted labels internally. A trainer is not included.

Model-weights licensing is left for the owner to choose when publishing. Using the Llama architecture here does not import Meta pretrained weights or their model license.