mrx46/Xenon-303

Project Genesis

Project Genesis is a research codebase for building a decoder-only large language model from randomly initialized weights with Python and PyTorch. Development proceeds in reviewed phases.

Phases 2 through 14 provide:

• Safe typed YAML configuration with strict dotted overrides.
• Runtime environment detection and configuration-relative paths.
• Immutable dataset records, schemas, metadata, and statistics.
• Deterministic local manifests, SHA-256 fingerprints, and integrity checks.
• Dataset registry, cache contract, and atomic local manifest storage.
• Deterministic readers for text, Markdown, JSON, JSONL, CSV, PDF, Git snapshots, and local HTML snapshots.
• Configurable normalization, filtering, exact deduplication, quality reports, and processed-data manifests.
• Custom byte-level BPE vocabulary training, reversible Unicode encoding, stable special-token IDs, tokenizer persistence, and quality metrics.
• PyTorch token and position embeddings, custom LayerNorm, feed-forward layers, causal multi-head self-attention, and strict residual addition.
• Opt-in RoPE, RMSNorm, SwiGLU, grouped-query attention, and native scaled dot-product attention while legacy bundles retain their original architecture.
• Pre-normalization transformer blocks and a GPT-style decoder.
• Deterministic next-token batching, AdamW, warmup/cosine scheduling, gradient accumulation and clipping, mixed precision, and atomic resumable checkpoints.
• Token-weighted validation, perplexity, named language and coding cases, throughput measurements, canonical reports, and regression gates.
• Cached autoregressive generation, greedy and stochastic sampling, stop/context handling, immutable conversations, prompt assembly, and completion benchmarks.
• Verified inference-only bundles, bounded generate/chat HTTP APIs, health/readiness checks, structured request logs, and a non-root CPU container.
• Required bundle provenance and semantic compatibility checks.
• Fatal inference failure isolation, a dependency-free concurrent load probe, operational recovery procedures, and checksummed attested releases.
• Atomic end-to-end experiment execution from verified local sources through a trained, evaluated, checkpointed, deployable inference bundle.
• A non-allocating training preflight for source integrity, configuration compatibility, parameter count, schedule size, and device capacity facts.
• Recoverable long runs with periodic checkpoints, latest-checkpoint resume, periodic validation, durable metrics, best-checkpoint selection, and retention.
• Deterministic seeded epoch shuffling with exact microbatch replay after resume.

Requirements

• Python 3.12 or 3.13
• uv

Setup

uv sync --locked --group dev

The default dataset configuration is configs/dataset/default.yaml. It deliberately contains no sources; experiments must declare their local inputs explicitly. Cleaning defaults are in configs/preprocessing/default.yaml. Tokenizer defaults are in configs/tokenizer/default.yaml. Model defaults are in configs/model/default.yaml. Training defaults are in configs/training/default.yaml. Evaluation defaults are in configs/evaluation/default.yaml. Inference defaults are in configs/inference/default.yaml.

The opt-in modern-super.yaml profile is a 303,350,784-parameter, 24-layer decoder with 16 query heads, 4 key/value heads, a 32,000-token vocabulary, and an 8,192-token context. Preflight estimates a 4.85 GB persistent training-state lower bound; activations and CUDA workspace require additional memory. Select batch size and training sequence length only after measuring the target GPU. Existing model configs and trained bundles continue to use the legacy architecture.

Run a complete experiment with a dataset configuration that declares both training and validation sources:

genesis-preflight \
  --dataset-config configs/dataset/experiment.yaml \
  --device cuda

genesis-train \
  --dataset-config configs/dataset/experiment.yaml \
  --output artifacts/runs/run-001 \
  --source-revision COMMIT_SHA \
  --training-run-id run-001 \
  --device cuda

An interrupted run prints its exact checkpoint. Resume with the same arguments and configuration:

genesis-train \
  --dataset-config configs/dataset/experiment.yaml \
  --output artifacts/runs/run-001 \
  --source-revision COMMIT_SHA \
  --training-run-id run-001 \
  --device cuda \
  --resume artifacts/runs/.run-001.in-progress/checkpoints/step-00001000.pt

Coding smoke test

Materialize the reviewed nanoGPT, minGPT, CodeSearchNet tooling, and lit-llama snapshots:

uv run --locked python scripts/prepare_coding_smoke.py

Validate the small CPU experiment:

uv run --locked genesis-preflight \
  --dataset-config configs/dataset/coding-smoke.yaml \
  --tokenizer-config configs/tokenizer/coding-smoke.yaml \
  --model-config configs/model/coding-smoke.yaml \
  --training-config configs/training/coding-smoke.yaml \
  --evaluation-config configs/evaluation/coding-smoke.yaml \
  --device cpu

Train it:

uv run --locked genesis-train \
  --dataset-config configs/dataset/coding-smoke.yaml \
  --tokenizer-config configs/tokenizer/coding-smoke.yaml \
  --model-config configs/model/coding-smoke.yaml \
  --training-config configs/training/coding-smoke.yaml \
  --evaluation-config configs/evaluation/coding-smoke.yaml \
  --output artifacts/runs/coding-smoke \
  --source-revision COMMIT_SHA \
  --training-run-id coding-smoke \
  --device cpu

This 20-step run verifies the system. It is not enough data or training to produce a useful coding assistant.

The CodeSearchNet Git repository contributes its MIT-licensed tooling source, not the separately hosted multi-gigabyte CodeSearchNet corpus. Full-corpus use requires a later sharded data pipeline.

CodeSearchNet Python CPU training

Materialize a deterministic 12,000-example training split and 1,000-example validation split from the preserved CodeSearchNet Python archive:

uv run --locked python scripts/prepare_codesearchnet_instructions.py

The script downloads the 897 MiB archive once, verifies Zenodo's published size and MD5, streams compressed JSONL without extracting it, filters oversized functions, and formats each documentation/function pair with the same role contract used by chat inference. The upstream functions carry mixed licenses; the generated manifest retains source URLs for review and attribution. Do not redistribute the generated sample without performing that review.

Validate and train the 4,935,680-parameter CPU profile:

uv run --locked genesis-preflight \
  --dataset-config configs/dataset/codesearchnet-python.yaml \
  --tokenizer-config configs/tokenizer/codesearchnet-python.yaml \
  --model-config configs/model/codesearchnet-cpu.yaml \
  --training-config configs/training/codesearchnet-cpu.yaml \
  --evaluation-config configs/evaluation/codesearchnet-cpu.yaml \
  --device cpu

uv run --locked genesis-train \
  --dataset-config configs/dataset/codesearchnet-python.yaml \
  --tokenizer-config configs/tokenizer/codesearchnet-python.yaml \
  --model-config configs/model/codesearchnet-cpu.yaml \
  --training-config configs/training/codesearchnet-cpu.yaml \
  --evaluation-config configs/evaluation/codesearchnet-cpu.yaml \
  --output artifacts/runs/codesearchnet-python-cpu-v1 \
  --source-revision COMMIT_SHA \
  --training-run-id codesearchnet-python-cpu-v1 \
  --device cpu

This profile schedules 5.12 million training tokens. It is a meaningful from-scratch CPU experiment, not a production coding assistant; much larger models, broader licensed corpora, and GPU training are required for that.

Small English conversation CPU run

Download and deterministically split the pinned, human-written Databricks Dolly-15k corpus (CC BY-SA 3.0):

py -m uv run --locked python scripts/prepare_english_conversations.py

The conversation run deliberately reuses the existing 4,935,680-parameter CPU model, tokenizer, training, and evaluation profiles:

py -m uv run --locked genesis-preflight --dataset-config configs/dataset/english-conversations.yaml --tokenizer-config configs/tokenizer/codesearchnet-python.yaml --model-config configs/model/codesearchnet-cpu.yaml --training-config configs/training/codesearchnet-cpu.yaml --evaluation-config configs/evaluation/codesearchnet-cpu.yaml --device cpu

py -m uv run --locked genesis-train --dataset-config configs/dataset/english-conversations.yaml --tokenizer-config configs/tokenizer/codesearchnet-python.yaml --model-config configs/model/codesearchnet-cpu.yaml --training-config configs/training/codesearchnet-cpu.yaml --evaluation-config configs/evaluation/codesearchnet-cpu.yaml --output artifacts/runs/english-conversations-cpu-v1 --source-revision bdd27f4d94b9c1f951818a7da7fd7aeea5dbff1a --training-run-id english-conversations-cpu-v1 --device cpu

This is a bounded from-scratch experiment for short English replies, not a general-purpose assistant. Keep the generated dataset attribution when sharing adaptations, as required by CC BY-SA 3.0.

For a broader second CPU run, add the pinned Apache-2.0 OpenAssistant English subset and train it together with Dolly rather than replacing the earlier data:

py -m uv run --locked python scripts/prepare_openassistant_conversations.py

py -m uv run --locked genesis-train --dataset-config configs/dataset/english-conversations-mixed.yaml --tokenizer-config configs/tokenizer/codesearchnet-python.yaml --model-config configs/model/codesearchnet-cpu.yaml --training-config configs/training/english-conversations-mixed-cpu.yaml --evaluation-config configs/evaluation/codesearchnet-cpu.yaml --output artifacts/runs/english-conversations-mixed-cpu-v2 --source-revision oasst1-fdf72ae+dolly-bdd27f4 --training-run-id english-conversations-mixed-cpu-v2 --device cpu

With genesis-serve running, open http://127.0.0.1:8000/ for the local chat interface. The OpenAPI testing page remains available at http://127.0.0.1:8000/docs.

Verify

uv run --locked ruff format --check .
uv run --locked ruff check .
uv run --locked mypy src
uv run --locked pytest
uv build

See the architecture, Phase 2 decisions, Phase 3 decisions, Phase 4 decisions, Phase 5 decisions, Phase 6 decisions, Phase 7 decisions, Phase 8 decisions, Phase 9 decisions, Phase 10 decisions, Phase 11 decisions, compatibility policy, Phase 12 decisions, Phase 13 decisions, Phase 14 decisions, Phase 15 decisions, Phase 16 decisions, production runbook, security policy, development standards, and the roadmap.

License

Project Genesis is licensed under the MIT License.