sriharithebest/nayara-v1

NAYARA v1.0

NAYARA is a from-scratch, decoder-only 351.57M-parameter code model and a complete 7×RTX 5090 training-to-CPU-demo pipeline. It does not download or initialize from pretrained model weights. The tokenizer is trained from the selected corpus.

The repository is an implementation, not a pretrained checkpoint. This checkout has no CUDA runtime or training corpus, so weights must be produced on the stated GPU host. “100% useful,” “flawless,” and a fixed quality level cannot be guaranteed before evaluation; the sprint has explicit quality gates instead.

For a notebook run today, open notebooks/NAYARA_Colab.ipynb and follow COLAB_QUICKSTART.md. Its deadline-safe one-GPU L4/A100 profile trains the same 351.57M architecture for 50M tokens and exports a working preview plus Gradio demo. It is a pipeline/behavior preview, not the production-quality 3.8B-token checkpoint.

For a Lightning AI Studio with one H100 80 GB, follow LIGHTNING_H100_QUICKSTART.md. It includes exact machine settings, a 2B-token overnight profile, compiled single-GPU training, automatic unique-data clamping, INT8 export, API verification, and Snapshot Deploy settings.

NAYARA_BUILD_PROMPT.md is the tightened execution prompt for a GPT-5.6 Codex SOL Ultra run. It replaces untestable superlatives with frozen-cohort quality and CPU performance gates.

Locked architecture

Item	Value
Parameters (tied embedding/head)	351,568,896
Layers / width	16 / 1024
Query / KV heads	8 / 2
Head dimension	128
SwiGLU intermediate	5632
Vocabulary / context	32,000 / 1024
Position / normalization	RoPE / RMSNorm ε=1e-5

The 5632 intermediate dimension is intentional. An intermediate size of 2816 would make the proposed network roughly 213M parameters, not 350M.

Budget truth table

The usual dense-transformer training estimate is 6 × parameters × tokens.

Token target	Approx. model FLOPs	Optimizer steps at 7×32×1024×GA2
3.8B (launch default)	8.016e18	8,284
4.0B (corpus capacity)	8.438e18	8,720

Four billion tokens leaves less than one percent of an 8.5e18-FLOP ceiling for approximation error and overhead. The 3.8B default preserves about 5.7% headroom. The wall-clock stop always wins over the token target.

At exactly 45% model-FLOP utilization, finishing 3.8B tokens in 3.5 hours requires about 636 TFLOP/s aggregate measured model throughput. Measure the first 100 steps; do not infer this from marketing TOPS.

Data, before the timed sprint

The timed 3.5-hour window cannot include acquiring, filtering, training a tokenizer, and packing four billion tokens. Prepare the three uint16 files on local NVMe first.

The included source manifest uses an immediately streamable baseline:

• 80% code: CodeParrot Clean Train, restricted to rows marked with permissive licenses and filtered/deduplicated again locally. If Software Heritage access has already been arranged, replace this with Stack-Edu/Stack v2 educational code.
• 15% reasoning: NVIDIA OpenCodeReasoning formatted with NAYARA's user/assistant tokens.
• 5% documentation: deduplicated FineWeb-Edu from the SmolLM corpus.

Review every upstream dataset card and license before distribution. Public code corpora can still contain PII, secrets, malicious code, attribution obligations, or incorrect license labels. The fetcher rejects common embedded-secret patterns and exact normalized duplicates, but it is not a compliance guarantee.

python -m venv .venv
source .venv/bin/activate
pip install -e '.[train,demo,dev]'

# This can take many hours and substantial disk/network bandwidth.
python -m nayara.prepare_data fetch --sources data/sources.example.json --raw-dir /mnt/nvme/nayara/raw
python -m nayara.prepare_data tokenizer --raw-dir /mnt/nvme/nayara/raw --output /mnt/nvme/nayara/tokenizer.json
python -m nayara.prepare_data pack --raw-dir /mnt/nvme/nayara/raw --tokenizer /mnt/nvme/nayara/tokenizer.json --output /mnt/nvme/nayara/packed

Copy tokenizer.json into every exported checkpoint directory before inference. The resulting manifest.json records the actual token count of every stream. Code documents receive a deterministic 50% fill-in-the-middle transform during packing. Do not start if any source has less than world_size × micro_batch × (sequence_length + 1) tokens.

Preflight

Use Linux, a current PyTorch/CUDA build that supports Blackwell (sm_120), one process per GPU, and a local PCIe Gen 5 NVMe path. FlashAttention-3's original package path was Hopper-specific; NAYARA instead forces PyTorch native fused flash SDPA and executes a real BF16 GQA probe at startup. It aborts rather than silently choosing the math kernel.

Before the real launch:

pytest -q
nvidia-smi topo -m
nvidia-smi --query-gpu=index,name,memory.total,pstate,temperature.gpu,power.draw --format=csv
torchrun --standalone --nproc-per-node=7 -m nayara.train \
  --data /mnt/nvme/nayara/packed --output /mnt/nvme/nayara/preflight \
  --micro-batch 2 --grad-accum 1 --max-tokens 143360 --max-minutes 2

The production launcher sets NCCL async error handling and a PCIe-appropriate P2P level. Confirm the actual topology first; consumer motherboards may split lanes or route peers through the CPU. If batch 32 OOMs, lower --micro-batch and increase accumulation so global tokens per optimizer step remain similar. “32/64 tokens per GPU” is not a valid batch unit here: the implementation interprets 32 as sequences, or 32,768 tokens per GPU per micro-step.

Timed launch

Edit the two NVMe paths in scripts/launch_train.sh, then:

bash scripts/launch_train.sh

The trainer uses FSDP FULL_SHARD, block-level wrapping, BF16 parameters/reductions/buffers, fused AdamW, activation checkpointing, cosine decay with 2% warmup, norm clipping at 1.0, and torch.compile(mode="reduce-overhead"). It checkpoints at whichever occurs first: 200 optimizer steps or 15 minutes. --stop-at 15:15 is interpreted in the host's local time and refuses to run if that time has passed. With an explicit deadline, the default 15-minute shutdown reserve starts the final gather at 15:00 so checkpoint export does not consume the conversion window.

Checkpoint export gathers a CPU full state on rank 0. Budget RAM and fast local storage for the temporary gather; the final SafeTensors model is approximately 671 MiB in BF16 before filesystem metadata. The optimizer checkpoint is for trusted local resume only.

Conversion and CPU demo

Copy the tokenizer into the selected final checkpoint, quantize during the reserved conversion window, then launch Gradio:

cp /mnt/nvme/nayara/tokenizer.json /mnt/nvme/nayara/checkpoints/step-XXXXXX/tokenizer.json
python -m nayara.quantize /mnt/nvme/nayara/checkpoints/step-XXXXXX
python app.py /mnt/nvme/nayara/checkpoints/step-XXXXXX --host 127.0.0.1 --port 7860

The demo uses TorchAO dynamic INT8 linear layers and a KV cache. CPU latency depends heavily on generation length, memory bandwidth, PyTorch build, and core count; benchmark on the target i5 before describing it as “ultra fast.” Keep the server bound to localhost unless authentication and sandboxing are added.

Direct CLI completion:

python -m nayara.generate /path/to/checkpoint "def stable_topological_sort(graph):" --completion --max-new-tokens 128

For a local OpenAI-style API, install .[serve] and run uvicorn api:app --host 127.0.0.1 --port 8080 with NAYARA_CHECKPOINT set to a local checkpoint or S3 prefix. The container and ECS Fargate instructions are in deploy/aws/README.md.

Release gates

Do not label a checkpoint v1.0 until it passes all of these:

1. Held-out loss is finite and beats a frequency/unigram baseline.
2. HumanEval+, MBPP+, and a decontaminated internal completion set are run with recorded sampling settings.
3. Secret-regurgitation, license-attribution, insecure-code, and prompt-abuse probes are reviewed.
4. Median time-to-first-token and tokens/second are recorded on the actual target i5 with INT8 enabled.
5. At least 100 manually reviewed prompts establish the intended concise, direct NAYARA tone. Tone is learned from the formatted reasoning stream; a system prompt cannot substitute for training at this model size.