nfeliperealbr/kai-antcore-118m-v1

KAI AntCore 118M

Experimental from-scratch checkpoint for KAI AntCore 118M.

KAI AntCore is a small decoder-only causal language model trained with a standard next-token objective plus Ant Trail auxiliary prediction during pre-training. Ant Trail asks the same hidden state to lightly predict farther future tokens, encouraging route planning without changing inference cost.

Aviso

Este checkpoint e de pre-treino. Ele ainda nao passou por SFT/DPO e nao deve ser tratado como assistente final.

Arquivos principais

• checkpoints/final.pt: checkpoint PyTorch
• tokenizer/kai_tokenizer.model: tokenizer SentencePiece
• tokenizer/kai_tokenizer.vocab: vocabulario
• configs/kai_antcore.generated.yaml: config usada no treino
• training_history.json: historico de loss, se disponivel

Resumo do checkpoint

{
  "global_step": 7630,
  "tokens_seen": 2000158720,
  "best_eval_loss": 2.5984683084487914,
  "model_config": {
    "name": "KAI-AntCore-118M",
    "hidden_dim": 768,
    "num_layers": 12,
    "num_heads": 12,
    "num_kv_heads": 4,
    "intermediate_dim": 2688,
    "vocab_size": 32000,
    "max_seq_len": 2048,
    "rope_base": 10000.0,
    "rms_norm_eps": 1e-05,
    "dropout": 0.0,
    "tie_embeddings": true
  },
  "training_config": {
    "optimizer": "adamw",
    "lr": 0.0003,
    "min_lr": 3e-05,
    "weight_decay": 0.1,
    "beta1": 0.9,
    "beta2": 0.95,
    "grad_clip": 1.0,
    "warmup_steps": 457,
    "lr_schedule": "cosine",
    "total_steps": 7630,
    "batch_size": 16,
    "gradient_accumulation": 8,
    "seq_length": 2048,
    "precision": "bf16",
    "seed": 42,
    "save_every": 250,
    "eval_every": 250,
    "log_every": 20,
    "trail_loss": {
      "enabled": true,
      "offsets": [
        2,
        4,
        8
      ],
      "weight": 0.12
    }
  }
}

Como carregar localmente

Este modelo usa a implementacao local do projeto NeuralSpark/KAI, nao o formato Transformers padrao.

python test_kai.py --checkpoint checkpoints/final.pt --tokenizer tokenizer/kai_tokenizer.model

Repo: https://huggingface.co/nfeliperealbr/kai-antcore-118m-v1