CharlesMiller

CharlesMiller is a Portuguese-only small language model (SLM), with approximately 880.8 million parameters, created for learning and experimentation. It explores tokenization, pretraining, evaluation, and the practical work of distributing a language model through Hugging Face Transformers.

Educational use only. This model was not created or validated for production use. Do not deploy it in production or rely on its outputs for consequential decisions. It can generate fluent but incorrect, repetitive, biased, or inappropriate text.

What kind of model is this?

An SLM (small language model) is relatively compact compared with larger language models; there is no universally fixed parameter-count boundary. CharlesMiller has 880,803,840 parameters (approximately 0.88B).

This release is the base-pretraining checkpoint at step 2,150,400. It is a decoder-only, autoregressive model that predicts the next token and continues text. It is not a chat assistant or an instruction-tuned model.

Training text was in Portuguese only, using TucanoBR/GigaVerbo. Portuguese is the intended language; multilingual capability is not a supported claim. The ability to tokenize other languages does not establish competence in them.

Architecture and model size

Setting	Value
Architecture	Dense, decoder-only causal Transformer derived from nanochat
Parameters	880,803,840
Layers	24
Hidden size	1,536
MLP intermediate size	6,144
Query / key-value heads	12 / 12 (multi-head attention)
Head dimension	128
Vocabulary	65,536 tokens
Supported context	2,048 tokens, including BOS
Attention	Full causal attention in every layer
Position encoding	Rotary positional embeddings (RoPE), base 10,000
Normalization	Parameter-free RMS normalization, including QK normalization
MLP activation	Squared ReLU
Embedding / output weights	Untied
Linear-layer biases	None
Logit softcap	15
Export format	One Safetensors file, approximately 3.32 GB

The integration preserves the original rotary convention, including rounding rotary sine/cosine values to BF16 even for FP32 CPU inference. The stored checkpoint uses mixed BF16/FP32 tensors, preserved without casting during export; it is not a quantized model. Loading all parameters as FP32 needs approximately 3.52 GB for parameters alone, plus runtime overhead and activations.

Training data and provenance

The sole training corpus for this base model was TucanoBR/GigaVerbo, a collection of Portuguese text. Exact dataset revisions, filtering, and sampling details are not fully captured in the supplied report; the upstream dataset may change.

The report records 70,464,307,200 training tokens (approximately 70.46B token positions processed), or 80 processed tokens per model parameter. These are training positions, not a count of unique tokens or documents.

Training code: marlosb/nanochat, branch gigaverbo, pinned to the branch HEAD inspected for this release, d5f6edb58322ba6a7937ca99e1a99d163d59f2b9. The original checkpoint metadata is included as training_meta.json.

Training setup and diagnostics

The following are reported historical values, not independently reproduced training results.

Setting	Reported value
Run	april1
Base-training report timestamp	May 6, 2026, 03:41:44 (timezone unspecified)
Final step	2,150,400
Resume point	Step 1,075,200
Sequence length	2,048
Global batch size	32,768 tokens per optimizer step
Per-device microbatch	8 sequences
Distributed world size	1
Optimizers	Muon for transformer matrices; AdamW for embedding/output weights
Embedding / output-head learning rates	0.2 / 0.004
Matrix learning rate	0.02
Weight decay	0.0
Gradient clipping	1.0
Warmup / warmdown ratios	0.0 / 0.2
Final learning-rate fraction	0.0
Reported training compute	3.936709e20 FLOPs
Reported model FLOPs utilization (MFU)	30.84%
Recorded training-loop time	21,205.69 minutes (approximately 353.43 hours)
Reported peak memory	34,766.90 MiB
Total wall-clock time	Unknown
GPU model and full hardware/software environment	Not specified in the supplied report

Diagnostic	Reported value
Minimum validation bits per byte during base training	0.7218
Final validation bits per byte during base training	0.9559
Separate base-loss evaluation: training bpb	0.7605
Separate base-loss evaluation: validation bpb	0.9266

Bits per byte (bpb) is predictive loss normalized by text byte length; lower is better on the same data and setup. Training-time and separately evaluated values are distinct measurements and should not be compared without checking their evaluation settings. The final checkpoint is not the checkpoint with the lowest recorded training-time validation bpb.

Load and generate text

Use PyTorch >= 2.9.1 and < 3**, **Transformers >= 4.57.3 and < 5, and Safetensors. Inference does not require nanochat, Rust tokenizer bindings, tiktoken, optimizer files, or pickle loading.

The following example downloads the model from marlosb/CharlesMiller. To load the local export instead, use model_id = r".\CharlesMiller".

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "marlosb/CharlesMiller"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
).eval()

inputs = tokenizer("O futebol brasileiro", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=48, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))

do_sample=False selects the highest-probability next token. For sampling experiments, set do_sample=True, temperature=0.8, and top_p=0.9. Completions are not guaranteed factual answers.

The tokenizer prepends <|bos|> automatically. Use add_special_tokens=False only when adding BOS yourself. Batch generation uses left padding; always pass the tokenizer's attention_mask when padding.

On CUDA, move the model and input tensors to the same device. BF16 loading can reduce memory use on supported hardware but casts the original mixed-precision parameters. FP32 CPU inference is the reference configuration; GPU inference has not yet been validated. FP16/FP64 inference is not supported by this integration.

The model uses custom Python code. Review it before enabling trust_remote_code=True; after publication, pin both loading calls to a reviewed Hugging Face commit using revision.

Benchmarks

Portuguese-oriented base-model evaluation

The evaluation reports dated 2026-10-20 give CharlesMiller a CORE score of 0.203252. The comparison below includes this model and five external reference models evaluated by the same pipeline. All numbers come exclusively from the completed runs supplied in the local reports directory, not from earlier training reports or published model leaderboards.

Model	CORE (mean centered accuracy)
Polygl0t/Tucano2-qwen-1.5B-Base	0.353404
Polygl0t/Tucano2-0.6B-Base	0.274381
TucanoBR/Tucano-1b1	0.232633
marlosb/CharlesMiller	0.203252
TucanoBR/Tucano-630m	0.194483
openai-community/gpt2	0.085468

Download all task-level scores used in this comparison. These are results from this evaluation pipeline, not claims of reproducing each reference model's official benchmark results.

Tasks and scoring

The evaluation implementation is marlosb/nanochat, branch eval_benchmark, whose inspected HEAD is pinned here to 00155c4bef1229cf1a9c16e37ed384d0178a33ca. The supplied logs confirm --eval core and the following few-shot settings. These results evaluate the published base checkpoint, not the later MID/SFT checkpoints described in the original training report.

Task (original benchmark)	Shots	CharlesMiller accuracy (%)	Centered score
HellaSwag, zero-shot	0	37.00	0.160039
ARC-Easy	10	42.97	0.239618
ARC-Challenge	10	26.51	0.020134
COPA	0	60.00	0.200000
PIQA	10	60.83	0.216540
Global PIQA	10	60.00	0.200000
OpenBookQA	0	29.60	0.061333
LAMBADA	0	35.86	0.358626
CALAME-PT	0	61.51	0.615125
HellaSwag, few-shot	10	37.33	0.164373
Winograd Schema Challenge	0	53.85	0.076923
WinoGrande	0	50.91	0.018153
BIG-bench: Dyck languages	10	9.40	0.094000
AGIEval: LSAT analytical reasoning	3	28.26	0.103261
BIG-bench: CS algorithms	10	37.95	0.379545
BIG-bench: operators	10	54.76	0.547619
BIG-bench: repeat/copy logic	10	0.00	0.000000

Raw accuracy is the fraction of correct examples. The evaluator computes centered = (accuracy - random_baseline) / (1 - random_baseline), then takes the unweighted mean of all 17 centered scores as CORE. Baselines are 0.25 for HellaSwag, ARC and OpenBookQA; 0.50 for COPA, PIQA, Global PIQA, Winograd and WinoGrande; 0.20 for LSAT; and zero for the language-modeling tasks. A centered score of zero equals the configured baseline; negative scores are possible. CORE is not percentage accuracy.

Multiple-choice and schema tasks select the candidate with the lowest mean continuation-token cross-entropy. Language-modeling tasks require every target continuation token to match its teacher-forced argmax prediction; this is not an unrestricted generation or chat evaluation. Task data is shuffled with seed 1337. Few-shot demonstrations are sampled from the same task's evaluation pool, excluding the current example, with seed 1234 + example_index. Overlong inputs are left-truncated to each model's context limit. No sampling temperature is used for these scores.

Dataset origins and translation credits

The evaluator reads packaged JSONL inputs from marlosb/auxiliary_data, eval_bundle.zip, a Portuguese adaptation of the nanochat evaluation bundle. Portuguese translations published under marlosb were produced by this model's author. This includes the translated bundle and the separately published marlosb/ai2_arc-pt.

Translations published under Polygl0t are the work of the Polygl0t team, not this model's author. Relevant Portuguese resources include Polygl0t/Hellaswag-poly, Polygl0t/ARC-poly and Polygl0t/LAMBADA-poly. Polygl0t/CALAME-PT adapts the originally Portuguese CALAME-PT dataset by Ricardo Lopes, Joao Magalhaes and David Semedo for an evaluation harness; it is not an English-to-Portuguese translation.

The task links above identify the original benchmarks. The evaluator loads the bundle, not the individual Hub repositories listed here. Neither the reports nor the bundle configuration provide a per-file source/revision manifest, so an exact mapping from each bundled file to a particular translated Hub release remains unverified; matching task names alone do not establish that mapping.

Interpretation and reproducibility limits

CharlesMiller's aggregate is above GPT-2 and Tucano-630m, but below Tucano-1b1 and the two Tucano2 references. Results vary by task: CALAME-PT accuracy is 61.51%, while ARC-Challenge and WinoGrande are close to their configured baselines, and repeat/copy logic scores zero. These models differ in size, architecture, training data and compute; the comparison is not a controlled scaling or training-efficiency experiment.

This is a custom 17-task CORE variant, not the original DCLM CORE suite. HellaSwag contributes twice, at zero and ten shots. The branch excludes Jeopardy, BIG-bench QA Wikidata, CommonsenseQA, SQuAD, CoQA, BoolQ and BIG-bench language identification. The historical base CORE of 0.1366 used a different evaluation setup and must not be interpreted as a directly comparable earlier score or as evidence that these unchanged weights improved.

The launcher defaults to all bundled examples (--max-per-task -1) and CUDA; the loader selects BF16 on supporting CUDA hardware, otherwise FP16. However, the reports do not record actual example counts/caps, an input split manifest, dataset/model revisions, runtime code hash, GPU or selected dtype. The code pin above documents the requested implementation, not a hash captured by the run, so the reports alone cannot guarantee exact replay. Different tokenizers and context limits can change truncation and remove demonstrations. No repeated-run uncertainty estimates or contamination/translation-quality audits are documented. These educational results do not establish production readiness.

Export correctness

verification.json, generated by the verification script, records tensor identity, tokenizer checks, native-model logit/generation comparisons, and padded-generation checks. These establish implementation compatibility, not general model quality.

Scope and limitations

• Created for learning and experimentation, not for production deployment.
• This is a Portuguese base completion model, not a safety-aligned chat assistant. It can hallucinate facts and generate repetitive, biased, or inappropriate text.
• Generation recomputes the full prefix. KV caching is not implemented; use_cache=True is rejected explicitly.
• Prompt plus generated tokens must fit within the 2,048-token supported context.
• No EOS token is inferred from chat delimiters; generation stops at the requested length unless the caller supplies a suitable stopping condition.
• Hidden-state/attention outputs, Transformers 5, and third-party serving engines are not supported.
• Do not rely on outputs for medical, legal, financial, or other consequential decisions.

Licensing

The model weights are released under the MIT License; see LICENSE. LICENSE_CODE retains the MIT notice for the nanochat-derived implementation. The educational-use and not-for-production warnings describe intended use and limitations, not additional restrictions on the MIT license. The dataset has its own licensing information and source-specific conditions.

Acknowledgments

Thank you, Andrej Karpathy, for nanoGPT and nanochat, which made learning the practical engineering of language models accessible.

Thank you, Nicholas Kluge Corrêa, and the collaborators behind TucanoBR/GigaVerbo, the Portuguese corpus used to train this model.

Citation Information

Please credit the upstream dataset research and software that made this project possible using the following BibTeX entries.

GigaVerbo / Tucano

The GigaVerbo dataset card provides both the published article and its earlier preprint. These are alternative references to the same work.

@article{correa2025tucanoadvancingneuraltext,
  title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
  author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
  journal = {Patterns},
  publisher = {Elsevier},
  year = {2025},
  doi = {10.1016/j.patter.2025.101325},
  url = {https://doi.org/10.1016/j.patter.2025.101325},
  issn = {2666-3899}
}

@misc{correa2024tucanoadvancingneuraltext,
  title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
  author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
  year = {2024},
  eprint = {2411.07854},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2411.07854}
}

nanoGPT and nanochat

The nanochat entry follows its upstream citation guidance. nanoGPT is cited as a software repository.

@misc{karpathy2022nanogpt,
  author = {Andrej Karpathy},
  title = {{nanoGPT}},
  year = {2022},
  howpublished = {GitHub repository},
  url = {https://github.com/karpathy/nanoGPT}
}

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best {ChatGPT} that \$100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

About the name

The name honors Charles William Miller (1874-1953), the Brazilian-born sportsman widely known as the father of football in Brazil. After studying in Southampton, England, he returned to Brazil in 1894 with two footballs and a rulebook, helping establish organized football in the country. This model celebrates that spirit of learning abroad, bringing knowledge home, and building something new in Brazil.