ArthurFriedenreich

ArthurFriedenreich is a Portuguese-only small language model (SLM), with approximately 2.09 billion parameters, created for learning and experimentation.

The project explores the complete language-model development process: preparing data, training a tokenizer, pretraining a decoder-only Transformer, evaluating checkpoints, and packaging a model for others to inspect and run. It builds on Andrej Karpathy's nanoGPT and nanochat, with modifications to the training code.

Educational use only. This model was not created or validated for production use. Do not deploy it in production or rely on its outputs for consequential decisions. It can produce fluent but incorrect, repetitive, biased, or inappropriate text.

What kind of model is this?

An SLM (small language model) is a language model with a relatively compact parameter count compared with much larger models. There is no universally fixed size boundary; here, "small" describes a roughly 2B-parameter model, not a promise that it will run cheaply on every device.

ArthurFriedenreich is a base, autoregressive text-completion model. It predicts the next token from the preceding text. This release contains the base-pretraining checkpoint, not an instruction-tuned or chat-aligned assistant: no supervised fine-tuning (SFT), reinforcement-learning stage, or chat template is included.

Training text was in Portuguese only. Portuguese is the intended language for prompts and completions; multilingual performance is not a goal or a supported capability. The ability to tokenize another language does not imply that the model was trained to understand or generate it reliably.

Model size and architecture

Setting	Value
Architecture	Dense, decoder-only causal Transformer derived from nanochat
Parameters	2,088,765,194 (approximately 2.09B)
Transformer layers	24
Hidden size	1,536
MLP intermediate size	6,144
Query / key-value heads	12 / 12 (multi-head attention)
Head dimension	128
Vocabulary	65,536 tokens
Supported context	2,048 tokens, including the beginning-of-sequence token
Position encoding	Rotary positional embeddings (RoPE), base 100,000
Attention pattern	Repeating SSSL; the final layer always uses long attention
Activation	Squared ReLU
Embedding / output weights	Untied
Checkpoint step	1,119,072
Distribution format	Two Safetensors shards, approximately 5.74 GB total

S denotes a short causal attention window and L denotes long causal attention over the supported context. At this context size, the short window includes up to 512 preceding tokens plus the current token.

The implementation also preserves nanochat's parameter-free RMS normalization, QK normalization and scaling, alternating-layer value embeddings, previous-token embedding mixing ("smear"), learned residual scalars, midpoint residual subtraction ("backout"), and a logit softcap of 15.

The exported tensors retain the native checkpoint's mixed BF16/FP32 storage precision; this is not a quantized release. Loading all parameters as FP32 requires approximately 8.36 GB for parameter storage alone, plus activations, temporary tensors, and framework overhead.

Training data and language

The project used Portuguese-language data from:

• TucanoBR/GigaVerbo
• Polygl0t/gigaverbo-v2
• Polygl0t/gigaverbo-v2-synth

The links above identify the upstream resources, which may have changed since training.

The report records 87,090,659,328 training tokens (approximately 87.09B token positions processed). This is not a count of unique tokens or unique documents; training can revisit data across epochs and resumed runs.

Training setup

The following details come from the training report and saved checkpoint metadata. They are historical run records, not results independently reproduced for this model card.

Setting	Reported value
Run	july-extended
Run start	July 30, 2026, 11:36:56 (timezone unspecified)
Final base-training report	August 7, 2026, 05:42:00 (timezone unspecified)
Hardware	2 NVIDIA H100 NVL GPUs; 186.0 GB total GPU memory reported
Host	Linux; 80 logical CPU cores; 629.9 GB RAM
Software	Python 3.10.12; PyTorch 2.9.1+cu128; CUDA 12.8
Distributed training	DDP, world size 2
Training precision	FP8 enabled, tensorwise recipe
Optimizers	nanochat's Muon/AdamW parameter-group setup
Sequence length	2,048
Global batch size	77,824 tokens per optimizer step
Per-device microbatch	19 sequences
Final step	1,119,072
Resume point for the final continuation	Step 1,002,444
Warmup	40 steps
Warmdown ratio / final learning-rate fraction	1.0 / 0.05
Reported training compute	4.421780e20 FLOPs
Reported model FLOPs utilization (MFU)	44.95%
Recorded training-loop time	9,810.11 minutes (approximately 163.50 hours)
Reported total wall-clock time	186 hours, 23 minutes
Reported peak memory	75,326.87 MiB; rank/aggregation scope unspecified

FP8 describes the training compute configuration; it does not mean that the published weights are FP8 or that FP8 hardware is required for inference. Training-loop time and total wall-clock time are separate report fields.

The recorded learning rates were 0.3 for embeddings, 0.008 for the output head, 0.02 for matrix parameters, and 0.5 for scalar parameters, with a configured weight decay of 0.28.

Training diagnostics

Metric	Reported value
Minimum validation bits per byte during training	0.4361
Final validation bits per byte during training	0.4714

Bits per byte (bpb) measures predictive loss normalized by text byte length; lower is better on the same evaluation data and setup. These values are training diagnostics, not general-purpose benchmark scores. The final checkpoint is not the checkpoint with the lowest recorded validation bpb.

Code provenance

The training implementation comes from the gigaverbo2 branch of marlosb/nanochat, not the default branch. As confirmed by the model author, the release is pinned to that branch's HEAD at the time of this correction: 00b6ab30091388186798741cfdcece7095f67e44. Use the pinned commit rather than a moving branch for reproducibility. The original checkpoint metadata is included as training_meta.json.

For transparency, the report's initial environment snapshot lists branch gigaverbo2 and commit 0f6b74c with a dirty working tree. This remains historical run metadata; the corrected release reference is the gigaverbo2 revision above. The report alone does not capture those uncommitted changes.

How to load and use the model

Use PyTorch >= 2.9.1 and < 3**, **Transformers >= 4.57.3 and < 5, and Safetensors. No nanochat installation, tiktoken, Rust tokenizer, or pickle loading is required for inference.

The example below downloads the model from marlosb/ArthurFriedenreich. To load the local export instead, use model_id = r".\ArthurFriedenreich".

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "marlosb/ArthurFriedenreich"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
).eval()

inputs = tokenizer("A historia do futebol brasileiro", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=48, do_sample=False)

print(tokenizer.decode(output[0], skip_special_tokens=True))

This produces a continuation of the prompt, not a guaranteed factual answer. do_sample=False uses deterministic greedy token selection for a fixed execution setup. To experiment with sampling, use do_sample=True, temperature=0.8, and top_p=0.9 in the generation call.

GPU use

The FP32 example runs on CPU. To use CUDA, move both the model and the tokenizer outputs to the same device:

device = torch.device("cuda")
model = model.to(device)
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=48, do_sample=False)

On compatible GPUs, loading with dtype=torch.bfloat16 can reduce memory use. That casts the original mixed-precision parameters and can change numerical results. GPU execution has not yet been validated for this integration.

Tokenization and reproducibility

The tokenizer automatically prepends the training document delimiter <|bos|>. Use add_special_tokens=False only when supplying that token yourself. Batch generation requires left padding (the tokenizer default). Pass the tokenizer's attention_mask when padding.

The model uses custom Python code: review it before enabling trust_remote_code=True. For a published repository, pass a reviewed commit hash as revision to both loading calls for reproducibility.

Benchmarks

Portuguese-oriented base-model evaluation

The evaluation reports dated 2026-10-20 give ArthurFriedenreich a CORE score of 0.297296, second among the seven models in this comparison. All numbers below come exclusively from the completed runs supplied in the local reports directory, not from earlier training reports or published model leaderboards.

Model	CORE (mean centered accuracy)
Polygl0t/Tucano2-qwen-1.5B-Base	0.353404
marlosb/ArthurFriedenreich	0.297296
Polygl0t/Tucano2-0.6B-Base	0.274381
TucanoBR/Tucano-1b1	0.232633
marlosb/CharlesMiller	0.203252
TucanoBR/Tucano-630m	0.194483
openai-community/gpt2	0.085468

Download all task-level scores used in this comparison. These are results from this evaluation pipeline, not claims of reproducing each reference model's official benchmark results.

Tasks and scoring

The evaluation implementation is marlosb/nanochat, branch eval_benchmark, whose inspected HEAD is pinned here to 00155c4bef1229cf1a9c16e37ed384d0178a33ca. The supplied logs confirm --eval core and the following few-shot settings for all seven models. These evaluate the published base models, not chat/SFT checkpoints.

Task (original benchmark)	Shots	ArthurFriedenreich accuracy (%)	Centered score
HellaSwag, zero-shot	0	47.97	0.306245
ARC-Easy	10	56.19	0.415825
ARC-Challenge	10	31.21	0.082774
COPA	0	63.00	0.260000
PIQA	10	64.20	0.284004
Global PIQA	10	70.00	0.400000
OpenBookQA	0	38.00	0.173333
LAMBADA	0	49.02	0.490200
CALAME-PT	0	62.14	0.621387
HellaSwag, few-shot	10	47.34	0.297865
Winograd Schema Challenge	0	65.20	0.304029
WinoGrande	0	53.99	0.079716
BIG-bench: Dyck languages	10	12.90	0.129000
AGIEval: LSAT analytical reasoning	3	23.48	0.043478
BIG-bench: CS algorithms	10	52.27	0.522727
BIG-bench: operators	10	58.10	0.580952
BIG-bench: repeat/copy logic	10	6.25	0.062500

Raw accuracy is the fraction of correct examples. The evaluator converts it to centered = (accuracy - random_baseline) / (1 - random_baseline) and reports CORE as the unweighted mean of the 17 centered scores. The configured baselines are 0.25 for HellaSwag, ARC and OpenBookQA; 0.50 for COPA, PIQA, Global PIQA, Winograd and WinoGrande; 0.20 for LSAT; and zero for the language-modeling tasks. A centered score of zero equals the configured baseline; negative scores are possible. CORE is not percentage accuracy.

Multiple-choice and schema tasks choose the candidate with the lowest mean continuation-token cross-entropy. Language-modeling tasks count an example as correct only when every target continuation token is the argmax prediction under teacher forcing; this is not an unrestricted free-generation or chat evaluation. The code shuffles each task with seed 1337, samples few-shot demonstrations from the same task's evaluation pool with seed 1234 + example_index while excluding the current example, and left-truncates overlong inputs to each model's context limit. No sampling temperature is used for these scores.

Dataset origins and translation credits

The evaluator reads packaged JSONL inputs from marlosb/auxiliary_data, eval_bundle.zip, a Portuguese adaptation of the nanochat evaluation bundle. Portuguese translations published under marlosb were produced by this model's author. This includes the translated bundle and the separately published marlosb/ai2_arc-pt.

Translations published under Polygl0t are the work of the Polygl0t team, not this model's author. Relevant Portuguese resources include Polygl0t/Hellaswag-poly, Polygl0t/ARC-poly and Polygl0t/LAMBADA-poly. Polygl0t/CALAME-PT is an evaluation-harness adaptation of the originally Portuguese CALAME-PT dataset by Ricardo Lopes, Joao Magalhaes and David Semedo, rather than an English-to- Portuguese translation.

The task links above identify the original benchmarks. The evaluator loads the bundle, not the individual Hub repositories listed here. Neither the reports nor the bundle configuration provide a per-file source/revision manifest, so an exact mapping from each bundled file to a particular translated Hub release remains unverified; matching task names alone do not establish that mapping.

Interpretation and reproducibility limits

ArthurFriedenreich scores above GPT-2, both first-generation Tucano references, and Tucano2-0.6B-Base on this aggregate, but below Tucano2-qwen-1.5B-Base. The task-level results are mixed: for example, WinoGrande and LSAT remain close to their configured baselines. Model size, architecture, training data and compute differ, so this is not a controlled scaling or training-efficiency experiment.

This is a custom 17-task CORE variant, not the original DCLM CORE suite or the historical training-report aggregate. HellaSwag is counted twice, at zero and ten shots. Seven tasks are excluded by this branch: Jeopardy, BIG-bench QA Wikidata, CommonsenseQA, SQuAD, CoQA, BoolQ and BIG-bench language identification. Do not compare its aggregate directly with CORE values from another task mix.

The launcher's defaults request all bundled examples (--max-per-task -1) and CUDA; the loader chooses BF16 on supporting CUDA hardware, otherwise FP16. However, these reports do not record the actual sample cap/counts, input split manifest, dataset/model revisions, runtime code hash, GPU or selected dtype. The code pin above documents the requested implementation, not a hash captured by the run. Consequently exact replay cannot be guaranteed from these artifacts alone. Tokenizers and context limits differ between models, and truncation can remove demonstrations. There are no repeated-run uncertainty estimates or documented contamination/translation-quality audits. These educational results do not establish production readiness.

Export correctness checks

These checks establish compatibility with the native implementation, not model quality or production readiness:

• All 175 exported tensors matched the original checkpoint in values and dtype.
• Full-model CPU logits matched native nanochat exactly for the checked prompt.
• Greedy generation and left-padded batch generation matched the reference checks.
• The tokenizer retained vocabulary and special-token IDs and passed 265 text cases.

Details are included in verification.json. These are bounded checks, not an exhaustive guarantee across hardware, dtypes, prompts, or sequence lengths.

Scope and limitations

• For learning, research, and experimentation only; not for production use.
• Portuguese-only training does not guarantee correct Portuguese facts, cultural knowledge, or reliable reasoning. The model can hallucinate and repeat itself.
• Generation recomputes the full prefix; KV caching is not implemented. use_cache=True is rejected rather than silently ignored. Generation is slower than nanochat's cached engine.
• This release provides text completion, not a claim of instruction-following or chat alignment. No chat template is provided.
• No EOS token is inferred from chat delimiters. Generation stops at the requested length unless the caller supplies a suitable stopping condition.
• Sequences exceeding 2,048 tokens are rejected. Keep prompt plus generated tokens within this limit.
• Hidden-state/attention outputs, Transformers 5, and third-party serving engines are not currently supported.
• The model is not a source of professional advice, a factual database, or a safety-aligned assistant. Do not use it for medical, legal, financial, or other consequential decisions.

Licensing

The model weights are released under the MIT License; see LICENSE. LICENSE_CODE retains the MIT notice for the nanochat-derived implementation. The educational-use and not-for-production warnings describe intended use and limitations, not additional restrictions on the MIT license.

The upstream datasets have their own licensing information and source-specific conditions. Acknowledgment does not replace reviewing those terms.

Acknowledgments

Thank you, Andrej Karpathy, for nanoGPT and nanochat. Both projects were used throughout this learning process, making the ideas and practical engineering behind language models accessible through clear, hands-on implementations.

Thank you, Nicholas Kluge Corrêa, and the collaborators behind TucanoBR/GigaVerbo, Polygl0t/gigaverbo-v2, and Polygl0t/gigaverbo-v2-synth. These resources were used in this project and made Portuguese-focused training possible.

Citation Information

Please credit the upstream dataset research and software that made this project possible using the following BibTeX entries.

GigaVerbo / Tucano

The GigaVerbo dataset card provides both the published article and its earlier preprint. These are alternative references to the same work.

@article{correa2025tucanoadvancingneuraltext,
  title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
  author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
  journal = {Patterns},
  publisher = {Elsevier},
  year = {2025},
  doi = {10.1016/j.patter.2025.101325},
  url = {https://doi.org/10.1016/j.patter.2025.101325},
  issn = {2666-3899}
}

@misc{correa2024tucanoadvancingneuraltext,
  title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
  author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
  year = {2024},
  eprint = {2411.07854},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2411.07854}
}

GigaVerbo-v2 and GigaVerbo-v2 Synth

Both the GigaVerbo-v2 dataset card and the GigaVerbo-v2 Synth dataset card request the following citation:

@misc{correa2026tucano2cool,
  title = {{Tucano 2 Cool: Better Open Source LLMs for Portuguese}},
  author = {Nicholas Kluge Corr{\^e}a and Aniket Sen and Shiza Fatimah and Sophia Falk and Lennard Landgraf and Julia Kastner and Lucie Flek},
  year = {2026},
  eprint = {2603.03543},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2603.03543}
}

nanoGPT and nanochat

The nanochat entry follows its upstream citation guidance. nanoGPT is cited as a software repository.

@misc{karpathy2022nanogpt,
  author = {Andrej Karpathy},
  title = {{nanoGPT}},
  year = {2022},
  howpublished = {GitHub repository},
  url = {https://github.com/karpathy/nanoGPT}
}

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best {ChatGPT} that \$100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

About the name

The name is a tribute to Arthur Friedenreich (1892-1969), a pioneering Brazilian footballer whose career extended into the 1930s. Known as El Tigre, he helped São Paulo Futebol Clube win the 1931 Campeonato Paulista and remains a legendary figure in Brazilian football. According to São Paulo FC's official scoring statistics, he scored 102 goals in 124 matches, averaging 0.82 goals per match: the highest average among the club's players with at least 100 goals. This model celebrates his talent, pioneering spirit, and enduring place in Brazilian sporting history.