guicybercode/br-sovereign-llm-research

BR Sovereign LLM

BR Sovereign LLM is a research protocol and implementation for training Brazilian Portuguese decoder-only language models from scratch on locally controlled and national high-performance computing infrastructure. The project tests whether data lineage, checkpoint custody, offline execution, scheduler portability, and open artifact formats can be maintained without sacrificing measured language-model quality or operational efficiency.

This repository is at the protocol and pipeline-validation stage. It does not yet contain trained research weights or results from a Brazilian supercomputer. Unmeasured claims remain explicitly marked not_measured.

Paper

The public 10-page English manuscript is available as paper/manuscript.pdf. Its deterministic arXiv source bundle, submission metadata, and author submission guide are published alongside it.

Status: public pre-arXiv manuscript version 1.0.0. An arXiv identifier and a Hugging Face Paper Page have not yet been assigned. Target-scale model quality, Brazilian-supercomputer execution, energy, and residency remain not_measured.

Research contribution

Portuguese language models already exist at several scales. TeenyTinyLlama, Tucano, Tucano 2, Sabiá, and related projects rule out broad novelty claims such as "the first Portuguese model". This project instead studies operational sovereignty as a set of independently verifiable capabilities:

1. complete source, rights, and transformation lineage;
2. local custody and hash verification of required artifacts;
3. training, recovery, evaluation, and export without a mandatory remote API;
4. reproducible environments and a rebuildable software supply chain;
5. equivalent execution across validated local and Slurm profiles;
6. deterministic checkpoint recovery and data-cursor restoration;
7. exportable tokenizer, configuration, and weights in open formats;
8. Brazilian Portuguese quality measured by bytes, compute, and domain;
9. explicit disclosure of vendor and jurisdictional dependencies.

No aggregate sovereignty score is reported. A single number could hide a critical dependency behind unrelated strengths. Every capability is paired with evidence, a target, and one of pass, fail, or not_measured.

Primary research question

Can a Brazilian institution train, recover, evaluate, and serve a Brazilian Portuguese language model from scratch without mandatory external cloud services, while retaining complete legal and technical lineage for every critical artifact?

The prospectively specified hypotheses test execution equivalence, checkpoint recovery, small-scale prediction, Brazilian data allocation, governance overhead, and offline operability. Exact outcomes, equivalence margins, exclusions, and statistical procedures are defined in docs/research/preregistration.md.

Model scale ladder

The final scale is conditional on licensed unique-token inventory, measured throughput, accelerator allocation, storage bandwidth, and scaling results. The planning ladder is not a promise of completed training.

Stage	Parameters	Planning tokens	Purpose
nano	20–30M	0.25–0.50B	pipeline, overfit, and recovery validation
micro	100–125M	2–3B	tokenizer, mixture, and seed experiments
small	about 350M	about 7B	single-node scaling holdout
medium	about 1.3B	about 26B	first conditional multi-node target
target	about 3B	about 60B	conditional national HPC study
stretch	about 7B	about 140B	future work outside the initial study

Each model uses the same Llama-style decoder family with causal attention, RMSNorm, RoPE, SwiGLU, and grouped-query attention. Tokenizers and weights are initialized and trained by this project. Scaling experiments keep a 2,048-token context to avoid confounding model scale with context length.

Corpus policy

Public accessibility is not evidence of permission. A source enters the training pool only when each released document has a rights basis, evidence, attribution, revision, content hash, and transformation history. The pipeline rejects unknown licenses, noncommercial restrictions, no-derivatives terms, missing evidence, and ambiguous aggregate web corpora.

The initial registry permits controlled acquisition from:

• official Portuguese Wikimedia dumps under the applicable CC BY-SA terms;
• Portal da OBMEP items whose CC BY 3.0 Brazil coverage is verified;
• Brazilian laws, decrees, regulations, and other official acts covered by Article 8(IV) of Law 9,610/1998;
• ProEdu resources with an exact item-level CC0, CC BY, or CC BY-SA license;
• SciELO Brasil articles with an exact item-level CC BY 4.0 license.

The current registry hashes its recorded rights assertions for tamper detection; those statement hashes are not independent legal proof. Before a scientific document is admitted, the pipeline additionally requires a source revision, snapshot hash, item attribution, and, for conditional sources, archived evidence-byte hash and media type. Final public release still requires legal review.

The main release excludes BrWaC, Carolina as an aggregate, CulturaX, mC4, OSCAR, HPLT, Common Crawl, GigaVerbo as an aggregate, and other collections whose package terms do not establish rights over every underlying text. Rejected sources can be recorded for audit but cannot produce publishable training shards.

Corpus documents retain their source-specific licenses. Apache-2.0 applies to the project-controlled code and documentation, not to a future mixed text collection.

System boundary

source registry and rights evidence
  -> immutable raw snapshots
  -> extraction and Unicode normalization
  -> PII, language, and quality checks
  -> exact and near deduplication
  -> benchmark decontamination
  -> deterministic group-aware splits
  -> licensed corpus snapshot in Parquet
  -> project-trained byte-level tokenizer
  -> content-addressed token shards
  -> local or Slurm training backend
  -> recoverable distributed checkpoints
  -> evaluation and sovereignty evidence
  -> reviewed publication mirror

Hugging Face is a publication mirror, not the authoritative archive. A controlled content-addressed store remains the source of record so that the research does not make an external commercial platform a mandatory custodian.

Quick start

Python 3.11, 3.12, or 3.13 is required. Synchronize the universal lockfile into an environment inside this repository:

python3.13 -m pip install 'uv==0.12.6'
uv sync --frozen --all-extras
source .venv/bin/activate
make check
make coverage sbom verify-wheel

Inspect the command surface:

br-sovereign-llm --help
br-sovereign-llm validate-sources
br-sovereign-llm fingerprint \
  --study configs/experiments/scaling-study.yaml \
  --runtime configs/runtime/local-cpu.yaml
br-sovereign-llm evidence-report --output artifacts/evidence-report.json

An installed wheel does not embed a mutable research corpus or site policy. When invoking it outside the checkout, identify the audited resource boundary:

br-sovereign-llm \
  --project-root /absolute/path/to/br-sovereign-llm \
  validate-project

Optional dependencies are installed by purpose:

python -m pip install -e '.[corpus,tokenizer]'
python -m pip install -e '.[training]'
python -m pip install -e '.[publication]'

Training dependencies are intentionally separate because cluster builds must pin PyTorch and the executor to versions validated for the target driver and accelerator. The generic Python metadata does not replace the reviewed container digest or the cluster software contract.

Configuration boundary

Scientific and execution settings have different fingerprints:

• study_fingerprint covers corpus snapshot, tokenizer, model, optimizer, batch order, seed, token budget, and numerical policy;
• execution_fingerprint covers container, hardware, world size, parallelism, drivers, kernels, and scheduler mechanism.

Runtime profiles may not silently override scientific parameters. Slurm accounts, partitions, hostnames, storage paths, and credentials belong in an ignored site.yaml or environment variables.

Validation gates

The release process is fail-closed. Machine-readable reports are emitted for repository integrity, legal admissibility, acquisition integrity, corpus quality, lineage, benchmark isolation, tokenizer quality, training semantics, HPC preflight, prospective registration, evaluation, offline reproduction, and release integrity. Public visibility for version 0.1.1 was a separate owner decision after the scoped review in docs/governance/public-release-review-v0.1.1.md. That decision does not authorize a future scientific corpus or checkpoint.

The current evidence table is in docs/governance/sovereignty-evidence.md. HPC measurements, energy, data residency, and multi-node equivalence remain not_measured until real infrastructure evidence exists.

Repository map

• registry/: source, license, and benchmark policy
• configs/: corpus, tokenizer, model, experiment, and runtime specifications
• src/br_sovereign_llm/: deterministic pipeline and evidence code
• infrastructure/: container and Slurm adapters without site secrets
• docs/research/: protocol, hypotheses, and statistical analysis
• docs/governance/: data and publication controls
• docs/operations/: local, Slurm, recovery, and offline runbooks
• paper/: English technical manuscript, arXiv metadata, and submission guide
• hub/: artifact-specific dataset and model cards
• tests/: unit, integration, recovery, and smoke validation

Current limitations

• No supercomputer allocation or cluster-specific configuration has been provided.
• No multi-node throughput, MFU, energy, or data-residency result is claimed.
• No full corpus snapshot or target-scale checkpoint has been released.
• A new item-level rights and legal review remains required before any public scientific corpus or target-scale weight release.
• Accelerator portability must be demonstrated independently for each tested CUDA or ROCm environment.
• The current runtime fingerprint does not yet commit the backend source pin, artifact digest, entrypoint, and arguments. G8 and scientific execution claims remain blocked until that contract is completed and tested.
• The planned paired TOST for H1 is not implemented in protocol version 0.1.0; the available normal-interval helper is descriptive only.

Author

Guilherme Monteiro, FIAP
ORCID: 0009-0008-5294-224X

License

Project-controlled code and documentation are released under Apache-2.0 to the extent of the owner's rights. Dataset records, benchmarks, model weights, and third-party components use artifact-specific terms described in their manifests and cards.