michelepao1993/mosar-final-checkpoints

Final Checkpoint Artifacts

This repository contains the final checkpoint artifacts for the submission.

The checkpoints are provided for reproducibility and correspond to the eight attention variants evaluated at training step 610000.

Contents

• checkpoints/*.tar: final Megatron checkpoint archives
• MANIFEST.tsv: archive names, variants, sizes, and SHA-256 hashes
• CHECKSUMS.sha256: full checksum file
• SIZES.txt: archive sizes

The included variants are:

• vanilla
• alibi
• prope075
• s_fixed
• m_fixed
• mosar0
• cost001
• rope_m_mask

Checkpoint format

Each archive contains one final Megatron checkpoint directory:

iter_0610000/

These checkpoints are not stored in Hugging Face Transformers format and are not intended to be loaded with:

AutoModel.from_pretrained(...)

They are intended to be used with the accompanying Megatron-based training and evaluation code.

Training setup summary

The checkpoints correspond to decoder-only language models trained from scratch using a Megatron-based runtime.

High-level configuration:

• Training step: 610000
• Sequence length: 2048
• Vocabulary size: 256000
• Tokenizer: Gemma tokenizer
• Source corpus: English Wikipedia
• Dataset split: 990,10,0
• Approximate indexed corpus size: 4,975,435,463 tokens

The exact code, configuration files, evaluation scripts, and reproduction instructions are provided in the accompanying code repository.

Integrity verification

After downloading the artifact, verify the files with:

sha256sum -c CHECKSUMS.sha256

To extract a checkpoint archive:

tar -xf checkpoints/mosar0_step610000_iter_0610000.tar

This will produce:

iter_0610000/

Notes

These files are provided as reproducibility artifacts for review. They are not packaged as general-purpose Hugging Face Transformers checkpoints.

Machine-specific paths, usernames, allocation names, job IDs, and node names are intentionally omitted from this artifact. File names and archive contents are organized using only variant names and checkpoint identifiers.