marin-community/marin-data-mix-tools

Marin data mix tools

This repository publishes the model materials used to score and cluster documents for the August 18, 2026 Harrier data mix.

Version 08.18.2026

• fast-transformer/ contains Mark Muchane's nemotron88k_v1 FastTransformer checkpoint. It consumes Nemotron-tokenized document text and a 1024-dimensional Harrier embedding, and emits a scalar document-quality score.
• kmeans/ contains 5,000 spherical K-means centroids. lookup_5000_to_40.npy maps each fine cluster ID to one of 40 coarse domain buckets.
• manifest.json records each file's upstream S3 URI, byte size, SHA-256 digest, and license identifier.

All model materials under this version are licensed under the OpenMDW License Agreement, version 1.1.