Nebula V18 — raw text and GPU experiment
Public experimental continuation of the owner's Nebula weights. Durable datasets, packed byte-token shards, source indexes, model/optimizer/scaler checkpoints and reports are uploaded directly from RAM. No dataset or checkpoint is written to the worker's filesystem. Python code, installed packages and the notebook itself can exist on disk. This runner does not ingest account chats or review feeds. Inherited weights and public web data are not guaranteed free of personal information. No website deployment or automatic quality approval is enabled.
Licenses
Each corpus retains its upstream terms and attribution; this collection does not relicense it. Pinned upstream cards are stored with each run; original FineWeb document URLs are retained in compressed indexes. The new textbooks component is synthetic Cosmopedia text seeded from OpenStax outlines. ODC-By: https://opendatacommons.org/licenses/by/1-0/ CDLA-Sharing-1.0: https://cdla.dev/sharing-1-0/ Apache-2.0: https://www.apache.org/licenses/LICENSE-2.0
- fineweb: HuggingFaceFW/fineweb-edu,
sample-10BT,odc-by. - cosmopedia: HuggingFaceTB/smollm-corpus,
cosmopedia-v2,odc-by. - stories: roneneldan/TinyStories,
default,cdla-sharing-1.0. - textbooks: HuggingFaceTB/cosmopedia,
openstax,apache-2.0.
Representation and limitations
Packed tokens are little-endian uint16: byte IDs 0–255, BOS=256, EOS=257. Whole UTF-8 documents are normalized-exact-deduplicated and assigned by hash to train or validation before packing. Credential-pattern/control/size filters are applied; this is not a full PII, factual or fuzzy-duplicate audit. The model may have seen overlapping public text before this experiment. Source mix percentages describe acquired UTF-8 bytes; training samples packed positions, including repeated exposure. Training batches use deterministic shard locality (64 updates per sampled shard) to bound network traffic. At most two training shards and two validation shards are cached in RAM; deduplication hashes also use RAM. Full optimizer snapshots consume substantial host RAM and public storage; GPU memory alone is not sufficient. Only confirmed public checkpoints survive worker loss. The history is retained and public storage is not unlimited.