toklens/fineweb_edu_12gb_tokenizer_split_on_whitespace

Byte-Level BPE Tokenizer: ['fw_edu'] (32K)

A Byte-Level BPE tokenizer trained on ['fw_edu'] data from Fineweb-2-HQ.

Training Details

Parameter	Value
Algorithm	Byte-Level BPE
Language	['fw_edu']
Target Vocab Size	32,000
Final Vocab Size	32,000
Pre-tokenizer	custom:_WS
Number handling	learned
Contraction handling	False
Normalizer	NFC
Special Tokens	<s>, </s>, <pad>, <unk>
Training Shards	2, ['fineweb_edu_10bt.chunk.00.jsonl', 'fineweb_edu_10bt.chunk.01.jsonl']

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("None")
tokens = tokenizer.encode("Hello, world!")

Files

• tokenizer.json — Full HuggingFace tokenizer
• vocab.json — Vocabulary mapping
• merges.txt — BPE merge rules

Sample Encoding

Text	Tokens	Token IDs
Hello, world! 12345 This is a test. こんにちは	Hel, lo,, Ġ, world, !, Ġ, 123, 45, Ġ, This, Ġ, is, Ġ, a, Ġ, test., Ġ, ãģ, ĵ, ãĤ	4082, 15644, 223, 812, 3, 223, 23512, 3536, 223, 519, 223, 271, 223, 67, 223, 9948, 223, 10115, 244, 17023

Command used to create this tokenizer:

['/home/gsa/tokenizers2/flexitok/tokenizer_training/train_tokenizers.py', 'algorithm=bpe', 'vocab_size=32_000', 'langs=[fw_edu]', 'data_dir=/scratch/gsa/data/toklens/tokenizer-training/', 'output_dir=/scratch/gsa/trained_tokenizers/toklens-whitespace', 'pretokenizer=custom:_WS', 'number_handling=learned', 'handle_contractions=false', 'unicode_normalization=nfc', 'use_byte_level_regex=false', 'strip_zero_width=false', 'max_lines=-1', 'hf.publish_to_hf=true', 'hf.hf_repo_id=toklens/fineweb_edu_12gb_tokenizer_whitespace', 'hf.collections=[toklens/fineweb2-hq-tokenizers,toklens/32_000-vocab]']