ravisonawane211/insurance-bpe-32k

Insurance BPE 32K Tokenizer

A domain-specific Byte-Level BPE tokenizer trained on English insurance-domain text.

Overview

This tokenizer is part of a domain-specific tokenizer research project that evaluates whether an insurance-focused vocabulary can improve tokenization efficiency compared with general-purpose tokenizers.

The tokenizer uses:

• Byte-Level BPE
• Vocabulary size: 32,000
• Minimum token frequency: 2
• add_prefix_space=True
• Case-sensitive tokenization
• <|endoftext|> as EOS token
• <pad> as padding token
• No <unk> token

Training Data

The training corpus combines three insurance-domain sources:

1. Insurance question-answer data
2. Insurance chatbot instruction/response data
3. Reinsurance contract text

The data was cleaned and split before tokenizer training.

Dataset Split

Dataset	Train	Held-out
Insurance QA	19,192	2,133
Insurance Chatbot	35,100	3,900
Reinsurance Contracts	892	100
Total	55,184	6,133

• Held-out ratio: 10%
• Random seed: 42
• Held-out data was not used for tokenizer training.

Tokenizer Evaluation

Five custom vocabulary sizes were evaluated:

• 16K
• 32K
• 50K
• 64K
• 100K

General-purpose tokenizers were also used as reference baselines:

• cl100k_base
• o200k_base

Tokenizer Evaluation

Five custom insurance-domain vocabulary sizes were evaluated:

• 16K
• 32K
• 50K
• 64K
• 100K

Two general-purpose tokenizers were also evaluated as reference baselines:

• cl100k_base
• o200k_base

Overall Results

Tokenizer	Vocabulary	Token Count	Fertility
Insurance BPE	16K	2,336,703	1.4079
Insurance BPE	32K	2,290,419	1.3800
Insurance BPE	50K	2,278,080	1.3726
Insurance BPE	64K	2,273,408	1.3698
Insurance BPE	100K	2,272,872	1.3695
cl100k_base	—	2,298,304	1.3848
o200k_base	—	2,285,445	1.3770

The custom insurance tokenizer improves tokenization efficiency compared with the 16K insurance baseline.

The 32K insurance tokenizer produces fewer tokens than cl100k_base (2,290,419 vs. 2,298,304), while o200k_base produces slightly fewer tokens than the 32K tokenizer (2,285,445 vs. 2,290,419).

The larger custom vocabularies achieve progressively lower token counts. However, the additional gains become increasingly small as vocabulary size increases.

For example:

• 16K → 32K: 46,284 fewer tokens
• 32K → 50K: 12,339 fewer tokens
• 50K → 64K: 4,672 fewer tokens
• 64K → 100K: only 536 fewer tokens

Therefore, 32K is currently the preferred configuration based on the balance between vocabulary size and tokenization efficiency. This selection will be validated further through downstream LLM fine-tuning experiments.

Comparison with General-Purpose Tokenizers

The general-purpose tokenizers provide useful reference points for evaluating whether an insurance-domain vocabulary provides measurable tokenization benefits.

The 32K insurance tokenizer has a fertility of 1.3800, compared with 1.3848 for cl100k_base and 1.3770 for o200k_base.

This indicates that the 32K insurance tokenizer is competitive with general-purpose tokenizers on the held-out insurance corpus, despite using a much smaller vocabulary than o200k_base.

Token-count efficiency alone does not establish downstream model improvement. LLM fine-tuning experiments are required to determine whether the domain-specific tokenizer provides practical benefits.

Insurance Domain-Term Evaluation

A curated set of 59 insurance-domain terms was used for additional evaluation.

The 32K tokenizer achieved:

• Average tokens per term: 1.5424
• Single-token coverage: 96.67%

The custom tokenizer achieved the same domain-term metrics across all tested custom vocabulary sizes.

Intended Use

This tokenizer is intended for:

• Insurance-domain NLP research
• Domain-specific tokenization experiments
• Insurance-focused LLM research
• LLM fine-tuning experiments
• Comparing domain-specific and general-purpose tokenizers

This repository contains a tokenizer, not a language model.

Limitations

This tokenizer has primarily been evaluated on English insurance-domain text.

Improved tokenization efficiency does not automatically imply improved downstream LLM performance. Further experiments are required to determine whether the tokenizer provides measurable benefits during model pretraining or fine-tuning.

The 32K vocabulary is the current preferred configuration based on the tokenization experiments. This selection may change after downstream LLM evaluation.

Usage

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained(
    "ravisonawane211/insurance-bpe-32k"
)

text = "The policyholder must pay the insurance premium."

tokens = tokenizer.tokenize(text)

print(tokens)

Output:
['ĠThe', 'Ġpolicyholder', 'Ġmust', 'Ġpay', 'Ġthe', 'Ġinsurance', 'Ġpremium', '.']

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")

text = "The policyholder must pay the insurance premium."

tokens = tok.tokenize(text)

print(tokens)

Output:
['The', 'Ġpolicy', 'holder', 'Ġmust', 'Ġpay', 'Ġthe', 'Ġinsurance', 'Ġpremium', '.']