semvec/pypi-packages huggingface dataset:
machinemind-bpe 1.0.1
A Byte Pair Encoding (BPE) tokenizer package for CPU and GPU.
License: MIT

BPE Tokenizer
This package provides an implementation of Byte Pair Encoding (BPE) tokenizer, designed to be efficient and fast. It includes training on both CPU and GPU, along with encoding and decoding functionalities.

Features
Train BPE tokenizer on CPU and GPU
Encode and decode text using the trained tokenizer
Save and load merges and vocabulary to/from JSON files
Support for special tokens and customizable regex patterns
Installation
To install the package, use the following command:

pip install machinemind_bpe