semvec/pypi-packages huggingface dataset:
ZCodeTokenizers 0.1.2
ZCodeTokenizers is the python wrapper of the sentencepiece-based tokenizers for ZCode.
License: MIT
Requires Python: >=3.6

[![Build Status](https://msasg.visualstudio.com/SDRG/_apis/build/status/ZCode/Tokenization/DeployZCodeTokenizers?branchName=main)](https://msasg.visualstudio.com/SDRG/_build/latest?definitionId=16202&branchName=main)
# ZCodeTokenizers 
This package provides a python wrapper for the ZCode tokenizers. It has been tested on Windows x64 and Linux x64. 

# Usage Instructions

The python interface implements the HuggingFace `PreTrainedTokenizer` class for ZCode.

```python
from ZCodeTokenizers import ZCodeTokenizer
# initialize tokenizer from checkpoint directory containing a spiece.model file.
tokenizer = ZCodeTokenizer.from_pretrained(os.path.join('models', 'zcodeexpert'))
text = "Hello, World!"
tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text, add_special_tokens=False)
decoded_text = tokenizer.decode(ids)
print(f"{text} => {tokens} => {ids} => {decoded_text}")
```

# Build and Test
To run the python tests, run the following from the project root:
```
$ python -m unittest discover ./test
```

# Contribute
TODO