semvec/pypi-packages huggingface dataset:
pyTokenizer 1.1.4
A streaming tokenizer.
Requires Python: >=3.5
Keywords: Python3 Token Tokenizer Streaming

[![Sourcecode on GitHub](https://img.shields.io/badge/Paebbels-pyTokenizer-323131.svg?logo=github&longCache=true)](https://github.com/Paebbels/pyTokenizer)
[![License](https://img.shields.io/badge/code%20license-Apache%20License%2C%202.0-lightgrey?logo=GitHub)](LICENSE.md)
[![GitHub tag (latest SemVer incl. pre-release)](https://img.shields.io/github/v/tag/Paebbels/pyTokenizer?logo=GitHub&include_prereleases)](https://github.com/Paebbels/pyTokenizer/tags)
[![GitHub release (latest SemVer incl. including pre-releases)](https://img.shields.io/github/v/release/Paebbels/pyTokenizer?logo=GitHub&include_prereleases)](https://github.com/Paebbels/pyTokenizer/releases/latest)
[![GitHub release date](https://img.shields.io/github/release-date/Paebbels/pyTokenizer?logo=GitHub&)](https://github.com/Paebbels/pyTokenizer/releases)  
[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/Paebbels/pyTokenizer/Test,%20Coverage%20and%20Release?label=Workflow&logo=GitHub)](https://github.com/Paebbels/pyTokenizer/actions?query=workflow%3A%22Test%2C+Coverage+and+Release%22)
[![PyPI](https://img.shields.io/pypi/v/pyTokenizer?logo=PyPI)](https://pypi.org/project/pyTokenizer/)
![PyPI - Status](https://img.shields.io/pypi/status/pyTokenizer?logo=PyPI)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pyTokenizer?logo=PyPI)
[![Dependent repos (via libraries.io)](https://img.shields.io/librariesio/dependent-repos/pypi/pyTokenizer)](https://github.com/Paebbels/pyTokenizer/network/dependents)  
[![Libraries.io status for latest release](https://img.shields.io/librariesio/release/pypi/pyTokenizer)](https://libraries.io/github/Paebbels/pyTokenizer)
[![Requires.io](https://img.shields.io/requires/github/Paebbels/pyTokenizer)](https://requires.io/github/Paebbels/pyTokenizer/requirements/?branch=master)  
[![Codacy - Quality](https://img.shields.io/codacy/grade/7ea6c9eea5b94493837ea55c07de3562?logo=Codacy)](https://www.codacy.com/manual/Paebbels/pyTokenizer)
[![Codacy - Coverage](https://img.shields.io/codacy/coverage/7ea6c9eea5b94493837ea55c07de3562?logo=Codacy)](https://www.codacy.com/manual/Paebbels/pyTokenizer)
[![Codecov - Branch Coverage](https://img.shields.io/codecov/c/github/Paebbels/pyTokenizer?logo=Codecov)](https://codecov.io/gh/Paebbels/pyTokenizer)
[![Libraries.io SourceRank](https://img.shields.io/librariesio/sourcerank/pypi/pyTokenizer)](https://libraries.io/github/Paebbels/pyTokenizer/sourcerank)  
[![Read the Docs](https://img.shields.io/readthedocs/pytokenizer)](https://pyTokenizer.readthedocs.io/en/latest/)

# pyTokenizer

A streaming tokenizer.


## Contributors

* [Patrick Lehmann](https://github.com/Paebbels) (Maintainer)


## License

This Python package (source code) is licensed under [Apache License 2.0](LICENSE.md).

<!-- The accompanying documentation is licensed under Creative Commons - Attribution-4.0 (CC-BY 4.0). -->

-------------------------

SPDX-License-Identifier: Apache-2.0