semvec/pypi-packages huggingface dataset:
pytoki 0.1.2
toki Python bindings
License: MIT

# About

Toki library was originally developed by Tomasz Śniatowski and Adam Radziszewski at Wroclaw University of Science and Technology. The main purpose of the library was to provide fast [SRX](https://en.wikipedia.org/wiki/Segmentation_Rules_eXchange)-based tokenizer. The following python library is a set of python bindings to C++ toki that has been further developed at [Alphamoon](https://alphamoon.ai/). 

Original toki has been released under GNU LGPL 3.0. The sources may be obtained from the git repositories:
```
git clone http://nlp.pwr.wroc.pl/corpus2.git # contains pwrutils library that is needed for building toki
git clone http://nlp.pwr.wroc.pl/toki.git
```
To build the codes you will need CMake 2.8 or later. Besides, you will need:

* ICU 4.2
* Boost 1.41 or later (tested with 1.41 and 1.42)
* Loki (libloki-dev)
* libxml++2.6 (for SRX support)
* libpwrutils from corpus2 repository (its build process is based on CMake, see the project site)

1. To create a working tokeniser, instantiate ```Toki::LayerTokenizer```. There are several constructors available; the simplest one assumes using the default configuration (for Polish). To access a named configuration, use ```Toki::get_named_config```(config_name) and pass the acquired object to ```Toki::LayerTokenizer``` constructor.
2. To create a working tokeniser with sentence-splitter, first instantiate a ```Toki::LayerTokenizer``` object and then wrap a ```Toki::SentenceSplitter``` around it. The sentencer object contains a convenient has_more-get_next_sentence interface. The default config loads sentence-splitting rules so is suitable for this purpose.
NOTE: when using a custom config, check whether it contains working sentence-splitting rules. If it doesn't, ```Toki::SentenceSplitter``` will buffer all the input and finally produce an enormous sentence containing all the tokens.

# Examples

For now, Python interface is simple and allows only for sentence splitting and tokenizing within the sequence with polish as a default language.

Sentence splitting: 
```python
import toki
tokenizer = toki.Toki()
tokenizer.get_all_sentences("To jest zdanie. To jest np. inne zdanie.")
```
Sentence tokenizing: 
```python
import toki
tokenizer = toki.Toki()
tokenizer.get_all_sentences_tokenized("To jest zdanie. To jest np. inne zdanie.")
```

More languages will be supported in upcoming releases.

It is recommended to build package from source if possible to make use of AVX and other CPU instruction. Package originally has been built with `core2` optimization so any CPU older than that or which does not have `MMX`, `SSE`, `SSE2`, `SSE3` and `SSSE3` must build package from source.