semvec/pypi-packages huggingface dataset:
tmatch 0.1.1
Super fast token matcher

# Super fast token matcher

```
pip3 install tmatch
```

Very fast matching of multiple sequences of text to a single text.

Use TokenMatcher to `add_tokens` for tokens that you want to match.

To match tokens all you need is to call `get_matches`.

```python
from tmatch import TokenMatcher
tm = TokenMatcher()
tm.add_tokens("token matcher".split(" "))
tm.add_tokens("very".split(" "))
tm.add_tokens("fast".split(" "))
tm.add_tokens("very fast token".split(" "))
tm.add_tokens("fast token matcher".split(" "))
# returns all matches of tokens in add_tokens
# Note that fast is not extracted below as ["fast"] is in ["very","fast","tokens"] However if fast is somewhere else in the text it will be extracted.
tm.get_matches("token matcher is a very fast token matcher implemented in pure python".split(" ")) 
>> [['token', 'matcher'], ['very', 'fast', 'token'], ['fast', 'token', 'matcher']]
```

# Implementation details
`add_tokens` add the token into a trie. This has time complexity of O(L).

`get_matches` find all matches in tokens from `add_tokens` this traverses both tokens and trie tokens at the same time. Has time complexity of O(NL).
Where N is the tokens length and L is the **max** length of tokens added.