added tokenizer from v4

#1
by neuralworm - opened
flozi00 changed pull request status to merged

thanks a lot

Sign up or log in to comment