semvec/pypi-packages huggingface dataset:
whoosh-igo 0.7
tokenizers for Whoosh designed for Japanese language
License: Apache License, Version 2.0
Keywords: japanese,tokenizer

================================
 Japanese Tokenizers for Whoosh
================================

About
=====

Tokenizers for Whoosh full text search library designed for Japanese language.
This package conteins two Tokenizers.

* IgoTokenizer

 + requires igo-python(http://pypi.python.org/pypi/igo-python/) and its dictionary.

* TinySegmenterTokenizer

 + requires TinySegmenter in Python(https://code.google.com/p/mhagiwara/source/browse/trunk/nltk/jpbook/tinysegmenter.py)

* MeCabTokenizer

 * requires MeCab python binding(http://mecab.sourceforge.net/bindings.html)


How To Use
==========

IgoTokenizer::

 import igo.Tagger
 import whooshjp
 from whooshjp.IgoTokenizer import IgoTokenizer

 tk = IgoTokenizer(igo.Tagger.Tagger('ipadic'))
 scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk))


TinySegmenterTokenizer::

 import tinysegmenter
 import whooshjp
 from whooshjp.TinySegmenterTokenizer import TinySegmenterTokenizer

 tk = TinySegmenterTokenizer(tinysegmenter.TinySegmenter())
 scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk))



Changelog for Japanese Tokenizers for Whoosh
============================================

2011-02-19 -- 0.1
    * first release.

2011-02-21 -- 0.2
    * add TinySegmenterTokenizer
    * change module name

2011-02-24 -- 0.3
    * add FeatureFilter

2011-02-27 -- 0.4
    * add MeCabTokenizer
    * add a mode for don't pickle igo tagger to minimize index.

2011-04-17 -- 0.5
    * correct char offsets

2011-04-17 -- 0.6
    * correct char offsets(TinySegmenterTokenizer)

2012-04-14 -- 0.7
    * rename package(WhooshJapaneseTokenizer to whooshjp)
    * no longer import sub modules automatically
    * Python3 compatibility(3.2, 3.3)
    * Drop Python2.5 support