fugashi, a Tool for Tokenizing J apanese in Python (2020)

Abstract: Recent years have seen an increase in the number of large-scale multilingual NLP projects. However, even in such projects, languages with special processing requirements are often excluded. One such language is Japanese. Japanese is written without spaces, tokenization is non-trivial, and while high quality open source tokenizers exist they can be hard to use and lack English documentation. This paper introduces fugashi, a MeCab wrapper for Python, and gives an introduction to tokenizing Japanese.

## sample code now we're ready to get started with converting plain japanese text into a list of words in python. this prints the original sentence with spaces inserted between words. in many cases, that's all you need, but fugashi provides a lot of other information, such as part of speech, lemmas, broad etymological category, pronunciation, and more. this information all comes from unidic (den et al., 2008), a dictionary provided by the national institute for japanese language and linguistics (ninjal). @xcite fugashi is a wrapper for mecab (kudo et al., 2004), a c++ japanese tokenizer. mecab is doing all the hard work here, but fugashi wraps it to make it more pythonic, easier to install, and to clarify some common error cases. you may wonder why part of speech and other information is included by default. in the classical nlp pipeline for languages like english, tokenization is a separate step before part of speech tagging. in japanese, however, knowing part of speech is important in getting tokenization right, so they're conventionally solved as a joint task. this is why japanese tokenizers are often referred to as "morphological analyzers" (形態素解析器 keitaisokaisekiki). ## notes on japanese tokenization there are several things about japanese tokenization that may be surprising if you're used to languages like english. ## comparison with other tokenizers there are a tremendous number of tokenizers for japanese, and a comprehensive comparison is beyond the scope of this paper. this is a short overview of other tokenizers usable in python. tokenizers usable in python may be broadly grouped into three categories: mecab wrappers, mecab-like tokenizers, and other tokenizers. ## summary as noted in (agirre et al., 2018), to document best practices is good, but to automate them is better. while it's hoped that newer tokenizers like sudachipy will be able to catch up in performance soon, at present fugashi is a good choice for many applications. the past year has seen many new developments in the world of japanese tokenizers. for more information on current japanese tokenizers in python, refer to konoha 25 or toiro 26 , which wrap multiple tokenizers and allow comparisons between them.