P y T hai NLP : T hai Natural Language Processing in Python (2023)

Abstract: We present PyThaiNLP, a free and open-source natural language processing (NLP) library for Thai language implemented in Python. It provides a wide range of software, models, and datasets for Thai language. We first provide a brief historical context of tools for Thai language prior to the development of PyThaiNLP. We then outline the functionalities it provided as well as datasets and pre-trained language models. We later summarize its development milestones and discuss our experience during its development. We conclude by demonstrating how industrial and research communities utilize PyThaiNLP in their work. The library is freely available at https://github.com/pythainlp/pythainlp.

## introduction in recent years, the field of natural language processing has witnessed remarkable advancements, catalyzing breakthroughs for various applications. however, thai has remained comparatively underserved due to the challenges posed by limited language resources @xcite . thai is the de facto national language of thailand. it belongs to tai linguistic group within the kradai language family. according to ethnologue @xcite , there are 60.2 million users of central thai, of which 20.8 million are native @xcite . if including the northern (6 million, 2004), northeastern (15 million, 1983), and southern (4.5 million, 2006) variants, there are estimated 85.7 million users of thais speakers around the world. thai is a scriptio continua or has neither spaces nor other marks between the words or sentences in its most common writing style. @xcite . the lack of clear word and sentence boundaries leads to ambiguity that cannot be disambiguated using merely just grammatical knowledge @xcite . although many closed-source open apis for nlp have an ability to process thai language foot_0 , we this paper introduces pythainlp, an opensource thai natural language processing library written in python programming language. its features span from a simple dictionary-based word tokenizer, to a statistical named-entity recognition, and an instruction-following large language model. the library was released in 2016 under an open source initiative-approved apache license 2.0 that allows free use and modification of software, including commercial use. ## open-source thai nlp before pythainlp before pythainlp started in 2016, some free and open-source software do exist for different thai nlp tasks, but there were no unified open-source toolkits that unified multiple tools or tasks in a single library, and the number of available thai nlp datasets was low compared to high-resource languages like chinese, english, or german. natural language toolkit (nltk) @xcite , one of the most comprehensive and most popular nlp libraries in python at the time, did not support thai. opennlp, another popular free and open-source nlp toolkit written in java, started having thai models in version @xcite 3 but in version 1.5 (2010) thai was no longer listed in its supported languages 4 . open thai language resources, like annotated corpora, were also limited in size and number. "publicly available" datasets tend to have restricted access, either through restrictive licenses 5 or the registration requirement, or both. because there is a few toolkits available, limited in documentation and performance, short of rigorous benchmarking, and/or lack of maintenance, thai nlp reseachers had to spend their limited time and resources building basic components and/or collecting a dataset before they could proceed further for more advanced problems. the limited availability of source codes and datasets also affects reproducibility. examples of thai nlp tools and datasets before pythainlp: apart from the ones listed above, more opensource thai word tokenizers were released after 2009 as a result of best (benchmark for enhancing the standard of thai language processing) evaluation for thai word segmentation organized by the national electronics and computer technology center (nectec) in 2009 @xcite , and 2010 foot_2 . unfortunately, these tokenizers are no longer maintained and are not accessible at the time of writing. the most impactful contribution from best, however, is the best-2010 word segmentation dataset that was publicly released. this dataset provides a basis for a lot of modern thai open-source word segmentation software. we should also mention the thai language toolkit (tltk) @xcite . while releasing its source code a few years after, it is richer in features than pythainlp at the time. its first release on python package index (version 0.3.4, february 2018) includes statistical syllable and word segmentation @xcite , pos tagging, and spelling suggestion. its latest version, as of writing, features discourse unit segmentation, ner, grapheme-to-phoneme conversion, ipa transcription, romanization, and more. to date, tltk and pythainlp are the only two comprehensive thai nlp libraries for python. however, tltk's documentation is still quite limited. ## pythainlp and its ecosystem our primary objective is to ensure the userfriendliness and simplicity of the library. drawing inspiration from nltk, we follow numerous established interfaces. for example, word_tokenize and pos_tag. in addition, we also create datasets and pre-trained models for the thai language. figure 1 illustrates the overview of pythainlp's functionalities and its ecosystem. we will discuss here only popular features and major datasets/models. ## conclusion and future works this paper introduces the pythainlp library, explains its features and datasets (as illustrated in figure 1 ), and discusses the community and the engineering project supporting the library. by 2023, we will have implemented the opensource version of most general nlp capabilities available in english for thai 27 . we see the following items as the next major milestones: