D adma T ools: Natural Language Processing Toolkit for P ersian Language (2022)

Abstract: We introduce DadmaTools, an open-source Python Natural Language Processing toolkit for the Persian language. The toolkit is a neural pipeline based on spaCy for several text processing tasks, including normalization, tokenization, lemmatization, part-of-speech, dependency parsing, constituency parsing, chunking, and ezafe detecting. DadmaTools relies on fine-tuning of ParsBERT using the PerDT dataset for most of the tasks. Dataset module and embedding module are included in DadmaTools that support different Persian datasets, embeddings, and commonly used functions for them. Our evaluations show that DadmaTools can attain state-of-the-art performance on multiple NLP tasks. The source code is freely available at https://github.com/Dadmatech/DadmaTools .

## introduction with the increased accessibility of open-source natural language processing toolkits, users are now able to more easily develop tools that perform sophisticated linguistic tasks. there are several persian nlp toolkits, such as stanza @xcite , hazm foot_0 , parsivar foot_1 , and jptdp @xcite . however, they suffer from several limitations. first, most persian toolkits are based on conventional non-neural models which prevents them from taking full advantage of the recent developments in the filed. examples include parsivar @xcite , step1 @xcite , virastar foot_2 , virastyar foot_3 , and parsianalyzer foot_4 . second, most persian toolkits either do not cover all the basic processing tools (e.g., perstem foot_5 and farsinlptools) or are not open-source (e.g., farsi-yar foot_6 ). third, there is no single toolkit that provides state-of-the-art results across different basic tasks. we introduce dadmatools, an open-source python natural language processing toolkit for persian. dadmatools provides the following advantages compared to existing toolkits: moreover, dadmatools is based on the spacy framework which allows users to integrate our toolkit with other piplelines implemented in spacy. we note that stanza is a widely used persian toolkit. hence, we mainly compare dadmatools to stanza. many of the standard tasks, such as constituency parsing, chunking, and ezafe detection 8 , are not supported by stanza for persian. in addition to covering these tasks, our toolkit also provides support for datasets and word embeddings. dadmatools is toolkit normalizer lemma pos dependency constintuency chunker ezafe ## system usage it is possible to use the normalizer directly without triggering the pipeline using the normalizer class. the dadmatools pipline can be also initialized with pipline class. by default, only the tokenizer with sentence splitting and mwt are loaded. however, it is possible to load custom processors by adding their names as arguments. the pipeline will generate a doc instance, which contains all the raw text's properties regarding the processes that have been called, in the form of the spacy doc. the following code snippet shows a minimal usage example of dadmatools. dadmatools is designed to run on different types of hardware (cuda and cpu). priority is given to cuda devices, if available. ## training datasets there are multiple persian datasets that provide training data for various nlp tasks. ## experiments models presented in lemmatizer. we compare dadmatools against stanza and hazm on the lemmatization task. as shown in pos tagger. for this experiment, we compared dadmatools against stanza given that both models use the same dataset for their training. as shown in ## conclusion and future work we introduced dadmatools, an open-source python natural language processing toolkit for the persian language. dadmatools supports different nlp tasks. moreover, it is based on the spacy framework which allows users to integrate the toolkit with other processors in a pipeline. as future work, we intend to extend the supported tasks by adding high-level nlp tasks, such as sentiment analysis, entailment, and summarization. we also plan to provide users with the ability to add new datasets and models to the toolkit. we hope that the toolkit can pave the way for research and development for the persian language.