S tanza: A Python Natural Language Processing Toolkit for Many Human Languages (2020)

Abstract: We introduce Stanza, an open-source Python natural language processing toolkit supporting 66 human languages. Compared to existing widely used toolkits, Stanza features a language-agnostic fully neural pipeline for text analysis, including tokenization, multi-word token expansion, lemmatization, part-of-speech and morphological feature tagging, dependency parsing, and named entity recognition. We have trained Stanza on a total of 112 datasets, including the Universal Dependencies treebanks and other multilingual corpora, and show that the same neural architecture generalizes well and achieves competitive performance on all languages tested. Additionally, Stanza includes a native Python interface to the widely used Java Stanford CoreNLP software, which further extends its functionality to cover other tasks such as coreference resolution and relation extraction. Source code, documentation, and pretrained models for 66 languages are available at https://stanfordnlp.github.io/stanza/ .

## introduction the growing availability of open-source natural language processing (nlp) toolkits has made it easier for users to build tools with sophisticated linguistic processing. while existing nlp toolkits such as corenlp @xcite , flair @xcite , spacy 1 , and udpipe @xcite have had wide usage, they also suffer from several limitations. first, existing toolkits often support only a few major languages. this has significantly limited the community's ability to process multilingual text. second, widely used tools are sometimes under-optimized for accuracy either due to a focus on efficiency (e.g., spacy) or use of less powerful models (e.g., corenlp), potentially mislead- ## system design and architecture at the top level, sta n z a consists of two individual components: (1) a fully neural multilingual nlp pipeline; (2) a python client interface to the java stanford corenlp software. in this section we introduce their designs. ## system usage sta n z a 's user interface is designed to allow quick out-of-the-box processing of multilingual text. to achieve this, sta n z a supports automated model download via python code and pipeline customization with processors of choice. annotation results can be accessed as native python objects to allow for flexible post-processing. ## performance evaluation to establish benchmark results and compare with other popular toolkits, we trained and evaluated sta n z a on a total of 112 datasets. all pretrained models are publicly downloadable. datasets. we train and evaluate sta n z a 's tokenizer/sentence splitter, mwt expander, pos/ufeats tagger, lemmatizer, and dependency parser with the universal dependencies v2.5 treebanks @xcite . for training we use 100 treebanks from this release that have non-copyrighted training data, and for treebanks that do not include development data, we randomly split out 20% of the training data as development data. these treebanks represent 66 languages, mostly european languages, but spanning a diversity of language families, including indo-european, afro-asiatic, uralic, turkic, sino-tibetan, etc. for ner, we train and evaluate sta n z a with 12 publicly available datasets covering 8 major languages as shown in training. on the universal dependencies treebanks, we tuned all hyper-parameters on several large treebanks and applied them to all other treebanks. we used the word2vec embeddings released as part of the 2018 ud shared task @xcite , or the fasttext embeddings @xcite whenever word2vec is not available. for the character-level language models in the ner component, we pretrained them on a mix of the common crawl and wikipedia dumps, and the news corpora released by the wmt19 shared task @xcite , except for english and chinese, for which we pretrained on the google one billion word @xcite and the chi-nese gigaword corpora foot_3 , respectively. we again applied the same hyper-parameters to models for all languages. ## conclusion and future work we introduced sta n z a , a python natural language processing toolkit supporting many human languages. we have showed that sta n z a 's neural pipeline not only has wide coverage of human languages, but also is accurate on all tasks, thanks to its language-agnostic, fully neural architectural design. simultaneously, sta n z a 's corenlp client extends its functionality with additional nlp tools. for future work, we consider the following areas of improvement in the near term: