Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021)

Abstract: We introduce Trankit, a light-weight Transformer-based Toolkit for multilingual Natural Language Processing (NLP). It provides a trainable pipeline for fundamental NLP tasks over 100 languages, and 90 pretrained pipelines for 56 languages. Built on a state-of-the-art pretrained language model, Trankit significantly outperforms prior multilingual NLP pipelines over sentence segmentation, part-of-speech tagging, morphological feature tagging, and dependency parsing while maintaining competitive performance for tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. Despite the use of a large pretrained transformer, our toolkit is still efficient in memory usage and speed. This is achieved by our novel plug-and-play mechanism with Adapters where a multilingual pretrained transformer is shared across pipelines for different languages. Our toolkit along with pretrained models and code are publicly available at: https://github.com/nlp-uoregon/trankit . A demo website for our toolkit is also available at: http://nlp.uoregon.edu/trankit . Finally, we create a demo video for Trankit at: https://youtu.be/q0KGP3zGjGc .

## introduction many efforts have been devoted to developing multilingual nlp systems to overcome language barriers @xcite @xcite @xcite . a large portion of existing multilingual systems has focused on downstream nlp tasks that critically depend on upstream linguistic features, ranging from basic information such as token and sentence boundaries for raw text to more sophisticated structures such as part-of-speech tags, morphological features, and dependency trees of sentences (called fundamental nlp tasks). as such, building effective multilingual systems/pipelines for fundamental upstream nlp tasks to produce such information has the potentials to transform multilingual downstream systems. there have been several nlp toolkits that concerns multilingualism for fundamental nlp tasks, featuring spacy foot_0 , udify @xcite , flair @xcite , corenlp @xcite , udpipe @xcite , and stanza @xcite . however, these toolkits have their own limitations. spacy is designed to focus on speed, thus it needs to sacrifice the performance. udify and flair cannot process raw text as they depend on external tokenizers. corenlp supports raw text, but it does not offer state-ofthe-art performance. udpipe and stanza are the recent toolkits that leverage word embeddings, i.e., word2vec @xcite and fasttext @xcite , to deliver current state-ofthe-art performance for many languages. however, stanza and udpipe's pipelines for different languages are trained separately and do not share any component, especially the embedding layers that account for most of the model size. this makes their memory usage grow aggressively as pipelines for more languages are simultaneously needed and loaded into the memory (e.g., for language learning apps). most importantly, none of such toolkits have explored contextualized embeddings from pretrained transformer-based language models that have the potentials to significantly improve the performance of the nlp tasks, as demonstrated in many prior works @xcite @xcite . shared multilingual pretrained transformer joint model for pos,morphological tagging, and dependency parsing joint token and sentence splitter multi-word token expander lemmatizer @xmath0,l hierarchical native python dictionary named entity recognizer output: input: raw sentence/document string figure 1: overall architecture of trankit. a single multilingual pretrained transformer is shared across three components (pointed by the red arrows) of the pipeline for different languages. comes such limitations. our toolkit can process raw text for fundamental nlp tasks, supporting 56 languages with 90 pre-trained pipelines on 90 treebanks of the universal dependency v2.5 (zeman et al., 2019). by utilizing the state-of-the-art multilingual pretrained transformer xlm-roberta (conneau et al., 2020), trankit advances state-of-theart performance for sentence segmentation, partof-speech (pos) tagging, morphological feature tagging, and dependency parsing while achieving competitive or better performance for tokenization, multi-word token expansion, and lemmatization over the 90 treebanks. it also obtains competitive or better performance for named entity recognition (ner) on 11 public datasets. unlike previous work, our token and sentence splitter is wordpiece-based instead of characterbased to better exploit contextual information, which are beneficial in many languages. considering the following sentence: "john donovan from argghhh! has put out a excellent slide show on what was actually found and fought for in fallujah." as such, trankit correctly recognizes this as a single sentence while character-based sentence splitters of stanza and udpipe are easily fooled by the exclamation mark "!", treating it as two separate sentences. to our knowledge, this is the first work to successfully build a wordpiece-based token and sentence splitter that works well for 56 languages. figure 1 presents the overall architecture of trankit pipeline that features three novel transformer-based components for: (i) the joint token and sentence splitter, (ii) the joint model for pos tagging, morphological tagging, dependency parsing, and (iii) the named entity recognizer. one potential concern for our use of a large pretrained transformer model (i.e., xml-roberta) in trankit involves gpu memory where different transformer-based components in the pipeline for one or multiple languages must be simultaneously loaded into the memory to serve multilingual tasks. this could extensively consume the memory if different versions of the large pre-trained transformer (finetuned for each component) are employed in the pipeline. as such, we introduce a novel plugand-play mechanism with adapters to address this memory issue. adapters are small networks injected inside all layers of the pretrained transformer model that have shown their effectiveness as a lightweight alternative for the traditional finetuning of pretrained transformers @xcite @xcite . in trankit, a set of adapters (for transfomer layers) and task-specific weights (for final predictions) are created for each transformer-based component for each language while only one single large multilingual pretrained transformer is shared across components and languages. adapters allow us to learn language-specific features for tasks. during training, the shared pretrained transformer is fixed while only the adapters and task-specific weights are updated. at inference time, depending on the language of the input text and the current active component, the corresponding trained adapter and task-specific weights are activated and plugged into the pipeline to process the input. this mechanism not only solves the memory problem but also substantially reduces the training time. ## related work there have been works using pre-trained transformers to build models for character-based word segmentation for chinese @xcite @xcite ; pos tagging for dutch, english, chinese, and vietnamese @xcite @xcite @xcite ; morphological feature tagging for estonian and persian @xcite ; and dependency parsing for english and chinese @xcite . however, all of these works are only developed for some specific language, thus potentially unable to support and scale to the multilingual setting. some works have designed multilingual transformer-based systems via multilingual training on the combined data of different languages @xcite @xcite . however, multilingual training is suboptimal (see section 5). also, these systems still rely on external resources to perform tokenization and sentence segmentation, thus unable to consume raw text. to our knowedge, this is the first work to successfully build a multilingual transformer-based nlp toolkit where different transformer-based models for many languages can be simultaneously loaded into gpu memory and process raw text inputs of different languages. ## design and architecture adapters. adapters play a critical role in making trankit memory-and time-efficient for training and inference. figure 2 shows the architecture and the location of an adapter inside a layer of transformer. we use the adapter architecture proposed by @xcite , which consists of two projection layers up and down (feed-forward networks), and a residual connection. multilingual encoder with adapters. this is our core component that is shared across different transformer-based components for different languages of the system. given an input raw text s, we first split it into substrings by spaces. afterward, sentence piece, a multilingual subword tokenizer @xcite , is used to further split each substring into wordpieces. by concatenating wordpiece sequences for substrings, we obtain an overall sequence of wordpieces @xmath1, w 2 , . . . , w k ] for s. in the next step, w is fed into the pretrained transformer, which is already integrated with adapters, to obtain the wordpiece representations: here, θ l,m ad represents the adapter weights for language l and component m of the system. as such, we have specific adapters in all transformer layers for each component m and language l. note that if k is larger than the maximum input length of the pretrained transformer (i.e., 512), we further divide w into consecutive chunks; each has the length less than or equal to the maximum length. the pretrained transformer is then applied over each chunk to obtain a representation vector for each wordpiece in w. finally, x l,m 1:k will be sent to component m to perform the corresponding task. ## usage detailed documentation for trankit can be found at: https://trankit.readthedocs.io . ## conclusion and future work we introduce trankit, a transformer-based multilingual toolkit that significantly improves the performance for fundamental nlp tasks, including sentence segmentation, part-of-speech, morphological tagging, and dependency parsing over 90 universal dependencies v2.5 treebanks of 56 different languages. our toolkit is fast on gpus and efficient in memory use, making it usable for general users. in the future, we plan to improve our toolkit by investigating different pretrained transformers such as mbert and xlm-roberta large . we also plan to provide named entity recognizers for more languages and add modules to perform more nlp tasks.