End-to-end NLP Pipelines in Rust (2020)

Abstract: The recent progress in natural language processing research has been supported by the development of a rich open source ecosystem in Python. Libraries allowing NLP practitioners but also non-specialists to leverage state-of-the-art models have been instrumental in the democratization of this technology. The maturity of the open-source NLP ecosystem however varies between languages. This work proposes a new open-source library aimed at bringing state-of-the-art NLP to Rust. Rust is a systems programming language for which the foundations required to build machine learning applications are available but still lacks ready-to-use, end-to-end NLP libraries. The proposed library, rust-bert, implements modern language models and ready-to-use pipelines (for example translation or summarization). This allows further development by the Rust community from both NLP experts and non-specialists. It is hoped that this library will accelerate the development of the NLP ecosystem in Rust. The library is under active development and available at https://github.com/guillaume-be/rust-bert .

## introduction natural language processing (nlp) has undergone a rapid transformation over the last few years. modern architectures based on the transformers @xcite , leveraging efficiently the large amount of data available for unsupervised pretraining, have enabled significant progress for a variety of tasks including sentiment analysis, question answering, summarization or translation. these research efforts have been accompanied by the development of a rich python ecosystem enabling a democratization of these technologies for both practitioners and users, from tokenization to deep learning architectures. the transformers library @xcite is an example of a library propos-ing apis at various levels to either promote further development of nlp or their integration in higher level applications. the adoption of these technologies in other programming languages has unfortunately not been as fast, for example in rust. rust @xcite is a promising modern static, strongly typed language that offers execution speeds similar to c. its built-in memory safety design makes it an attractive alternative to c++ for the development of productive machine learning systems. rust does not include a garbage collector but instead relies on strict ownership rules for the variables, dropping them when going out of scope. its modern implementation of the strings data model that complies with utf-8 standards is especially relevant to nlp applications. finally, rust includes a powerful utility called cargo to manage external dependencies. this allows the development of open-source ecosystems, similar to python's pypi (python packaging authority, 2000) or java's maven @xcite . rust is a modern programming language for which the foundations of a machine learning ecosystem are still being built. a number of initiatives including array manipulation (rust-ndarray team, 2011), low-level cuda libraries and deep learning framework bindings for tensorflow (tensorflow project, 2016) or torch @xcite ) are now maturing. however, there is still a lack of end-to-end, ready to use libraries leveraging stateof-the-art nlp models. the proposed library aims at filling this gap and exposes both transformersbased architectures to nlp practitioners in rust and pipelines that are ready for integration in rust-based back-ends. the proposed library, rust-bert, is available at https://github.com/ guillaume-be/rust-bert or https://crates. io/crates/rust-bert and is shared under apache 2.0 license. ## related work this work leverages the rich open-source resources available in python. especially relevant is the transformers library @xcite , of which large sections of the proposed rust library were ported from. the model architectures and layers naming have been aligned with the transformers implementation, and rust-compatible pre-trained weights are available in hugging @xcite . the general api for the high-level and ready-to-use pipelines has been strongly inspired by the spacy library @xcite . a number of specialized libraries have been developed for rust, including high performance tokenizers ((hugging face, 2020) or @xcite ) and language detection libraries @xcite . ## architecture design the library exposes three main features: library root pipelines models bert translation summarization bert.rs embedding.rs attention.rs transformer.rs the language models and the pipelines are separated in different modules. within the models, a sub-module is defined for each model (for example, bert) with individual files for the major model components (for example, its attention mechanism). this promotes readability and modularity of the code base (figure 1 ). an important design aspect of the library is related to the choice of abstractions. rust does not implement the concept of classes and inheritance in a similar way to python. rather, data is arranged in structs that may implement associated methods in an impl block or shared behaviour via traits. as opposed to python, layers do not inherit from a shared nn.module because rust requires a strict definition of the names and types of the inputs and outputs (those may differ significantly from model to model). as a consequence the registration of the model parameters in the variable store is done manually: ## capabilities overview the library exposes an api at two different levels: the language models themselves, allowing to build nlp pipelines from scratch, and end-to-end pipelines that can readily be integrated in higher level applications. a rust implementation for a wide range of language models has been implemented, including bert @xcite , distilbert @xcite , roberta @xcite , gpt @xcite , gpt2 @xcite , al-bert @xcite , bart @xcite , marian (junczys-dowmunt et al., 2018), xlm-roberta @xcite and t5 @xcite . for each of these models, pre-trained weights have been converted to a c-array format and are hosted alongside the python version on hugging face's model hub @xcite . a large user base of nlp technologies also benefits from the availability of state of the art, end-toend pipelines requiring little to no familiarity with nlp to be integrated in higher level applications. to answer these needs of the rust community, the following capabilities have been implemented: these pipelines can be created and used in a few lines of code without prior knowledge in nlp. while the implementation of the language models is a prerequisite, the availability of powerful end-to-end pipelines is key to a broader adoption of nlp technology in rust. these pipelines can easily be integrated with server back-ends running rust with queuing and batching of incoming requests (walsh, 2020). ## benchmarks this library was developed with the primary goal of making state of the art nlp capabilities available to the rust community rather than speeding up inference. nevertheless, rust is a high performance language with execution speeds matching c or c++. efficient predictions using nlp systems has become a key subject of research and engineering development over the past few months. several methods have been investigated to improve the 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 relative execution time (@xmath0 bart large model loading (gpu) sst2 forward pass (2000 ex.) squad forward pass (1000 ex.) t5-base translation (3 ex.) t5-base translation (10 ex.) bart summarization (1 ex.) marian translation (1 ex.) tensor multiplication (15k repeat) task python time (s) 7.73 1.62 58.6 3.08 4.65 1.68 1.21 0.34 python rust c++ figure 2: rust-python benchmark results. it should be noted that the proposed library is based on bindings @xcite to libtorch @xcite , and therefore limited benefits can be expected from the tensor operations. these are executed in the cuda layer that is effectively shared with the python-based models. the following investigates if these high performance features of the language translate into benefits for the proposed nlp pipelines. benchmarks between python and rust are shown in figure 2 using a turing rtx2070 gpu with a amd 2700x cpu. for all experiments the average time relative to python is reported with the standard deviation. for all prediction tasks, the transformers @xcite ) library (v3.2.0) is used as a reference. all experiments are run for 10 iterations, with various number of samples (provided in brackets). for reference, the python absolute execution time per iteration is provided. the loading benchmarks represent the average time required to load models into the gpu buffer. significant benefits can be observed for rust. this is probably caused by the simpler serialization format based on c-arrays for rust, and may be advantageous for event-driven applications loading models on a per-request basis (short warm-up time). the forward pass results vary between applications. as expected, pipelines with very simple pre-and post-processing steps offer virtually identical performance (for example sentiment analysis). significant benefits can be observed for question answering, coming entirely from the tokenization process (at the time this document was prepared, the transformers' @xcite question answering pipeline did not leverage rust-based tokenizers yet). the performance of pipelines involving complex post-processing steps (text generation with sampling and beam search) can show significant benefits. marian-based translation models (tiedemann and thottingal, 2020) exhibit a 40% speedup (in line with the native c++ implementation @xcite ). the t5 implementation is faster for small effective batch sizes (with a beam size of 6) but slower for larger batches, indicating optimization potential remains. in general it was observed that the actual model forward pass (tensor operations) is comparable albeit slightly slower in rust than in python. a last experiment (large matrix multiplication) shows the rust libtorch bindings seem to be 1 to 2% slower than the pytorch equivalent. ## conclusion rust is a promising language for the development of nlp systems. its concurrency capabilities, memory safety features and modern strings data model make it a good alternative to c++ for production systems. while evolving quickly, the rust nlp open-source ecosystem still lags behind python rich set of libraries. complementing the availability of high performance tokenizers, rust-bert makes state-of-the-art language models and end-to-end nlp pipelines available to the rust community.