D eep Z ensols: A Deep Learning Natural Language Processing Framework for Experimentation and Reproducibility (2023)

Abstract: Given the criticality and difficulty of reproducing machine learning experiments, there have been significant efforts in reducing the variance of these results. The ability to consistently reproduce results effectively strengthens the underlying hypothesis of the work and should be regarded as important as the novel aspect of the research itself. The contribution of this work is an open source framework that has the following characteristics: a) facilitates reproducing consistent results, b) allows hot-swapping features and embeddings without further processing and re-vectorizing the dataset, c) provides a means of easily creating, training and evaluating natural language processing deep learning models with little to no code changes, and d) is freely available to the community.

## introduction consistently reproducing results is a fundamental criterion of the scientific method, without which, a hypothesis may be weakened or even invalidated @xcite . reproduction of results becomes even more necessary as a growing number of publications are inflated by false positives @xcite . efforts to abate this trend include introducing new statistical methods to detect false findings @xcite . the inability to reproduce results has been referred to as the "replication crisis" @xcite . the problem of reproducibility in results is becoming more acknowledged as a serious issue in the machine learning (ml) community with efforts to understand and overcome the challenge @xcite . not only has the community addressed the issue in the literature, it has endeavored to assess if experiments are reproducible and provide recommendations to enhance reproducibility as with the reproducibility challenge 1 . to address these issues, we present deepzensols, 1 https://www.cs.mcgill.ca/.../reproducibilitychallenge.html a freely available foot_0 deep learning (dl) framework for nlp research by and for the academic research community including citizen scientists, academic researchers, and students. it has been used for research projects @xcite funded by the national institute of health (nih) 3 . a key feature that sets deepzensols apart from others is a novel method to rapidly and easily swap features sets and compare performance across models (see section 2.4). other systems must re-parse and re-vectorize each mini-batch over each epoch. while there exist similar frameworks to ours @xcite @xcite , none of these provides this batch strategy, vectorization of natural language text features and reproducibility of results across advanced programming interfaces (apis) and datasets in one framework. popular neural network (nn) architectures are available out of the box and easily configurable with little to no coding necessary (see section 2.2 for nlp specific framework details). ## library design deepzensols is a combination of python apis built on top of pytorch that provide a means of easily and quickly creating nlp task specific pipelines. the framework's source code and installable libraries are released under the mit open source license, and includes extensive and in depth overview and api documentation, tutorials, jupyter notebook examples and class diagrams for nlp reference models and datasets. the framework is validated with 381 unit tests and six integration tests, which are automated using continuous integration. ## runtime analysis runtime analysis was performed for parsing, feature vectorization (see section 2.3), batching (see section 2.4), training and testing three different types of models using a nvidia titan rtx graphics processor on an intel 3.6ghz cpu using the following criteria: data model duration batch gpu both ner bert 1:06:04 04:23 00:12 04:35 glove 34:08 04:19 05:41 10:00 mov bert 21:19 02:04 -00:26 01:38 glove 05:03 03:07 01:20 04:27 cb bert 05:48 01:50 -00:01 01:49 glove 05:45 01:51 03:03 04:54 the datasets used in the runtime analysis include the connl 2003 @xcite for ner, the movie review corpus @xcite for sentiment, and the clickbate corpus @xcite ) for text classification. the results show significant processing improvements in all three datasets with the glove model leading. this is likely due to how the static embedding are not computed for each sentence (unlike bert). the ner dataset with the bert model was faster by 4.5 minutes and the glove model was 10 minutes faster (1.4x speedup). however, the movie review sentiment dataset shows the best improvement (7.8x speedup) on the glove model. this is primarily from the batch caching 2.6x speedup, but benefited from a gpu 1.3x caching speedup. we hypothesize that the gpu slowdowns for the movie review and clickbate datasets are potentially due to larger bert (768d vs 300d embeddings) mini-batches copied from the cpu. ## related frameworks popular dl frameworks such as tensorflow foot_7 have a dashboard that provides metrics, such as training and validation loss. however, these general purpose frameworks offer basic performance metrics and do not provide a means of producing higher abstraction level nlp specific models. more specifically, frameworks such as keras, supply a very coarse api allowing solely for cookie-cutter models. they lack the ability to easily create and evaluate models past this surface interface. frameworks such as pytorch foot_8 , which are more common in academia, provide a more straightforward simple api that is similar to the core tensor-flow libraries, and thus have the same shortcomings as a tool to bridge the gap between pure research and reproducibility. specifically, they do not provide batching for accessible feature swapping and ablation studies, or retention of ml algorithm state necessary to reproduce results. allennlp @xcite is a flexible configuration driven framework that provides construction of nlp nn architectures and is the closest framework to ours. however, it does not have fast feature swapping (see section 2.4) and batch creation capability, and lacks most of the components necessary to consistently reproduce results foot_9 . popular packages providing support for transformer architectures such as bert @xcite include huggingface foot_10 . however, this framework only provides transformer models for contextual word embeddings. ## conclusion and limitations the deepzensols framework is a viable solution to easily create nlp specific models with apis and analysis tools to produce consistent results. such frameworks create the types of models that give confidence and legitimacy by providing a way to produce reliable reproducible results for researchers not familiar with deep learning tools, practitioners, medical personnel, students, and those new to the field. runtime analysis shows the framework offers significant processing time savings compared to systems that do not provide feature caching with stable results, but not all huggingface pretrained models foot_11 have been tested. the following have been tested: bert, roberta @xcite , distil-bert @xcite , big bird @xcite , biobert @xcite , xml-r (conneau et al., 2020), clinicalbiobert @xcite , and gatortron @xcite have been tested. a planned future work is to integrate the framework with tensorboard foot_12 .