torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP (2023)

Abstract: Reproducibility in scientific work has been becoming increasingly important in research communities such as machine learning, natural language processing, and computer vision communities due to the rapid development of the research domains supported by recent advances in deep learning. In this work, we present a significantly upgraded version of torchdistill, a modular-driven coding-free deep learning framework significantly upgraded from the initial release, which supports only image classification and object detection tasks for reproducible knowledge distillation experiments. To demonstrate that the upgraded framework can support more tasks with third-party libraries, we reproduce the GLUE benchmark results of BERT models using a script based on the upgraded torchdistill, harmonizing with various Hugging Face libraries. All the 27 fine-tuned BERT models and configurations to reproduce the results are published at Hugging Face, and the model weights have already been widely used in research communities. We also reimplement popular small-sized models and new knowledge distillation methods and perform additional experiments for computer vision tasks.

## introduction the rapid developments of various research domains such as natural language procession (nlp), computer vision, and speech recognition @xcite @xcite @xcite @xcite have been supported by advances in deep learning @xcite @xcite @xcite @xcite . while it has been developed rapidly, poor reproducibility of deep learning-based studies is a severe problem that research communities have been facing @xcite @xcite , and the reproducibility has been attracting significant attention from researchers @xcite @xcite @xcite . to address the serious problem, research communities introduced reproducibility checklists. at the time of writing, some venues require authors to complete checklists when submitting their work e.g., responsible nlp research checklist foot_0 @xcite at nlp venues (acl, naacl, arr) and paper checklist at neurips. 4 matsubara (2021) developed torchdistill, a modular, configuration-driven knowledge distillation framework built on pytorch @xcite for reproducible deep learning research. knowledge distillation @xcite is a well known model compression method usually to train a small model (called student) leveraging outputs from a more complex model (called teacher) as part of loss functions to be minimized. recent knowledge distillation approaches are more complex e.g., using intermediate layers' outputs (embeddings or feature maps) besides the final output (logits) of teacher models with auxiliary module branches attached to teacher and/or student models during training @xcite @xcite , using multiple teachers @xcite , and training multilingual or non-english models solely with an english teacher model @xcite @xcite . in this work, we significantly upgrade torchdistill from the initial framework @xcite to enable further generalized implementations, supporting more flexible module abstractions and enhance the advantage of decralative pyyaml configuration files to design experiments with third-party packages of user's choice, as promised in @xcite . using glue tasks @xcite as an example, we demonstrate that the upgraded torchdistill and a new script harmonize with hugging face transformers @xcite , datasets @xcite , accelerate @xcite , and evaluate @xcite to reproduce the glue test results reported in @xcite by fine-tuning pretrained bert-base and bert-large models with the upgraded torchdistill. we also conduct knowledge distillation experiments using the fine-tuned bert-large models as teachers to train bert-base models. all these experiments are performed on google colaboratory. 7 we also publish all the code and configuration files at github 1 and trained model weights and training logs at hugging face 2 for reproducibility and helping researchers build on this work. our bert models fine-tuned for the glue tasks have already been downloaded 138,000 times in total and widely used in research communities not only in research papers but also in tutorials of deep learning frameworks and acl 2022. besides the nlp tasks, we reimplement popular small-sized computer vision models and a few more recent knowledge distillation methods as part of torchdistill, and perform additional experiments to demonstrate that the upgraded torchdistill still supports computer vision tasks. ## related work in this section, we briefly summarize related work on open source software that supports end-to-end research frameworks. @xcite propose anserini, an information retrieval toolkit built on lucene foot_5 for reproducible information retrieval research. pyserini @xcite is a python toolkit built on pytorch @xcite and faiss @xcite for reproducible information retrieval research with sparse and dense representations, and the sparse representation-based retrieval support comes from lucene via anserini. allennlp @xcite is a toolkit built on pytorch for research on deep learning methods in nlp and designed to lower barriers to high quality nlp research e.g., useful nlp module abstractions and defining experiments using declarative configuration files. highly inspired by allennlp, @xcite design torchdistill, a module, configuration-driven framework built on pytorch for reproducible knowledge distillation studies. similar to allennlp, torchdistill enables users to design experiments by declarative pyyaml configuration files and supports highlevel module abstractions. for image classification and object detection tasks, its generalized starter scripts and configurations help users implement knowledge distillation methods without much coding cost. matsubara (2021) also reimplement 18 knowledge distillation methods with torchdistill and point out that the standard knowledge distillation @xcite can outperform many of the recent state of the art knowledge distillation methods for a popular teacher-student pair with ilsvrc 2012 dataset @xcite . in section 3, we describe major upgrades in torchdistill from the initial release @xcite . ## major upgrades from the initial release in this section, we summarize the major upgrades from the initial release of torchdistill (matsubara, 2021). figure 1 highlights high-level differences between the initial design @xcite of torchdistill and a largely upgraded version in this work. the initial torchdistill is dependent on pytorch and torchvision and contains key modules and functionalities specifically designed to support image classification and object detection tasks. for example, dataset modules that the initial version officially supports are only those in torchvision, and some of dataset-relevant functionalities such as building a sequence of data transforms and dataset loader are based on datasets in torchvision. in this work, we make torchdistill less dependent on torchvision and support more tasks with third-party packages of users' choice, by generalizing some of the key components in the framework and exporting task-specific implementations to the corresponding executable scripts and local packages. we also reimplement popular small-sized models whose official pytorch implementations are not either available or maintained. ## google colab demos in this section, we demonstrate that the upgraded torchdistill can collaborate with third-party libraries for supporting more tasks. we also attempt to reproduce the cifar-10 and cifar-100 results reported in the original papers. to lower the barrier to reusing and building on the scripts with torchdistill, we conduct all the experiments on google colaboratory 7 , which gives users access to gpus free of charge. we publish the jupyter notebook foot_9 files to run the experiments as part of torchdistill repository 1 so that researchers can easily use them. ## ilsvrc 2012 as highlighted in section 3, torchdistill was initially focused on supporting implementations of diverse knowledge distillation in a unified way and dependent on torchvision to specifically support image classification and object detection tasks with its relevant modules (see fig. 1 ). to demonstrate that the upgraded torchdistill still preserves the feature, we reimplement a few more knowledge distillation methods with the upgraded torchdistill: knowledge review (kr) framework @xcite and knowledge translation and adaptation with affinity distillation (ktaad) @xcite . note that @xcite present the results of various knowledge distillation methods reimplemented with the initial version of torchdistill for ilsvrc 2012 and coco 2017 @xcite datasets. those results are not included in this work, and we refer interested readers to @xcite . t: resnet-34 s: resnet-18 ce ce kr (original) kr (ours) 73.31 69.75 71.61 71.64 ## pascal voc 2012 & coco 2017 the initial torchdistill @xcite supports image classification and object detection tasks. as mentioned in section 3.2, we also provide a starter script for semantic segmentation tasks. using two popular datasets, pascal voc 2012 @xcite and coco 2017 @xcite , we demonstrate that the upgraded torchdistill supports semantic segmentation tasks as well. in the experiments with pascal voc 2012 dataset, we use deeplabv3 @xcite with resnet-50 and resnet-101 backbones @xcite , using torchvision's pretrained model weights for coco 2017 dataset. we choose hyperparameters such as learning rate policy and crop size based on the original study of deeplabv3 @xcite . our results are shown in we also examine our reimplemented ktaad method @xcite for the lite r-aspp model (lraspp in torchvision) @xcite as a student model, using the coco 2017 dataset and the pretrained deeplabv3 with resnet-50 in torchvision as a teacher model, whose miou and global pixelwise accuracy are 66.4 and 92.4, respectively. since the ktaad method is not tested on coco 2017 dataset for lraspp with mobilenetv3-large backbone in the original paper of ktaad @xcite , our hyperparameter choice is based on torchvision's reference script. 21 as with other experiments, the trained model weights and configuration used in this section are published as part of the torchdistill repository. 1 ## conclusion in this work, we significantly upgraded torchdistill @xcite , a modular, configurationdriven framework built on pytorch @xcite for reproducible deep learning and knowledge distillation studies. we enhanced pyyamlbased instantiation, generalized internal modules for supporting more tasks, and reimplemented popular models and methods. to demonstrate that the upgraded framework can support more tasks as we claim, we provided starter scripts for new tasks based on the upgraded framework. one of the new starter scripts supports glue tasks @xcite and harmonizes with hugging face transformers @xcite , datasets @xcite , accelerate @xcite , and evaluate @xcite . using the script on google colaboratory, we reproduced the glue test results of fine-tuned bert models @xcite and performed knowledge distillation experiments with our fine-tuned bert-large models as teacher models. similarly, we reproduced cifar-10 and -100 results of popular small-sized models we reimplemented, using google colaboratory. furthermore, we reproduced the result of resnet-18 trained with the reimplemented kr method @xcite for the imagenet dataset. we also demonstrated a new starter script for semantic segmentation tasks using pascal voc 2012 and coco 2017 datasets, and the reimplemented ktaad method @xcite improves a pretrained semantic segmentation model in torchvision. in this study, we also published 27 trained models for nlp tasks 2 and 14 trained models for computer vision tasks. 1 according to hugging face model repositories, the bert models fine-tuned for the glue tasks have already been downloaded about 138,000 times in total at the time of writing. research communities leverage torchdistill not only for knowledge distillation studies @xcite @xcite @xcite , but also for machine learning reproducibility challenge (mlrc) @xcite and reproducible deep learning studies @xcite @xcite . torchdistill is publicly available as a pip-installable pypi package and will be maintained and upgraded for encouraging coding-free reproducible deep learning and knowledge distillation studies.