jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020)

Abstract: We introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. jiant enables modular and configuration driven experimentation with state-of-the-art models and a broad set of tasks for probing, transfer learning, and multitask training experiments. jiant implements over 50 NLU tasks, including all GLUE and SuperGLUE benchmark tasks. We demonstrate that jiant reproduces published performance on a variety of tasks and models, e.g., RoBERTa and BERT.

## introduction this paper introduces jiant, 1 an open source toolkit that allows researchers to quickly experiment on a wide array of nlu tasks, using state-of-the-art nlp models, and conduct experiments on probing, transfer learning, and multitask training. jiant supports many state-of-the-art transformer-based models implemented by huggingface's transformers package, as well as non-transformer models such as bilstms. packages and libraries like huggingface's transformers @xcite and allennlp @xcite have accelerated the process of experimenting and iterating on nlp models by both abstracting out implementation details, and simplifying the model training pipeline. jiant extends the capabilities of both toolkits by presenting a wrapper that implements a variety of complex experimental pipelines in a scalable and easily controllable setting. jiant contains a task bank of over 50 tasks, including all the tasks presented in glue @xcite , superglue @xcite , the edge-probing suite @xcite , and the senteval probing suite @xcite , as well as other individual tasks including ccg supertagging @xcite , socialiqa @xcite , and commonsenseqa @xcite . jiant is also the official baseline codebase for the super-glue benchmark. early versions of jiant have already been used in multiple works, including probing analyses @xcite @xcite @xcite , transfer learning experiments @xcite , and dataset and benchmark construction @xcite @xcite . ## background transfer learning is an area of research that uses knowledge from pretrained models to transfer to new tasks. in recent years, transformer-based models like bert @xcite and t5 @xcite have yielded state-of-the-art results on the lion's share of benchmark tasks for language understanding through pretraining and transfer, often paired with some form of multitask learning. ## benchmark experiments to benchmark jiant, we perform a set of experiments that reproduce external results for single fine-tuning and transfer learning experiments. jiant has been benchmarked extensively in both published and ongoing work on a majority of the implemented tasks. we benchmark single-task fine-tuning configurations using commonsenseqa @xcite and socialiqa @xcite . on common-senseqa with roberta large , jiant achieves an accuracy of 72.2, comparable to 72.1 reported by @xcite . on socialiqa with bertlarge, jiant achieves a dev set accuracy of 65.8, comparable to 66.0 reported in @xcite . next, we benchmark jiant's transfer learning regime. we perform transfer experiments from mnli to boolq with bert-large. in this configuration @xcite demonstrated an accuracy improvement of 78.1 to 82.2 on the dev set, and jiant achieves an improvement of 78.1 to 80.3.