T ext B ox 2.0: A Text Generation Library with Pre-trained Language Models (2022)

Abstract: To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2.0, focusing on the use of pre-trained language models (PLMs). To be comprehensive, our library covers 13 common text generation tasks and their corresponding 83 datasets and further incorporates 45 PLMs covering general, translation, Chinese, dialogue, controllable, distilled, prompting, and lightweight PLMs. We also implement 4 efficient training strategies and provide 4 generation objectives for pre-training new PLMs from scratch. To be unified, we design the interfaces to support the entire research pipeline (from data loading to training and evaluation), ensuring that each step can be fulfilled in a unified way. Despite the rich functionality, it is easy to use our library, either through the friendly Python API or command line. To validate the effectiveness of our library, we conduct extensive experiments and exemplify four types of research scenarios. The project is released at the link: https://github.com/RUCAIBox/TextBox#2.0 .

## library design in order to facilitate plm-based text generation research, textbox 2.0 has introduced various new features, mainly from three aspects: generation tasks, generation models, and training strategies. ## library usage in this section, we introduce how to use our library in four different kinds of research scenarios by showing the example codes. reproducing existing models. textbox 2.0 includes various plms and supports many text generation tasks and datasets. it is convenient for users pre-training a new model. in textbox 2.0, we provide several pre-training objectives for users to pre-train new models from scratch. specifically, users just need to specify the pre-training task, pre-training corpus, and architecture by setting pretrain_task, dataset, and model. figure 1(d) shows an example that pre-trains a chinese bart on the wudaocorpora @xcite using the denoising pre-training objective. to improve the pre-training efficiency, textbox 2.0 supports distributed data parallel and efficient decoding (section 2.3). figure 1 (e) shows an illustrative example of how users can use the accelerate command to set configurations of multiple devices and launch the training code. ## experiments in this section, we conduct extensive experiments to verify the generation abilities of textbox 2.0. ## conclusion this paper presented textbox 2.0, a comprehensive and unified library for conducting research on plm-based text generation. our library makes significant extensions in three major aspects, namely generation tasks (13 tasks and 83 datasets), generation models (45 plms), and training strategies (e.g., distributed data parallel and efficient decoding). results from extensive test experiments demonstrate that our library can accurately reproduce existing models. besides, we also provide a series of utility tools to better analyze and explore the generated results. to summarize, our library can be very useful to facilitate text generation research, and our team will improve this library with regular updates.