T ext B ox: A Unified, Modularized, and Extensible Framework for Text Generation (2021)

Abstract: In this paper, we release an open-source library, called TextBox, to provide a unified, modularized, and extensible text generation framework. TextBox aims to support a broad set of text generation tasks and models. In our library, we implement 21 text generation models on 9 benchmark datasets, covering the categories of VAE, GAN, and pretrained language models. Meanwhile, our library maintains sufficient modularity and extensibility by properly decomposing the model architecture, inference, and learning process into highly reusable modules, which allows users to easily incorporate new models into our framework. The above features make TextBox especially suitable for researchers and practitioners to quickly reproduce baseline models and develop new models. TextBox is implemented based on PyTorch, and released under Apache License 2.0 at the link https://github.com/RUCAIBox/TextBox .

## introduction text generation, which has emerged as an important branch of natural language processing (nlp), is often formally referred as natural language generation (nlg) @xcite . it aims to produce plausible and understandable text in human language from input data (e.g., a sequence, keywords) or machine representation. because of incredible performance of deep learning models, many classic text generation tasks have achieved rapid progress, such as machine translation @xcite , dialogue systems @xcite ), text summarization @xcite , graph-to-text generation @xcite , and more. to facilitate the development of text generation models, a few remarkable open-source libraries † equal contribution. * corresponding author. in order to alleviate the above issues, we initiate a project to provide a unified framework for text generation algorithms. we implement an opensource text generation library, called textbox, aiming to enhance the reproducibility of existing text generation models, standardize the implementation and evaluation protocol of text generation algorithms, and ease the development process of new algorithms. our work is also useful to support several real-world applications in the field of text generation. we have extensively surveyed related text generation libraries and broadly fused their merits into textbox. the key features and capabilities of our library are summarized in the following three aspects: in our library, it is convenient to compare different text generation model module seq2seq gan vae lm encoder decoder attention embedder data dataloader … … dataset evaluation task-agnostic metric task-specific evaluator preprocessing strategy greedy search top-k search beam search ## architecture and design figure 1 presents the illustration of the main functionalities and modules in our library textbox. the configuration module at the bottom helps users set up the experimental environment (e.g., hyperparameters and running details). built upon the configuration module, the data, model, and evaluation modules form the core elements of our library. in the following, we describe the detailed structure of these three modules. ## system usage in this section, we show a detailed guideline to use our system library. users can run the existing models or add their own models as needed. ## performance evaluation to evaluate the models in textbox, we conduct extensive experiments to compare their performance on unconditional and conditional generation tasks. ## related work several toolkits have been released focusing on one or a few specific text generation tasks or techniques. for example, tensor2tensor @xcite , @xcite and opennmt @xcite are designed for machine translation task, while parlai @xcite and plato @xcite ized for dialog research in this field. there are two text generation libraries closely related to our library, including texygen @xcite and texar @xcite focusing on gan technique and high modularization, respectively. textbox has drawn inspirations from these toolkits when designing relevant functions. compared with them, textbox covers more text generation tasks and models, which is useful for reproducibility. besides, we implement standardized evaluation to compare different models. also, our library provides various common modules for convenience. it has a proper focus on text generation field, and provide a comprehensive set of modules and functionalities. ## conclusion this paper presented a unified, modularized, and extensible text generation library, called textbox. so far, we have implemented 21 text generation models, including vae-based, gan-based, pretrained language models, sequence-to-sequence and 9 benchmark datasets for unconditional and conditional text generation tasks. moreover, our library is modularized to easily plug in or swap out components, and extensible to support seamless incorporation of other external modules. in the future, features and functionalities will continue be added to our library, including more models and datasets, diverse inputs such as graph and table, and distributed training in multiple machines. we invite researchers and practitioners to join and enrich textbox, and help push forward the research on text generation.