AUTOSUMM : Automatic Model Creation for Text Summarization (2021)

Abstract: Recent efforts to develop deep learning models for text generation tasks such as extractive and abstractive summarization have resulted in state-of-the-art performances on various datasets. However, obtaining the best model configuration for a given dataset requires an extensive knowledge of deep learning specifics like model architecture, tuning parameters etc., and is often extremely challenging for a non-expert. In this paper, we propose methods to automatically create deep learning models for the tasks of extractive and abstractive text summarization. Based on the recent advances in Automated Machine Learning and the success of large language models such as BERT and GPT-2 in encoding knowledge, we use a combination of Neural Architecture Search (NAS) and Knowledge Distillation (KD) techniques to perform model search and compression using the vast knowledge provided by these language models to develop smaller, customized models for any given dataset. We present extensive empirical results to illustrate the effectiveness of our model creation methods in terms of inference time and model size, while achieving near state-of-the-art performances in terms of accuracy across a range of datasets.

## introduction machine learning algorithms, particularly, deep learning techniques have led to the simplification of several computationally expensive tasks. however, training and optimizing these models for different tasks demand the experienced engineering resources and require expertise, making it difficult for non-experts. automated machine learning is a strategy to automate this pipeline for model creation including automated generation of the model itself. in the case of natural language processing and text analysis, the advent of large language models such as bert @xcite , gpt2 @xcite , and more recently gpt3 @xcite have created resources that can be exploited for the creation of robust models for several downstream nlp tasks. however, the need for ml expertise creates a bottleneck. further, these deep learning models have thousands of parameters and need fairly large datasets and computational resources for training. we focus on providing algorithms for autogeneration of ml models for complex nlp tasks such as extraction and generation, making them accessible to non experts. our proposed approaches feed off the knowledge available in large pretrained models to auto-generate new, smaller, customized models for a custom dataset. specifically, the major contributions are as follows. to the best of our knowledge, this is the first effort towards automatically building customized and compressed models for text generation tasks, specifically summarization. ## related work neural architectural search is a trending area in automl, which automates the process of model creation by searching efficient model architectures, without human expertise. a typical nas problem involves identifying the search space, employing a search strategy to find the best task-specific model architecture, and training the model from scratch. most of the nas experiments are done on images - @xcite , @xcite , @xcite using neuro-evolutional and genetic algorithms, which are computationally very expensive and time consuming. recently, gradient based methods like darts @xcite , snas @xcite and @xcite are proposed to speed-up the search strategy. but, the explorations of using nas for language-related tasks are very limited. @xcite propose textnas with a search space for better understanding of the text representations. they use a simple and efficient, enas @xcite , which is guided by reinforcement learning for model generation. however, these models mainly focus on text understanding and do not directly extend to the generation-related tasks like summarization. ## methodology figure 1 shows the overview of our model creation framework. the input is the dataset and task specifications (summary type, size) and the output is a custom trained summarization model, which can be further used to create text summaries. in this paper, we generate models for both extractive and abstractive summarization tasks, with the former being a binary classification task to extract summary sentences from the input, while the latter aims to generate summaries containing novel words and phrases that may not be present in the input text. our proposed approaches distills knowledge from a language-model based teacher network to generate an encoder-decoder-based child model. we present two algorithms that aid in auto-creation of different types of resulting 'child' models -(1) a model with convolutional and recurrent units and (2) a mini-transformer based model. the first is achieved by our approach autosumm-create and the second using autosumm-distill, which are detailed as follows. ## experiments we evaluate our proposed framework by performing experiments that test the performance of the newly created models against benchmark summarization datasets on both extractive and abstractive tasks. ## results and discussion the other dataset for extreme summarization is the contract dataset. ## conclusions we present a framework for auto-generation of ml models for extract and generate tasks by leveraging knowledge distillation, nas, and transformerdistillation techniques. the proposed approach successfully creates new model architectures that are more efficient in terms of inference time and space while achieving near state-of-the-art performance in terms of accuracies across datasets for extractive and abstractive summarization. we believe our work can help create the foundation towards democratizing the use of deep-learning for nlp applications for non-experts in practice.