E ns LM : Ensemble Language Model for Data Diversity by Semantic Clustering (2021)

Abstract: Natural language processing (NLP) often faces the problem of data diversity such as different domains, themes, styles, and so on. Therefore, a single language model (LM) is insufficient to learn all knowledge from diverse samples. To solve this problem, we firstly propose an autoencoding topic model with a mixture prior (mATM) to perform clustering for the data, where the clusters defined in semantic space describes the data diversity. Having obtained the clustering assignment for each sample, we develop the ensemble LM (EnsLM) with the technique of weight modulation. Specifically, EnsLM contains a backbone that is adjusted by a few modulated weights to fit for different sample clusters. As a result, the backbone learns the shared knowledge among all clusters while modulated weights extract the cluster-specific features. EnsLM can be trained jointly with mATM with a flexible LM backbone. We evaluate the effectiveness of both mATM and EnsLM on various tasks.

## introduction it is common knowledge in modern natural language processing (nlp) that natural language varies greatly across domains, themes, styles, genres and many other linguistic nuances @xcite @xcite . generally, we call such nature of language as data diversity. many existing works @xcite @xcite have illustrated that data diversity will affect the performance of lms if we just train a single lm over the entire dataset, even though fine-tuning a pretrained lm (that has been pre-training on a very large corpus) such as bert @xcite on current task @xcite . the domain diversity in dataset is a very common type of data diversity. in some cases, if we can obtain a well-defined domain label for each sample, some works @xcite @xcite try to consider the multi-domain property of data in developing the lms. however, these pre-defined domain labels are not always accurate or even available @xcite , especially for the wild datasets, in which data come from different sources, such as internet news, product reviews, and daily conversation. to this end, we hope to develop a lm that can explore the diversity from data automatically. data selection is a commonly used strategy to handle diversity in data @xcite @xcite @xcite . this kind of method is developed from an assumption that samples belonging to the same cluster should own similar characteristics. according to the clustering assignment, models can select suitable data for training a lm for each cluster separately. although, to some extend, data selection is an efficient strategy to alleviate the problem of data diversity, it may bring two disadvantages as follows. firstly, the process of data selection is independent of the lm learning. in other words, the gradient signal generated by lm's training loss can not affect the data selection. secondly, data selection only tells the hard cluster belongings of samples, ignoring a fact that some samples may belong to more than one clusters with soft (weighted) assignment. inspired by their works and to move beyond, in this paper, we find the semantics learned by topic modeling @xcite can infer sample clusters to a certain extent via k-means, but is not good enough, as shown in fig. 1a . to jointly consider the clustering and topic modeling for better clustering (as shown in fig. 1b ) and for joint training with the following lm, we firstly introduce an autoencoding topic model with mixture priors (matm). for each sample in the corpus, matm can infer a soft clustering assignment. in order to jointly consider the learning of matm with various lms, we employ the weight modulation methods @xcite . specifically, as shown in fig. 3 , given a lm as backbone, for each layer (convolutional or fully-connected), we introduce some modulated parameters. guided by clustering assignment inferred from matm, these parameters modulate the backbone single lm to multiple lms, corresponding to different clusters. therefore, our proposed model can be seen as a type of ensemble learning, and hence we call it ensemble language model (enslm). our proposed matm and enslm enjoy the following distinguished properties: ## related work for nlp, topic modeling (tm) @xcite and lms are two common regimes with their own advantages. tm can discover the interpretable global semantics that are topics, while with pre-training on large corpus, lms recently achieve the sota performance on many nlp tasks with more focuses on local dependencies. therefore, some works consider to combine them to obtain benefits from both. dieng et al. (2016) and wang et al. (2020) incorporate the tm with rnn-based model to capture the longrange dependencies. to move beyond single-layer tm for rnns, guo et al. (2020) propose the recurrent hierarchical topic-guided rnn with the help of multi-layer tm @xcite . to extract explicit document semantics for summarization, @xcite propose three different modules to plug knowledge from tm into transformer-based lms @xcite . our work can be seen as a parallel work to combine their advantages together but focuses on dealing with data diversity in nlp without the ground-truth information such as domain labels. meanwhile, our work can be applied for different lms including cnns, rnns, and transformer-based models. ## autoencoding topic model with mixture prior we firstly describe one of the most popular topic models, latent dirichlet allocation (lda) @xcite , and its autoencoding inference @xcite . inspired by them, in order to jointly consider topic learning and sample clustering, we propose the autoencoding topic model with mixture prior (matm). ## ensemble language model recently, various advanced lms for language understanding and generation have been introduced, most of which do not consider the data diversities in the corpus. in this paper, having obtained the clustering assignment vector z from matm, given a single lm as backbone, we propose the ensemble lm (enslm) via z-guided weight modulation. in other words, the enslm can modulate the backbone single lm to fit for different clusters. ## experiments in this section, we evaluate the effectiveness and efficiency of our proposed matm and enslm on different nlp tasks including document clusters, text classification, language generation and abstractive document summarization. our code is available at https://github.com/bochengroup/enslm ## conclusion in this paper, we first propose matm to infer latent semantic clusters from raw text corpus, and then combine it with lm with efficient weight modulation, resulting in a more powerful enslm, which can be naturally extended to other lms. in the future, we will study the effectiveness of enslm on other nlp tasks, such as the multi domain translation, and investigate whether enslm can be applied to the pre-training stage of transformer. cnn/dm cnn/dm consists of news and associated sentence highlights, that is a brief overview composed of a few sentences. following the standard training/validation/testing splits in hermann et al. (2015) without anonymizing entities, we perform our experiments. we splits sentences using the stanford corenlp toolkit foot_6 and pre-process the dataset following @xcite . . xsum xsum includes 226, 711 news articles, each of which is associated with a one-sentence summary. we use the standard training/validation/testing splits (204, 045/11, 332/11, 334) and follow the pre-processing in @xcite . to satisfy the maximum capacity of the encoder in the base model, such as 512 for bertsum, we use truncated document as the encoder input. statistics of summarization datasets is shown in table . 7.