Can Data Diversity Enhance Learning Generalization? (2022)

Abstract: This paper introduces our Diversity Advanced Actor-Critic reinforcement learning (A2C) framework (DAAC) to improve the generalization and accuracy of Natural Language Processing (NLP). We show that the diversification of training samples alleviates overfitting and improves model generalization and accuracy. We quantify diversity on a set of samples using the max dispersion, convex hull volume, and graph entropy based on sentence embeddings in high-dimensional metric space. We also introduce A2C to select such a diversified training subset efficiently. Our experiments achieve up to +23.8 accuracy increase (38.0% relatively) in sentiment analysis, -44.7 perplexity decrease (37.9% relatively) in language modeling, and consistent improvements in named entity recognition over various domains. In particular, our method outperforms both domain adaptation and generalization baselines without using any target domain knowledge.

## introduction we introduce the diversity advanced actor-critic reinforcement learning framework (daac) to improve the generalization and accuracy of natural language processing @xcite . training data plays a crucial role in deep learning (dl)-based nlp. investigations @xcite @xcite show that training data quality is imperative to the machine learning model's performance. good data are effective and easy to generalize, while bad data bring noise and overfits on out-of-domain test sets. decades of work @xcite @xcite have been devoted to finding the best data, with the most commonly used method we call domain adaptation @xcite that uses target domain knowledge. despite large improvements on specific domains, these adapted models lack robustness and are prone to rare events. for example, when models shift towards one target domain like twitter, they may become erroneous for other domains like medical or travel, and real-world applications on news report data, for example, do not know which queries to receive before they launch, resulting in a critical performance gap between laboratory findings and reality. therefore, we move away from the independent and identically distributed assumption (i.i.d.) and focus on reducing generalization errors by studying the inter-dependencies among samples. in this paper, we ask three questions (rqs) and propose our answers. the first question (rq 1) is: "does higher diversity among training samples lead to better generalized nlp model learning?". semantic diversity is a desirable aspect in human annotation, where training data sets should be large and diverse enough to learn the many ways the objects differ @xcite @xcite , like intrinsic facial diversity in face recognition data, since every face is different. we show that diversity ensures that the training data can provide sufficiently discriminative information for the model @xcite , and thus give a more accurate prediction. intuitively, for a fixed number of samples, the more semantic meaning they cover, the more information they contain, and thus the more effective they are for learning. we will use two examples to elaborate on our rationale. let us first take one explanatory example of @xmath0, cheerful, happy, stay}, a four single-word dataset that forms two semantically close clusters: @xmath1, stay} and @xmath2, cheerful}. our hypothetical nlp task is to generate sentences from v . from 4 2 word combinations, four sentences are meaningful: "stay happy", "stay cheerful", "be happy", "be cheerful". our dataset selection task is to find two single-word samples from v that generate the most sentences. if we select both single-word samples from a, which are semantically close, we will generate no meaningful sentence (analogously that from b). however, we can generate one of the four meaningful sentences if we select one word from a and the other word from b, e.g., "stay" and "happy", which are semantically distant. in fact, since these four sentences can be treated as paraphrasing, generating any of them will roughly cover the semantic meaning for all. in this example, the diversified subset {stay, happy} is more effective for learning generalization. our second example is conducted on a real dataset. figure 1 shows our experimental results on the penn treebank dataset. we equipartition 42k training sentences into five subsets and compute their diversity using the sum of pairwise cosine distances of sentence embeddings (i.e., our first method, see next paragraph and section 2.2.1). we sort the subsets according to their diversity scores. the first subset has the lowest diversity (3.18e 7 ), the second subset has the second-lowest diversity (3.26e 7 ), etc. we then train five language models on each subset foot_0 , and computes the perplexity (ppl) and out-of-vocabulary (oov) on ten randomly sampled subsets of the test set. ppl and oov are typical indicators of how training data is generalized @xcite . our results show that the diversity of the subset strongly negatively correlates to both ppl and oov rate, where the subset with the highest diversity is more likely to generalize on a random test and has fewer unseen words. with the preliminary evidence showing that a training set with higher diversity leads to more gen-eralized learning, we propose our second question (rq 2): how to measure the diversity of a sample set? we introduce three methods to quantify training dataset diversity. the first method is based on the notion of dispersion, also known as max-sum diversification @xcite . by viewing each training sample as a data point in the sentence embedding space, we maximize the dispersion of the training set, which is the sum of pairwise cosine distances of sentence embeddings, as in figure 2 (2). the larger the sum, the more informative the subset is. the pairwise geometric distance method has a worst-case complexity of o(n foot_1 ), and to accelerate the computation, we introduce our second method (figure 2 (3)) that measures diversity using the volume of a convex hull around the samples, namely the volume that encloses all vertices, taking each sentence embedding as a vertex. its quickhull @xcite implementation has an averagecase complexity of o(n log n) and a worst-case complexity of o(n 2 ). the above two methods compute diversity based on distances in the metric space. we can obtain global information content by measuring the uncertainty of the semantic distances between samples based on frequencies. our third method combines our view of geometric distance and entropy in nlp. as depicted in figure 2 (4), we apply graph entropy, h(g, p ), an information-theoretic function on graph g with a probability distribution p on its vertex set @xcite . here, each sentence in the subset is a node in the graph, and the cosine distance of the twosentence embeddings is the weighted edge between the two nodes. empirically, all these three diversity measures consistently outperform the baselines. the max dispersion (md) and graph entropy (ge) are more robust in various domains with the best performance overall because they account for every pair of distances in the training set. after we have defined our diversity measure, we ask our third question (rq 3): "how can we optimize the subset to maximize the information content?" subset selection is np-hard in general @xcite . we conjecture that this holds true for every objective function, including the notion of diversity introduced in this paper, and is the reason why we use actor-critic reinforcement learning @xcite . we equipartition the training data into mini-batches nlp model encoder cnn/lstm/ transformer batch 1 selected reward embedding batch 2 policy reward function batch action [diversity measures] [actor] [actor] [critic] value advantage state finetune nlp model … value update update 2 3 4 1 convex hull volume i like nlp. thank you. hello . good morning. how are you? bye bye. see you. max dispersion i like nlp. thank you. hello . good morning. how are you? bye bye. see you. graph entropy distance edge and simultaneously learn a policy network (as an actor) to select data and a value network (as a critic) to estimate future returns using the diversity measure as an evaluation reward. our actor-critic data selection method has the advantages of low variance updates as well as credit assignment and significantly outperforms domain adaptation @xcite on various domains without requiring any target domain knowledge. the architecture is shown in figure 2 and section 2. in summary, our work mainly contributes to the following: (1) modeling sample dependency for effective data sampling using diversity; (2) measure dataset diversity using max dispersion, convex hull volume, and graph entropy; (3) introduction of the advantage actor-critic reinforcement learning framework to select informative samples and enhance nlp accuracy and generalization. the rest of the paper is organized as follows. in section 2.1, we detail diversity advanced actor-critic reinforcement learning (daac). then in section 2.2, we introduce different diversity measures. afterward, in section 3, we empirically verify the generalization and accuracy improvement using daac. we discuss related work in section 4 and conclude the paper in the last section. ## diversity advanced actor-critic we now present the details of our diversity advanced actor-critic (daac) algorithms. the details of data selection and fine-tuning process are depicted in figure 2 . our task model f (any deep learning-based nlp model, such as transformer, etc.,) is pre-trained on the full training data set @xmath3, where x i is a sentence, n is training set size. then, as in @xcite , we shuffle and randomly partition x into t disjoint data batches so that @xmath4, b 2 , ..., b t }, with b @xmath5, x (t-1)n|t +2 , ..., x tn|t }. n|t is the integer division of n by t , and @xmath6. if mod (n, t ) ̸ = 0, then the last batch has a variable size of mod (n, t ) and collects the remaining sentences. for each batch, we select and denote @xmath7. after obtaining bt , we use bt to fine-tune f. f and its encoder g is updated on bt for t times in an epoch, and each update is based on the previous checkpoint. our goal is to improve our task model f on any test domain whose distribution is different from the source training set x , by learning an effective subset of x to fine-tune f. figure 2-(1 ) shows our new model. our framework is based on actor-critic reinforcement learning. our actor-critic method has the following properties, which are very desirable to achieve our goals: variance deduction, efficient sampling, and credit assignment @xcite . our framework consists of policy and value networks jointly and dynamically learned together with the task model using the advantage error computed from the reward function. in the following context, we will first introduce our actor-critic algorithm (section 2.1) and then our three different reward functions based on diversity (section 2.2). ## experiments we describe our experimental details and demonstrate that our daac improves baselines in three nlp applications, including two recognition tasks: sentiment analysis, named entity recognition (ner), and one generation task of language modeling, without the knowledge of the target domain. we give hyperparameters of daac in appendix. finetuning vs. data selection we introduce our four models based on daac. the first three models finetune a pretrained nlp model using max dispersion md, convex hull volume cv and graph entropy ge as reward function. as a comparison, we add the fourth model trained on selected data from scratch that does not rely on a pretrained model, denoted as data selection ds. specifically, we train a2c using md as reward function for 150 epochs first, and use the trained a2c to select data samples from all source data, then use only the selected data to train the nlp model from scratch. we select 3833 (63.8%) and 7630 (54.3%) samples from the source domain of sentiment analysis and ner respectively, and 21017 (49.9%) samples ## related work existing domain generalization methods related to data focus on data augmentation @xcite and data generation @xcite . these methods increase the quantity of training set while our method utilizes existing data maximally by modeling the intrinsic sample dependency. there have been several influential works @xcite @xcite on data selection that significantly contributed to today's nlp state-of-the-arts. @xcite proposes a data filter based on deep reinforcement learning on image and sentiment classification task. @xcite implemented an instance selector using reinforcement learning to filter noisy data to improve the accuracy on natural language inference @xcite , sentiment analysis, part-of-speech tagging and dependency parsing @xcite . these work select the training data close to a given target domain for domain adaptation. in contrast, we aim to enhance the model generalization and increase the accuracy on any arbitrary domain. as for diversity measures, ruder and plank (2017b) examines the effects of shannon entropy in the transfer learning setting, but they do not consider content semantic meaning. @xcite uses determinantal point processes to select diverse data to reduce the labor of annotating training examples for dependency parsing, but the proposed diversity measure cannot be generalizaed to other nlp tasks. other works incorporate the notion of diversity into topics like language representation @xcite , and ensemble language modeling @xcite . ## conclusion we introduce actor-critic reinforcement learning rewarded with diversity measures to select effective training data that significantly enhances the accuracy of sentiment analysis, named entity recognition, and language modeling tasks across various domains. without any target domain knowledge, our method outperforms the cnn, transformer and bert baselines. our experiments show that modeling sample dependency by increasing training set diversity enhances the learning generalization and prediction accuracy.