Probabilistic Robustness for Data Filtering (2023)

Abstract: We introduce our probabilistic robustness rewarded data optimization (PRoDO) approach as a framework to enhance the model’s generalization power by selecting training data that optimizes our probabilistic robustness metrics. We use proximal policy optimization (PPO) reinforcement learning to approximately solve the computationally intractable training subset selection problem. The PPO’s reward is defined as our ( \alpha,\epsilon, \gamma )-Robustness that measures performance consistency over multiple domains by simulating unknown test sets in real-world scenarios using a leaving-one-out strategy. We demonstrate that our PRoDO effectively filters data that lead to significantly higher prediction accuracy and robustness on unknown-domain test sets. Our experiments achieve up to +17.2% increase of accuracy (+25.5% relatively) in sentiment analysis, and -28.05 decrease of perplexity (-32.1% relatively) in language modeling.In addition, our probabilistic ( \alpha,\epsilon, \gamma )-Robustness definition serves as an evaluation metric with higher levels of agreement with human annotations than typical performance-based metrics.

## introduction modern machine learning works with massive amounts of data on a range of tasks like language modeling, object detection, and data mining. using large amounts of training set to build machine learning systems requires extensive computational resources and creates problems like domain shifts and input noise. these unfiltered training data harm model learning robustness @xcite that leads to prediction errors and serious consequences like self driving car fatality and medical misdiagnosis @xcite . one problem causing this model instability is that the model learning is opt for the system's quality, which is typically evaluated by measuring how close this system's output of a test set is from its human label using metrics such as accuracy, error rate, perplexity, human evaluation score, and so on. however, such a system performance metric highly depends on the test set's choice and is thus unreliable. for instance, if our training set is drawn from the news domain, then the performance on a test set from the news domain (in-domain test set) is usually much higher than that from the twitter domain (out-of-domain test set). as a result, in nlp, while some systems produce human parity results like the use of a pre-trained transformer @xcite on in-domain test sets, these systems are easily corrupted by out-of-domain (ood) samples from the real world. existing studies on data selection and robust learning demonstrate a need for test domain knowledge during training. some data selection work @xcite @xcite @xcite @xcite chooses critical in-domain data for domain adaptation, and other work defends against adversarial attacks but offers little help for out-of-domain robustness @xcite under natural distributional shifts @xcite that occurs more frequently than extreme adversarial cases. this outof-domain robustness is often measured by testing on a specific domain and a single task like sentiment classification @xcite . the problem with these existing approaches is that the target domain knowledge is often unknown, as is the case for most real-world applications that do not know the domain of test data they will receive before launch. to address these challenges, our goal is to select training data to achieve high accuracy on ood test set, without requiring any target domain insight during data selection or model training process. we distinguish the out-of-domain and unknowndomain test sets by assigning the out-of-domain as the test domain known during the training, and the nlp model 1 2 6 4 3 finetune nlp model with evaluate nlp model on all test domains eg. transformer, cnn… encoder . . . . . . . . . . . . compute robustness blind test set training data test data pool unknown world domain 1 domain 2 domain n domain x batch 1 batch t batch t rl agent policy value ppo update selected data clipped advantage test bootstraps (subsampled from domain1, …, domain n) score x eg. accuracy -robustness reward function: 5 advantage clip figure 1: probabilistic robustness rewarded data optimization (prodo) framework. in our approach to the measurement of robustness on unknown domains, we define robustness as the consistency of the behavior of a machine learning system. the more a machine learning system's behavior deviates from the typical, the less robust the system is defined to be. notice that this definition does not necessarily give a notion of whether system performance is good or bad. for example, in terms of sentiment analysis, this definition refers to consistency in prediction accuracy for a trained classifier. to measure a system's performance consistency, we combine test sets for evaluation from various domains, like news, biomedical, and twitter; randomly sample their subsets, and take each subset as this system's input and obtain an output. then, we measure the performance of each output. if these output performances are close to each other, we say they are consistent. to quantify this consistency, we define our robustness metric as a notion of a probabilistic definition on the distribution of performance across different test domains, called (α, ϵ, γ)-robustness, where the higher the probability of the consistent prediction performance, the more robust the system. our objective is to measure the probability of upper bounding the prediction accuracy gap between any test subset and its average. more specifically, we call an nlp system (α, ϵ, γ)-robust, if for every subset uniform-randomly drawn from a distribution, its prediction error, a combination of the target domain error and the source domain error weighted by the parameter α, is centered around the mean error, which is bounded through a parameter ϵ with a probability depending on γ, an indicator of how robust a system is, see definition in figure 1 (in green) and details in section 2.2. in our approach, we do not need any targetdomain data since we "simulate" unknown targetdomain test sets using the leave-one-out error stability @xcite . we assume that the non-left-out test sets are the simulated target domain while the left-out test is the real target domain for evaluation. for example, given biomecial as our unknown target domain to evaluate, we take the training data as the source-domain set and sample different subsets from a combination of other test domain data (e.g., news, ted talks, etc.) to simulate our target-domain test sets. the hyperparameter α is the target-domain error weight and offers flexibility to balance the trade-offs between source-and target-domain errors. the (α, ϵ, γ)-robustness takes a new direction away from adversarial robustness to a general consistency of a model's quality as meta-evaluation methods that measure the consistency of userdefined quality metrics. thus, any standard performance evaluation metrics, such as accuracy, can be used within the definition of our robustness. after defining our robustness metric as our data optimization goal, we ask, how should we select a subset of data that can maximize the robustness? we assert that in general, the subset selection problem is computationally intractable. we conjecture that this condition holds true for every objective function, including our notion of probabilistic robustness, and is the reason why we use proximal policy optimization (ppo) @xcite deep reinforcement learning to optimize the training set, which has the advantages of low variance, monotonic policy improvement, and sampling efficiency @xcite @xcite and policy gradient @xcite . our probabilistic robustness rewarded data optimization (prodo) framework equipartitions the training data into mini-batches and simultaneously learns a policy network to select data iteratively and a value network to estimate future returns using our (α, ϵ, γ)-robustness as the reward functions illustrated in figure 1 . our empirical results on sentiment analysis and language modeling show that the use of our robustness definition consistently and significantly enhances a model's out-of-domain performance. the main contributions of this work include: the rest of the paper is as follows: in section 2.1, we describe our prodo framework. in section 2.2, we introduce the (α, ϵ, γ)-robustness. section 3 introduces experimental details including baselines, nlp tasks and ablation study. in section 4, we discuss the previous literature on robustness in machine learning. section 5 concludes the paper. ## method our goal is to enable our task model f (any deep learning-based nlp model, such as transformer, etc.,) to achieve consistent while superior performance on any unknown test domain d x whose distribution is different from the source training set x , by learning an effective subset of x that can maximize the robustness of model f. the entire process details are depicted in figure 1. our method consists of reinforcement learning (rl), data selection, and training nlp models using the selected data. specifically, we use reinforcement learning to train a data selection policy, and we use the data selection policy to select a subset of training data to fine-tune nlp models. following @xcite , we pre-train the task model f on the full training data set @xmath0, where x i is a sentence, n is training set size. then, the training set x is shuffled and randomly partitioned into t disjoint data batches such that in the following sections, we will first introduce how our prodo framework learns to select data (section 2.1) and then our reward function based on the probabilistic robustness definition (section 2.2). ## related work in previous years, a crucial direction of work on robustness of nlp models lies on the vulnerability of nlp models to input perturbations, such as crafted noises @xcite @xcite . for instance, @xcite proposes an adversarial stability training objective to enable neural machine translation models robust to input perturbations. @xcite evaluates robustness to input perturbations for neural machine translation. compared to them, our approach handles new test inputs with distribution shifts from any unknown domains. some literature propose evaluation metrics for robustness from the perspectives of statistics or input perturbations @xcite @xcite . however, they either focus on the worst-case scenario of adversarial inputs or sampling single instances without considering the full distribution of the system performance. ## conclusion we introduce probabilistic robustness rewarded proximal policy data optimization (prodo) framework to improve nlp model's generalization by selecting training data. our framework is rewarded by the (α, ϵ, γ)-robustness to measure an nlp model's performance consistency over multiple domains. our experiments show the effectiveness of probabilistic robustness measure to enhance learning generalization and prediction accuracy. our work also demonstrates a successful step towards general robustness evaluation and data selection without target domain insight.