Original: Improving Fine-tuning on Low-resource Corpora with Information Bottleneck
Revised: V ARIATIONAL I NFORMATION B OTTLENECK FOR E FFECTIVE L OW -R ESOURCE F INE -T UNING

Original: Abstract
Revised: A BSTRACT

Original: Large-scale pre-trained language models act as general-purpose feature extractors, but not all the features are relevant for a given target task.This can cause problems in low-resource scenarios, where fine-tuning such large-scale models often over-fits on the small training set.
Revised: While large-scale pretrained language models have obtained impressive results when fine-tuned on a wide variety of tasks, they still often suffer from overfitting in low-resource scenarios. Since such models are general-purpose feature extractors, many of these features are inevitably irrelevant for a given target task.

Original: We propose to use the information bottleneck principle to improve generalization in this scenario. We apply the variational information bottleneck method  to remove task-irrelevant and redundant features from sentence embeddings during the fine-tuning of BERT.
Revised: We propose to use Variational Information Bottleneck (VIB) to suppress irrelevant features when fine-tuning on low-resource target tasks, and show that our method successfully reduces overfitting. Moreover, we show that our VIB model finds sentence representations that are more robust to biases in natural language inference datasets, and thereby obtains better generalization to out-of-domain datasets.

Original: Evaluation on seven low-resource datasets for different tasks shows that our method significantly improves transfer learning in low-resource scenarios and obtains better generalization on 11 out ofout-of-domain textual entailment datasets.
Revised: Evaluation on seven low-resource datasets in different tasks shows that our method significantly improves transfer learning in low-resource scenarios, surpassing prior work. Moreover, it improves generalization on 13 out of 15 out-of-domain natural language inference benchmarks. Our code is publicly available in https://github.com/rabeehk/vibert .

Original:  Introduction
Revised:  I NTRODUCTION

Original: Transfer learning has emerged as the de facto standard technique in natural language processing (NLP), where large-scale language models are pre-trained on an immense amount of text to learn a general-purpose representation, which is then transferred to the target domain.
Revised: Transfer learning has emerged as the de facto standard technique in natural language processing (NLP), where large-scale language models are pretrained on an immense amount of text to learn a general-purpose representation, which is then transferred to the target domain with fine-tuning on target task data.

Original: This method has exhibited state-of-the-art results on various NLP benchmarks (Devlin et al., 2019; 
Revised: This method has exhibited state-of-the-art results on a wide range of NLP benchmarks (Devlin et al., 2019; Liu et al., 2019;

Original: Radford et al.,2018; Liu et al., 2019; Radford et al., 2019).
Revised: Radford et al., 2019).

Original: However,  the universal nature of largescale pre-trained representations poses a problem in low-resource scenarios, which is an important topic in
Revised: However, such pretrained models have a huge number of parameters, potentially making fine-tuning susceptible to overfitting. In particular, the task-universal nature of large-scale pretrained sentence representations means that much of the information in these representations is irrelevant to a given target task.

Original: NLP (Cherry et al., 2019). Much of the information in such sentence embeddings is irrelevant to the  target task, and it can be difficult to distinguish relevant from irrelevant information when fine-tuning the languagemodels with a large number of parameters on a small amount of target task data, resulting in over-fitting. For many real-world applications, it can be difficult and expensive tosolve this problem by collecting sufficient annotated data for these large neural models to excel.
Revised: If the amount of target task data is small, it can be hard for fine-tuning to distinguish relevant from irrelevant information, leading to overfitting on statistically spurious correlations between the irrelevant information and target labels. Learning low-resource tasks is an important topic in NLP (Cherry et al., 2019) because annotating more data can be very costly and time-consuming, and because in several tasks access to data is limited.

Original: In this paper, we propose to use the Information 
Revised: In this paper, we propose to use the Information Bottleneck (IB) principle (Tishby et al., 1999) to address this problem of overfitting.

Original: [Figure]
Revised: More specifically, we propose a fine-tuning method that uses Variational

Original:  Bottleneck (IB) principle (Tishby et al., 1999) to improve transfer learning in resource-limited scenarios.
Revised: Information Bottleneck (VIB; Alemi et al. 2017) to improve transfer learning in low-resource scenarios.

Original: We apply a Variational Information Bottleneck (VIB) (Alemi et al., 2017) approach, as illustrated in Figure 1,  to remove task-irrelevant and redundant information from the sentence embeddings.
Revised: VIB addresses the problem of overfitting by adding a regularization term to the training loss that directly suppresses irrelevant information. As illustrated in Figure 1, the VIB component maps the sentence embedding from the pretrained model to a latent representation z , which is the only input to the taskspecific classifier. The information that is represented in z is chosen based on the IB principle, namely that all the information about the input that is represented in z should be necessary for the task. In particular, VIB directly tries to remove the irrelevant information, making it easier for the task classifier to avoid overfitting when trained on a small amount of data.

Original: Removing task-irrelevant information helps to mitigate over-fitting when fine-tuning on thesmall target task training set.
Revised: We find that in low-resource scenarios, using VIB to suppress irrelevant features in pretrained sentence representations substantially improves accuracy on the target task. µ x ) and eliminates irrelevant and redundant information through the Gaussian noise with variance Σ( x ) . Removing unnecessary information from the sentence representation also implies removing redundant information. VIB tries to find the most concise representation which can still solve the task, so even if a feature is useful alone, it may be removed if it isn’t useful when added to other features because it is redundant.

Original: Additionally, removing redundant information helps to avoid learning superficial correlations that  do not generalize well to out-of-domain datasets.
Revised: We hypothesize that this provides a useful inductive bias for some tasks, resulting in better generalization to out-of-domain data. In particular, it has recently been demonstrated that annotation biases and artifacts in several natural language understanding benchmarks (Kaushik & Lipton, 2018; Gururangan et al., 2018; Poliak et al., 2018; Schuster et al., 2019) allow models to exploit superficial shortcuts during training to perform surprisingly well without learning the underlying task. However, models that rely on such superficial features do not generalize well to out-of-domain datasets, which do not share the same shortcuts (Belinkov et al., 2019a).

Original: We  evaluate the effectiveness of our method on fine-tuning BERT (Devlin et al., 2019) on seven different datasets for text classification, natural language inference (NLI), similarity, and paraphrase tasks.
Revised: We investigate whether using VIB to suppress redundant features in pretrained sentence embeddings has the effect of removing these superficial shortcuts and keeping the deep semantic features that are truly useful for learning the underlying task. We find that using VIB does reduce the model’s dependence on shortcut features and substantially improves generalization to out-of-domain datasets. We evaluate the effectiveness of our method on fine-tuning BERT (Devlin et al., 2019), which we call the VIBERT model (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning). On seven different datasets for text classification, natural language inference, similarity, and paraphrase tasks,

Original: Our method shows greater robustness to over-fitting than conventional fine-tuning, improving the accuracy on low-resource datasets.
Revised: VIBERT shows greater robustness to overfitting than conventional fine-tuning and other regularization techniques, improving accuracies on low-resource datasets.

Original: Moreover, our method obtains better generalization to out-of-domain NLI datasets.
Revised: Moreover, on NLI datasets, VIBERT shows robustness to dataset biases, obtaining substantially better generalization to out-of-domain NLI datasets.

Revised: Further analysis demonstrates that VIB regularization results in less biased representations.

Original: Our approach is highly effective and simple to implement, involving a small additional MLP classifier on top of the sentence embeddings.
Revised: Our approach is highly effective and simple to implement, involving a small additional MLP classifier on top of the sentence embeddings.

Original: It is model agnostic and end-to-end trainable.
Revised: It is model agnostic and end-to-end trainable.

Original: In summary, we make the following contributions: 1) Proposingthe IB principle, and specifically a VIB model, as an effective approach to address over-fitting when fine-tuning large-scale pre-trained models on low-resource scenarios.
Revised: In summary, we make the following contributions: 1) Proposing VIB for low-resource fine-tuning of large pretrained language models.

Original: An empirical evaluation of the method on seven low-resource benchmarks, obtaining substantial generalization gains in all settings, including over 2 point gains on the STS-B and MRPC datasets. Providing analysis on how VIB reduces over-fitting.
Revised: 2) Showing empirically that VIB reduces overfitting, resulting in substantially improved accuracies on seven low-resource benchmark datasets against conventional fine-tuning and prior regularization techniques.

Original: Demonstrating significantly improved generalizationby our trained NLI models to11 out of 13 out-of-domain NLI datasets.
Revised: Showing empirically that training with VIB is more robust to dataset biases in NLI, resulting in significantly improved generalization to out-of-domain NLI datasets.

Original: To facilitate future work, we will release our code.
Revised: To facilitate future work, we will release our code.

Original:  Fine-tuning on Low-resource Settings Problem Formulation
Revised:  F INE - TUNING IN L OW - RESOURCE S ETTINGS

Original: We consider a general multi-class classification problem with a dataset D = { x i ,y i } Ni =1 consisting of inputs x i ∈X , and labels y i ∈ Y .
Revised: The standard fine-tuning paradigm starts with a large-scale pretrained model such as BERT, adds a task-specific output component which uses the pretrained model’s sentence representation, and trains this model end-to-end on the task data, fine-tuning the parameters of the pretrained model.

Original: We assume we are also given a pre-trained encoder f ϕ ( . ) parameterized by ϕ that computes sentence embeddings for the input x i .
Revised: As depicted in Figure 1, we propose to add a VIB component that controls the flow of information from the representations of the pretrained model to the output component.

Original: Our goal is to improve the fine-tuning of powerful pre-trained models f ϕ ( . ) on a low-resource target dataset D . 2. Mitigating Over-fitting
Revised: The goal is to address overfitting in resource-limited scenarios by removing irrelevant and redundant information from the pretrained representation.

Original: In general, pre-trained language models are powerful general-purpose feature extractors with a large number of parameters. We hypothesize that only a small subset of this encoded information is needed to perform the target task, meaning that there are many irrelevant and redundant features in the embedding.
Revised: Problem Formulation We consider a general multi-class classification problem with a low-resource dataset D = { x i ,y i } Ni =1 consisting of inputs x i ∈X , and labels y i ∈Y . We assume we are also given a large-scale pretrained encoder f ϕ ( . ) parameterized by ϕ that computes sentence embeddings for the input x i .

Original: When the dataset for the target task is small, it is hard for fine-tuning to distinguish the required information from the irrelevant one, resulting in over-fitting on the target dataset. The key idea of our approach is to specifically optimize for the removal of irrelevant and redundant information from the input embeddings.
Revised: Our goal is to fine-tune f ϕ ( . )

Original: As depicted in Figure 1, we introduce a variational information bottleneck module on top of the pre-trained input embeddings. During fine-tuning, this VIB module tries to compress the sentence embeddings x into an embedding z that keeps only the information necessary to predict y . 2. Information Bottleneck
Revised: on D to maximize generalization.

Original:  To specifically optimize for the removal of irrelevant and redundant information from the input embeddings, we adopt the Information Bottleneck principle.
Revised: Information Bottleneck To specifically optimize for the removal of irrelevant and redundant information from the input representations, we adopt the Information Bottleneck principle.

Original: The objective of IB is to find a maximally compressed representation Z of the input representation X (compression loss) that maximally preserves information about the output Y (prediction loss),  by minimizing:
Revised: The objective of IB is to find a maximally compressed representation Z of the input representation X (compression loss) that maximally preserves information about the output Y (prediction loss), 1 by minimizing:

Original: [Equation]
Revised: [Equation]

Original: whereI ( .,. ) is the mutual information, and β ≥ 0 controls the balance between compression and prediction.
Revised: where β ≥ 0 controls the balance between compression and prediction, and I ( .,. ) is the mutual information.

Original: Variational Information Bottleneck Alemi et al.
Revised: Variational Information Bottleneck Alemi et al.

Original: (2017) derive an efficient variational estimate of (1):
Revised: (2017) derive an efficient variational estimate of (1):

Original: [Equation]
Revised: [Equation]

Original: where q φ ( y | z ) is an estimate of p ( y | z ) , r ( z ) is an estimate of the prior  p ( z )  , and p θ ( z | x ) is an estimate of the posterior probability of z .
Revised: where q φ ( y | z ) is a parametric approximation of p ( y | z ) , r ( z ) is an estimate of the prior probability p ( z ) of z , and p θ ( z | x ) is an estimate of the posterior probability of z .

Original: During training, the compressed sentence representation z is sampled from the distribution p θ ( z | x ) , making dimensions with high variance less informative for predicting the output class.
Revised: During training, the compressed sentence representation z is sampled from the distribution p θ ( z | x ) , meaning that a specific pattern of noise is added to the input of the output classifier q φ ( y | z ) . Increasing this noise decreases the information conveyed by z .

Original: Thus, the VIB module canlearn to block thetrained output classifier q φ ( y | z ) from using specific dimensions of z .
Revised: In this way, the VIB module can block the output classifier q φ ( y | z ) from learning to use specific information.

Original: At test time, the expected value of z is used for predicting labels with q φ ( y | z ) . 
Revised: At test time, the expected value of z is used for predicting labels with q φ ( y | z ) . We refer to the dimensionality of z as K , which specifies the bottleneck size.

Revised: Note that there is an interaction between decreasing K and increasing the compression by increasing β (Shamir et al., 2010; Harremo ¨ es & Tishby, 2007).

Original:  We assume parametric Gaussian distributions for  r ( z ) and p θ ( z | x ) to allow an analytic computation for their KullbackLeibler divergence,  namely r ( z )=
Revised: K and β are hyper-parameters (Alemi et al., 2017). We consider parametric Gaussian distributions for prior r ( z ) and p θ ( z | x ) to allow an analytic computation for their Kullback-Leibler divergence, 2 namely r ( z ) =

Original: N ( z | µ 0 , Σ 0 ) and p θ ( z | x )=
Revised: N ( z | µ 0 , Σ 0 ) and p θ ( z | x ) =

Original: N ( z | µ ( x ) , Σ( x )) , wherethe µ and µ 0 are K-dimensional mean vectors, the Σ and Σ 0 are diagonal covariance matrices, and K is the dimensionality of z  .
Revised: N ( z | µ ( x ) , Σ( x )) , where µ and µ 0 are K − dimensional mean vectors, and Σ and Σ 0 are diagonal covariance matrices. We use the reparameterization trick (Kingma & Welling, 2013) to estimate the gradients, namely z = µ ( x )+Σ( x ) (cid:12) (cid:15) , where (cid:15) ∼N (0 ,I ) .

Original: For p θ ( z | x ) ,  µ ( x ) and Σ( x ) are estimated using an MLP from the input sentence representations f ϕ ( x ) , and q φ ( y | z ) is another MLP.
Revised: To compute the compressed sentence representations p θ ( z | x ) , as shown in Figure 1, we first feed sentence embeddings f ϕ ( x ) through a shallow MLP. It is then followed by two linear layers, each with K hidden units to compute µ ( x ) and Σ( x ) (after a softplus transform to ensure non-negativity). We also use another linear layer to approximate q φ ( y | z ) .

Original:  Experiments Datasets
Revised:  E XPERIMENTS

Original:  We evaluate the performance on sevenbenchmarks on different  tasks, including text classification, natural language inference, similarity, and paraphrase detection.
Revised: Datasets We evaluate the performance on seven different benchmarks for multiple tasks, in particular text classification, natural language inference, similarity, and paraphrase detection.

Original: For NLI, we experiment with the SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) benchmarks.
Revised: For NLI, we experiment with two well-known NLI benchmarks, namely SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018).

Original: For text classification, we evaluate on two sentiment analysis datasets, namely IMDB (Maas et al., 2011) and Yelp (YELP)
Revised: For text classification, we evaluate on two sentiment analysis datasets, namely IMDB (Maas et al., 2011) and Yelp2013 (YELP)

Original: (Zhang et al., 2015).
Revised: (Zhang et al., 2015).

Original: We additionally evaluate on three low-resource datasets in the GLUE benchmark (Wang et al., 2019). These include paraphrase detection using MRPC (Dolan and Brockett, 2005), semantic textual similarity using STS-B (Cer et al., 2017), and textual entailment using RTE (Dagan et al., 2006).
Revised: We additionally evaluate on three low-resource datasets in the GLUE benchmark (Wang et al., 2019): 3 paraphrase detection using MRPC (Dolan & Brockett, 2005), semantic textual similarity using STS-B (Cer et al., 2017), and textual entailment using RTE (Dagan et al., 2006).

Original: We evaluate on the standard validation and test splits.
Revised: For the GLUE benchmark, SNLI, and Yelp, we evaluate on the standard validation and test splits.

Original: Since the test sets are not available for MNLI, we tune on the matched dev set and evaluate on the mismatched dev set (MNLI-M) or vice versa (see Appendix A for datasets statistics and more details on the experimental setups).
Revised: For MNLI, since the test sets are not available, we tune on the matched dev set and evaluate on the mismatched dev set (MNLI-M) or vice versa. See Appendix A for datasets statistics and Appendix B for hyper-parameters of all methods.

Original:  We use the BERT base uncased (Devlin et al., 2019) implementation of Wolf et al.
Revised: Base Model We use the BERT Base (12 layers, 110M parameters) and BERT Large (24 layers, 340M parameters) uncased (Devlin et al., 2019) implementation of Wolf et al.

Original: as [Table]
Revised: as our base models, 4 known to work well for these tasks.

Original: [Table]
Revised: We use the default hyper-parameters of BERT, i.e., we use a sequence length of 128 , with batch size 32 . We use the stable variant of the Adam optimizer (Zhang et al., 2021; Mosbach et al., 2021) with the default learning rate of 2e − 5 through all experiments.

Original: in GLUE. We report Pearson/Spearman correlation for
Revised: We do not use warm-up or weight decay.

Original: STS-B, F1/accuracy for MRPC, and accuracy for RTE. our base model, known to work well for these tasks.
Revised: Baselines We compare against prior regularization techniques, including previous state-of-the-art, Mixout:

Original: We use the default hyper-parameters of BERT.
Revised: [Equation]

Original: We compare against two baselines 1) Fine-tuning with cross-entropy loss (CE), and 2) Dropout (Srivastava et al., 2014), a widely used stochastic regularization technique to mitigate over-fitting in neural networks.
Revised: • Dropout (Srivastava et al., 2014), a widely used stochastic regularization techniques used in multiple large-scale language models (Devlin et al., 2019; Yang et al., 2019; Vaswani et al., 2017) to mitigate overfitting. Following Devlin et al. (2019), we apply dropout on all layers of BERT.

Original: 3. Results on the GLUE Benchmark
Revised: • Mixout (Lee et al., 2019) is a stochastic regularization technique inspired by Dropout with the goal of preventing catastrophic forgetting during fine-tuning. Mixout regularizes the learning to minimize the deviation of a fine-tuned model from the pretrained initialization. It replaces the model parameters with the corresponding value from the pretrained model with probability p .

Original: Table 1 shows results on the low-resource datasets in GLUE.
Revised: • Weight Decay (WD) is a common regularization technique to improve generalization (Krogh & Hertz, 1992). It regularizes the large weights w by adding a penalization term λ 2 (cid:107) w (cid:107) to the loss, where λ is a hyperparameter specifying the strength of regularization. Chelba & Acero (2004) and Daum´e III (2007) adapt WD for fine-tuning of the pretrained models, and propose to replace this regularization term with λ (cid:107) w − w 0 (cid:107) , where w 0 are the weights of the pretrained models. Recently, Lee

Revised: et al.

Original:  Our VIB model substantially outperforms the baselines on all the datasets,improving the results on MRPC, STS-B, and RTE by 1.43/2.03, 2.70/2.97, and 2.60 absolute points respectively, demonstrating the effectiveness of the proposed method.
Revised: ) demonstrated that the latter formulation of WD works better for finetuning of BERT than conventional WD and can improve generalization on small training sets. 3. R ESULTS ON THE GLUE B ENCHMARK Table 1 shows results on the low-resource datasets in GLUE. 5 We find that a) Our VIBERT model substantially outperforms the baselines on all the datasets, demonstrating the effectiveness of the proposed method.

Original: We find that dropout decreases the performancecompared to CE on low-resource datasets.
Revised: b) Dropout decreases the performance on low-resource datasets.

Original: We conjecture that regularization techniques relying on stochasticity without considering the relevance to the output, in contrast to VIB, can make it more difficult for learning to extract relevant information from a small amount of data.
Revised: We conjecture that regularization techniques relying on stochasticity without considering the relevance to the output, in contrast to VIB, can make it more difficult for learning to extract relevant information from a small amount of data.

Original: Igl et al.
Revised: Igl et al.

Original: (2019) observe similar effects in another application.
Revised: (2019) observe similar effects in another application.

Revised: c) Similar to the results of Zhang et al.

Revised: (2021), we find less pronounced benefits of the previously suggested methods than the results originally published.

Revised: This can be explained by using a more stable version of Adam (Zhang et al., 2021) suggested by the very recent work in our experiments, which decreases the added benefits of previously suggested regularization techniques on top of a stable optimizer.

Revised: In contrast, our VIBERT model still substantially improves the results and surpasses the prior work in all settings for both BERT Base and BERT Large models.

Revised: Due to the computational overhead of BERT Large , for the rest of this work, we stick to BERT Base .

Original: See Appendix B for more details and the selected hyper-parameters.
Revised: ∆ shows the absolute difference between the results of VIBERT model BERT. [Table]

Revised: Impact of Random Seeds: Following Dodge et al.

Revised: (2020), we examine the choice of random seed and evaluate the performance of VIBERT and BERT by fine-tuning them across 50 random seeds on GLUE.

Revised: To comply with the limited access to the GLUE benchmark online system

Revised: [Table]

Revised: we split the original validation sets into half and consider one half as the validation set and use the other half as the test set.

Revised: We first perform model selection on the validation set to fix the hyper-parameters and then fine-tune the selected models for 50 different seeds.

Revised: Figure 2 shows the expected test performance (Dodge et al., 2019) as the function

Revised: 50 # of Random Seeds

Original: 3. Varying-resource Results
Revised: E xp . T e s t A cc . MRPC

Revised: VIBERT

Revised: BERT     50 # of Random Seeds 88 .88 .88 .88 .89 .89 .89 .89 .

Revised: E xp .

Revised: T e s t P

Revised: CC .

Revised: STS-B

Revised: VIBERT

Revised: BERT     50 # of Random Seeds

Revised: E xp .

Revised: T e s t

Revised: A cc .

Original: To analyze the performance of  our method as a function of dataset size, we use large-resource datasets and subsample the training data with varying sizes.
Revised: RTE VIBERT BERT of random trials. The results demonstrate that our VIBERT model consistently obtains better performance than BERT on all datasets.

Revised: As anticipated, the expected test performance monotonically increases with more random trials (Dodge et al., 2020) till it reaches a plateau, such as after 30 trials on STS-B.

Revised: 3. V ARYING - RESOURCE R ESULTS

Revised: To analyze the performance of our method as a function of dataset size, we use four large-resource NLI and sentiment analysis datasets, namely SNLI, MNLI, IMDB, and YELP to be able to subsample the training data with varying sizes.

Original: Results are shown in Figure 2.
Revised: Table 2 shows the obtained results.

Original: VIB consistentlyimproves the fine-tuning of BERT and outperforms dropout on lowresource scenarios, but the advantages are reduced or eliminated as we approach a medium-resource scenario. 
Revised: VIBERT consistently outperforms all the baselines on low-resource scenarios, but the advantages are reduced or eliminated as we approach a medium-resource scenario. Also, the improvements are generally larger when the datasets are smaller, showing that our method successfully addresses low-resource scenarios. [Table]

Original: Also, the improvements are generally larger when the datasets are smaller.
Revised: [Table] [Table] [Table] [Table]

Original: See detailed test, validation results, and hyper-parameters in Appendix C.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  3. Out-of-domain Generalization
Revised: [Table] [Table] [Table] 3. O UT - OF - DOMAIN G ENERALIZATION

Original: Besides improving fine-tuning on low-resource data by removing irrelevant features, we expect VIB to improve on out-of-domain data because it removes redundant features.
Revised: Besides improving fine-tuning on low-resource data by removing irrelevant features, we expect VIB to improve on out-of-domain data because it removes redundant features.

Original: In particular, annotation artifactsin a specific dataset are known to create shortcut features, which are superficial cues correlated with[Figure] a label (Gururangan et al., 2018).
Revised: In particular, annotation artifacts create shortcut features, which are superficial cues correlated with a label (Gururangan et al., 2018; Poliak et al., 2018) that do not generalize well to out-of-domain datasets (Belinkov et al., 2019a).

Original:  We hypothesize that these shortcuts are easy to learn (especially when the amount of data is not sufficient), but are actually redundant with deeper features, which capture the true generalizations in the task.
Revised: Since solving the real underlying task can be done without these superficial shortcuts, they must be redundant with the deep semantic features that are truly needed. We hypothesize that many more superficial shortcut features are needed to reach the same level of performance as a few deep semantic features.

Original: By compressing the input embeddings, VIB encourages learning these desirable general features(Shamir et al., 2010; Tishby and Zaslavsky, 2015). By removing the redundant features,  the trained model generalizes better  to out-of-domain datasets that lack these shortcut features (Belinkov et al., 2019).
Revised: If so, then VIB should prefer to keep the concise deep features and remove the abundant superficial features, thus encouraging the classifier to rely on the deep semantic features, and therefore resulting in better generalization to outof-domain data.

Original: To evaluate out-of-domain generalization, we take NLI models trained on medium-sized 6K subsampled SNLI and MNLI in Section 3.2 and test their generalization to several NLI datasets.
Revised: To evaluate out-of-domain generalization, we take NLI models trained on medium-sized 6K subsampled SNLI and MNLI in Section 3.2 and evaluate their generalization on several NLI datasets.

Original:  We consider a total of 12 different NLI datasets used in Mahabadi et al.
Revised: Datasets: We consider a total of 15 different NLI datasets used in Mahabadi et al.

Original: (2020), including SICK (Marelli et al., 2014), ADD1 (Pavlick and Callison-Burch, 2016), JOCI (Zhang et al., 2017), MPE (Lai et al., 2017), MNLI, SNLI, SciTail (Khot et al., 2018), and three datasets from White et al. 
Revised: (2020), including SICK (Marelli et al., 2014), ADD1 (Pavlick & Callison-Burch, 2016), JOCI (Zhang et al., 2017), MPE (Lai et al., 2017), MNLI, SNLI, SciTail (Khot et al., 2018), and three datasets from White et al. (2017) namely

Original: (2017) namely DPR (Rahman and Ng, 2012), FN+ (Pavlick et al., 2015),and SPR (Reisinger et al., 2015).
Revised: DPR (Rahman & Ng, 2012), FN+ (Pavlick et al., 2015), SPR (Reisinger et al., 2015), and Quora Question

Original: We also consider SNLI hard set (Gururangan et al., 2018), a subset of SNLI that avoids its known biases, and Quora Question Pairs (QQP) interpreted as an NLI task as by Gong et al.
Revised: Pairs (QQP) interpreted as an NLI task as by Gong et al.

Original: We use the same split used in Wang et al.
Revised: We use the same split used in Wang et al.

Revised: We also consider SNLI hard and MNLI(-M) Hard sets (Gururangan et al., 2018), a subset of SNLI/MNLI(-M) where a hypothesis-only model cannot correctly predict the labels and the known biases are avoided.

Original: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space.
Revised: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (Appendix C).

Original: Following prior work (Belinkov et al., 2019), we perform model selection on the development set of each [Table]
Revised: Following prior work (Belinkov et al., 2019a;

Original:  target dataset and report  results on the test set.See Appendix E for hyper-parameters and more details.
Revised: Mahabadi et al., 2020), we select hyper-parameters based on the development set of each target dataset and report the results on the test set.

Original: Results: {+Table 
+}
Revised: Results: {+Table 
+}

Original: Table 2 shows the results of the models trained with VIB and  the baseline with BERT.
Revised: shows the results of VIBERT and BERT. We additionally include WD, the baseline that performed the best on average on SNLI and MNLI in Table 2.

Original: On models trained on SNLI, VIB improves the transfer on 11 out of 13 datasets, obtaining a substantial average improvement of 5.25 points.
Revised: On models trained on SNLI, VIBERT improves the transfer on 13 out of 15 datasets, obtaining a substantial average improvement of 5.51 points.

Original: On models trained on 
Revised: The amount of improvement on different datasets varies, with the largest improvement on SPR and SciTail with +15.5, and +12.5 points respectively, while WD on average obtains only 0.99 points improvement.

Original:  MNLI, VIB improves the transfer on 11 datasets, obtaining an average improvement of 4.28 points.
Revised: On models trained on MNLI, VIBERT improves the transfer on 13 datasets, obtaining an average improvement of 3.83 points.

Revised: The improvement varies across the datasets, with the largest on ADD1 and JOCI with 16.8 and 8.3 points respectively, substantially surpassing WD.

Original: Additionally, VIB improves the results on the SNLI  hard set, resulting in models  more robust to known biases.
Revised: Interestingly, VIBERT improves the results on the SNLI and MNLI(-M) hard sets, resulting in models that are more robust to known biases.

Original: These results support the claim that VIB motivates learning more general features, rather than redundant superficial features, leading to an improved generalization to datasets without these superficial biases. 
Revised: These results support our claim that VIBERT motivates learning more general features, rather than redundant superficial features, leading to an improved generalization to datasets without these superficial biases. In the next section, we analyze this phenomenon more. [Table] +4.

Revised: ADD 78.

Revised: +5.

Original: Analysis Impact
Revised: +1. +16. +1. DPR 50.

Revised: SPR 50.

Revised: +15.

Revised: +1.

Revised: +6.

Revised: FN+ 50.

Revised: +2.

Revised: +3.

Revised: +0.

Revised: JOCI 42.

Revised: +8.

Revised: +1.

Revised: +8.

Revised: +0.

Revised: MPE 58.

Revised: +3.

Revised: SCITAIL 62.

Revised: +12.

Revised: +2.

Revised: +3.

Revised: +3.

Revised: QQP 65.

Revised: +5.

Revised: +0.

Revised: +2.

Revised: +0.

Revised: SNLI Hard 65.

Revised: +2.

Revised: +1.

Revised: +3.

Revised: +0.

Revised: MNLI Hard 46.

Revised: +6.

Revised: +1.

Revised: +1.

Revised: +0.

Revised: MNLI-M Hard 46.

Original: of VIB
Revised: +6. + [Table] — — +5.

Original: on Over-fitting
Revised: — +0. — — +3. — +0.  A NALYSIS

Revised: Analysis of the Removed Features Elazar & Goldberg (2018) propose a challenging framework to evaluate if debiasing methods have succeeded in removing biases from the sentence representation.

Revised: After debiasing, the trained encoder is frozen and the classifier is retrained to try to extract the biases.

Revised: If the classifier reaches high accuracy given only bias features, then the encoder’s representation has not been successfully debiased.

Revised: We follow the framework of Elazar & Goldberg (2018) to analyze whether known biases in NLI data have been removed in the trained sentence representations.

Revised: In particular, following Belinkov et al.

Revised: (2019b), we train a classifier which only sees the representation of the hypothesis sentence and see if it can predict the class of the sentence pair, which is an established criterion to measure known biases in NLI datasets (Gururangan et al., 2018).

Revised: Thus, we freeze the trained encoders from our model and the BERT baseline and retrain a hypothesis-only classifier on hypotheses from the SNLI and

Original: As further evidence on the effect of VIB to reduce over-fitting, we analyze the effects of the β parameter on training and validation error.
Revised: MNLI datasets. 6 For reference, we compare to a hypothesis-only model with a BERT encoder trained end-to-end. Table 4 shows the results. With the baseline (BERT), the retrained classifier is not able to recapture all the biases (H-only), but it captures much more than with our method (VIBERT).

Original: The β parameter  controls the trade-off between removing information from the sentence embedding  and keeping information that is predictive of the output.
Revised: VIBERT is so successful at reducing biases that performance of the hypothesis-only classifier is close to chance (33%). [Table] VIBERT 38. Impact of VIB on Overfitting To analyze the effect of VIB on reducing overfitting, we analyze the effect of the β parameter on training and validation error since β controls the trade-off between removing information from the sentence embedding (high β ) and keeping information that is predictive of the output (low β ).

Original: We  train a BERT model using VIB on the MRPC dataset for varying values of β and plot theresulting accuracies on the training and validation sets in Figure 3.
Revised: We fix the bottleneck size ( K ) based on the models selected in Section 3.1, and we train VIBERT on the GLUE benchmark for varying values of β and plot the validation and training loss in Figure 3.

Original: For small values of β , where VIB has little effect, the validation error is substantially higher than the training error, indicating over-fitting.
Revised: For small values of β , where VIB has little effect, the validation loss is substantially higher than the training loss, indicating overfitting.

Original: This is because the network learns to be more deterministic ( Σ ≈ 0 ), thereby retaining too much irrelevant information.
Revised: This is because the network learns to be more deterministic ( Σ ≈ 0 ), thereby retaining too much irrelevant information.

Original: As we increase β , where[Figure] VIB has an effect, we observe better generalization performance with less over-fitting.
Revised: As we increase β , where VIB has an effect, we observe better generalization performance with less overfitting.

Original: As β becomes too large, both the training and validation errors shoot up because the amount of preserved information is insufficient to differentiate between the classes.
Revised: As β becomes too large, both the training and validation losses shoot up because the amount of preserved information is insufficient to differentiate between the classes.

Original: We have consistently observed this pattern  in other datasets.
Revised: This pattern is observable in the MRPC and RTE datasets, with a similar pattern in the STS-B dataset. [Figure]

Original: Ablation Study
Revised: Efficiency Evaluation Table 5 presents the efficiency evaluation in terms of memory, number of parameters, and time for all the methods measured on RTE. Our approach has several attractive properties. First, while our method is slightly larger in terms of parameters compared to the other standard regularization approaches due to an additional MLP layer (Figure 1), the difference is still marginal, and for BERT Base model with 109.48M trainable parameters, that is less than 1.22% more parameters. Second,

Original: As an ablation, we consider our model without the compression loss (VIB ( β = 0 )), which reduces to deterministic dimensionality reduction with an MLP, since the nonzero variance of embedding z is only useful for the compression loss.
Revised: [Table] our approach presents a much better memory usage with low-overhead, close to Dropout, while WD and especially Mixout cause substantial memory overhead. In dealing with large-scale transformer models like BERT, efficient memory usage is of paramount importance.

Revised: Third, in terms of training time, our method is similar to Dropout and much faster than the other two baselines.

Revised: Relative to BERT, VIBERT increases the training time by 3.77%, while WD and Mixout cause the substantial training overhead of 9.11% and 14.44%.

Original: Despite optimizing the reduced dimensionality  K  as a hyper-parameter for both methods, this ablation does reduce performance on all considered datasets of GLUE, demonstrating the usefulness of the compression loss of VIB (see details in Appendix D).
Revised: Note that our method and other baselines require hyper-parameter tuning. Ablation Study As an ablation, Table 6 shows results for our model without the compression loss (VIBERT ( β =0 )), in which case there is no incentive to introduce noise, and the VIB layer reduces to deterministic dimensionality reduction with an MLP. We optimize the dimensionality of the MLP layer ( K ) as a hyper-parameter for both methods. This ablation does reduce performance on all considered datasets, demonstrating the added benefit of the compression loss of VIBERT.

Original: [Table]
Revised: [Table]

Revised:  R ELATED W ORK

Revised: Low-resource Setting Recently, developing methods for low-resource NLP has gained attention (Cherry et al., 2019).

Revised: Prior work has investigated improving on low-resource datasets by injecting large unlabeled in-domain data and pretraining a unigram document model using a variational autoencoder and use its internal representations as features for downstream tasks (Gururangan et al., 2019).

Revised: Other approaches propose injecting a million-scale previously collected phrasal paraphrase relations (Arase & Tsujii, 2019) and data augmentation for translation task (Fadaee et al., 2017).

Revised: Due to relying on the additional source and in-domain corpus, such techniques are not directly comparable to our model.

Revised: Information Bottleneck IB has recently been adopted in NLP in applications such as parsing (Li &

Revised: Eisner, 2019), and summarization (West et al., 2019).

Revised: Voita et al.

Revised: (2019) use the mutual information to study how token representations evolve across layers of a Transformer model (Vaswani et al., 2017).

Revised: This paper – to the best of our knowledge – is the first attempt to study VIB as a regularization technique to improve the fine-tuning of large-scale language models on low-resource scenarios.

Revised: Regularization Techniques for Fine-tuning Language models In addition to references given throughout, Phang et al.

Revised: (2018) proposed to perform an extra data-rich intermediate supervised task pretraining followed by fine-tuning on the target task.

Revised: They showed that their method leads to improved fine-tuning performance on the GLUE benchmark.

Revised: However, their method requires pretraining with a large intermediate task.

Revised: In contrast, our goal is to use only the provided low-resource target datasets.

Original:  Conclusion and Future Directions
Revised:  C ONCLUSION AND F UTURE D IRECTIONS

Original: We propose to apply a VIB module to reduce over-fitting when fine-tuning large-scale pre-trained language models on low-resource datasets.
Revised: We propose VIBERT, an effective model to reduce overfitting when fine-tuning large-scale pretrained language models on low-resource datasets.

Original:  VIB  finds the simplest sentence embedding, predictive of the target labels, by removing task-irrelevant and task-redundant information.
Revised: By leveraging a VIB objective, VIBERT finds the simplest sentence embedding, predictive of the target labels, while removing task-irrelevant and redundant information.

Original: Our approach is model agnostic, simple to implement, and highly effective.
Revised: Our approach is model agnostic, simple to implement, and highly effective.

Original: Extensive experiments  show that our method substantially improves transfer performance in low-resource scenarios, including a 2.97 point gain on STS-B and a 2.03 point gain on MRPC.
Revised: Extensive experiments and analyses show that our method substantially improves transfer performance in low-resource scenarios.

Original: Furthermore, we demonstrate that our model results in a  better generalization to out-of-domain NLI datasets.
Revised: We demonstrate our obtained sentence embeddings are robust to biases and our model results in a substantially better generalization to out-of-domain NLI datasets.

Original: Future work includes exploring incorporating VIB on multiple layers of pre-trained language models and using it to jointly learn relevant features and relevant layers.
Revised: Future work includes exploring incorporating VIB on multiple layers of pretrained language models and using it to jointly learn relevant features and relevant layers.

Original: A Experimental Details Datasets Statistics
Revised: A CKNOWLEDGEMENTS

Original: Table 4 shows the statistics of the datasets used in our experiments. [Table]
Revised: We would like to thank Maksym Andriushchenko for his helpful comments.

Original: For IMDB, SNLI, MNLI, and tasks in the GLUE benchmark, we do not use any prepossessing. For the considered target NLI datasets in Section 3.3, we use the data in Mahabadi et al.
Revised: Rabeeh Karimi was supported by the Swiss National Science Foundation under the project Learning Representations of Abstraction for Opinion Summarisation (LAOS), grant number “FNS-30216”.

Original: (2020), where the data are first processed with an NLTK tokenizer.
Revised: Yonatan Belinkov was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20).

Original: We use the Yelp dataset in Zhang et al.
Revised: Datasets Statistics Table 7 shows the statistics of the datasets used in our experiments.

Original: (2015), which is also preprocessed with an NLTK tokenizer. Computing Infrastructure
Revised: [Table]

Original:  We run all experiments on one GTX1080Ti GPU with 12 GB of RAM.
Revised: Computing Infrastructure We run all experiments on one GTX1080Ti GPU with 11 GB of RAM.

Original: We use the BERT base uncased model with default hyper-parameters through all experiments, i.e., we use a sequence length of 128 , with batch size 8 and Adam optimizer with a learning rate of 2e − 5 .
Revised: VIBERT Architecture The MLP module used to compute the compressed sentence representations (Figure 1) is a shallow MLP with 768 , 2304+ K, 768+ Khidden units with a ReLU non-linearity, where

Original: We do not use warm-up or weight decay. VIB We compute the compressed sentence representations using p θ ( z | x ) =
Revised: K is the bottleneck size.

Original: N ( z | µ ( x ) , Σ( x )) . For computing µ ( x ) and Σ( x ) , we first feed sentence embeddings f ϕ ( x ) through a shallow nonlinear classifier with 768 , 2304+ K, 768+ Khidden units with a ReLU non-linearity. It is then followed by two linear layers, each with K hidden units to compute µ ( x ) and Σ( x ) (after a softplus transform to ensure non-negativity).
Revised: Following Alemi et al.

Original: We average over 5 posterior samples, i.e., we compute p ( y | x ) = 15 Σ 5 i =1 q φ ( y |
Revised: (2017), we average over 5 posterior samples, i.e., we compute p ( y | x )= 15 Σ 5 i =1 q φ ( y |

Original: z i ) , where z i ∼ p θ ( z | x ) .
Revised: z i ) , where z i ∼ p θ ( z | x ) .

Original: We consider Gaussian distributions for prior r ( z ) and p θ ( z | x ) because the Kullback-Leibler divergence between two Gaussian distributions has a closed-form solution 3 .

Original: We use the reparameterization trick (Kingma and Welling, 2013) to estimate the gradients, namely

Original: − 1 1 Σ 0 )+( µ 1 − µ 0 ) T Σ − 1 1 ( µ 1 − µ 0 ) − K + log( det(Σ 1 ) det(Σ 0 ) )) .

Original: z = µ ( x ) + Σ( x ) (cid:12) (cid:15) , where (cid:15) ∼ N (0 ,I ) .

Original: Similar to Bowman et al.
Revised: Similar to Bowman et al.

Original: (2016), we use a linear annealing schedule for β and set it as min(1 , epoch × β 0 ) in each epoch, where β 0 is the initial value.Dropout
Revised: (2016), we use a linear annealing schedule for β and set it as min(1 , epoch × β 0 ) in each epoch, where β 0 is the initial value.

Original: As a convention, we apply dropout to the last hidden layer of the BERT encoder. Similar Training Time Table 5 shows the finetuning time for all models. Our VIB method almost has the same training time as the baselines, but our method and dropout require hyper-parameter tuning.
Revised: B H YPER - PARAMETERS

Original: [Table] Number of Parameters Our VIB model requires (768+ K ) K + (2304+ K ) 2parameters for the MLP used to estimate µ ( x ) and Σ( x ) and an additional 2 K parameters for the parameters of the prior r ( z ) , namely µ 0 and the diagonal covariance Σ 0 .
Revised: The GLUE Benchmark Experiment Results on GLUE benchmark are reported in Table 1.

Original: We consider K min =12 and K max =384 in this work; for the BERT base model with 110M trainable parameters, it results in 0 . 61% − 1 . 22% of the BERT encoder’s parameters.
Revised: We fine-tune all the models for 6 epochs to allow them to converge.

Original: Wefine-tune BERT forepochs and use early stopping for all models by choosing the model performing the best on the validation set.
Revised: We use early stopping for all models by choosing the model performing the best on the validation set with the evaluation criterion of average F1 and accuracy for MRPC, accuracy for RTE, and average Pearson and Spearman correlations for STS-B.

Original: We sweep β over { 1 e − 4 , 1 e − 5 , 1 e − 6 } and K over { 144 , 192 , 288 , 384 } .
Revised: For VIBERT, we sweep β over { 10 − 4 , 10 − 5 , 10 − 6 } and K over { 144 , 192 , 288 , 384 } .

Original: We usethe dropout with dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .
Revised: For dropout, we use dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .

Original: Dropout with probability  0 .25 performs the best on all datasets. Table 6 shows the selected hyper-parameters for VIB, Table 7 shows the corresponding validation accuracies, and Table 8 demonstrates the standard deviations for the reported average results in Table 1.
Revised: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } . For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .

Original: [Table] C Varying-resource  Results Hyper-parameters
Revised: Varying-resource Experiment Results on varying sizes of training data are reported in Table 2.

Original: We fine-tune all models for 25 epochs.
Revised: We fine-tune all models for 25 epochs to allow them to converge.

Original: We use early stopping for all models [Table] [Table] [Table]
Revised: We use early stopping for all models based on the performance on the validation set.

Original: [Table]

Original: [Table]

Original: Dropout 0.2/0.

Original: VIB 0.0/0.

Original: based on the performance on the validation set.

Original: Hyper-parameter tuningis done on the validation set.
Revised: We also perform hyper-parameter tuning on the validation set.

Original: Since we consider datasets of a different number of training samples, we need to account for a suitable range of dimensions and we sweep K over { 12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 } and β over { 1 e − 4 , 1 e − 5 } .
Revised: Since we consider datasets of a different number of training samples, we need to account for a suitable range of bottleneck size and we sweep K over { 12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 } and β over { 10 − 4 , 10 − 5 } .

Original: For dropout, we consider dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .
Revised: For dropout, we consider dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .

Original: Dropout probability of  0 . 25 results in the best validation performance on all datasets. Hyper-parameters used to report the final results of VIB on each dataset are shown in Table 9. [Table]
Revised: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: [Table]

Original: [Table] [Table] [Table] used to report the results of the VIB method.
Revised: For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .

Original: Table 10 shows the obtained test accuracies, and Table 11 shows their corresponding validation resultsfor varying number of training samples. Since we are dealing with small datasets, we run each experiment with three random seeds, each with a different randomly sampled training instances. D Analysis
Revised: Ablation Experiment Ablation results are shown in Table 6.

Original: The results of VIB reported in Table 3 are obtained given the described setup in Appendix B. For VIB ( β =0), we sweep K over the same range of values as VIB, i [Table] e., {12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 }.
Revised: For VIBERT ( β =0), we sweep K over the same range of values as VIBERT, i.e., { 144 , 192 , 288 , 384 }

Original: Table 12 shows the standard deviations over 3 runs for the reported results in Table 3, Table 13 shows the validation results [Table] and Table 14 shows the selected hyper-parameters. [Table]
Revised: C M APPING

Original: [Table] [Table] E Out-of-domain Generalization Mapping: We train all models on SNLI or MNLI datasets and evaluate their performance on other target datasets.
Revised: We train all models on SNLI or MNLI datasets and evaluate their performance on other target datasets.

Original: The SNLI and MNLI datasets contain three labels of contradiction, neutral, and entailment.
Revised: The SNLI and MNLI datasets contain three labels of contradiction, neutral, and entailment.

Original: However, some of the considered target datasets

Original: Data [Table] [Table] have only two labels, such as DPR or SciTail.
Revised: However, some of the considered target datasets have only two labels, such as DPR or SciTail.

Original: When the target dataset has two labels of entailed and not-entailed , as in DPR, we consider the predicted contradiction and neutral labels as the not-entailed label.

Original: In the case the target dataset has two labels of entailment and neutral , as in SciTail, we consider the predicted contradiction label as neutral.

Original: Hyper-parameters We consider the models trained on 6K subsampled SNLI and MNLI explained in Appendix C. Table 15 shows the selected hyperparameters for each target dataset

Original: [Table]

Original: and Tableshows their corresponding validation accuracies

Original: [Table]

Original: F High-resource Out-of-domain Generalization Hyper-parameters

Original: we train all models for the default 3 epochs as in

Original: Devlin et al.

Original: Hyper-parameter tuning is done on the validation set.

Original: We sweep K over { 12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 } and β over { 1 e − 4 , 1 e − 5 } .

Original: Results

Original: Table 17 shows the transfer performance on the high-resource setting.

Original: Models are trained on the full SNLI or MNLI datasets and transferred to

Original: the various target datasets.

Original: The model trained on

Original: SNLI with VIB improves the transfer on average by 1.54 absolute points; the improvement for the model trained on MNLI with VIB is lower and is 0.25 points.

Original: We discuss the transfer performance from the medium-sized SNLI and MNLI datasets in

Original: Section 3.3 and observe that VIB is effective in this scenario.

Original: We conjecture that when the available data is not sufficient for the models to learn more general features, it is easier for them to learn easy-to-learn SNLI MNLI Data
Revised: When the target dataset has two labels of entailed and not-entailed , as in DPR, we consider the predicted contradiction and neutral labels as the not-entailed label.

Original: K β

Original: K β

Original: SICK 1e1e-

Original: ADD  1e1e-

Original: DPR 1e1e-

Original: SPR 1e1e-

Original: FN+ 1e1e-

Original: JOCI 1e1e-

Original: MPE 1e1e-

Original: SCITAIL 1e1e-

Original: QQP 1e1e-

Original: SNLIHard 1e1e-

Original: SNLI 1e1e-

Original: MNLI-M 1e1e-

Original: MNLI 1e1e-

Original: [Table]

Original: shortcut features; learning these shortcuts is reduced by motivating the network towards learning more general features through a VIB module.

Original: However, in high-resource datasets, there are already enough data available to learn the general features; therefore, the impact of VIB is reduced, and one needs other methods specifically targeting bias reduction like Mahabadi et al.

Original: (2020) and Clark et al.

Original: (2019) to address this issue.

Original: Investigating this phenomenon further is future work.

Original: We report the hyper-parameters in Table 19 and their corresponding validation accuracies in Table [Table] [Table] SNLI
Revised: In the case the target dataset has two labels of entailment and neutral , as in SciTail, we consider the predicted contradiction label as neutral.

Original: MNLI

Original: Data

Original: K β

Original: K β

Original: SICK 1e1e-

Original: ADD  1e1e-

Original: DPR 1e1e-

Original: SPR 1e1e-

Original: FN+ 1e1e-

Original: JOCI 1e1e-

Original: MPE 1e1e-

Original: SCITAIL 1e1e-

Original: QQP 1e1e-

Original: SNLIHard 1e1e-

Original: SNLI 1e1e-

Original: MNLI

Original: [Table]