Original: V ARIATIONAL I NFORMATION B OTTLENECK FOR E FFECTIVE L OW -R ESOURCE F INE -T UNING
Revised: V ARIATIONAL I NFORMATION B OTTLENECK FOR E FFECTIVE L OW -R ESOURCE F INE -T UNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While large-scale pretrained language models have obtained impressive results when fine-tuned on a wide variety of tasks, they still often suffer from overfitting in low-resource scenarios.
Revised: While large-scale pretrained language models have obtained impressive results when fine-tuned on a wide variety of tasks, they still often suffer from overfitting in low-resource scenarios.

Original: Since such models are general-purpose feature extractors, many of these features are inevitably irrelevant for a given target task.
Revised: Since such models are general-purpose feature extractors, many of these features are inevitably irrelevant for a given target task.

Original: We propose to use Variational Information Bottleneck (VIB) to suppress irrelevant features when fine-tuning on low-resource target tasks, and show that our method successfully reduces overfitting.
Revised: We propose to use Variational Information Bottleneck (VIB) to suppress irrelevant features when fine-tuning on low-resource target tasks, and show that our method successfully reduces overfitting.

Original: Moreover, we show that our VIB model finds sentence representations that are more robust to biases in natural language inference datasets, and thereby obtains better generalization to out-of-domain datasets.
Revised: Moreover, we show that our VIB model finds sentence representations that are more robust to biases in natural language inference datasets, and thereby obtains better generalization to out-of-domain datasets.

Original: Evaluation on seven low-resource datasets in different tasks shows that our method significantly improves transfer learning in low-resource scenarios, surpassing prior work.
Revised: Evaluation on seven low-resource datasets in different tasks shows that our method significantly improves transfer learning in low-resource scenarios, surpassing prior work.

Original: Moreover, it improves generalization on 13 out of 15 out-of-domain natural language inference benchmarks.
Revised: Moreover, it improves generalization on 13 out of 15 out-of-domain natural language inference benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transfer learning has emerged as the de facto standard technique in natural language processing (NLP), where large-scale language models are pretrained on an immense amount of text to learn a general-purpose representation, which is then transferred to the target domain with fine-tuning on target task data.
Revised: Transfer learning has emerged as the de facto standard technique in natural language processing (NLP), where large-scale language models are pretrained on an immense amount of text to learn a general-purpose representation, which is then transferred to the target domain with fine-tuning on target task data.

Original: This method has exhibited state-of-the-art results on a wide range of NLP benchmarks (Devlin et al., 2019; Liu et al., 2019;
Revised: This method has exhibited state-of-the-art results on a wide range of NLP benchmarks (Devlin et al., 2019; Liu et al., 2019;

Original: Radford et al., 2019).
Revised: Radford et al., 2019).

Original: However, such pretrained models have a huge number of parameters, potentially making fine-tuning susceptible to overfitting.
Revised: However, such pretrained models have a huge number of parameters, potentially making fine-tuning susceptible to overfitting.

Original: In particular, the task-universal nature of large-scale pretrained sentence representations means that much of the information in these representations is irrelevant to a given target task.
Revised: In particular, the task-universal nature of large-scale pretrained sentence representations means that much of the information in these representations is irrelevant to a given target task.

Original: If the amount of target task data is small, it can be hard for fine-tuning to distinguish relevant from irrelevant information, leading to overfitting on statistically spurious correlations between the irrelevant information and target labels.
Revised: If the amount of target task data is small, it can be hard for fine-tuning to distinguish relevant from irrelevant information, leading to overfitting on statistically spurious correlations between the irrelevant information and target labels.

Original: Learning low-resource tasks is an important topic in NLP (Cherry et al., 2019) because annotating more data can be very costly and time-consuming, and because in several tasks access to data is limited.
Revised: Learning low-resource tasks is an important topic in NLP (Cherry et al., 2019) because annotating more data can be very costly and time-consuming, and because in several tasks access to data is limited.

Original: In this paper, we propose to use the Information Bottleneck (IB) principle (Tishby et al., 1999) to address this problem of overfitting.
Revised: In this paper, we propose to use the Information Bottleneck (IB) principle (Tishby et al., 1999) to address this problem of overfitting.

Original: More specifically, we propose a fine-tuning method that uses Variational
Revised: More specifically, we propose a fine-tuning method that uses Variational

Original: Information Bottleneck (VIB; Alemi et al. 2017) to improve transfer learning in low-resource scenarios.
Revised: Information Bottleneck (VIB; Alemi et al. 2017) to improve transfer learning in low-resource scenarios.

Original: VIB addresses the problem of overfitting by adding a regularization term to the training loss that directly suppresses irrelevant information.
Revised: VIB addresses the problem of overfitting by adding a regularization term to the training loss that directly suppresses irrelevant information.

Original: As illustrated in Figure 1, the VIB component maps the sentence embedding from the pretrained model to a latent representation z , which is the only input to the taskspecific classifier.
Revised: As illustrated in Figure 1, the VIB component maps the sentence embedding from the pretrained model to a latent representation z , which is the only input to the taskspecific classifier.

Original: The information that is represented in z is chosen based on the IB principle, namely that all the information about the input that is represented in z should be necessary for the task.
Revised: The information that is represented in z is chosen based on the IB principle, namely that all the information about the input that is represented in z should be necessary for the task.

Original: In particular, VIB directly tries to remove the irrelevant information, making it easier for the task classifier to avoid overfitting when trained on a small amount of data.
Revised: In particular, VIB directly tries to remove the irrelevant information, making it easier for the task classifier to avoid overfitting when trained on a small amount of data.

Original: We find that in low-resource scenarios, using VIB to suppress irrelevant features in pretrained sentence representations substantially improves accuracy on the target task.
Revised: We find that in low-resource scenarios, using VIB to suppress irrelevant features in pretrained sentence representations substantially improves accuracy on the target task.

Revised: µ x ) and eliminates irrelevant and redundant information through the Gaussian noise with variance Σ( x ) .

Original: Removing unnecessary information from the sentence representation also implies removing redundant information.
Revised: Removing unnecessary information from the sentence representation also implies removing redundant information.

Original: VIB tries to find the most concise representation which can still solve the task, so even if a feature is useful alone, it may be removed if it isn’t useful when added to other features because it is redundant.
Revised: VIB tries to find the most concise representation which can still solve the task, so even if a feature is useful alone, it may be removed if it isn’t useful when added to other features because it is redundant.

Original: We hypothesize that this provides a useful inductive bias for some tasks, resulting in better generalization to out-of-domain data.
Revised: We hypothesize that this provides a useful inductive bias for some tasks, resulting in better generalization to out-of-domain data.

Original: In particular, it has recently been demonstrated that annotation biases and artifacts in several natural language understanding benchmarks (Kaushik & Lipton, 2018; Gururangan et al., 2018;
Revised: In particular, it has recently been demonstrated that annotation biases and artifacts in several natural language understanding benchmarks (Kaushik & Lipton, 2018; Gururangan et al., 2018;

Original: Poliak et al., 2018; Schuster et al., 2019) allow models to exploit superficial shortcuts during training to perform surprisingly well without learning the underlying task.
Revised: Poliak et al., 2018; Schuster et al., 2019) allow models to exploit superficial shortcuts during training to perform surprisingly well without learning the underlying task.

Original: However, models that rely on such superficial features do not generalize well to out-of-domain datasets, which do not share the same shortcuts (Belinkov et al., 2019a).
Revised: However, models that rely on such superficial features do not generalize well to out-of-domain datasets, which do not share the same shortcuts (Belinkov et al., 2019a).

Original: We investigate whether using VIB to suppress redundant features in pretrained sentence embeddings has the effect of removing these superficial shortcuts and keeping the deep semantic features that are truly useful for learning the underlying task.
Revised: We investigate whether using VIB to suppress redundant features in pretrained sentence embeddings has the effect of removing these superficial shortcuts and keeping the deep semantic features that are truly useful for learning the underlying task.

Original: We find that using VIB does reduce the model’s dependence on shortcut features and substantially improves generalization to out-of-domain datasets.
Revised: We find that using VIB does reduce the model’s dependence on shortcut features and substantially improves generalization to out-of-domain datasets.

Original: We evaluate the effectiveness of our method on fine-tuning BERT (Devlin et al., 2019), which we call the VIBERT model (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning).
Revised: We evaluate the effectiveness of our method on fine-tuning BERT (Devlin et al., 2019), which we call the VIBERT model (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning).

Original: On seven different datasets for text classification, natural language inference, similarity, and paraphrase tasks,
Revised: On seven different datasets for text classification, natural language inference, similarity, and paraphrase tasks,

Original: VIBERT shows greater robustness to overfitting than conventional fine-tuning and other regularization techniques, improving accuracies on low-resource datasets.
Revised: VIBERT shows greater robustness to overfitting than conventional fine-tuning and other regularization techniques, improving accuracies on low-resource datasets.

Original: Moreover, on NLI datasets, VIBERT shows robustness to dataset biases, obtaining substantially better generalization to out-of-domain NLI datasets.
Revised: Moreover, on NLI datasets, VIBERT shows robustness to dataset biases, obtaining substantially better generalization to out-of-domain NLI datasets.

Original: Further analysis demonstrates that VIB regularization results in less biased representations.
Revised: Further analysis demonstrates that VIB regularization results in less biased representations.

Original: Our approach is highly effective and simple to implement, involving a small additional MLP classifier on top of the sentence embeddings.
Revised: Our approach is highly effective and simple to implement, involving a small additional MLP classifier on top of the sentence embeddings.

Original: It is model agnostic and end-to-end trainable.
Revised: It is model agnostic and end-to-end trainable.

Original: In summary, we make the following contributions: 1) Proposing VIB for low-resource fine-tuning of large pretrained language models.
Revised: In summary, we make the following contributions: 1) Proposing VIB for low-resource fine-tuning of large pretrained language models.

Original: 2) Showing empirically that VIB reduces overfitting, resulting in substantially improved accuracies on seven low-resource benchmark datasets against conventional fine-tuning and prior regularization techniques.
Revised: 2) Showing empirically that VIB reduces overfitting, resulting in substantially improved accuracies on seven low-resource benchmark datasets against conventional fine-tuning and prior regularization techniques.

Original: Showing empirically that training with VIB is more robust to dataset biases in NLI, resulting in significantly improved generalization to out-of-domain NLI datasets.
Revised: Showing empirically that training with VIB is more robust to dataset biases in NLI, resulting in significantly improved generalization to out-of-domain NLI datasets.

Original: To facilitate future work, we will release our code.
Revised: To facilitate future work, we will release our code.

Original:  F INE - TUNING IN L OW - RESOURCE S ETTINGS
Revised:  F INE - TUNING IN L OW - RESOURCE S ETTINGS

Original: The standard fine-tuning paradigm starts with a large-scale pretrained model such as BERT, adds a task-specific output component which uses the pretrained model’s sentence representation, and trains this model end-to-end on the task data, fine-tuning the parameters of the pretrained model.
Revised: The standard fine-tuning paradigm starts with a large-scale pretrained model such as BERT, adds a task-specific output component which uses the pretrained model’s sentence representation, and trains this model end-to-end on the task data, fine-tuning the parameters of the pretrained model.

Original: As depicted in
Revised: As depicted in

Original: Figure 1, we propose to add a VIB component that controls the flow of information from the representations of the pretrained model to the output component.
Revised: Figure 1, we propose to add a VIB component that controls the flow of information from the representations of the pretrained model to the output component.

Original: The goal is to address overfitting in resource-limited scenarios by removing irrelevant and redundant information from the pretrained representation.
Revised: The goal is to address overfitting in resource-limited scenarios by removing irrelevant and redundant information from the pretrained representation.

Original: Problem Formulation We consider a general multi-class classification problem with a low-resource dataset D = { x i ,y i } Ni =1 consisting of inputs x i ∈X , and labels y i ∈Y .
Revised: Problem Formulation We consider a general multi-class classification problem with a low-resource dataset D = { x i ,y i } Ni =1 consisting of inputs x i ∈X , and labels y i ∈Y .

Original: We assume we are also given a large-scale pretrained encoder f ϕ ( . ) parameterized by ϕ that computes sentence embeddings for the input x i .
Revised: We assume we are also given a large-scale pretrained encoder f ϕ ( . ) parameterized by ϕ that computes sentence embeddings for the input x i .

Original: Our goal is to fine-tune f ϕ ( . )
Revised: Our goal is to fine-tune f ϕ ( . )

Original: on D to maximize generalization.
Revised: on D to maximize generalization.

Original: Information Bottleneck To specifically optimize for the removal of irrelevant and redundant information from the input representations, we adopt the Information Bottleneck principle.
Revised: Information Bottleneck To specifically optimize for the removal of irrelevant and redundant information from the input representations, we adopt the Information Bottleneck principle.

Original: The objective of IB is to find a maximally compressed representation Z of the input representation X (compression loss) that maximally preserves information about the output Y (prediction loss), 1 by minimizing:
Revised: The objective of IB is to find a maximally compressed representation Z of the input representation X (compression loss) that maximally preserves information about the output Y (prediction loss), 1 by minimizing:

Original: [Equation]
Revised: [Equation]

Original: where β ≥ 0 controls the balance between compression and prediction, and I ( .,. )
Revised: where β ≥ 0 controls the balance between compression and prediction, and I ( .,. )

Original: is the mutual information.
Revised: is the mutual information.

Original: Variational Information Bottleneck Alemi et al.
Revised: Variational Information Bottleneck Alemi et al.

Original: (2017) derive an efficient variational estimate of (1):
Revised: (2017) derive an efficient variational estimate of (1):

Original: [Equation]
Revised: [Equation]

Original: where q φ ( y | z ) is a parametric approximation of p ( y | z ) , r ( z ) is an estimate of the prior probability p ( z ) of z , and p θ ( z | x ) is an estimate of the posterior probability of z .
Revised: where q φ ( y | z ) is a parametric approximation of p ( y | z ) , r ( z ) is an estimate of the prior probability p ( z ) of z , and p θ ( z | x ) is an estimate of the posterior probability of z .

Original: During training, the compressed sentence representation z is sampled from the distribution p θ ( z | x ) , meaning that a specific pattern of noise is added to the input of the output classifier q φ ( y | z ) .
Revised: During training, the compressed sentence representation z is sampled from the distribution p θ ( z | x ) , meaning that a specific pattern of noise is added to the input of the output classifier q φ ( y | z ) .

Original: Increasing this noise decreases the information conveyed by z .
Revised: Increasing this noise decreases the information conveyed by z .

Original: In this way, the VIB module can block the output classifier q φ ( y | z ) from learning to use specific information.
Revised: In this way, the VIB module can block the output classifier q φ ( y | z ) from learning to use specific information.

Original: At test time, the expected value of z is used for predicting labels with q φ ( y | z ) .
Revised: At test time, the expected value of z is used for predicting labels with q φ ( y | z ) .

Original: We refer to the dimensionality of z as K , which specifies the bottleneck size.
Revised: We refer to the dimensionality of z as K , which specifies the bottleneck size.

Original: Note that there is an interaction between decreasing K and increasing the compression by increasing β (Shamir et al., 2010; Harremo¨es & Tishby, 2007).
Revised: Note that there is an interaction between decreasing K and increasing the compression by increasing β (Shamir et al., 2010; Harremo ¨ es & Tishby, 2007).

Original: K and β are hyper-parameters (Alemi et al., 2017).
Revised: K and β are hyper-parameters (Alemi et al., 2017).

Original: We consider parametric Gaussian distributions for prior r ( z ) and p θ ( z | x ) to allow an analytic computation for their Kullback-Leibler divergence, 2 namely r ( z ) =
Revised: We consider parametric Gaussian distributions for prior r ( z ) and p θ ( z | x ) to allow an analytic computation for their Kullback-Leibler divergence, 2 namely r ( z ) =

Original: N ( z | µ 0 , Σ 0 ) and p θ ( z | x ) =
Revised: N ( z | µ 0 , Σ 0 ) and p θ ( z | x ) =

Original: N ( z | µ ( x ) , Σ( x )) , where µ and µ 0 are K − dimensional mean vectors, and Σ and Σ 0 are diagonal covariance matrices.
Revised: N ( z | µ ( x ) , Σ( x )) , where µ and µ 0 are K − dimensional mean vectors, and Σ and Σ 0 are diagonal covariance matrices.

Original: We use the reparameterization trick (Kingma & Welling, 2013) to estimate the gradients, namely z = µ ( x )+Σ( x ) (cid:12) (cid:15) , where (cid:15) ∼N (0 ,I ) .
Revised: We use the reparameterization trick (Kingma & Welling, 2013) to estimate the gradients, namely z = µ ( x )+Σ( x ) (cid:12) (cid:15) , where (cid:15) ∼N (0 ,I ) .

Original: To compute the compressed sentence representations p θ ( z | x ) , as shown in Figure 1, we first feed sentence embeddings f ϕ ( x ) through a shallow MLP.
Revised: To compute the compressed sentence representations p θ ( z | x ) , as shown in Figure 1, we first feed sentence embeddings f ϕ ( x ) through a shallow MLP.

Original: It is then followed by two linear layers, each with K hidden units to compute µ ( x ) and Σ( x ) (after a softplus transform to ensure non-negativity).
Revised: It is then followed by two linear layers, each with K hidden units to compute µ ( x ) and Σ( x ) (after a softplus transform to ensure non-negativity).

Original: We also use another linear layer to approximate q φ ( y | z ) .
Revised: We also use another linear layer to approximate q φ ( y | z ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets We evaluate the performance on seven different benchmarks for multiple tasks, in particular text classification, natural language inference, similarity, and paraphrase detection.
Revised: Datasets We evaluate the performance on seven different benchmarks for multiple tasks, in particular text classification, natural language inference, similarity, and paraphrase detection.

Original: For NLI, we experiment with two well-known NLI benchmarks, namely SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018).
Revised: For NLI, we experiment with two well-known NLI benchmarks, namely SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018).

Original: For text classification, we evaluate on two sentiment analysis datasets, namely IMDB (Maas et al., 2011) and Yelp2013 (YELP)
Revised: For text classification, we evaluate on two sentiment analysis datasets, namely IMDB (Maas et al., 2011) and Yelp2013 (YELP)

Original: (Zhang et al., 2015).
Revised: (Zhang et al., 2015).

Original: We additionally evaluate on three low-resource datasets in the GLUE benchmark (Wang et al., 2019): 3 paraphrase detection using MRPC (Dolan & Brockett, 2005), semantic textual similarity using STS-B (Cer et al., 2017), and textual entailment using RTE (Dagan et al., 2006).
Revised: We additionally evaluate on three low-resource datasets in the GLUE benchmark (Wang et al., 2019): 3 paraphrase detection using MRPC (Dolan & Brockett, 2005), semantic textual similarity using STS-B (Cer et al., 2017), and textual entailment using RTE (Dagan et al., 2006).

Original: For the GLUE benchmark, SNLI, and Yelp, we evaluate on the standard validation and test splits.
Revised: For the GLUE benchmark, SNLI, and Yelp, we evaluate on the standard validation and test splits.

Original: For MNLI, since the test sets are not available, we tune on the matched dev set and evaluate on the mismatched dev set (MNLI-M) or vice versa.
Revised: For MNLI, since the test sets are not available, we tune on the matched dev set and evaluate on the mismatched dev set (MNLI-M) or vice versa.

Original: See Appendix A for datasets statistics and Appendix B for hyper-parameters of all methods.
Revised: See Appendix A for datasets statistics and Appendix B for hyper-parameters of all methods.

Original: Base Model We use the BERT Base (12 layers, 110M parameters) and BERT Large (24 layers, 340M parameters) uncased (Devlin et al., 2019) implementation of Wolf et al.
Revised: Base Model We use the BERT Base (12 layers, 110M parameters) and BERT Large (24 layers, 340M parameters) uncased (Devlin et al., 2019) implementation of Wolf et al.

Original: as our base models, 4 known to work well for these tasks.
Revised: as our base models, 4 known to work well for these tasks.

Original: We use the default hyper-parameters of BERT, i.e., we use a sequence length of 128 , with batch size 32 .
Revised: We use the default hyper-parameters of BERT, i.e., we use a sequence length of 128 , with batch size 32 .

Original: We use the stable variant of the Adam optimizer (Zhang et al., 2020; Mosbach et al., 2020) with the default learning rate of 2e − 5 through all experiments.
Revised: We use the stable variant of the Adam optimizer (Zhang et al., 2020; Mosbach et al., 2020) with the default learning rate of 2e − 5 through all experiments.

Original: We do not use warm-up or weight decay.
Revised: We do not use warm-up or weight decay.

Original: Baselines We compare against prior regularization techniques, including previous state-of-the-art, Mixout:
Revised: Baselines We compare against prior regularization techniques, including previous state-of-the-art, Mixout:

Original:  • Dropout (Srivastava et al., 2014), a widely used stochastic regularization techniques used in multiple large-scale language models (Devlin et al., 2019; Yang et al., 2019; Vaswani et al., 2017) to mitigate overfitting.
Revised: [Equation] • Dropout (Srivastava et al., 2014), a widely used stochastic regularization techniques used in multiple large-scale language models (Devlin et al., 2019; Yang et al., 2019; Vaswani et al., 2017) to mitigate overfitting.

Original: Following Devlin et al.
Revised: Following Devlin et al.

Original: (2019), we apply dropout on all layers of BERT.
Revised: (2019), we apply dropout on all layers of BERT.

Original: • Mixout
Revised: • Mixout

Original: (Lee et al., 2019) is a stochastic regularization technique inspired by Dropout with the goal of preventing catastrophic forgetting during fine-tuning.
Revised: (Lee et al., 2019) is a stochastic regularization technique inspired by Dropout with the goal of preventing catastrophic forgetting during fine-tuning.

Original: Mixout regularizes the learning to minimize the deviation of a fine-tuned model from the pretrained initialization.
Revised: Mixout regularizes the learning to minimize the deviation of a fine-tuned model from the pretrained initialization.

Original: It replaces the model parameters with the corresponding value from the pretrained model with probability p .
Revised: It replaces the model parameters with the corresponding value from the pretrained model with probability p .

Original: [Equation] • Weight Decay (WD) is a common regularization technique to improve generalization (Krogh & Hertz, 1992).
Revised: • Weight Decay (WD) is a common regularization technique to improve generalization (Krogh & Hertz, 1992).

Original: It regularizes the large weights w by adding a penalization term λ 2 (cid:107) w (cid:107) to the loss, where λ is a hyperparameter specifying the strength of regularization.
Revised: It regularizes the large weights w by adding a penalization term λ 2 (cid:107) w (cid:107) to the loss, where λ is a hyperparameter specifying the strength of regularization.

Original: Chelba & Acero (2004) and Daum´e III (2007) adapt WD for fine-tuning of the pretrained models, and propose to replace this regularization term with λ (cid:107) w − w 0 (cid:107) , where w 0 are the weights of the pretrained models.
Revised: Chelba & Acero (2004) and Daum´e III (2007) adapt WD for fine-tuning of the pretrained models, and propose to replace this regularization term with λ (cid:107) w − w 0 (cid:107) , where w 0 are the weights of the pretrained models.

Original: Recently, Lee et al.
Revised: Recently, Lee et al.

Original: (2019) demonstrated that the latter formulation of WD works better for finetuning of BERT than conventional WD and can improve generalization on small training sets.
Revised: ) demonstrated that the latter formulation of WD works better for finetuning of BERT than conventional WD and can improve generalization on small training sets.

Original: 3. R ESULTS ON THE GLUE B ENCHMARK
Revised: 3. R ESULTS ON THE GLUE B ENCHMARK

Original: Table 1 shows results on the low-resource datasets in GLUE.
Revised: Table 1 shows results on the low-resource datasets in GLUE.

Original: 5 We find that a) Our VIBERT model substantially outperforms the baselines on all the datasets, demonstrating the effectiveness of the proposed method.
Revised: 5 We find that a) Our VIBERT model substantially outperforms the baselines on all the datasets, demonstrating the effectiveness of the proposed method.

Original: b) Dropout decreases the performance on low-resource datasets.
Revised: b) Dropout decreases the performance on low-resource datasets.

Original: We conjecture that regularization techniques relying on stochasticity without considering the relevance to the output, in contrast to VIB, can make it more difficult for learning to extract relevant information from a small amount of data.
Revised: We conjecture that regularization techniques relying on stochasticity without considering the relevance to the output, in contrast to VIB, can make it more difficult for learning to extract relevant information from a small amount of data.

Original: Igl et al.
Revised: Igl et al.

Original: (2019) observe similar effects in another application.
Revised: (2019) observe similar effects in another application.

Original: c) Similar to the results of Zhang et al.
Revised: c) Similar to the results of Zhang et al.

Original: (2020), we find less pronounced benefits of the previously suggested methods than the results originally published.
Revised: (2020), we find less pronounced benefits of the previously suggested methods than the results originally published.

Original: This can be explained by using a more stable version of Adam (Zhang et al., 2020) suggested by the very recent work in our experiments, which decreases the added benefits of previously suggested regularization techniques on top of a stable optimizer.
Revised: This can be explained by using a more stable version of Adam (Zhang et al., 2020) suggested by the very recent work in our experiments, which decreases the added benefits of previously suggested regularization techniques on top of a stable optimizer.

Original: In contrast, our VIBERT model still substantially improves the results and surpasses the prior work in all settings for both BERT Base and BERT Large models.
Revised: In contrast, our VIBERT model still substantially improves the results and surpasses the prior work in all settings for both BERT Base and BERT Large models.

Original: Due to the computational overhead of BERT Large , for the rest of this work, we stick to BERT Base .
Revised: Due to the computational overhead of BERT Large , for the rest of this work, we stick to BERT Base .

Original: ∆ shows the absolute difference 
Revised: ∆ shows the absolute difference between the results of

Revised: VIBERT model

Revised: BERT

Original: [Table]
Revised: [Table]

Original: Impact of Random Seeds: Following Dodge et al.
Revised: Impact of Random Seeds: Following Dodge et al.

Original: (2020), we examine the choice of random seed and evaluate the performance of VIBERT and BERT by fine-tuning them across 50 random seeds on GLUE.
Revised: (2020), we examine the choice of random seed and evaluate the performance of VIBERT and BERT by fine-tuning them across 50 random seeds on GLUE.

Original: To comply with the limited access to the GLUE benchmark online system, we split the original validation sets into half and consider one half as the validation set and use the other half as the test set.
Revised: To comply with the limited access to the GLUE benchmark online system [Table] we split the original validation sets into half and consider one half as the validation set and use the other half as the test set.

Original: We first perform model selection on the validation set to fix the hyper-parameters and then fine-tune the selected models for 50 different seeds.
Revised: We first perform model selection on the validation set to fix the hyper-parameters and then fine-tune the selected models for 50 different seeds.

Original: Figure 2 shows the expected test performance (Dodge et al., 2019) as the functionof random trials.
Revised: Figure 2 shows the expected test performance (Dodge et al., 2019) as the function

Original: The results demonstrate that our VIBERT model consistently obtains better performance than BERT on all datasets.

Original: As anticipated, the expected test performance monotonically increases with more random trials (Dodge et al., 2020) till it reaches a plateau, such as after 30 trials on STS-B.

Original: 3. V ARYING - RESOURCE R ESULTS

Original: To analyze the performance of our method as a function of dataset size, we use four large-resource NLI and sentiment analysis datasets, namely SNLI, MNLI, IMDB, and YELP to be able to subsample the

Original: 50 # of Random Seeds
Revised: 50 # of Random Seeds

Original: E xp .
Revised: E xp .

Original: T e s t
Revised: T e s t

Original: A cc .
Revised: A cc .

Original: MRPC
Revised: MRPC

Original: VIBERT
Revised: VIBERT

Original: BERT     50 # of Random Seeds 88 .88 .88 .88 .89 .89 .89 .89 .
Revised: BERT     50 # of Random Seeds 88 .88 .88 .88 .89 .89 .89 .89 .

Original: E xp .
Revised: E xp .

Original: T e s t P
Revised: T e s t P

Original: CC .
Revised: CC .

Original: STS-B
Revised: STS-B

Original: VIBERT
Revised: VIBERT

Original: BERT     50 # of Random Seeds
Revised: BERT     50 # of Random Seeds

Original: E xp .
Revised: E xp .

Original: T e s t
Revised: T e s t

Original: A cc .
Revised: A cc .

Original: RTE
Revised: RTE

Original: VIBERT
Revised: VIBERT

Revised: BERT of random trials.

Original:  BERT  training data with varying sizes.
Revised: The results demonstrate that our VIBERT model consistently obtains better performance than BERT on all datasets. As anticipated, the expected test performance monotonically increases with more random trials (Dodge et al., 2020) till it reaches a plateau, such as after 30 trials on STS-B. 3. V ARYING - RESOURCE R ESULTS To analyze the performance of our method as a function of dataset size, we use four large-resource NLI and sentiment analysis datasets, namely SNLI, MNLI, IMDB, and YELP to be able to subsample the training data with varying sizes.

Original: Table 2 shows the obtained results.
Revised: Table 2 shows the obtained results.

Original: VIBERT consistently outperforms all the baselines on low-resource scenarios, but the advantages are reduced or eliminated as we approach a medium-resource scenario.
Revised: VIBERT consistently outperforms all the baselines on low-resource scenarios, but the advantages are reduced or eliminated as we approach a medium-resource scenario.

Original: Also, the improvements are generally larger when the datasets are smaller, showing that our method successfully addresses 
Revised: Also, the improvements are generally larger when the datasets are smaller, showing that our method successfully addresses low-resource scenarios.

Original: [Table]
Revised: [Table]

Original: 3. O UT - OF - DOMAIN G ENERALIZATION
Revised: [Table] [Table] [Table] [Table]

Original: Besides improving fine-tuning on low-resource data by removing irrelevant features
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: we expect VIB to improve on out-of-domain data because it removes redundant features.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: In particular, annotation artifacts create shortcut features, which are superficial cues correlated with a label (Gururangan et al., 2018; Poliak et al., 2018) that do not generalize well to out-of-domain datasets (Belinkov et al., 2019a).
Revised: [Table] [Table] [Table] [Table]

Original: Since solving the real underlying task can be done without these superficial shortcuts
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  they must be redundant with the deep semantic features that are truly needed.
Revised: Besides improving fine-tuning on low-resource data by removing irrelevant features, we expect VIB to improve on out-of-domain data because it removes redundant features. In particular, annotation artifacts create shortcut features, which are superficial cues correlated with a label (Gururangan et al., 2018; Poliak et al., 2018) that do not generalize well to out-of-domain datasets (Belinkov et al., 2019a). Since solving the real underlying task can be done without these superficial shortcuts, they must be redundant with the deep semantic features that are truly needed.

Original: We hypothesize that many more superficial shortcut features are needed to reach the same level of performance as a few deep semantic features.
Revised: We hypothesize that many more superficial shortcut features are needed to reach the same level of performance as a few deep semantic features.

Original: If so, then VIB should prefer to keep the concise deep features and remove the abundant superficial features, thus encouraging the classifier to rely on the deep semantic features, and thus resulting in better generalization to out-of-domain data.
Revised: If so, then VIB should prefer to keep the concise deep features and remove the abundant superficial features, thus encouraging the classifier to rely on the deep semantic features, and thus resulting in better generalization to out-of-domain data.

Original: To evaluate out-of-domain generalization, we take NLI models trained on medium-sized 6K subsampled SNLI and MNLI in Section 3.2 and evaluate their generalization on several NLI datasets.
Revised: To evaluate out-of-domain generalization, we take NLI models trained on medium-sized 6K subsampled SNLI and MNLI in Section 3.2 and evaluate their generalization on several NLI datasets.

Original: Datasets: We consider a total of 15 different NLI datasets used in Mahabadi et al.
Revised: Datasets: We consider a total of 15 different NLI datasets used in Mahabadi et al.

Original: (2020), including
Revised: (2020), including

Original: SICK (Marelli et al., 2014), ADD1 (Pavlick & Callison-Burch, 2016), JOCI (Zhang et al., 2017), MPE (Lai et al., 2017), MNLI, SNLI, SciTail (Khot et al., 2018), and three datasets from White et al. (2017) namely
Revised: SICK (Marelli et al., 2014), ADD1 (Pavlick & Callison-Burch, 2016), JOCI (Zhang et al., 2017), MPE (Lai et al., 2017), MNLI, SNLI, SciTail (Khot et al., 2018), and three datasets from White et al. (2017) namely

Original: DPR (Rahman & Ng, 2012), FN+ (Pavlick et al., 2015), SPR (Reisinger et al., 2015), and Quora Question
Revised: DPR (Rahman & Ng, 2012), FN+ (Pavlick et al., 2015), SPR (Reisinger et al., 2015), and Quora Question

Original: Pairs (QQP) interpreted as an NLI task as by Gong et al.
Revised: Pairs (QQP) interpreted as an NLI task as by Gong et al.

Original: We use the same split used in Wang et al.
Revised: We use the same split used in Wang et al.

Original: We also consider SNLI hard and MNLI(-M) Hard sets (Gururangan et al., 2018), a subset of SNLI/MNLI(-M) where a hypothesis-only model cannot correctly predict the labels and the known biases are avoided.
Revised: We also consider SNLI hard and MNLI(-M) Hard sets (Gururangan et al., 2018), a subset of SNLI/MNLI(-M) where a hypothesis-only model cannot correctly predict the labels and the known biases are avoided.

Original: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (Appendix C).
Revised: Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (Appendix C).

Original: Following prior work (Belinkov et al., 2019a;
Revised: Following prior work (Belinkov et al., 2019a;

Original: Mahabadi et al., 2020), we select hyper-parameters based on the development set of each target dataset and report the results on the test set.
Revised: Mahabadi et al., 2020), we select hyper-parameters based on the development set of each target dataset and report the results on the test set.

Original: Results: Table 

Revised: Results: Table

Original: shows the results of VIBERT and BERT.
Revised: shows the results of VIBERT and BERT.

Original: We additionally include WD, the baseline that performed the best on average on SNLI and MNLI in Table 2.
Revised: We additionally include WD, the baseline that performed the best on average on SNLI and MNLI in Table 2.

Original: On models trained on SNLI, VIBERT improves the transfer on 13 out of 15 datasets, obtaining a substantial average improvement of 5.51 points.
Revised: On models trained on SNLI, VIBERT improves the transfer on 13 out of 15 datasets, obtaining a substantial average improvement of 5.51 points.

Original: The amount of improvement on different datasets varies, with the largest improvement on SPR and SciTail with +15.5, and +12.5 points respectively, while WD on average obtains only 0.99 points improvement.
Revised: The amount of improvement on different datasets varies, with the largest improvement on SPR and SciTail with +15.5, and +12.5 points respectively, while WD on average obtains only 0.99 points improvement.

Original: On models trained on MNLI, VIBERT improves the transfer on 13 datasets, obtaining an average improvement of 3.83 points.
Revised: On models trained on MNLI, VIBERT improves the transfer on 13 datasets, obtaining an average improvement of 3.83 points.

Original: The improvement varies across the datasets, with the largest on ADD1 and JOCI with 16.8 and 8.3 points respectively, substantially surpassing WD.
Revised: The improvement varies across the datasets, with the largest on ADD1 and JOCI with 16.8 and 8.3 points respectively, substantially surpassing WD.

Original: Interestingly, VIBERT improves the results on the SNLI and MNLI(-M) hard sets, resulting in models that are more robust to known biases.
Revised: Interestingly, VIBERT improves the results on the SNLI and MNLI(-M) hard sets, resulting in models that are more robust to known biases.

Original: These results support our claim that VIBERT motivates learning more general features, rather than redundant superficial features, leading to an improved generalization to datasets without these superficial biases.
Revised: These results support our claim that VIBERT motivates learning more general features, rather than redundant superficial features, leading to an improved generalization to datasets without these superficial biases.

Original: In the next section, we analyze this phenomenon more.
Revised: In the next section, we analyze this phenomenon more.

Original: [Table]
Revised: [Table]

Revised: +4.

Revised: ADD 78.

Revised: +5.

Revised: +1.

Revised: +16.

Revised: +1.

Revised: DPR 50.

Revised: SPR 50.

Revised: +15.

Revised: +1.

Revised: +6.

Revised: FN+ 50.

Revised: +2.

Revised: +3.

Original: +0.
Revised: +0.

Original: JOCI 42.
Revised: JOCI 42.

Original: +8.
Revised: +8.

Original: +1.
Revised: +1.

Original: +8.
Revised: +8.

Original: +0.
Revised: +0.

Original: MPE 58.
Revised: MPE 58.

Original: +3.
Revised: +3.

Original: SCITAIL 62.
Revised: SCITAIL 62.

Original: +12.
Revised: +12.

Original: +2.
Revised: +2.

Original: +3. +3.
Revised: +3. +3.

Original: QQP 65.
Revised: QQP 65.

Original: +5.
Revised: +5.

Original: +0.
Revised: +0.

Original: +2.
Revised: +2.

Original: +0.
Revised: +0.

Original: SNLI Hard 65.
Revised: SNLI Hard 65.

Original: +2.
Revised: +2.

Original: +1.
Revised: +1.

Original: +3.
Revised: +3.

Original: +0.
Revised: +0.

Original: MNLI Hard 46.
Revised: MNLI Hard 46.

Original: +6.
Revised: +6.

Original: +1. +1.
Revised: +1. +1.

Original: +0.
Revised: +0.

Original: MNLI-M Hard 46.
Revised: MNLI-M Hard 46.

Original: +6.
Revised: +6.

Original: +0. +0. SNLI 80. +1.
Revised: +

Original: +0. +3. +1. MNLI-M 60.
Revised: [Table]

Original: +4. +1.
Revised: — — +5.

Original: +0. +0.
Revised: — +0.

Original: MNLI 61.

Original: +5. +1. +1.
Revised: — — +3.

Original: +0.

Original: Average — — +5.

Original: — +0.
Revised: — +0.

Original: [Table]

Original:  A NALYSIS
Revised:  A NALYSIS

Original: Analysis of the Removed Features Elazar & Goldberg (2018) propose a challenging framework to evaluate if debiasing methods have succeeded in removing biases from the sentence representation.
Revised: Analysis of the Removed Features Elazar & Goldberg (2018) propose a challenging framework to evaluate if debiasing methods have succeeded in removing biases from the sentence representation.

Original: After debiasing, the trained encoder is frozen and the classifier is retrained to try to extract the biases.
Revised: After debiasing, the trained encoder is frozen and the classifier is retrained to try to extract the biases.

Original: If the classifier reaches high accuracy given only bias features, then the encoder’s representation has not been successfully debiased.
Revised: If the classifier reaches high accuracy given only bias features, then the encoder’s representation has not been successfully debiased.

Original: We follow the framework of Elazar & Goldberg (2018) to analyze whether known biases in NLI data have been removed in the trained sentence representations.
Revised: We follow the framework of Elazar & Goldberg (2018) to analyze whether known biases in NLI data have been removed in the trained sentence representations.

Original: In particular, following Belinkov et al.
Revised: In particular, following Belinkov et al.

Original: (2019b), we train a classifier which only sees the representation of the hypothesis sentence and see if it can predict the class of the sentence pair, which is an established criterion to measure known biases in NLI datasets (Gururangan et al., 2018).
Revised: (2019b), we train a classifier which only sees the representation of the hypothesis sentence and see if it can predict the class of the sentence pair, which is an established criterion to measure known biases in NLI datasets (Gururangan et al., 2018).

Original: Thus, we freeze the trained encoders from our model and the BERT baseline and retrain a hypothesis-only classifier on hypotheses from the SNLI and
Revised: Thus, we freeze the trained encoders from our model and the BERT baseline and retrain a hypothesis-only classifier on hypotheses from the SNLI and

Original: MNLI datasets.
Revised: MNLI datasets.

Original: 6 For reference, we compare to a hypothesis-only model with a BERT encoder trained end-to-end.
Revised: 6 For reference, we compare to a hypothesis-only model with a BERT encoder trained end-to-end.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: With the baseline (BERT), the retrained classifier is not able to recapture all the bias (H-only), but it captures much more than with our method (VIBERT).
Revised: With the baseline (BERT), the retrained classifier is not able to recapture all the bias (H-only), but it captures much more than with our method (VIBERT).

Original: VIBERT is so successful at reducing biases that performance of the hypothesis-only classifier is close to chance (33%).
Revised: VIBERT is so successful at reducing biases that performance of the hypothesis-only classifier is close to chance (33%).

Original: [Table]
Revised: [Table]

Original:  Impact of VIB on Overfitting To analyze the effect of VIB on reducing overfitting, we analyze the effect of the β parameter on training and validation error since β controls the trade-off between removing information from the sentence embedding (high β ) and keeping information that is predictive of the output (low β ).
Revised: VIBERT 38. Impact of VIB on Overfitting To analyze the effect of VIB on reducing overfitting, we analyze the effect of the β parameter on training and validation error since β controls the trade-off between removing information from the sentence embedding (high β ) and keeping information that is predictive of the output (low β ).

Original: We fix the bottleneck size ( K ) based on the models selected in Section 3.1, and we train VIBERT on the GLUE benchmark for varying values of β and plot the validation and training loss in Figure 3.
Revised: We fix the bottleneck size ( K ) based on the models selected in Section 3.1, and we train VIBERT on the GLUE benchmark for varying values of β and plot the validation and training loss in Figure 3.

Original: For small values of β , where VIB has little effect, the validation loss is substantially higher than the training loss, indicating overfitting.
Revised: For small values of β , where VIB has little effect, the validation loss is substantially higher than the training loss, indicating overfitting.

Original: This is because the network learns to be more deterministic ( Σ ≈ 0 ), thereby retaining too much irrelevant information.
Revised: This is because the network learns to be more deterministic ( Σ ≈ 0 ), thereby retaining too much irrelevant information.

Original: As we increase β , where VIB has an effect, we observe better generalization performance with less overfitting.
Revised: As we increase β , where VIB has an effect, we observe better generalization performance with less overfitting.

Original: As β becomes too large, both the training and validation losses shoot up because the amount of preserved information is insufficient to differentiate between the classes.
Revised: As β becomes too large, both the training and validation losses shoot up because the amount of preserved information is insufficient to differentiate between the classes.

Original: This pattern is observable in the MRPC and RTE datasets, with a similar pattern in the STS-B dataset.
Revised: This pattern is observable in the MRPC and RTE datasets, with a similar pattern in the STS-B dataset.

Original: [Figure]
Revised: [Figure]

Original: Efficiency Evaluation Table 5 presents the efficiency evaluation in terms of memory, number of parameters, and time for all the methods measured on RTE.
Revised: Efficiency Evaluation Table 5 presents the efficiency evaluation in terms of memory, number of parameters, and time for all the methods measured on RTE.

Original: Our approach has several attractive properties.
Revised: Our approach has several attractive properties.

Original: First, while our method is slightly larger in terms of parameters compared to the other standard regularization approaches due to an additional MLP layer (Figure 1), the difference is still marginal, and for BERT Base model with 109.48M trainable parameters, that is less than 1.22% more parameters. 
Revised: First, while our method is slightly larger in terms of parameters compared to the other standard regularization approaches due to an additional MLP layer (Figure 1), the difference is still marginal, and for BERT Base model with 109.48M trainable parameters, that is less than 1.22% more parameters. Second, [Figure]

Original: Second, our approach presents a much better memory usage with low-overhead, close to Dropout, while WD and especially Mixout cause substantial memory overhead.
Revised: [Table] our approach presents a much better memory usage with low-overhead, close to Dropout, while WD and especially Mixout cause substantial memory overhead.

Original: In dealing with large-scale transformer models like
Revised: In dealing with large-scale transformer models like

Original: BERT, efficient memory usage is of paramount importance.
Revised: BERT, efficient memory usage is of paramount importance.

Original: Third, in terms of training time, our method is similar to Dropout and much faster than the other two baselines.
Revised: Third, in terms of training time, our method is similar to Dropout and much faster than the other two baselines.

Original: Relative to BERT, VIBERT increases the training time by 3.77%, while WD and Mixout cause the substantial training overhead of 9.11% and 14.44%.
Revised: Relative to BERT, VIBERT increases the training time by 3.77%, while WD and Mixout cause the substantial training overhead of 9.11% and 14.44%.

Original: Note that our method and other baselines require hyper-parameter tuning.
Revised: Note that our method and other baselines require hyper-parameter tuning.

Original: [Table] Ablation Study As an ablation, Table 6 shows results for our model without the compression loss (VIBERT ( β =0 )) [Table] in which case there is no incentive to introduce noise
Revised: Ablation Study As an ablation, Table 6 shows results for our model without the compression loss (VIBERT ( β =0 )), in which case there is no incentive to introduce noise, and the VIB layer reduces to deterministic dimensionality reduction with an MLP.

Original: [Table]

Original: and the VIB layer reduces to deterministic dimensionality reduction with an MLP.

Original: We optimize the dimensionality of the MLP layer ( K ) as a hyper-parameter for both methods.
Revised: We optimize the dimensionality of the MLP layer ( K ) as a hyper-parameter for both methods.

Original: This ablation does reduce performance on all considered datasets, demonstrating the added benefit of the compression loss of VIBERT.
Revised: This ablation does reduce performance on all considered datasets, demonstrating the added benefit of the compression loss of VIBERT.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Low-resource Setting Recently, developing methods for low-resource NLP has gained attention (Cherry et al., 2019).
Revised: Low-resource Setting Recently, developing methods for low-resource NLP has gained attention (Cherry et al., 2019).

Original: Prior work has investigated improving on low-resource datasets by injecting large unlabeled in-domain data and pretraining a unigram document model using a variational autoencoder and use its internal representations as features for downstream tasks (Gururangan et al., 2019).
Revised: Prior work has investigated improving on low-resource datasets by injecting large unlabeled in-domain data and pretraining a unigram document model using a variational autoencoder and use its internal representations as features for downstream tasks (Gururangan et al., 2019).

Original: Other approaches propose injecting a million-scale previously collected phrasal paraphrase relations (Arase & Tsujii, 2019) and data augmentation for translation task (Fadaee et al., 2017).
Revised: Other approaches propose injecting a million-scale previously collected phrasal paraphrase relations (Arase & Tsujii, 2019) and data augmentation for translation task (Fadaee et al., 2017).

Original: Due to relying on the additional source and in-domain corpus, such techniques are not directly comparable to our model.
Revised: Due to relying on the additional source and in-domain corpus, such techniques are not directly comparable to our model.

Original: Information Bottleneck IB has recently been adopted in NLP in applications such as parsing (Li &
Revised: Information Bottleneck IB has recently been adopted in NLP in applications such as parsing (Li &

Original: Eisner, 2019), and summarization (West et al., 2019).
Revised: Eisner, 2019), and summarization (West et al., 2019).

Original: Voita et al.
Revised: Voita et al.

Original: (2019) use the mutual information to study how token representations evolve across layers of a Transformer model (Vaswani et al., 2017).
Revised: (2019) use the mutual information to study how token representations evolve across layers of a Transformer model (Vaswani et al., 2017).

Original: This paper – to the best of our knowledge – is the first attempt to study VIB as a regularization technique to improve the fine-tuning of large-scale language models on low-resource scenarios.
Revised: This paper – to the best of our knowledge – is the first attempt to study VIB as a regularization technique to improve the fine-tuning of large-scale language models on low-resource scenarios.

Original: Regularization Techniques for Fine-tuning Language models In addition to references given throughout, Phang et al.
Revised: Regularization Techniques for Fine-tuning Language models In addition to references given throughout, Phang et al.

Original: (2018) proposed to perform an extra data-rich intermediate supervised task pretraining followed by fine-tuning on the target task.
Revised: (2018) proposed to perform an extra data-rich intermediate supervised task pretraining followed by fine-tuning on the target task.

Original: They showed that their method leads to improved fine-tuning performance on the GLUE benchmark.
Revised: They showed that their method leads to improved fine-tuning performance on the GLUE benchmark.

Original: However, their method requires pretraining with a large intermediate task.
Revised: However, their method requires pretraining with a large intermediate task.

Original: In contrast, our goal is to use only the provided low-resource target datasets.
Revised: In contrast, our goal is to use only the provided low-resource target datasets.

Original:  C ONCLUSION AND F UTURE D IRECTIONS
Revised:  C ONCLUSION AND F UTURE D IRECTIONS

Original: We propose VIBERT, an effective model to reduce overfitting when fine-tuning large-scale pretrained language models on low-resource datasets.
Revised: We propose VIBERT, an effective model to reduce overfitting when fine-tuning large-scale pretrained language models on low-resource datasets.

Original: By leveraging a VIB objective, VIBERT finds the simplest sentence embedding, predictive of the target labels, while removing task-irrelevant and redundant information.
Revised: By leveraging a VIB objective, VIBERT finds the simplest sentence embedding, predictive of the target labels, while removing task-irrelevant and redundant information.

Original: Our approach is model agnostic, simple to implement, and highly effective.
Revised: Our approach is model agnostic, simple to implement, and highly effective.

Original: Extensive experiments and analyses show that our method substantially improves transfer performance in low-resource scenarios.
Revised: Extensive experiments and analyses show that our method substantially improves transfer performance in low-resource scenarios.

Original: We demonstrate our obtained sentence embeddings are robust to biases and our model results in a substantially better generalization to out-of-domain NLI datasets.
Revised: We demonstrate our obtained sentence embeddings are robust to biases and our model results in a substantially better generalization to out-of-domain NLI datasets.

Original: Future work includes exploring incorporating VIB on multiple layers of pretrained language models and using it to jointly learn relevant features and relevant layers. 
Revised: Future work includes exploring incorporating VIB on multiple layers of pretrained language models and using it to jointly learn relevant features and relevant layers. A CKNOWLEDGEMENTS

Revised: Rabeeh Karimi was supported by the Swiss National Science Foundation under the project Learning

Revised: Representations of Abstraction for Opinion Summarisation (LAOS), grant number “FNS-30216”.

Revised: Yonatan

Revised: Belinkov was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20)

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: Datasets Statistics Table 7 shows the statistics of the datasets used in our experiments.
Revised: Datasets Statistics Table 7 shows the statistics of the datasets used in our experiments.

Original: [Table]
Revised: [Table]

Original: Computing Infrastructure We run all experiments on one GTX1080Ti GPU with 12 GB of RAM.
Revised: Computing Infrastructure We run all experiments on one GTX1080Ti GPU with 12 GB of RAM.

Original:  The MLP module used to compute the compressed sentence representations (Figure 1) is a shallow MLP with 768 , 2304+ K, 768+ Khidden units with a ReLU non-linearity, where
Revised: VIBERT Architecture The MLP module used to compute the compressed sentence representations (Figure 1) is a shallow MLP with 768 , 2304+ K, 768+ Khidden units with a ReLU non-linearity, where

Original: K is the bottleneck size.
Revised: K is the bottleneck size.

Original: Following Alemi et al.
Revised: Following Alemi et al.

Original: (2017), we average over 5 posterior samples, i.e., we compute p ( y | x )= 15 Σ 5 i =1 q φ ( y |
Revised: (2017), we average over 5 posterior samples, i.e., we compute p ( y | x )= 15 Σ 5 i =1 q φ ( y |

Original: z i ) , where z i ∼ p θ ( z | x ) .
Revised: z i ) , where z i ∼ p θ ( z | x ) .

Original: Similar to Bowman et al.
Revised: Similar to Bowman et al.

Original: (2016), we use a linear annealing schedule for β and set it as min(1 , epoch × β 0 ) in each epoch, where β 0 is the initial value.
Revised: (2016), we use a linear annealing schedule for β and set it as min(1 , epoch × β 0 ) in each epoch, where β 0 is the initial value.

Original: B H YPER - PARAMETERS
Revised: B H YPER - PARAMETERS

Original: The GLUE Benchmark Experiment Results on GLUE benchmark are reported in Table 1.
Revised: The GLUE Benchmark Experiment Results on GLUE benchmark are reported in Table 1.

Original: We fine-tune all the models for 6 epochs to allow them to converge.
Revised: We fine-tune all the models for 6 epochs to allow them to converge.

Original: We use early stopping for all models by choosing the model performing the best on the validation set with the evaluation criterion of average
Revised: We use early stopping for all models by choosing the model performing the best on the validation set with the evaluation criterion of average

Original: F1 and accuracy for MRPC, accuracy for RTE, and average Pearson and Spearman correlations for STS-B.
Revised: F1 and accuracy for MRPC, accuracy for RTE, and average Pearson and Spearman correlations for STS-B.

Original: For VIBERT, we sweep β over { 10 − 4 , 10 − 5 , 10 − 6 } and K over { 144 , 192 , 288 , 384 } .
Revised: For VIBERT, we sweep β over { 10 − 4 , 10 − 5 , 10 − 6 } and K over { 144 , 192 , 288 , 384 } .

Original: For dropout, we use dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .
Revised: For dropout, we use dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .

Original: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .
Revised: For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .

Original: Varying-resource Experiment Results on varying sizes of training data are reported in Table 2.
Revised: Varying-resource Experiment Results on varying sizes of training data are reported in Table 2.

Original: We fine-tune all models for 25 epochs to allow them to converge.
Revised: We fine-tune all models for 25 epochs to allow them to converge.

Original: We use early stopping for all models based on the performance on the validation set.
Revised: We use early stopping for all models based on the performance on the validation set.

Original: We also perform hyper-parameter tuning on the validation set.
Revised: We also perform hyper-parameter tuning on the validation set.

Original: Since we consider datasets of a different number of training samples, we need to account for a suitable range of bottleneck size and we sweep K over { 12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 } and β over { 10 − 4 , 10 − 5 } .
Revised: Since we consider datasets of a different number of training samples, we need to account for a suitable range of bottleneck size and we sweep K over { 12 , 18 , 24 , 36 , 48 , 72 , 96 , 144 , 192 , 288 , 384 } and β over { 10 − 4 , 10 − 5 } .

Original: For dropout, we consider dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .
Revised: For dropout, we consider dropping probabilities of { 0 . 25 , 0 . 45 , 0 . 65 , 0 . 85 } .

Original: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: For Mixout, we consider mixout probability of { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .
Revised: For WD, we consider weight decay of { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } .

Original: Ablation Experiment Ablation results are shown in Table 6.
Revised: Ablation Experiment Ablation results are shown in Table 6.

Original: For VIBERT ( β =0), we sweep K over the same range of values as VIBERT, i.e., { 144 , 192 , 288 , 384 }
Revised: For VIBERT ( β =0), we sweep K over the same range of values as VIBERT, i.e., { 144 , 192 , 288 , 384 }

Original: C M APPING
Revised: C M APPING

Original: We train all models on SNLI or MNLI datasets and evaluate their performance on other target datasets.
Revised: We train all models on SNLI or MNLI datasets and evaluate their performance on other target datasets.

Original: The SNLI and MNLI datasets contain three labels of contradiction, neutral, and entailment.
Revised: The SNLI and MNLI datasets contain three labels of contradiction, neutral, and entailment.

Original: However, some of the considered target datasets have only two labels, such as DPR or SciTail.
Revised: However, some of the considered target datasets have only two labels, such as DPR or SciTail.

Original: When the target dataset has two labels of entailed and not-entailed , as in DPR, we consider the predicted contradiction and neutral labels as the not-entailed label.
Revised: When the target dataset has two labels of entailed and not-entailed , as in DPR, we consider the predicted contradiction and neutral labels as the not-entailed label.

Original: In the case the target dataset has two labels of entailment and neutral , as in SciTail, we consider the predicted contradiction label as neutral.
Revised: In the case the target dataset has two labels of entailment and neutral , as in SciTail, we consider the predicted contradiction label as neutral.