The biases present in training datasets have been shown to be affecting models for a number of tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate such biases , a common issue is that of catastrophic URLetting of the original task . In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases for NLI and fact verification while being less susceptible to catastrophic URLetting. In our evaluation on fact verification systems , we show that fine-tuning with EWC Pareto dominates standard fine-tuning, yielding models  lower levels of URLetting on the original task for equivalent gains in accuracy on the fine-tuned task. Additionally, we show that systems trained on NLI can be fine-tuned to improve their accuracy on stress test challenge tasks with minimal loss in accuracy on the MultiNLI datasetdespite greater domain shift .
The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate them , a common issue is that of catastrophic URLetting of the original training dataset . In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases while being less susceptible to catastrophic URLetting. In our evaluation on fact verification and NLI stress tests , we show that fine-tuning with EWC dominates standard fine-tuning, yielding models with lower levels of URLetting on the original (biased) dataset for equivalent gains in accuracy on the fine-tuning (unbiased) dataset .
R
be affecting models for a number of
affect models for sentence pair classification
R
such biases
them
R
task
training dataset
D
for NLI and fact verification
R
systems
and NLI stress tests
D
Pareto
A
with
R
task
(biased) dataset
R
fine-tuned task. Additionally, we show that systems trained on NLI can be fine-tuned to improve their accuracy on stress test challenge tasks with minimal loss in accuracy on the MultiNLI datasetdespite greater domain shift
fine-tuning (unbiased) dataset