Regularising Fisher Information Improves Cross-lingual Generalisation (2021)

Abstract: Many recent works use ‘consistency regularisation’ to improve the generalisation of fine-tuned pre-trained models, both multilingual and English-only. These works encourage model outputs to be similar between a perturbed and normal version of the input, usually via penalising the Kullback–Leibler (KL) divergence between the probability distribution of the perturbed and normal model. We believe that consistency losses may be implicitly regularizing the loss landscape. In particular, we build on work hypothesising that implicitly or explicitly regularizing trace of the Fisher Information Matrix (FIM), amplifies the implicit bias of SGD to avoid memorization. Our initial results show both empirically and theoretically that consistency losses are related to the FIM, and show that the flat minima implied by a small trace of the FIM improves performance when fine-tuning a multilingual model on additional languages. We aim to confirm these initial results on more datasets, and use our insights to develop better multilingual fine-tuning techniques.

## introduction multilingual pre-trained representations @xcite @xcite are ubiquitous in state-of-the-art methods for cross-lingual transfer @xcite . these methods learn from raw textual data in up to hundreds of languages. a typical pipeline transfers to another language by fine-tuning a downstream task in a high-resource language, often english. many recent works use 'consistency regularisation' to improve the generalisation of fine-tuned pre-trained models, both multilingual and englishonly @xcite @xcite @xcite . these works encourage model outputs to be similar between a perturbed and normal version of the input, usually via penalising the kullback-leibler (kl) divergence between the probability distribution of the perturbed and normal model. 'generic' perturbations can be adversarial inputs @xcite or inputs with gaussian or uniform noise @xcite . for cross-lingual generalisation in particular, probabilistic subword segmentations @xcite of the input or translations of the input generated by machine translation can be used @xcite . other work has found improvement by enforcing consistency for perturbations within models in addition to at the input @xcite . while these works show improved generalisation compared to 'vanilla' fine-tuning, they present multiple explanations of the effectiveness of consistency regularisation. they also rarely compare to traditional regularisation methods like dropout or l2 regularisation. finally such methods either require a complex adversarial training step @xcite , or tuning many hyper-parameters like type of noise, level of noise, and weight given to the consistency loss term. we believe that consistency losses may be implicitly regularizing the loss landscape. in particular, we build on the work of @xcite , who hypothesize that implicitly or explicitly regularizing trace of the fisher information matrix (fim), tr(f ), amplifies the implicit bias of sgd to avoid memorization. briefly, the fim is defined as f(θ) = e x∼x ,y∼p θ (y|x) [g(x, y)g(x, y) t ], where g(x, y) is the gradient w.r.t to θ on the loss for label y and input x. @xcite propose directly penalising a proxy of tr(f ), the fisher penalty defined as in the multilingual setting we may wish to first fine-tune on a high-resource language like english, then further fine-tune on a smaller amount data in a lower-resource language, a 'two-stage' fine-tuning procedure. the fim is a measure of the local curvature, and a small tr(f ) at the end of training implies a flatter minimum. intuitively, such flat minima imply we can 'travel further' in parameter space before reaching a region of high loss, allowing for better performance in the two-stage fine-tuning setting. figure 1 : english validation set accuracy on a subset of the xnli dataset vs. trace of the fim at the end of training, for models with different weight given to consistency regularisation (crosses) and fisher penalty (circles) losses. for clarity we leave various models with different l2 penalties (squares) off the legend. @xmath0. @xmath1. ## initial results we first present a theoretical argument that in some simple situations, consistency losses penalize the trace of the fim. we perturb model parameters θ with small, zero-mean, i.i.d. noise . for small , we can taylor expand the kl divergence between models with these parameters: @xcite . taking expectations w.r.t. and writing t f as a sum, we have, figure 1 presents results from an experiment on a subset (20k examples, with the small size chosen due to compute constraints) of the xnli dataset @xcite . it shows 1) a correlation between generalisation and small tr(f ), and 2) decreasing tr(f ) with increasing weight given to the figure 2 : increase in target language (either de, es or zh) validation set accuracy when fine-tuning on data in the target language vs. trace of the fim after training on english, for the paws-x dataset @xcite . figure 2 shows the effect of small tr(f ), i.e. flat minima, on fine-tuning a model trained on english data on another language. to obtain non-english training data, we split the 2000 dev set examples in two, leaving 1000 training examples in each language and 1000 new dev examples. we see that improvements on the non-english language are correlated with flat minima. we aim to confirm these initial results on more datasets, and use our insights to develop better multilingual fine-tuning techniques.