We present a focused study of few-shot crosslingual transfer , a recently proposed NLP scenario: a pretrained multilingual encoder is first finetuned on many annotations in a high resource language (typically English), and then finetuned on a few annotations (the ``few shots'') in a target language. Few-shot transfer brings large improvements over zero-shot transfer. However, we show that it inherently has large variance and it is necessary to report results on multiple sets of few shots for stable results and to guarantee fair comparison of different algorithms. To address this problem, we publish our few-shot sets. In a study of why few-shot learning outperforms zero-shot transfer, we show that large models heavily rely on lexical hints when finetuned on a few shots and then overfit quickly. We evaluate different methods that use few-shot annotations, but do not observe significant improvements over the baseline. This calls for better ways of utilizing the few-shot annotations .
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available .
R
We present a focused study of few-shot crosslingual transfer , a recently proposed NLP scenario: a pretrained multilingual encoder is first finetuned on many annotations in a high resource language (typically English), and then finetuned on a few annotations (the ``few shots'') in a target language. Few-shot transfer brings large improvements over
Few-shot crosslingual transfer has been shown to outperform its
R
transfer. However, we show that it inherently has large variance and it is necessary to report results on multiple sets of
counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled
R
stable results and to guarantee fair comparison of different algorithms. To address this problem, we publish our
six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of
R
sets. In a study of why few-shot learning outperforms zero-shot transfer,
transfer, which highlights the role of lexical features. Additionally,
R
large models heavily rely on lexical hints when finetuned on a few shots and then overfit quickly. We evaluate different methods that use
a straightforward full model finetuning approach is quite effective for
R
annotations, but do not observe significant improvements over the baseline. This calls for better ways of utilizing the
transfer, outperforming several state-of-the-art
R
annotations
approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available