TAPE : Assessing Few-shot R ussian Language Understanding (2022)

Abstract: Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation suites for non-English languages, hindering progress outside the Anglo-centric paradigm. To address this line of research, we propose TAPE (Text Attack and Perturbation Evaluation), a novel benchmark that includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. The TAPE’s design focuses on systematic zero-shot and few-shot NLU evaluation: (i) linguistic-oriented adversarial attacks and perturbations for analyzing robustness, and (ii) subpopulations for nuanced interpretation. The detailed analysis of testing the autoregressive baselines indicates that simple spelling-based perturbations affect the performance the most, while paraphrasing the input has a more negligible effect. At the same time, the results demonstrate a significant gap between the neural and human baselines for most tasks. We publicly release TAPE (https://tape-benchmark.com) to foster research on robust LMs that can generalize to new tasks when little to no supervision is available.

## introduction the ability to acquire new concepts from a few examples is central to human intelligence @xcite . recent advances in the nlp field have fostered the development of language models (lms; @xcite ) that exhibit such generalization capacity under a wide range of few-shot learning and prompting methods @xcite . the community has addressed various aspects of few-shot learning, such as efficient model application @xcite , adaptation to unseen tasks and domains @xcite , and cross-lingual generalization @xcite . the latest research has raised an essential question of standardized evaluation protocols to assess few-shot generalization from multiple perspectives. the novel tool-kits and benchmarks mainly focus on systematic evaluation design @xcite , cross-task generalization @xcite , and real-world scenarios @xcite . however, this rapidly developing area fails to provide similar evaluation suites for non-english languages, hindering progress outside the anglo-centric paradigm. motivation and contributions. in this paper, we introduce tape foot_0 , a novel benchmark for few-shot russian language understanding evaluation. our objective is to provide a reliable tool and methodology for nuanced assessment of zero-shot and few-shot methods for russian. the objective is achieved through two main contributions. contribution 1. our first contribution is to create six more complex question answering (qa), winograd schema, and ethics tasks for russian. the tasks require understanding many aspects of language, multi-hop reasoning, logic, and commonsense knowledge. the motivation behind this is that there are systems that match or outperform human baselines on most of the existing qa tasks for russian, e.g., the ones from russian superglue @xcite : danetqa @xcite , muserc and rucos @xcite . to the best of our knowledge, datasets on ethical concepts have not yet been created in russian. to bridge this gap, we propose one of the first russian datasets on estimating the ability of lms to predict human ethical judgments about various text situations. contribution 2. our second contribution is to develop a framework for multifaceted zero-shot and few-shot nlu evaluation. the design includes (i) linguistic-oriented adversarial attacks and perturbations for testing robustness, and (ii) subpopulations for nuanced performance analysis. here, we follow the methodological principles and recommendations by @xcite and @xcite , which motivate the need for systematic benchmark design and adversariallyconstructed test sets. findings. our findings are summarized as fivefold: (i) zero-shot evaluation may outperform fewshot evaluation, meaning that the autoregressive baselines fail to utilize demonstrations, (ii) fewshot results may be unstable and sensitive to prompt changes, (iii) negative result: zero-shot and fewshot generation for open-domain and span selection qa tasks leads to near-zero performance, (iv) the baselines are most vulnerable to spelling-based and emoji-based adversarial perturbations, and (v) human annotators significantly outperform the neural baselines, indicating that there is still room for developing robust and generalizable systems. few-shot benchmarking. research in few-shot benchmarking has evolved in several directions. @xcite create fewglue by sampling small fixed-sized training datasets from superglue. variance w.r.t to training dataset size and sampling strategy is not reported. later works overcome these issues by exploring evaluation strategies by k-fold cross-validation @xcite , bagging, and multi-splits, introduced in fewnlu @xcite . additionally, fewnlu explores correlations between performance on development and test sets and stability w.r.t. the number of runs. crossfit @xcite studies cross-task generalization by unifying task formats and splitting tasks into training, development, and test sets. flex @xcite covers the best practices and provides a unified interface for different types of transfer and varying shot sizes. finally, to the best of our knowledge, the only non-english dataset for few-shot benchmarking is few-clue in chinese @xcite . tape is the first few-shot benchmark for russian, which introduces variations at the data level by creating adversarial test sets. ## task formulations tape includes six novel datasets for russian, each requiring modeling "intellectual abilities" of at least two skills: logical reasoning ( §3.1; extended winograd schema challenge), reasoning with world knowledge ( §3.2; chegeka, ruopen-bookqa and ruworldtree), multi-hop reasoning ( §3.2; multiq), and ethical judgments ( §3.3; ethics 1/2 ). this section describes the task formulations, general data collection stages, and dataset examples. appendix a provides the general dataset statistics, while appendix e.1 includes details on dataset collection and extra validation stage via a crowd-sourcing platform toloka foot_1 @xcite . ## conclusion and future work zero-shot and few-shot methods have evolved as a new paradigm in nlp. addressing the best practices, we introduced tape, a text attack and perturbation evaluation benchmark for russian. tape combines the general language understanding evaluation techniques with the greener no-tuning approach, allowing the evaluation of lms' robustness on complex intellectual tasks. we present six new datasets and a framework for generating adversarial attacks and perturbations, which can also be used as a standalone tool for practical purposes. in future, we plan to incorporate more lms with various architectures and prompting-based methods into the framework. another direction is to evaluate the cross-lingual generalization capabilities of autoregressive lms. we hope to encourage the community to foster evaluation of lms' generalization capacity in non-english languages, leading to the development of more robust and reliable lms.