Multilingual C heck L ist: Generation and Evaluation (2022)

Abstract: Multilingual evaluation benchmarks usually contain limited high-resource languages and do not test models for specific linguistic capabilities. CheckList is a template-based evaluation approach that tests models for specific capabilities. The CheckList template creation process requires native speakers, posing a challenge in scaling to hundreds of languages. In this work, we explore multiple approaches to generate Multilingual CheckLists. We device an algorithm –Template Extraction Algorithm (TEA) for automatically extracting target language CheckList templates from machine translated instances of a source language templates. We compare the TEA CheckLists with CheckLists created with different levels of human intervention. We further introduce metrics along the dimensions of cost, diversity, utility, and correctness to compare the CheckLists. We thoroughly analyze different approaches to creating CheckLists in Hindi. Furthermore, we experiment with 9 more different languages. We find that TEA followed by human verification is ideal for scaling Checklist-based evaluation to multiple languages while TEA gives a good estimates of model performance. We release the code of TEA and the CheckLists created at aka.ms/multilingualchecklist

## introduction multilingual transformer based models @xcite @xcite have demonstrated commendable zero & few-shot capabilities. their performance is typically evaluated on benchmarks like xnli @xcite , xglue @xcite , xtreme @xcite ) & xtreme-r @xcite . however, this evaluation paradigm has a number of limitations including: first, most of these datasets are limited to a few high resource languages @xcite @xcite , except for a few tasks (e.g., ner, pos @xcite ). second, creating high quality test sets of substantial size for many tasks and languages is prohibitively expensive. third, state-of-art models are known to learn spurious patterns to achieve high accuracies, saturating performance on these test-benches, yet performing poorly on often much simpler real world cases @xcite @xcite @xcite . fourth, these benchmarks do not evaluate models for language specific nuances @xcite . lastly, this evaluation approach does not provide any insights into where the model is failing @xcite . these limitations lead to the need of interactive, challenging, and much larger testing datasets (like @xcite ) and more holistic approaches to evaluation @xcite ). checklist @xcite ) is an evaluation paradigm that systematically tests the various (linguistic) capabilities required to solve a task. it allows creation of large and targeted test sets easily using various abstractions. specifically, users can generate templates, essentially sentences with slots that can be filled in with a dictionary of lexicons to generate test instances. checklist templates are created by native speakers. @xcite introduce multilingual checklists created by human translation from english checklist for 50 languages for a subset of tests on question answering. however, since checklists are task & language specific, human creation or translation of checklists remains extremely resource-intensive. in this paper, we introduce an automatic approach to creating multilingual checklists. we devise the template extraction algorithm (tea) for extracting templates in a target language from the translated instances of a source language checklist (here english) automatically ( §2). we also experiment with semi-automatic and manual approaches for multilingual checklist creation ( §3). in the semi-automatic approach (tea-ver), we ask human annotators to verify and correct the templates created by tea. in the manual approach, we ask annotators to create checklists in two ways: first, by translation of english checklist to the target language (t9n) (same as @xcite ); second, by giving a description of the task and capabilities to create checklists from scratch (scr) (same as original english checklists creation @xcite ).using these four approaches, we create checklists for sentiment analysis (sa) and natural language inference (nli) in hindi ( §5). we demonstrate broad applicability of tea by generating checklists in additional 9 typologically diverse languages (gujarati, french, swahili, arabic, german, spanish, russian, vietnamese, japanese) and tea-ver checklists in 3 of them ( §6). evaluation of checklists is non-trivial. for thorough comparisons, we propose evaluation metrics along four axes: utility, diversity, cost & correctness ( §4). our evaluation indicates that checklists created using tea are not only cost-effective but also useful and diverse, with comparable quality to the manually and semi-automatically created checklists. experiments on typologically diverse languages show that tea checklists provide a good estimate of the failures of the model, and thus can be used even in the absence of resources to verify them or create human-annotated gold test-sets. to summarize, our contributions are: a) we propose tea (template extraction algorithm) to extract templates in a target language using translated instances of a source checklist. b) we experiment with varying degrees of human intervention, comparing semi-automatic & manual approaches of multilingual checklist creation with tea, to understand the best utilization of the human effort. c) we introduce evaluation metrics along the axes of utility, diversity, cost, and correctness for indepth comparison of the the checklists. d) we will release all the 4 checklists in hindi for sa and nli, tea checklists in 9 languages for sa and tea-ver checklists in 3 languages for sa. we release the code of tea and the checklists created at aka.ms/multilingualchecklist ## tea: template extraction algorithm terminology (consistent with ribeiro et al. ( 2020 )): linguistic capabilities: these are capabilities tested for a particular task. for e.g, negation. templates: these are sentences with slots. for e.g, '{city} is beautiful'. here, '{city}' is a slot. templates can have any number of slots. lexicon keys and values: this a dictionary of values. in the above example, 'city' is the key. values are the words that would be filled in the slots (replacing the keys) like 'new delhi', 'new york', 'london', etc. we use the notation '@xmath0, 'london', 'new york'] ' for lexicons. instances: these are test sentences created by inserting lexicon values in templates . in the above example, the instances formed are: 'new york is beautiful', 'london is beautiful', etc. the checklist paradigm allows creation of large number of test instances. for multilingual evaluation, these can then be translated to the target languages using machine translation. however, there are limitations to this approach. firstly, a large machine translated test set is difficult to be verified by humans, as one would have to go through every example. second, it defeats the purpose of abstraction that checklists facilitates. and third, the quality of this test set will be directly impacted by the quality of the mt system. this results in the need to generate templates in the target language so that these can be utilized and verified in the same fashion as the template sets in the source language. our early experiments suggested that due to word order and syntactic differences between languages, both: 1) a word-to-word or heuristic translation of the template and 2) extraction of template from a single source instance (such as by simply replacing one word with other in a single translated instance) do not work well for template translation. this necessities a non-trivial algorithm that can extract templates given a set of instances. we propose the template extraction algorithm or tea, to automatically extract template sets given an input a set of instances. in this paper, these input instances for tea are obtained by machine translating instances created from the source checklist template sets. we use machine translation to reduce cost and human effort, but the algorithm can be used with any input set of instances, i.e it would work with human-translated instances. briefly, tea is a recursive approach to extract templates from input instances by treating every input instance as directed acyclic graph of the words. tea combines the instances with similar structure into a single template by recursively merging instances and replacing terminals (or lexicon values) with non-terminals (or lexicon keys). figure 1 shows how tea creates lexicon keys by combining instances from the translated instances. we assume english (en) to be the source language and hindi (hi) to be the target language. the pipeline starts with an en template, the instances are created by replacing the lexicon values in the templates, that are then machine translated to get hi instances. these instances function as input to tea which then recursively groups instances using non-terminals to form templates. the entire process of template extraction is repeated for every en template, resulting in the hi template set. the tea has 3 steps which we describe as follows (pseudocode and details are in appendix a): step 1: grouping terminals into non-terminals: first, we convert the hi instances into a directed acyclic graph whose nodes are unique words (or tokens). there is an edge from node a to b if word b follows word a in at least one of the input instances. in this directed graph (see fig. 1 ), between any two nodes, if there are multiple paths of length less than equal to k + 1 (we set k to 2), we concatenate the intermediate words in the path (with space in between them) and treat them as terminals. this set of terminals, between the two nodes, are grouped together represented by a nonterminal symbol (for example key_1 and key_2 in fig. 1 ). this step corresponds to lexicon formation; the non-terminal extracted here are essentially keys of the lexicon & the terminals constituting them are the lexicon values for the slots of a template. step 2: template extraction using a set of hi instances, @xmath1, s 2 , . . . s n }, and all nonterminals v @xmath2, w i2 , . . .], where w ij are terminals (obtained step 1), tea outputs a set of templates @xmath3, t 2 , ...} such that t can generate all the examples in s using the given non-terminal and their corresponding terminals. for each sentence s i , we generate a set of candidate templates, t @xmath4, t i2 , ...}, such that s i belongs to the set of examples generated by each t ij . to find the minimal template set, i.e t that covers all examples is treated as a set cover problem and we use a greedy approximation to find this set. ## multilingual checklist generation we now describe the various ways in which multilingual checklists can be generated, ranging from fully automatic to fully manual approaches. using tea we start with a source language (en) checklist template and generate instances by replacing lexicon values in templates. these instances are translated using an mt system. the translated instances now serve as the input to the tea and target language (hi) checklist template is extracted. the process (fig. 1 ) is repeated for all en templates to form the complete hi checklist. tea with verification (tea-ver) this is a semi-automatic approach, where we ask a humanverifier to verify and correct the checklists generated using tea. the verifiers (or annotators) are provided with a set of templates and lexicons generated using the tea pipeline, along with the original source langauge checklist and description of the capabilities. the annotators are instructed to verify the target language templates for (grammatical) correctness. they can delete or edit the incorrect templates. they can also add any missing templates that they think are significantly important (cover too many missed instances). translating source checklist (t9n) this is a completely manual approach, but relies on a source language (here, en) checklist. the annotators are provided with the en templates, lexicons and the descriptions of capabilities. they are tasked to translate the templates and lexicons into the target language. if a source template cannot be translated to a single target template (such as due to divergent grammatical agreement patterns), annotators are instructed to include as many variants as necessary. this approach is same as that used by @xcite to create multilingual checklist. ## evaluation metrics comparison of checklists is non-trivial. firstly, checklists cannot be evaluated using absolute metrics, comparisons can only be relative @xcite . further, the question of what constitutes a better checklist can be answered in multiple ways. for example, if a checklist a can help discover (and/or fix) more bugs than checklist b, checklist a could be more useful. on the other hand, variability of instances may be desirable. if checklist b generates more diverse instances as compared checklist a, even though it discovers less bugs, b could be considered better as it allows testing of the system on a broader variety of instances. finally, in practical scenarios, cost and correctness are both important factors for generating the checklist. we thus propose evaluation metrics along 4 dimensions: a) utility for discovery and fixing bugs; b) diversity in the generated instances; c) cost of generating templates. d) correctness of templates. ## hindi checklists and results we start with hindi (hi) as the target language, create checklists using all 4 methods from §3 and evaluate them using the metrics from §4. hi has significant syntactic divergence from the source language (here english (en)) and uses a different script. hi is a mid-resource language with reasonably good publicly available en-hi mt systems. we argue that if tea works well in the en-hi pipeline, it would also work for most other high to mid resource languages with reasonable mt systems and similar or less syntactic divergence from en, which we also substantiate by performing additional multilingual experiments in §6. ## checklists in multiple languages so far, we see that tea is cost-efficient in producing effective hi checklists. we now experiment with 9 more typologically diverse languages -arabic, french, german, gujarati, japanese, russian, spanish, swahili and vietnamese to evaluate the efficacy of scaling tea to may languages. we use tea to automatically generate checklists across these languages from the same set of source templates in english for sa across 6 capabilities: vocabulary, temporal, fairness, negation, semantic role labeling (srl) and robustness. we use the same source en checklist from the ribeiro et al. (2020) and use bing translator in the tea pipeline to translate en instances to the target language. in for 3 of the target languages, namely french, gujarati and swahili, native speakers verified the generated templates and thus, we also report the fr for tea-ver. 5 we observe that the pearson (spearman) correlation between tea and tea-ver fr values for french, gujarati and swahili are 0.99 (1.0), 0.98 (0.89) and 0.97 (0.94) respectively. furthermore, the difference between fr (fr-diff) is also low. this implies, similar to our observations from section 5, that one can obtain an extremely accurate assessment of the capabilities of multilingual models just from tea checklists even for low resource languages like swahili. this re-affirms that despite noise, tea is able to generate checklists that are useful without any human supervision. ## limitations in this paper, we introduced the tea to generate target language checklist (templates + lexicon) from the translated instances of source language checklist. we show that with drastically reduced human effort required for creating checklist in a new language, the tea checklists provide an accurate estimate of the models' capabilities. however, some of the generated templates/lexicons are noisy and were removed or edited by humans through the tea-ver process. in this section, we summarize the limitations, common error patterns and suggest some possible ways to resolve them. agnostic to semantics tea is agnostic of the semantics of the lexicon keys. so, when faced with a set of sentences: las vegas is good., new york is good., new delhi is good. and las palmas is good., it is unclear whether it should design 1 template city is good. with lexicon @xmath5, new york, new delhi, las pal-mas} or 2 templates: las city1 is good., @xmath6, palmas} and new city2 is good., @xmath7, delhi}. this problem is hard to solve without heuristics. one possibility is to use the translation alignment information however, such alignments are often imperfect even for high-resource languages. we leave improvements to tea for handling this to future work. handling morphology creating good templates for morphologically rich languages @xcite is more challenging due to inflections. for e.g, in hindi a verb may take different form for different tenses and gender. while tea can handle such cases by creating multiple templates, but with still a third of hi templates needed correcting. we leave morphologically informed checklist creation to future work. translation errors translation errors are a frequent pattern, affecting the input target language instances. in some cases, due to the statistical nature of tea, we are able to naturally filter out such erroneous templates. for e.g, for an en template 'i used to think this {air_noun} was {neg_adj}, {change} now i think it is {pos_adj}', translated hi templates 'mujhe lagta hai ki us {udaan} {ghatia} tha, ab mujhe lagta hai ki yeh asadharan hai' (correct) matches 187 translations, and 'mujhe lagta hai ki us {udaan} {ghatia} tha karte the, ab mujhe lagta hai ki yeh bohut achha hai' (noisy) matches only 35. while tea can remove some noisy patterns, errors due to misunderstood context are much harder to fix. for e.g 'the service is poor' translated as 'vah seva garib hai' but 'garib' in hindi means "lacking sufficient money" and not "lower or insufficient standards". we leave comparisons of tea for human v.s machine translated input instances and methods to measure and reduce the effect of translation errors on tea to future work. ## conclusion in this paper we proposed tea (template extraction algorithm) to automatically generate multilingual checklists in a target language without any human supervision ( §2). this algorithm recursively extracts templates and lexicon from an input set of instances by treating sentences as a directed acyclic graph of words and combining them. we additionally experimented with 3 other approaches with varying degrees of human intervention, 2 manual and 1 semi-automatic for checklist generation ( §3). for comparing these checklists, we introduced metrics along the dimensions of utility, diversity, cost and correctness ( §4). we performed in-depth analysis of all the 4 methods, with varying degree of human interventions, to create checklists for sentiment analysis and nli in hindi ( §5). in addition to hindi, we experimented with 9 more typologically diverse languages to demonstrate the efficacy of tea along with comparison with human-verified checklists in 3 of them ( §3). we found that tea is costeffective, useful, and diverse in the checklists that it generates. while around one-third of the tea templates required correction by humans, making the semi-automatic approach more reliable, we find that the model performance estimates provided by unverified checklists are very close to that of the human-verified (or semi-automatically created) checklists and are also significantly correlated to it. we also substantiated the finding of tea being effective as well as reliable in the other languages. our overall recommendation is that tea followed by human verification is the most reliable and cost-effective way to scale checklist evalu-ation to multiple languages. but in case of very limited resources, tea is still good enough to test system performance. we end with a discussion on the limitations of this work and propose directions that will, hopefully, inspire research in scaling and improving multilingual evaluation using check-lists. finally, we note that tea is general purpose algorithm of template extraction that can be used for other template-based evaluations such as bias evaluation @xcite a details of tea