Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks (2022)

Abstract: Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today’s NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.

## introduction quantifier words-such as each or most or more than three-have been extensively studied, both in logic and in linguistics @xcite , going all the way back to @xcite . in this paper, we examine the extent to which they present a challenge to modern nlu systems. our analysis is motivated by three observations: quantifier words are abstract unlike nouns, verbs and adjectives, quantifier words do not have referents out in the world. rather, quantifier words specify relationships between sets of entities, events and properties. to provide intuitions about the semantics of quantifier words, and to be able to refer to quantifiers in a language-independent way, we rely on the notion of generalized quantifiers @xcite , as described in §2. quantifier words vary across languages quantifier word inventories differ across languages. ## background generalized quantifiers (gqs) are developed upon first-order predicate logic, denoting relations between sets @xcite . given a universe e, a quantifier q would be treated as a mapping q e from the cartesian product of powersets p(e) × p(e) to the set {false,true} or, as a binary relation on subsets of e @xcite . gqs are generalizations of the ∀,∃ quantifiers from first-order predicate logic @xcite @xcite @xcite . a generalized quantifier is, abstractly, a relation between sets. generalized quantifier theory, while developed by logicians, is used by formal linguists to analyze the 1 https://github.com/ruixiangcui/gqnli meaning of quantifier words in combination with referential expressions @xcite . most human languages contain ways of expressing generalized quantifiers, and their semantics exhibit striking similarities across languages @xcite @xcite . at the same time, generalized quantifiers can be instantiated very differently across languages due to pragmatic considerations @xcite or cognitive economy and costbenefit optimisation in the exchange of information @xcite @xcite . quantifier words also exhibit syntactic differences, e.g., with some languages having specialized words to express quantity, while others rely on metaphorical usage of common nouns @xcite . in english, most is a determiner, but spanish and french express the same concept through common nouns, la mayoría and la majorité. the relative stability of the core semantics of quantifiers makes a cross-linguistic comparison possible, but the syntactic and pragmatic variation associated with the expression of generalized quantifiers poses a challenge for multilingual nlu. we consult quantifier taxonomy studies @xcite @xcite and derive a categorization set for quantifier analysis in nlu benchmarks. in ## nlu benchmarks we conduct an error analysis focusing on the role of generalized quantifiers in two nlu tasks, natural language inference (nli) and question answering (qa), which generally require understanding of quantifiers. for each type of task, both monolingual and cross-lingual evaluation are conducted. we focus on generalized quantifiers in the hypotheses in nli examples-and on generalized quantifiers in the question fields in question answering. to this end, we identify quantifiers by the lemma and the universal dependency relation @xcite of a quantifier after preprocessing the sentences using stanza @xcite . take the sentence "the yiddish culture has survived for more than a thousand years.", we annotate it as 2 note that generalized quantifiers are not always explicit in discourse. the sentence inadequate sleep causes obesity should be interpreted as most of those who do not sleep adequately, gain weight @xcite . such implicit quantifiers related to pragmatic variation are important for language understanding, but will be ignored in this work. "the/det yiddish/amod culture/nsubj have/aux survive/root for/case more/advmod than/fixed a/det thousand/nummod year/obl ./punct". by matching the regex pattern of the quantifier "more than k", in this case "((more|great)\/advmod than\/(fixed|case)|at\/case least\/nmod) .+\/nummod .+\/(nsubj|obj|obl)", we approximate the surface form of the type "more than k".through matching quantifier patterns, we are able to find entries in which quantifiers are instantiated. see appendix a for the list of regex patterns we write to identify gqs. in ## quantifiers in english nli benchmarks nli is commonly framed as a three-way classification task with labels entailment, contradiction and neutral @xcite . while sota models exhibit low error rates on nli benchmarks, it is unclear when they succeed or fail in their underlying reasoning. we are interested in whether generalized quantifers challenge modern nli models. in our error analysis, we initially focus on three english nli datasets, multinli (mnli; @xcite , @xcite and anli @xcite ) as testbeds. ## quantifiers in cross-lingual nlu benchmarks quantifiers are acquired in similar orders across languages @xcite , although languages express quantifiers in different ways. for example, there are eight different universal quantifiers with different level of distributivity in malagasy @xcite . this poses challenges to training multilingual lms and transfer learning. we are interested in whether quantifiers are universally and evenly challenging for all languages. quantifiers in cross-lingual nli we choose xnli @xcite , a manual translation of the development and test set of mnli into 15 languages, for this multilingual error analysis. we should clarify that for xnli, the authors annotate entailment labels for the english data only and apply them to the other languages. we do not assume label changes due to translation in this study, but it is worth investigate in the future. we choose five languages belonging to different language families, namely arabic, chinese, german, spanish and vietnamese as targets. the last column in we fine-tune mbert foot_0 @xcite and xlm foot_1 (lample and conneau, 2019) on the mnli training set and evaluate them on xnli. we report the results in similar to our results for english, we can see that the lowest accuracies in xnli are with proportional quantifiers, such as most and few. but the gap in non-english languages is wider for these two categories, especially for chinese, the difference reaches 30%. other hard quantifiers include all, > k, < k, and each other. quantifiers in cross-lingual qa cross-lingual question answering (xqa) is another important nlu task that evaluates the cross-lingual transferability of lms. we evaluate the effect of quantifiers on system errors across two xqa datasets, namely xquad @xcite and mlqa @xcite . as demonstrated in figure 1 , quantifier word distributions in xqa tasks also follow zipf's law, as in nli tasks, but k is more frequent (perhaps because of a traditional emphasis on numerical reasoning), and we see less variance across languages. this is probably because question answering is targeting quantification less directly. to evaluate cross-lingual qa performance on gqs, we fine-tune mbert and xlm-r foot_2 (conneau et al., 2020) using hu et al. (2020)'s architecture. we present results for mbert in just as with xnli, lms suffer from performance drops across all languages for almost all gq phenomena with significant, cross-lingual variation. the most distinguished is that exact match (em) suffers from a greater deterioration than f1 scores for all languages. for example, the weighted em difference for mbert on mlqa is 2.9% while the weighted f1 is 1%. as one example in ## gqnli we have seen how quantifiers present challenges to nli and qa models. using an approach similar to anli @xcite and dynabench @xcite , we use model difficulty (roberta's) as a heuristic to select hard examples for a challenge dataset that can hopefully be used to evaluate any future progress on this. we propose gqnli, a generalized quantifier nli challenge dataset, consisting of 30 premises and 300 hypotheses. the average sentence lengths of hypothesis and premises are 15.97 and 7.35, respectively. both numbers are comparable to those of mnli, but lower than anli's @xcite . it should be noted that gqnli is designed for evaluating future models; obviously not for benchmarking roberta. dataset creation firstly, we manually create 100 premise-hypothesis pairs, in which various types of gqs appear. for each premise and hypothesis, the number of gqs varies from one to three. to choose the premises, we randomly sampled 100 premises with gqs from snli and anli test sets, respectively, and selected 10 premises in total, that we consider are semantically adequate for adding gqs and making simple hypotheses. to construct the hypotheses, we rely on roberta fine-tuned on mnli and manually select examples about which the model is unsure or incorrect. to focus on gqs, we keep the challenge examples otherwise simple @xcite , and avoid lexical variations in the hypotheses. hard examples were found to be characterized by (i) mixing generalized quantifiers with other logical operators, such as subsumption or negation, and (ii) combining multiple different generalized quantifiers. we discuss these observations in section 7. two of the authors annotated the examples. the inter-annotator agreement (fleiss' kappa) was 0.895, substantially higher than anli's (0.672-0.740). it is worth noting that the level of semantic or pragmatic interpretation difference of gqs is reflected in the measurement. we augmented the examples by substituting non-quantifier words (e.g., replacing "dogs" with "cats") while keeping the labels, to exclude the effect of specific lexical items. the resulting labels are uniformly distributed. ## discussion negation the interaction between negation words and quantifiers increases semantic complexity @xcite . we investigate whether this holds for nli tasks, using negation cue detection to find all cases where a negation word and a quantifier appear in the hypotheses. we break down the performances on negation of the seven models in appendix f. as indicated, lms overall have polarized results for negation cases comparing to the entire dataset. we can see a majority of the models even predicted opposite labels for some gq categories, with 0% accuracy. bart is no longer the second best model, replaced by roberta. the improvement by training with more data is overall consistent for reasoning over gqs with negation. for a cross-lingual investigation of the interaction of gqs and negation, we find that in xnli, the number of cases combining both phenomena is insufficient: we identified four such cases, involving only the quantifiers "all" and "more than." for english, mbert predicted two cases successfully. for chinese, german, vietnamese and arabic, one is correct. for spanish, all are wrongly predicted. it is evident that nlu models suffer from reasoning difficulties in certain cases when negation interacts with gqs, especially in cross-lingual evaluation. in future work, we are interested in expanding gqnli to more instances and more languages to facilitate qualitative investigations. ## related work we examine the sensitivity of nlu models to generalized quantifiers. these models are designed to induce correlations from large volumes of data, not to reason symbolically with logical quantifiers. such models have, nevertheless, been probed for logical knowledge. mul and zuidema (2019), for example, show neural networks encode fragments of first-order logic and exhibit zero-shot generalization ability. @xcite present a neural architecture that improves performance on propositional logical inference. @xcite also suggest neural networks learn semantic representations for logical inference in natural languages. however, on the same task, @xcite find neural networks fail to do so on a more stringent test. @xcite also show that neural networks fail to exhibit robust logical inference. @xcite use semantic parsers to encode quantifiers and improve zero-shot learning in classification tasks. @xcite present a system that computes logical inference over gqs and see improvements on two specialized datasets, fracas @xcite and med @xcite . none of these papers explicitly discussed generalized quantifiers, and all were limited to studying the ability of neural networks to capture the logical semantics of english. many studies have instead focused on lms' ability to capture negation @xcite @xcite @xcite or coreference @xcite @xcite . others have focused on lms' ability to reason with numbers @xcite . drop @xcite , for example, is a question answering dataset designed specifically to probe lms' ability to count, add and subtract for answering factoid questions. models have also been tailored for numerical reasoning @xcite . @xcite proposes to use a verification task during pretraining of lms to improve their ability to solve math word problems. others have studied monotonicity inference @xcite @xcite , and @xcite recently focused on the two quantifier words part and whole in an error analysis for named entity recognition. many nlu benchmarks contain quantifier words, but their influence on performance has not been studied systematically. one exception to this is that generalized quantifiers have been used to generate adversarial examples in the context of numerical reasoning @xcite . taxinli @xcite , which categorizes 15 types of reasoning abilities, is a dataset drawn from mnli. in their taxonomy, the quantifier category only refers to universal and existential quantifiers, not to generalized quantifiers, and ditto for @xcite . all of the above focused on english, but in an extension to @xcite incorporated quantifiers into the logic class and found a large cross-lingual transfer gap on lms. ## conclusion quantifiers lie in the intersection of logic, linguistics and nlp research. it is essential for nlu systems to learn quantifier reasoning. we examined generalized quantifiers in multilingual nlu tasks with regards to their expressiveness and logical reasoning requirement. our survey and experiments indicate quantifiers are neglected to a degree and cause significant performance drops for neural lms. to better understand lms' reasoning abilities, we release gqnli, a novel generalized quantifier nli challenge dataset. with the pervasiveness of generalized quantifiers, we stress that more efforts are necessary to investigate: (1) when and why models systematically fail when quantifiers interact with other operators; (2) how to improve cross-lingual transferability of quantifiers; (3) how we can exploit the theoretical results about generalized quantifiers from logic and linguistic studies, so as to improve the logical inference ability of neural lms.