Generalized Quantiﬁers as a Source of Error in Multilingual NLU Benchmarks
Abstract
Logical approaches to representing language have developed and evaluated computational models of quantiﬁer words since the 19th century, but today’s NLU models still struggle to capture their semantics.
We rely on
Generalized Quantiﬁer Theory for languageindependent representations of the semantics of quantiﬁer words, to quantify their contribution to the errors of NLU models.
We ﬁnd that quantiﬁers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops.
Multilingual models also exhibit unsatisfying quantiﬁer reasoning abilities, but not necessarily worse for non-English languages.
To facilitate directlytargeted probing, we present an adversarial generalized quantiﬁer NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantiﬁer reasoning.
Introduction
Quantiﬁer words—such as each or most or more than three —have been extensively studied, both in logic and in linguistics (Westerståhl, 1989; Peters and Westerståhl, 2006), going all the way back to Frege (1879).
In this paper, we examine the extent to which they present a challenge to modern
NLU systems.
Our analysis is motivated by three observations:
Quantiﬁer words are abstract Unlike nouns, verbs and adjectives, quantiﬁer words do not have referents out in the world.
Rather, quantiﬁer words specify relationships between sets of entities, events and properties.
To provide intuitions about the semantics of quantiﬁer words, and to be able to refer to quantiﬁers in a language-independent way, we rely on the notion of generalized quantiﬁers (Mostowski, 1957), as described in §2.
Quantiﬁer words vary across languages
Quantiﬁer word inventories differ across languages.
[Table]
Often what is considered rough translation equivalents also differ in syntax, ﬁne-grained semantics or pragmatics.
Stateva et al.
(2019) show, e [Table] g., that perceptions of the numerical bounds of existential quantiﬁers differ across speakers of English, French, Slovenian, and German.
Other papers showing discrepancies between quantiﬁer systems include comparisons of Salish to English (Matthewson, 2001), Adyghe to English (Nikolaeva, 2012) [Table] or of Dutch, Hebrew and Bengali (Gil, 1982).
The cross-linguistic differences in how generalized quantiﬁers are expressed motivates a cross-lingual error analysis, since quantiﬁers may contribute more to error when processing some languages rather than others
[Table]
Quantiﬁer words are important Quantiﬁer words are extremely important for tasks that require inference, including natural language inference, question answering, fact-checking, etc.
Datasets have, for example, been developed for numerical reasoning in English (Dua et al., 2019).
Several researchers have identiﬁed quantiﬁer words as important sources of errors for natural language processing systems (Joshi et al., 2020); see Table
[Table]
some [Table] [Table]
=
[Table]
A
[Table]
This process is known to increase security in several ways.
all (A)(B) =
A ⊆ B
Everyone agreed the food was terrible.
more than k the(A)(B) =|
A ∩ B | > k
They do let them go more than twice a week.
less than k the(A)(B) =|
A ∩ B | < k
San Augustin Acolman has less than 1,000 residents.
k (A)(B) =|
A ∩ B | = k
Please donate 100 million to the School of Nursing.
between p and k the(A)(B) =
p < | A ∩ B | < k
The USA added ten states to its nation between 1800 and 1850.
the p/k (A)(B) =|
A ∩ B | = p · ( | A | /k )
Captain Blood has 20/20 vision.
the k% (A)(B) =|
A ∩ B | = k · ( | A | / 100)
The lending fund is always guaranteed 9% interest.
most (A)(B) =|
[Table]
∩ B | > | A \ B |
Most ZIP Codes cover roughly ten thousand addresses.
few
[Table]
=
[Table]
|
Only a few teenagers were still listening to Rock ’n’ Roll. each other
[Table]
for examples of such errors.
Unfortunately, most efforts have concentrated on subsets of quantiﬁer words and on English.
Contributions We analyze how quantiﬁers are represented in NLU benchmarks, and how their occurrence at test time contributes to errors by neural language models (LMs).
We derive a linguistically motivated 11-way categorization set for generalized quantiﬁers and look into their distribution in three steps: (a) monolingual NLI; (b) cross-lingual
NLI; (c) cross-lingual question answering.
We also propose GQNLI 1 , an adversarial generalized quantiﬁer NLI challenge dataset.
Our work shows that (i) generalized quantiﬁers are pervasive and cause overall performance drops in NLU benchmarks; (ii) the contribution of quantiﬁer words to system error varies across languages; and (iii) generalized quantiﬁers are particularly difﬁcult for LMs in interaction with negation and subsumption.
Background
Generalized quantiﬁers (GQs) are developed upon ﬁrst-order predicate logic, denoting relations between sets (Mostowski, 1957).
Given a universe
E , a quantiﬁer Q would be treated as a mapping Q E from the Cartesian product of powersets
P ( E ) × P ( E ) to the set { false,true } or, as a binary relation on subsets of E (Dvoˇrák and Holˇcapek, 2015).
GQs are generalizations of the ∀ , ∃ quantiﬁers from ﬁrst-order predicate logic (Mostowski, 1957; Lindström, 1966; Montague, 1973; Bach et al., 1995; Keenan and Paperno, 2012).
A generalized quantiﬁer is, abstractly, a relation between sets.
Generalized quantiﬁer theory, while developed by logicians, is used by formal linguists to analyze the meaning of quantiﬁer words in combination with referential expressions (Barwise and Cooper, 1981;
Higginbotham and May, 1981).
Most human languages contain ways of expressing generalized quantiﬁers, and their semantics exhibit striking similarities across languages (Matthewson, 2004; Fintel and Matthewson, 2008;
Steinert-Threlkeld, 2019).
At the same time, generalized quantiﬁers can be instantiated very differently across languages due to pragmatic considerations (Grice, 1989) or cognitive economy and costbeneﬁt optimisation in the exchange of information (Levinson et al., 2000; Steinert-Threlkeld, 2021; Uegaki, 2022).
Quantiﬁer words also exhibit syntactic differences, e.g., with some languages having specialized words to express quantity, while others rely on metaphorical usage of common nouns (Katsos et al., 2012).
In English, most is a determiner, but Spanish and French express the same concept through common nouns, la mayoría and la majorité .
The relative stability of the core semantics of quantiﬁers makes a cross-linguistic comparison possible, but the syntactic and pragmatic variation associated with the expression of generalized quantiﬁers poses a challenge for multilingual NLU.
We consult quantiﬁer taxonomy studies (Keenan and
Westerståhl, 1997; Peters and Westerståhl, 2006;
Szymanik and Thorne, 2015; Szymanik, 2016) and derive a categorization set for quantiﬁer analysis in NLU benchmarks.
In Table 2, we list the 11way quantiﬁer categorization set and their logical denotation based on set theory.
While other foci of formal linguistics have attracted the attention of NLP researchers—including
Quantiﬁer
English
Cross-lingual
M N L I _ m M N L I _ mm
S N L
I
AN L I _
R
AN L I _
R
AN L I _
R
XN L I some
[Table]
166 > [Table] [Table] [Table]
[Table]
between
[Table] [Table]
coreference (Ogrodniczuk et al., 2019, 2020), negation (Hossain et al., 2020; Hartmann et al., 2021), and consistency (Li et al., 2019; Ribeiro et al., 2019; Asai and Hajishirzi, 2020; Geva et al., 2022)—there has been little work on generalized quantiﬁers as a source of error in NLU, let alone in multilingual NLU.
It remains an open problem whether LMs represent the semantics of quantiﬁers words adequately, or if they provide a basis for resolving scopal ambiguities.
NLU Benchmarks
We conduct an error analysis focusing on the role of generalized quantiﬁers in two NLU tasks, Natural
Language Inference (NLI) and Question Answering (QA), which generally require understanding of quantiﬁers.
For each type of task, both monolingual and cross-lingual evaluation are conducted.
We focus on generalized quantiﬁers in the hypotheses in NLI examples—and on generalized quantiﬁers in the question ﬁelds in question answering.
To this end, we identify quantiﬁers by the lemma and the universal dependency relation (Nivre et al., 2020) of a quantiﬁer after preprocessing the sentences using Stanza (Qi et al., 2020).
Take the sentence “The Yiddish culture has survived for more than a thousand years.”
, we annotate it as “The/ det Yiddish/ amod culture/ nsubj have/ aux survive/ root for/ case more/ advmod than/ ﬁxed a/ det thousand/ nummod year/ obl ./ punct ”.
By matching the regex pattern of the quantiﬁer “more than k”, in this case “((more|great)\/advmod than\/(ﬁxed|case)|at\/case least\/nmod) .+\/nummod .+\/(nsubj|obj|obl)”
, we approximate the surface form of the type “more than k”.
Through matching quantiﬁer patterns, we are able to ﬁnd entries in which quantiﬁers are instantiated.
See Appendix A for the list of regex patterns we write to identify
GQs.
In Table 3 and Table 6, we present the statistics of the quantiﬁer distributions in NLI and QA tasks, respectively.
As can be seen, quantiﬁers are indeed widespread in NLU tasks, accounting for roughly 10% in NLI tasks and 5% in QA tasks.
We will further discuss the statistics and experiments in the following section.
Quantiﬁers in English NLI Benchmarks
NLI is commonly framed as a three-way classiﬁcation task with labels entailment , contradiction and neutral (Bowman et al., 2015a).
While SOTA models exhibit low error rates on NLI benchmarks, it is unclear when they succeed or fail in their underlying reasoning.
We are interested in whether generalized quantifers challenge modern NLI models.
In our error analysis, we initially focus on three
English NLI datasets, MultiNLI (MNLI; Williams et al., 2018), SNLI (Bowman et al., 2015a) and
ANLI (Nie et al., 2020) as testbeds.
Table 3 presents statistics of quantiﬁer distri-
Quantiﬁer
BERT
RoBERTa
M [Table]
M _ mm
S N L
I
A _ R
[Table]
A _ R
A _
Rw e i g .
M _ m
M _ mm
S N L
I
A _ R A _ R
[Table]
A _ R [Table] w e i
[Table]
bution in these datasets, where we observe that, across, about 10% of all hypotheses contain quantiﬁer words, indicating the pervasiveness of quantiﬁcation.
We also plot the frequency of quantiﬁers in NLI in Figure 1 and ﬁnd the quantiﬁer word distribution follows Zipf’s law (Zipf, 1949).
Note the top three most common quantiﬁers account for more than 90% of all.
Experiments and Results
In order to investigate whether NLU systems can solve quantiﬁers in NLI, we experiment with two pretrained LMs: BERT(Devlin et al., 2019) and RoBERTa 4 (Liu et al., 2019).
We use the codebase by Nie et al.
The training data combines SNLI, MNLI, FEVERNLI (Nie et al., 2019) and ANLI.
In Table 4, we report the test set performance on SNLI and ANLI, and the dev set performance on MLNI matched and mismatched sections.
We can observe that SOTA models suffer from performance drops across almost all quantiﬁcation phenomena in every task.
When it comes to performance over all quantiﬁers, the improvement from
RoBERTa to BERT (2.2%) is less prominent than that over full datasets (2.9%), suggesting RoBERTa is particularly challenged.
Taking a closer look at error by category, proportional quantiﬁers seem harder to solve than Aristotelian/counting quantiﬁers.
Except for k% , all proportional quantiﬁers— p/k , most , and few —are about 10% lower than the ﬁve counting quantiﬁers (except less than k ) with BERT; and about 5% lower with RoBERTa.
RoBERTa is not generally superior to BERT; e.g., for k% , BERT outperforms it by 22%.
We show a pairwise analysis of how
GQs affect performance when they appear in both the premises and hypotheses in the Appendix B.
Generally, our results attest to the difﬁculty of resolving GQs in NLI benchmarks.
Quantiﬁers in Cross-lingual NLU Benchmarks
Quantiﬁers are acquired in similar orders across languages (Katsos et al., 2016), although languages express quantiﬁers in different ways.
For example, there are eight different universal quantiﬁers with different level of distributivity in Malagasy (Matthewson, 2008).
This poses challenges to training multilingual LMs and transfer learning.
We are interested in whether quantiﬁers are universally and evenly challenging for all languages.
Quantiﬁers in Cross-lingual NLI
We choose
XNLI (Conneau et al., 2018), a manual translation of the development and test set of MNLI into 15 languages, for this multilingual error analysis.
We should clarify that for XNLI, the authors annotate entailment labels for the English data only and apply them to the other languages.
We do not assume label changes due to translation in this study, but it is worth investigate in the future.
We choose ﬁve languages belonging to different language families, namely Arabic, Chinese, German, Spanish and Vietnamese as targets.
The last column in Table 3 shows the numbers of quantiﬁers in XNLI.
The distribution rate is 10%.
Note that the universal [Table] quantiﬁer is the most common quantiﬁer in XNLI.
We ﬁne-tune mBERT 5 (Devlin et al., 2019) and
XLM 6 (Lample and Conneau, 2019) on the MNLI training set and evaluate them on XNLI.
We report the results in Table 5.
We ﬁnd that performance varies across languages.
For Chinese and Vietnamese, we see signiﬁcant drops in performance for examples with GQs, whereas for Arabic and
German, we see improvements.
The results per quantiﬁer are more homogeneous, however.
Similar to our results for English, we can see that the lowest accuracies in XNLI are with proportional quantiﬁers, such as most and few .
But the gap in non-English languages is wider for these two categories, especially for Chinese, the difference reaches 30%.
Other hard quantiﬁers include all , > k , < k , and each other .
Quantiﬁers in Cross-lingual QA
Cross-lingual question answering (XQA) is another important
NLU task that evaluates the cross-lingual transferability of LMs.
We evaluate the effect of quantiﬁers on system errors across two XQA datasets, namely
XQuAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020).
As demonstrated in Figure 1, quantiﬁer word distributions in XQA tasks also follow
Zipf’s law, as in NLI tasks, but k is more frequent (perhaps because of a traditional emphasis on numerical reasoning), and we see less variance across languages.
This is probably because question answering is targeting quantiﬁcation less directly.
To evaluate cross-lingual QA performance on GQs, we ﬁne-tune mBERT and XLM-R 7 (Conneau et al., [Table] [Table] [Table]
[Table] [Table] 2020) using Hu et al. (2020)’s architecture.
We present results for mBERT in Table 7; for XLM-R results, please refer to Appendix D.
Just as with XNLI, LMs suffer from performance drops across all languages for almost all GQ phenomena with signiﬁcant, cross-lingual variation.
The most distinguished is that Exact Match (EM) suffers from a greater deterioration than F1 scores for all languages.
For example, the weighted EM difference for mBERT on MLQA is 2.9% while the weighted F1 is 1%.
As one example in Table 1, we observe that the plausible answers selected by models, while being incorrect, result in a sharper decrease of EMs comparing to F1s.
Questions containing GQs also tend to have less verbal answers comparing to those without GQs, and therefore require higher precision.
Regarding cross-lingual comparisons, Chinese and Arabic are the two languages that do not have Quantiﬁer [Table] lower performance over GQs compared to the performance over the complete dataset.
Despite the overall trends, subtle differences from XNLI performance still exist.
For example, XLM-R is worse than mBERT on quantiﬁer reasoning on XQuAD
Chinese, especially at proportional quantiﬁers, but this is not the case on MLQA Chinese.
GQNLI
We have seen how quantiﬁers present challenges to NLI and QA models.
Using an approach similar to ANLI (Nie et al., 2020) and DynaBench (Kiela et al., 2021), we use model difﬁculty (RoBERTa’s) as a heuristic to select hard examples for a challenge dataset that can hopefully be used to evaluate any future progress on this.
We propose GQNLI, a generalized quantiﬁer NLI challenge dataset, consisting of 30 premises and 300 hypotheses.
The average sentence lengths of hypothesis and premises are 15.97 and 7.35, respectively.
Both numbers are comparable to those of MNLI, but lower than
ANLI’s
(Williams et al., 2020).
It should be noted that GQNLI is designed for evaluating future models; obviously not for benchmarking RoBERTa.
Dataset Creation
Firstly, we manually create 100 premise-hypothesis pairs, in which various types of GQs appear.
For each premise and hypothesis, the number of GQs varies from one to three.
To choose the premises, we randomly sampled 100 premises with GQs from SNLI and ANLI test sets, respectively, and selected 10 premises in total, that we consider are semantically adequate for adding GQs and making simple hypotheses.
To construct the hypotheses, we rely on
RoBERTa ﬁne-tuned on MNLI and manually select examples about which the model is unsure or incorrect.
To focus on GQs, we keep the challenge examples otherwise simple (Ribeiro et al., 2020), and avoid lexical variations in the hypotheses.
Hard examples were found to be characterized by (i) mixing generalized quantiﬁers with other logical operators, such as subsumption or negation, and (ii) combining multiple different generalized quantiﬁers.
We discuss these observations in Section 7.
Two of the authors annotated the examples.
The inter-annotator agreement (Fleiss’ kappa) was 0.895, substantially higher than ANLI’s (0.672– 0.740).
It is worth noting that the level of semantic or pragmatic interpretation difference of GQs is reﬂected in the measurement.
We augmented the examples by substituting non-quantiﬁer words (e.g., replacing “dogs” with “cats”) while keeping the labels, to exclude the effect of speciﬁc lexical items.
The resulting labels are uniformly distributed.
Table 8 presents GQNLI statistics.
Since the dataset is curated to probe the ability to reason with quantiﬁers, the distribution of generalized quantiﬁers does not follow Zipf’s law; see §4.
A list of GQNLI examples per category is shown in Appendix E.
Experiments and Results
We evaluate seven types of models on GQNLI, ﬁne-tuned with different combinations of NLI datasets.
As data creation only relied on RoBERTa and MNLI, nothing prevents that models with different architectures and training data will perform well.
They do not, however.
The results are shown in Table 8.
We see that all models have great difﬁculty with
GQNLI.
With more training data, models improve, but the best performance is 42.7%, less thanpoints above chance level.
In general, the counting quantiﬁers, especially the existential and universal quantiﬁers, are easier than proportional quantiﬁers.
Particularly, most models struggle with less than k and few .
This is in some contrast with the NLU tasks studied above, where these quantiﬁers were among the easiest.
We also observe unstable GQ reasoning ability in simple word substitution cases.
For instance, it happens for DeBERTa ﬁne-tuned with M, F, Ling,
DocNLI that it predicted correctly the contradiction [Table] relation between “There are six children standing on top of a yellow mountain.
Two thirds wear red tops and one third wear green.”
and “Between 80% and 90% children do not wear red tops.”, but incorrectly when “red” is substituted with “beige” and “green” with “cyan”.
We are yet to study what kind of cues lead to the instability.
Our experiments suggest a lack of testing proportionality reasoning and robustness in existing benchmarks.
Discussion Negation
The interaction between negation words and quantiﬁers increases semantic complexity (Partee, 1970; Horn, 2010).
We investigate whether this holds for NLI tasks, using negation cue detection to ﬁnd all cases where a negation word and a quantiﬁer appear in the hypotheses.
We break down the performances on negation of the seven models in Appendix F. As indicated,
LMs overall have worse results for negation cases comparing to the entire dataset.
We can see a majority of the models even predicted opposite labels for some GQ categories, with 0% accuracy.
DeBERTav3 is no longer the best model, replaced by BERT and BART.
The improvement by training with more data becomes unstable for reasoning over GQs with negation.
For a cross-lingual investigation of the interaction of GQs and negation, we ﬁnd that in XNLI, the number of cases combining both phenomena is insufﬁcient: we identiﬁed four such cases, involving only the quantiﬁers “all” and “more than.”
For
English, mBERT predicted two cases successfully.
For Chinese, German, Vietnamese and Arabic, one is correct.
For Spanish, all are wrongly predicted.
It is evident that NLU models suffer from reasoning difﬁculties in certain cases when negation interacts with GQs, especially in cross-lingual evaluation.
In future work, we are interested in expanding GQNLI to more instances and more languages to facilitate qualitative investigations.
Subsumption
In generalized term subsumption languages (TSLs; Yen, 1991; Ali and Shapiro, 1993), a term a subsumes another term b if and only if the extension of a is a superset of the extension of b .
Rather than surface number comparison, subsumption reasoning requires knowledge of the relations between supersets and subsets.
For example, to decide whether “There are six dogs.
Three brown dogs, a black dog and a white dog run along the green grass” entails “One dog sits”, LMs should be aware that “six dogs” is a superset of the extension of the “brown dogs”, “black dog” and “white dog”.
Another example in GQNLI is to infer whether “There are twelve singers on a stage, less than half from Argentina and one from Cape Verde” entails “Several singers do not come from Chile”.
We annotate 63 cases out of the ﬁrst 100 in
GQNLI requiring subsumption reasoning.
We show the statistics and results regarding subsumption in Appendix G. It can be seen that more training data leads to higher accuracies.
Especially,
DeBERTa ﬁne-tuned with DocNLI, which uniﬁes the two classes “neutral” and “contradict” into a new class “not entail”, has a signiﬁcant improvement on subsumption cases with neutral label.
The training bias give an advantage to the model on the subsumption subset, half cases of which are labelled neutral.
But such bias has a negative effect on non-subsumption cases; the accuracy drops by 16.2% comparing to the model without training with DocNLI.
It is worth investigating whether
DocNLI is truly helping subsumption reasoning in future work.
Subsumption is a key concept in the study of knowledge representation (Woods, 1991), but is neglected in current NLP research.
The fact that LMs struggle to perform subsumption reasoning asserts the necessity to explicit tackle the problem.
Related Work
We examine the sensitivity of NLU models to generalized quantiﬁers.
These models are designed to induce correlations from large volumes of data, not to reason symbolically with logical quantiﬁers.
Such models have, nevertheless, been probed for logical knowledge.
Mul and Zuidema (2019), for example, show neural networks encode fragments of ﬁrst-order logic and exhibit zero-shot generalization ability.
Evans et al.
(2018) present a neural architecture that improves performance on propositional logical inference.
Bowman et al.
(2015b) also suggest neural networks learn semantic representations for logical inference in natural languages.
However, on the same task, Veldhoen and Zuidema (2017) ﬁnd neural networks fail to do so on a more stringent test.
Geiger et al.
(2019) also show that neural networks fail to exhibit robust logical inference.
Srivastava et al.
(2018) use semantic parsers to encode quantiﬁers and improve zero-shot learning in classiﬁcation tasks.
Haruta et al.
(2020) present a system that computes logical inference over GQs and see improvements on two specialized datasets,
FraCaS (Cooper et al., 1994) and MED (Yanaka et al., 2019).
None of these papers explicitly discussed generalized quantiﬁers, and all were limited to studying the ability of neural networks to capture the logical semantics of English.
Many studies have instead focused on LMs’ ability to capture negation (Gururangan et al., 2018;
Naik et al., 2018; Hossain et al., 2020; Ettinger, 2020; Hartmann et al., 2021) or coreference (Ye et al., 2020; Varkel and Globerson, 2020; Abdou et al., 2020).
Others have focused on LMs’ ability to reason with numbers (Johnson et al., 2020).
DROP (Dua et al., 2019), for example, is a question answering dataset designed speciﬁcally to probe
LMs’ ability to count, add and subtract for answering factoid questions.
Models have also been tailored for numerical reasoning (Geva et al., 2020;
Zhang et al., 2020).
Cobbe et al.
(2021) proposes to use a veriﬁcation task during pretraining of LMs to improve their ability to solve math word problems.
Others have studied monotonicity inference (Hu et al., 2019; Yanaka et al., 2019, 2020), and
Fang and Lou (2021) recently focused on the two quantiﬁer words part and whole in an error analysis for named entity recognition.
Many NLU benchmarks contain quantiﬁer words, but their inﬂuence on performance has not been studied systematically.
One exception to this is that generalized quantiﬁers have been used to generate adversarial examples in the context of numerical reasoning (Naik et al., 2018; Nie et al., 2020).
TaxiNLI (Joshi et al., 2020), which categorizes 15 types of reasoning abilities, is a dataset drawn from MNLI.
In their taxonomy, the Quantiﬁer category only refers to universal and existential quantiﬁers, not to generalized quantiﬁers, and ditto for Kim et al.
All of the above focused on English, but in an extension to TaxiNLI, K et al.
(2021) incorporated quantiﬁers into the Logic class and found a large cross-lingual transfer gap on LMs.
Conclusion
Quantiﬁers lie in the intersection of logic, linguistics and NLP research.
It is essential for NLU systems to learn quantiﬁer reasoning.
We examined generalized quantiﬁers in multilingual NLU tasks with regards to their expressiveness and logical reasoning requirement.
Our survey and experiments indicate quantiﬁers are neglected to a degree and cause signiﬁcant performance drops for neural LMs.
To better understand LMs’ reasoning abilities, we release GQNLI, a novel generalized quantiﬁer NLI challenge dataset.
With the pervasiveness of generalized quantiﬁers, we stress that more efforts are necessary to investigate: (1) when and why models systematically fail when quantiﬁers interact with other operators; (2) how to improve cross-lingual transferability of quantiﬁers; (3) how we can exploit the theoretical results about generalized quantiﬁers from logic and linguistic studies, so as to improve the logical inference ability of neural LMs.
Acknowledgements
We would like to thank Miryam de Lhoneux, Constanza Fierro, Desmond Elliott and the anonymous reviewers for their valuable feedback.
Appendices A Regular Expressions for Generalized Quantiﬁers
Table 9 lists the regex we use to parse generalized quntiﬁers in sentences augmented with universal dependency tags.
The approach does not ﬁnd all the generalized quantiﬁers exhuastively but rather approximates the common distributions.
B Pairwise Observation
While the analysis in Section 4 is based on quantiﬁers in hypotheses, next we consider the interaction of quantiﬁers in hypotheses and quantiﬁers in premises.
To this end, we calculate the difference between overall performance and performance for premise-hypothesis pairs of GQs.
In Figure 2, we visualize the results as heatmaps (see Table 10 for exact numbers of occurences and accuracies).
Surprisingly, whenever quantiﬁers appear in both the premise and the hypothesis, LMs largely fail to predict the entailment.
Percentage quantiﬁers, supposed to be semantically more complex than counting quantiﬁers, are not de facto harder in NLI.
We studied all 27 cases of percentage quantiﬁers in the English NLI datasets, and found that in most cases, percentage quantiﬁers occurrences are identical across premises and hypotheses, i.e., triggering little or no inference.
The other two proportional quantiﬁers, most and few , are hard for
LMs to resolve, e.g., in some quantiﬁer pairs, models yield 0% accuracy.
Although each other is supposed to be hardest to resolve due to the complex semantics of reciprocals (Szymanik and Thorne, 2015), it is not reﬂected in NLI tasks as such.
The reason is similar to percentage quantiﬁers, while annotators intend to alter counting quantiﬁers when writing hypotheses, reciprocality is seldomly considered a linguistic ability that needs testing for NLU systems.
And the annotation for Ramsey quantiﬁer is simply a knockoff, making reciprocal relation identiﬁcation unwarranted through shallow correlations.
C Fine-grained NLI Analysis D XQA Result: mBERT and XLM-R
Table 11 compares the results of mBERT and XLM-R on two XQA tasks, XQuAD and MLQA.
E GQNLI Examples
Table 12 list one example per category in GQNLI.
F GQNLI Negation Cases
We present the results of seven models’ performance on cases with negation cues in GQNLI in Table 13.
G GQNLI Subsumption Cases
See Table 14 for models ’performance on cases requiring subsumption reasoning in GQNLI.
We also break down subsumption results by entailment labels into two categories: neutral and non-neutral.
H GQNLI Experiment Details
We reused the ﬁne-tuned BERT and RobERTa in Section 4.
The other ﬁne-tuned LMs are from Hugging
Face.
We list the models and thier links in Table 15.
[Table]
MNLI_m_dev MNLI_mm_dev
SNLI_test
ANLI_R1_test ANLI_R2_test
ANLI_R3_test
Total #occurrence %Acc #occurrence %Acc #occurrence %
Acc #occurrence %Acc #occurrence %Acc #occurrence %Acc #occurrence %Acc #correctpred some some 93.
92 all 87.
14 >k<kk            85.
58 betweenmostfew    1 p/kk%eachother "unmatched" 79.
281 all some          26 all 82.
132 >k    0 <kk 85.
68 between      1 mostfew      6 p/k
k%eachother         1 "unmatched" 86.
303 >k some        2 all        1 >k        4 <kk       66.
16 between    1 mostfew    1 p/kk%eachother "unmatched"       66.
41 <k someall>k<kk 33.
6 betweenmostfew    0 p/kk%
[Figure]
"unmatched"     33.
[Figure]
some   78.
42 all 83.
45 >k 66.
2 <k      3 k 84.
810 between
[Figure] [Figure]
/k
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
Quant.
[Table] [Table]
[Table]
35.3 S,M,F,ANLI 0.
ALBERT
S,M,F,ANLI 44.
DeBERTa-
[Table]
MNLI 33.
23.5 M,
[Table]
,ANLI
[Table] [Table]
[Table] [Table]
[Table] [Table]
,Ling,DocNLI
[Table]
[Table]
Model
Training Data
Model’s
[Table]
RoBERTa
MNLI https:
[Table] [Table]
BART
MNLI https://huggingface.co/facebook/bart
[Table]
large
[Table]
mnli
[Table]