GQG : Generalized Quantifier Generalization - A Dataset for Evaluating Quantifier Semantics Understanding in Language Models (2023)

Abstract: We present a new dataset consisting of various quantifier expressions to evaluate the generalization abilities of language models. The dataset contains 18,360 prompts encompassing diverse quantifiers, forming the basis of a new framework for assessing semantic understanding in this domain. We test the effectiveness of our dataset using Pythia models, ranging from 410 million to 6.9 billion, showing that quantifier-based tasks can be challenging for current language models. We make our code and data publicly available, such that the dataset can be easily extended or updated based on different evaluation needs.

## introduction in recent years, the natural language processing (nlp) community has witnessed the rise of increasingly larger and more sophisticated language models (lms) capable of generating coherent texts over extended passages. however, the ability of these models to understand and generate human language that aligns with the underlying semantics remains a topic of debate @xcite . neural language models may rely on heuristics learned from the training data to generate seemingly coherent texts, but fail to generalize to scenarios that are more complex and cannot be solved by simple heuristics @xcite . whether language models can acquire meaning when trained only on text is also a topic of ongoing debate. @xcite have argued, through thought experiments, that lms cannot learn semantics through texts since they lack access to explicit representations of the external world. we hope to contribute to this ongoing dicussion by releasing this dataset on quantifiers, which will enable more research on this direction. in this paper, we introduce a new framework that uses formal semantics to test the generalization capabilities of language models, by developing a dataset that assesses lms' understanding of quantifier semantics. we ask the question -to what extent do language models capture the semantics of quantifiers? quantifiers are well-suited for evaluating language model generalization because compared to other linguistics objects, their meanings are more abstract, and can be fully specified in theoretical terms that do not require grounding. common examples of quantifiers include some, all, a few, many, etc. to construct the dataset, we use a deterministic algorithm to generate prompts and gold labels automatically, covering 15 different quantifiers in the english language. in each prompt, we ask the lm to give a truth value judgment (true or false) to a statement about a given quantifier in a constructed scenario. this dataset does not exhaust common quantifiers in english, but is designed to enable more research on evaluating lms' understanding of semantic objects. more specifically, the dataset will allow further investigation into different axes of generalization in this domain, i.e. for the same quantifier, does using different nouns in the prompts affect lm's acquisition of the semantics of the quantifier? and how do different number ranges or different word orders affect lm's understanding of the same quantifier? the dataset presented in this paper provides a valuable framework for researchers to study these types of questions, grounded by rich formal semantics literature on quantifiers. as demonstrated in section 3, the data generation pipeline can be easily extended to study new quantifiers, nouns, and number ranges. given initial native speaker annotated prompts and quantifiers, the data can also be easily extended to different languages. this paper is structured as follows. in section 2, we discuss the literature on quantifiers in formal semantics and demonstrate how they can be useful for evaluating lms. in section 3, we discuss how the dataset is constructed and how lms are evaluated. we discuss future work directions in section 4 and remark on limitations at the end. ## conclusion this paper presents a new dataset to evaluate language models' understanding of quantifier semantics. the dataset can be easily extended to different languages and quantifiers, enabling more research on assessing language models' understanding of semantic objects and investigation into different axes of generalization. the poor performance of the pythia models during evaluation shows the dataset can be challenging for neural language models, but more research is required to understand how instruction-tuned lms (and more sophisticated prompt-engineering) will perform on this dataset. we also note the limitations of our study, particularly on evaluation methods, and hope that this dataset will be a basis well-grounded in theoretical literature for more research on lms and semantics. future work includes developing datasets in more diverse prompt formats, analyzing how lms' performance can differ based on different types of quantifiers or linguistic objects, and investigating how finetuning can affect model performance.