Not all quantifiers are equal: Probing Transformer-based language models’ understanding of generalised quantifiers (2023)

Abstract: How do different generalised quantifiers affect the behaviour of transformer-based language models (TLMs)? The recent popularity of TLMs and the central role generalised quantifiers have traditionally played in linguistics and logic bring this question into particular focus. The current research investigating this subject has not utilised a task defined purely in a logical sense, and thus, has not captured the underlying logical significance of generalised quantifiers. Consequently, they have not answered the aforementioned question faithfully or adequately. Therefore, we investigate how different generalised quantifiers affect TLMs by employing a textual entailment problem defined in a purely logical sense, namely, model-checking with natural language. Our approach permits the automatic construction of datasets with respect to which we can assess the ability of TLMs to learn the meanings of generalised quantifiers. Our investigation reveals that TLMs generally can comprehend the logical semantics of the most common generalised quantifiers, but that distinct quantifiers influence TLMs in varying ways.

## introduction generalised quantifiers have been a topic of much interest for more than a century in logic and linguistics @xcite @xcite . by capturing the interplay between quantity and cardinality, they provide a useful lens through which to understand human language and cognition @xcite . since transformer-based language models (tlms) strive to stimulate humanlike language understanding @xcite @xcite @xcite , it is essential to determine the extent to which they can comprehend generalised quantifiers. assessing the depth of understanding that tlms possess for any given concept is best achieved by evaluating their proficiency in applying it. in the case of generalised quantifiers, the most suitable evaluation task is textual entailment. this is particularly relevant because altering quantifiers can fundamentally change the logical inferences derived from a given text, reinforcing the integral role that quantifiers occupy within the scope of the textual entailment task. when discussing entailment, it is vital to acknowledge two distinct strands of research in the literature. the first strand incorporates background knowledge and common sense into entailment, imbuing it with a probabilistic character @xcite @xcite . the second strand examines textual entailment in a purely logical sense, eliminating the influence of background knowledge and common sense @xcite @xcite . while the first form of entailment proves beneficial for a multitude of practical applications, it is not ideal in an investigation centred on the impact of linguistic properties with logical significance, such as generalised quantifiers and negation. the empirical evaluation of linguistic constructs under this kind of entailment gets compromised due to its intricate association with other concepts. consequently, it is challenging to differentiate the performance variation due to linguistic properties from those attributable to concepts like common sense and background knowledge. however, prior literature has only investigated generalised quantifiers in the context of entailment that incorporates background knowledge and common sense @xcite and naturally suffers from the same predicament. the second strand of textual entailment by defining entailment in a purely logical sense circumvents the aforementioned shortcoming. consequently, it offers a conducive environment for conducting evaluations figure 1 : an instance of the model-checking with natural language problem, the sentence "at least 3 musicians are guitarists" is t rue according to the structure since the set musicians @xmath0, solomon, ava, aria} are also guitarists and |x| ≥ 3. however, the sentence " all bee-keepers are scientists" are f alse as the set of bee-keepers {t alia, solomon} are not scientists centred around linguistic constructs. the logical problem that is most suited to study the influence of language constructs of logical significance is that of model-checking: given a formula ϕ and a structure a, determine whether ϕ is true in a (a |= ϕ). in the context of natural language, we are interested in a variant of the model-checking problem where the structure and the formula are translated into natural language. an instance of model-checking with natural language problem is depicted in figure 1 . from a complexity-theoretic point-of-view, modelchecking in most formal languages is, comparatively speaking, straightforward. indeed the modelchecking problem with a fixed number of free variables and a finite structure is in ptime. this is in contrast to other logical problems, such as satisfiability, whose problems for various fragments of logic can belong to different computational complexity classes @xcite ). yet, solving instances of the model-checking problem with natural language requires a comprehensive understanding of the logical semantics of the expressions involved. thus, it provides an ideal test environment to faithfully evaluate the extent to which generalised quantifiers affect transformer-based language models. in this study, we embark on an in-depth investigation into tlms' understanding of generalised quantifiers utilising the model checking problem, juxtaposing this with cognitive science research on quantifier verification tasks @xcite . a critical part of our exploration involves the evaluation of pre-trained models prior to any fine-tuning. thus, allowing us to discern whether any differences identified are intrinsic to the models themselves or introduced through the process of finetuning. additionally, we consider the complexities arising from the integration of boolean con-junctions and negation with generalised quantifiers. this aspect of our study sheds light on the intricate dynamics that exist between these linguistic elements and the challenges they pose to tlms. this comprehensive analysis paves the way for a more nuanced understanding of how tlms handle intricate linguistic constructs such as generalised quantifiers. the key contributions of the present research can be summarised as follows: (1) to the best of our knowledge, this study represents the first exploration into the effects of generalised quantifiers within a logical entailment context; (2) we analyse the effect on tlms when quantifiers are paired with diverse logical constructs like negation and boolean conjunctions; (3) we compare and contrast the behaviour of tlms with quantifiers with that of quantifier verification experiments done with human beings; and (4) we delve into how well tlms comprehend generalised quantifiers in a zero-shot context employing prompt engineering approaches such as chain-of-thought-prompting @xcite and provide comparisons between pre-trained and fine-tuned models. ## related work our work follows the literature on probing how different linguistic properties affect the behaviour of neural approaches such as transformer-based language models @xcite @xcite @xcite . specifically, our investigation is closely related to the literature whose linguistic properties of interest are generalised quantifiers @xcite . our exploration differentiates from prior research in two key ways. first, we explore generalised quantifiers employing a task that is defined purely in a logical sense. thus, we provide a more faithful investigation into how tlms comprehend ## results and discussion the ability of transformer-based language models to solve instances of the model-checking problem is differentially influenced by various generalised quantifiers. as demonstrated in as expressed by their precision and recall values, tlms often predict t rue for quantifiers such as "all" and "less than k", but often predict f alse with respect to quantifiers like "some" and "more than k". we attribute this to be an overcorrection introduced during fine-tuning. consider the sentence with the quantifier "less than k": "less than k artists are engineers". this sentence is more likely to be f alse in the context of the real world for k values we consider in this study (8 ≤ @xmath1. this proposition remains true even when negations are introduced to the sentences. thus, we speculate that tlms overcorrect during fine-tuning and predict t rue or f alse accordingly. tlms show evidence of learning to understand the logical semantics of generalised quantifiers. as illustrated in figure 3 , when tested with a dataset containing higher k values than that of the train set, the accuracy of tlms only decreases slightly for all numerical quantifiers. therefore, we posit that tlms possess the capacity to learn the logical semantics associated with generalised quantifiers. our conclusions regarding generalisation bear resemblances to prior work conducted on model-checking with natural language @xcite . their research also supports the premise that tlms are capable of comprehending the logical semantics of natural language. additionally, we highlight the contrast between the demonstrated ability of tlms to generalise in the context of model-checking problems, and their apparent lack of such generalisation when solving satisfiability problems @xcite . we hypothesise that this distinction is due to the different complexity levels associated with these two types of problems and the necessity to understand complex inference rules when solving satisfiability problems. the boolean conjunctions have a significant effect, while negation has much less effect on fine-tuned tlms when coupled with generalised quantifiers. as demonstrated in the number of parameters, training process and type of prompting can influence the tlms' performance when solving modelchecking problem instances in zero-shot settings. the performance for flan-t5 models exhibits the power law relationship with the number of parameters, as illustrated in figure 4 (a ). this empirical finding is consistent with prior research analysing language models' performance variation with factors such as the number of parameters, dataset size and computational resources @xcite . however, upon breaking down the performance metrics based on the quantifiers, the resulting graph (figure 4 (b) ) is observed to be less uniform compared to the representation of overall performance.we attribute this behaviour to the inherent probabilistic aspect of the predictions formulated by tlms since language models are trained to find the most probable next word given a set of words. this probabilistic nature of language models can lead to inaccurate predictions, especially in a logical context. the flan-t5 model with fewer parameters outperformed the chatgpt and llama models in a zero-shot setting, as depicted in accuracy values for tlms in zero-shot settings vary drastically with different quantifiers. as depicted in ## conclusion we investigated how generalised quantifiers affect the behaviour of transformer-based language mod-els by employing the problem of model-checking. we found that different generalised quantifiers affect tlms in varying ways when solving modelchecking problems in both fine-tuned and zero-shot settings. based on empirical findings on generalisation, we posited tlms can learn to understand the logical semantics of generalised quantifiers. moreover, our experimental setup in the zero-shot setting demonstrated that a multitude of factors, such as the training process, size of the models and type of prompts, can affect the ability of tlms to solve a simple entailment task. thus, a compelling avenue for future research is to probe how varying factors affect transformer-based language models when solving a more complex entailment task, like determining satisfiability.