Many speakers, many worlds: Interannotator variations in the quantification of feature norms (2016)

Abstract: Quantification (see e.g. Peters and Westerst ̊ahl, 2006) is probably one of the most extensively studied phenomena in formal semantics. But because of the specific representation of meaning assumed by modeltheoretic semantics (one where a true model of the world is a priori available), research in the area has primarily focused on one question: what is the relation of a quantifier to the truth value of a sentence? In contrast, relatively little has been said about the way the underlying model comes about, and its relation to individual speakers’ conceptual knowledge. In this paper, we make a first step in investigating how native speakers of English model relations between non-grounded sets, by observing how they quantify simple statements. We first give some motivation for our task, from both a theoretical linguistic and computational semantic point of view (§2). We then describe our annotation setup (§3) and follow on with an analysis of the produced dataset, conducting a quantitative evaluation which includes inter-annotator agreement for different classes of predicates (§4). We observe that there is significant agreement between speakers but also noticeable variations. We posit that in settheoretic terms, there are as many worlds as there are speakers (§5), but the overwhelming use of underspecified quantification in ordinary language covers up the individual differences that might otherwise be observed.

## introduction quantification (see e.g. @xcite is probably one of the most extensively studied phenomena in formal semantics. but because of the specific representation of meaning assumed by modeltheoretic semantics (one where a true model of the world is a priori available), research in the area has primarily focused on one question: given a model, what does it mean for a speaker to utter a statement of the form qx[p (x)], where q is a natural language quantifier such as no, few, some, many, most, all, at least 3... (or even a null quantifier ∅)? what is the relation of a quantifier to the truth value of a sentence? in contrast, relatively little has been said about the way the underlying model comes about, and its relation to individual speakers' conceptual knowledge. consider for instance the simple model in fig. 1 . given this state-ofaffairs, where the set of cats and the set of black things overlap, the sentences some cats are black and at least one cat is black can be said to be logically true, while all cats are black is logically false. expanding on this purely logical interpretation, researchers (e.g. @xcite , @xcite and @xcite ) have further shown how quantifier preferences depend on a variety of factors ranging from the cardinality of the set involved to the 'question under discussion' in the discourse. that is, depending on the number of cats involved in the state-of-affairs under consideration, humans may prefer the description two cats are black to some cats are black, despite both sentences being true. what semantics and pragmatics have so far failed to provide is an account of models themselves: how is it that in the first place, a speaker might model the world in a way that the set of cats and the set of black things overlap? in grounded situations, perceptual input arguably constrains the beliefs of the speaker about the observed state-of-affairs. if we stand in a room with five cats, two of which are black, it seems intuitive to infer that our model of the state-of-affairs includes a set of five cats, and the overlap between the set cats and the set of black things has cardinality 2. it is much less clear how the model comes about in non-grounded situations (which constitute the majority of an adult's utterances). having encountered only a small proportion of all cats in the world, and perhaps no unicorn, why is it that we confidently utter sentences such as most cats have four legs or all unicorns look like horses, pointing at an underlying model where indeed, most cats have four legs and all unicorns look like horses? in this paper, we make a first step in investigating how native speakers of english model relations between non-grounded sets, by observing how they quantify simple statements. for instance, we ask how an individual might quantify bats are blind (some? all?), hoping to gain a representation of their underlying model of bats (and blindness). note that explicit quantification is an unusual phenomenon in that it cannot directly be studied from corpora, being relatively rare in naturally occurring text: underspecified constructions like bare plurals and (in)definites starting with a/the are much more frequent than the equivalent some/most/all -quantified nps. 1 herbelot and copestake (2011) estimate that around 7% of noun phrases are explicitly quantified. this means that we are unlikely to find out from a corpus study, for instance, that all cats are mammals: the generic cats are mammals is the standard way to express the predication. our main contribution is to remedy this lack of data by releasing an annotation layer for a well-known set of feature norms (the 'mcrae norms @xcite consisting of over 7,000 concept-feature pairs, labelled by 3 native speakers of english. for each pair in the norms, coders have provided a natural language quantifier, resulting in new statements such as all tricycles have three wheels or few apes are blind to extend the data for pertinent computational tasks. this paper is structured as follows. we first give some motivation for our task, from both a theoretical linguistic and computational semantic point of view ( §2). we then describe our annotation setup ( §3) and follow on with an analysis of the produced dataset, conducting a quantitative evaluation which includes inter-annotator agreement for different classes of predicates ( §4). we observe that there is significant agreement between speakers but also noticeable variations. we posit that in settheoretic terms, there are as many worlds as there are speakers ( §5), but the overwhelming use of underspecified quantification in ordinary language covers up the individual differences that might otherwise be observed. ## motivation although quantification is rarely explicit in naturally occurring text, it is intrinsic to most utterances. any reference act picks out some set of individuals x in a world and, by associating a predicate p with it, builds a model which is interpretable in terms of a quantified relation: some, most, all individuals in x do p . this process happens intuitively so that, when someone utters mice are in the cellar, we don't assume that all mice in the world have gathered in the speaker's cellar, only some of them. in this paper, we will regard this process as generating a natural language quantifier with a set-theoretic interpretation. note that we are not making any claims about whether speakers have set-theoretic models 'in their head'. we only wish to argue that their interpretation of a statement involves building some kind of model of the state-of-affairs described by the speaker, and that this model gives them some (rough) information about the proportion of instances of a may 2016 concept involved in the situation. we use set theory as a shorthand to express the end-product of this intuitive process. being able to generate a quantifier for a given subject-predicate pair is a prerequisite for many lexical semantics and inference tasks. it is arguably part of the concept cat that some cat instances are tabbies and all are mammals, and knowing this allows us to infer that given a cat taken at random, that cat is most definitely a mammal and may be a tabby. a lot of work in computational semantics has focused on extracting specific set relations from text, in particular those involving set identity or set inclusion (e.g. synonymy, hyponymy: @xcite through to @xcite @xcite . but much less research has looked into the problem of generally inducing a set-theoretic model from corpus data, including set relations which can range from small to consequent -but not necessarily universal -overlap @xcite . the dataset we release with this paper has two motivations. the first is to gather linguistic data to help us understand, from a theoretical point of view, to which extent humans agree on a single model of the world. the second is to provide a large gold standard of quantified predications which can be used as training/test data in computational tasks such as entailment, inference, concept modelling, etc. ## quantifying the mcrae norms the mcrae norms (2005) are a set of feature norms elicited from 725 human participants for 541 concepts. the annotators were asked to provide features for each concept, covering physical, functional and other properties. the result is a set of 7257 concept-feature pairs such as airplane used-for-passengers or bear is-brown. we conducted the quantification of the mcrae data in the following way. we recruited three native english speakers (one southeast-asian and two american speakers, henceforth denoted as a1 and a2/a3 ), all computer science students. for each concept-feature pair (c, f ) in the norms, they were asked to provide a label expressing the ratio of instances of c having the feature f . the allowable classes were no, few, some, most, all. our investigation required minimising the pragmatic interferences observed in quantifier selection. note that the way a speaker quantifies concept feature ape is muscular all is wooly most lives on coasts some is blind few tricycle has 3 wheels all used by children most is small some used for transportation few a bike no the first aspect concerns matters of lexical semantics and, broadlyspeaking, world knowledge. does the speaker understand blindness as complete lack of sight or (just) poor sight? what do they know about bats? the second aspect relates to the pragmatics of quantifier semantics: we straightforwardly observe, for example, that all has a much wider meaning than ∀ suggests (as in all my friends say i'm right, which typically does not imply universal quantification). these two aspects had to be clearly separated in our study, as we focused on the first question, i.e. what people believe about the actual state of the world (regardless of their way of expressing it), and how this relates to their conceptual and lexical knowledge. in order to reduce such interferences, we gave clear instructions to the coders on how to use the annotation labels (reproduced in the appendix). we defined the label all as a 'true universal' which either a) doesn't allow exceptions (as in the pair cat is-mammal) or b) may allow some conceivable but 'unheard-of' exceptions. in other words, we wanted all to refer to near-definitional features and tried to prevent participants from worrying about far-fetched exceptions to the norm. the label most was used for all majority cases, including those where the annotator knew of actual real-world exceptions to a neardefinitional norm. the no/few distinction was defined as mirroring all/most. some was not associated with any specific instructions. to further minimise potential disagreements, we introduced extra instructions for cases where the participants might hesitate between two labels. we encouraged them to choose the label corresponding to lower set overlap (i.e. prefer some to most, most to all, etc). this ordering preference was set up with a view to use the dataset in computational inference tasks, where truth preservation is important: while cats have four legs should ideally be annotated as most, the label some still results in a true predication (some cats have four legs). compare with cats are black, where choosing most over some would result in a false sentence (most cats are black ). clearly, fixing the interpretation of the labels affects the type of information encoded by the dataset. we are not modelling the way that speakers naturally use the determiners some, most, all, etc. rather, we are modelling the perceived overlap between the set denoted by a noun and the set denoted by a predicate (simplifying somewhat and regarding predicates as sets rather than functions). our use of a script font for our allowable labels (e.g. some, most) reflects the fact that we have fixed the meaning of the corresponding quantifier in one of their possible interpretations. fixing the labels' interpretation, of course, does not completely suppress all unwanted effects. for instance, understanding all as a clear universal does not prevent annotators from falling into the 'generic' trap identified by @xcite , whereby people often agree to false statements such as all ducks lay eggs due to the straightforward availability of the corresponding generics (ducks lay eggs). the order of the data was randomised prior to distributing it to the annotators. participants took 20 or less hours to complete the task, which they did at their own pace, in as many sessions as they wished. while the task was a significant time investment, having a full set of labels for each participant allowed us to compare agreement over different classes of predicates ( §4.3) and thus obtain an insight into significant variations at the individual level. ## data analysis this section describes the annotated dataset, concentrating on three aspects: the overall distribution of the six labels, the overall interannotator agreement, and specific variations in agreement across conceptual feature classes. ## conclusion in this paper, we have presented an annotation layer for the mcrae feature norms @xcite , which shows quantifier labels of each concept-feature pair in the norms, as given by three native speakers of english. we are freely releasing this data for future research. 5 a subset of the dataset totalling 6120 instances contains all cases of majority agreement and can easily be used as gold standard for any computational application requiring examples of explicitly quantified statements about a range of concepts. for evaluation purposes, we systematically matched the quantifiers under study to a range of prevalence estimates and calculated the corresponding weighted kappas over our data. following the assumption that more accurate estimates should result in better kappa agreement, we derived prevalence figures for each one of our five quantifiers. for most, we found that, within the scope allowed by the guidelines (simple majority to near-universal), annotators applied the label to cases with prevalence close (but not equal) to 100%, clearly reading the quantifier as a 'universal with exceptions'. in spite of this, some did not end up covering cases of 'logical' majority (i.e. anything over 50%): with a prevalence of 35%, it was interpreted as a simple existential. this result is interesting, as it suggests that coders may not have felt the need for a quantificational category covering generally high prevalence. we also showed that agreement is not correlated with the frequency 4 for some discussion on the relation between event and individual quantification in generic sentences, see @xcite . 5 the full dataset, the majority cases and our annotation script are available at http://www.aurelieherbelot.net/research/ computational-linguistics-resources/. while overall, we observe good agreement on our quantification task (reaching κ a w =.59 for instances with a majority opinion), it seems unwarranted to assume that generalised quantifiers are entirely and reliably shared amongst speakers. rather, we must posit a 'many speakers, many worlds' hypothesis: individuals share some generic conceptual knowledge which helps them efficiently communicate, but their actual models of the world (what exactly counts as a cloak, and which proportion of those are coats) can differ fairly significantly. in addition, it is unclear how cognitively complex the quantification process is for a given speaker. while we did not monitor the difficulty of our task per se, it seems fair to assume that it is non-trivial: a simple pair such as bowl used-for-eating requires retrieving all subkinds of bowls the speaker may be familiar with, building a model of the entire set (across subkinds) and finally making a decision about the quantifier. it is not unreasonable to think that this process is only called upon when absolutely needed. this has consequences for the way we formalise models and, more specifically, quantification and inference. our observations above point at models which, most of the time, are left underspecified but can be specified when necessary -although at some cost and with some interspeaker variations (for an example of a formalisation of underspecified quantification, see @xcite . such variations will presumably affect agreement in logical inference tasks. for instance, inferring the probable colour of a hypothetical bathtub may turn out to be non-trivial: the fact that speakers produce the norm is-white for the corresponding concept may not be correlated with any expectation with regard to individuals (leading to the three annotations some, most and all in our data). we conclude by arguing in favour of a speaker-dependent notion of model which accommodates variations in people's beliefs about the world, while satisfying the requirement for broad general agreement, necessary for successful communication. we hope, at any rate, that the dataset we are releasing will be of use for further investigations of this question.