Categorisation, Typicality & Object-Specific Features in Spatial Referring Expressions (2020)

Abstract: Various accounts of cognition and semantic representations have highlighted that, for some concepts, different factors may influence category and typicality judgements. In particular, some features may be more salient in categorisation tasks while other features are more salient when assessing typicality. In this paper we explore the extent to which this is the case for English spatial prepositions and discuss the implications for pragmatic strategies and semantic models. We hypothesise that object-specific features — related to object properties and affordances — are more salient in categorisation, while geometric and physical relationships between objects are more salient in typicality judgements. In order to test this hypothesis we conducted a study using virtual environments to collect both category and typicality judgements in 3D scenes. Based on the collected data we cannot verify the hypothesis and conclude that object-specific features appear to be salient in both category and typicality judgements, further evidencing the need to include these types of features in semantic models.

## introduction various accounts of cognition and semantic representations have highlighted that, for some concepts, different factors may influence category and typicality judgements @xcite . in particular, some features may be more salient in categorisation tasks while other features are more salient when assessing typicality. in this paper we explore the extent to which this is the case for english spatial prepositions and discuss the implications for pragmatic strategies and semantic models. we hypothesise that object-specific features -related to object properties and affordancesare more salient in categorisation, while geometric and physical relationships between objects are more salient in typicality judgements. in order to test this hypothesis we conducted a study using virtual environments to collect both category and typicality judgements in 3d scenes. based on the collected data we cannot verify the hypothesis and conclude that object-specific features appear to be salient in both category and typicality judgements, further evidencing the need to include these types of features in semantic models. the spatial prepositions we analyse are those considered to have a functional component as well as those prepositions that seem to act as their geometric counterpart. for the 'functional' prepositions, object affordances and functional relationships, such as support and location control, appear to be salient @xcite compared to the geometric counterparts where geometric features and relative positions of objects appear to be more salient. in english, we consider the functional prepositions to be: 'in @xcite their respective geometric counterparts to be: 'inside', 'on top of', 'above' and 'below'. we also consider 'against' to be a functional preposition @xcite , without a clear geometric counterpart. to clarify the distinction between category and typicality judgements, we suppose that a category decision is when an entity is labelled with a category or concept and, though priming and context may certainly be factors, the judgement is not made in direct comparison with another entity. for example, a categorisation judgement occurs when an agent is asked whether 'the apple is in the bowl'. in order to reply, the agent judges the membership of the instance in the relevant category and the wider context plays a relatively minor role. typicality usually refers to the extent to which an entity is a good example of a concept -how similar it is to some ideal conceptual representation. in our study we ground the notion of typicality in comparison and preference -an entity, x, is more typical of a category than entity y if, when by assessing typicality in this way we distinguish the notion of typicality from graded category membership. the typicality data that we collect does not arise from graded membership judgements where study participants are asked to assign a value of how well the concept fits the category, e.g. in @xcite , but instead from tasks in which participants select the best fitting instance from a given description. in existing models of spatial language (and semantic models more generally), it is generally assumed that the underlying semantics of categorisation and typicality are essentially the same. however, as we will discuss in section 4, appropriately distinguishing categorisation and typicality judgements is important when generating and processing referring expressions. ## background following various criticisms of definitional representations of concepts in human cognition @xcite ), rosch's prototype theory @xcite provided an approach based on family resemblance which does not presuppose that concepts have necessary and sufficient conditions for making category judgements. by relying on a degree of resemblance to some prototypical notion of a concept, category membership in this account may be treated as a matter of degree. with such an account it becomes appealing to conflate the notions of categorisation and typicality -the more an entity resembles a prototype the more likely it is to be labelled with the category and the more typical it is. there are however various accounts of concept analysis which suggest that category and typicality judgements are fundamentally different. @xcite consider the influences of category decisions and propose a model to account for experimental findings. central to the model is the 'characteristic feature assumption', which supposes that features vary in the extent to which they define a concept. smith et al. suppose that there is a distinction in types of features -'defining' features which strongly influence category judgements and 'characteristic' features which strongly influence typicality judgements -and give the example of 'robin' to illustrate this. for the concept 'robin', 'have wings' is an important defining feature relating to the categorisation of an entity as 'a robin', while 'perches in trees' is a characteristic feature which relates to how typical an entity is of 'a robin'. @xcite argued that categorisation of some entity is more than simply a judgement of how similar the entity is to some typical representation of the category. rips provides support for this in an experiment where participants are asked to imagine an object of a given size and are asked which of two concepts, a, b say, the object is more similar to and which the object is most likely to be. for the given concept pairs, e.g. 'pizza' and 'quarter', the hypothetical object may be considered more similar to one of the concepts yet more likely to be the other. in the case of the pizza and quarter, a round object with a three inch diameter is regarded as more similar to a quarter as pizzas are rarely so small, but more likely to be a pizza as the size of a quarter is generally fixed and is less than 3 inches. this issue is explored further in @xcite where it is again argued that concept membership and typicality are distinct phenomena. based on a study in @xcite , osherson and smith argue that the notions differ using the seemingly extreme example of the concept 'red', even though it may be hard to imagine distinct defining and characteristic features for a concept with such simple semantics. however, the noted difference in judgements arises as people recognise particular wavelengths of light as being unambiguously red yet less typical than prototypical red. this difference in judgements preserves a monotonic relationship between category and typicality judgements i.e. if an entity, x, is a better category instance than y, then it is not less typical than y. we believe that such monontonicity results offer a trivial case which may be explained without any fundamental modification of the underlying semantics or how they are processed. in the simple case of the colour red, we may represent the semantics in both categorisation and typicality judgements by considering the dominant wavelength -an instance is more or less typical and a better or worse category instance based on the similarity of the dominant wavelength to prototypical red and the distinction in category and typicality judgements is explained via a threshold which is applied in category judgements. however, in the case of spatial language, the semantics are more complex and are influenced by a variety of geometric, functional and objectspecific features. we believe that as a result, the relationship between typicality and categorisation may in fact be non-monotonic -there may be instances of a spatial preposition, i 1 , i 2 , such that i 1 is a better category member but less typical than i 2 . ## hypothesis our main hypothesis is that categorisation and typicality judgements may differ for spatial prepositions in the following manner: given configurations c 1 , c 2 and preposition p , participants may be more likely to categorise c 1 with p yet more likely to select c 2 as a better instance of p . we hypothesise that this may arise in part because particular features are salient in category judgements which become less salient in typicality judgements. note that under this hypothesis the relationship between categorisation and typicality is nonmonotonic, making this in a sense stronger than the findings related to 'red' discussed by @xcite . in general, we expect that object-specific features will be more salient in categorisation while geometric and physical relationships, such as containment or support, will be more salient in typicality judgements. furthermore, we believe that this particular distinction may be more pronounced for the functional prepositions than for their geometric counterparts. this would be somewhat a corollary of the assumption that functionality in general is more salient for the former, for which tentative evidence is provided in (richard-bollans et al., 2020a) for these prepositions and @xcite in the case of 'over/above' and 'under/below'. ## generating & understanding referring expressions in this section we discuss the motivation for this investigation and the possible implications of a significant distinction between category and typicality judgements. in particular, we consider the ramifications for the field of referring expression generation and comprehension (reg/c), where noun phrases are used to identify entities e.g. 'the box under the table'. humans often prefer brief ambiguous descriptions over lengthy unambiguous descriptions @xcite , and expressions involving spatial prepositions often fulfil this desire for brevity. for example, rather than referring to objects based on elaborate visual attributes like 'the yellow cup with two pink dots on it', humans often refer to objects using simple locative expressions, say 'the cup next to the stapler'. we also see many examples of expressions containing spatial prepositions in the semeval-2014 corpus @xcite and the huric corpus @xcite , both of which consider natural language commands given to robots. to explain why a distinction in category and typicality judgements is important in these scenarios, suppose we have a speaker and listener, intended referent, r, set of distractor objects foot_1 and suppose that the speaker is generating a description, d. when the speaker generates an utterance in order to refer to r, there are various semantic and pragmatic considerations they must make. a naive model of such a speaker may simply find a concept within its vocabulary which is most suitable for r. this would clearly be a flawed strategy, however, as there may be other entities in the scene which better fit the concept -an expression may be true but not satisfy the speaker's communicative goals and so pragmatics must be considered. a more refined speaker model may find a description which best distinguishes r from the distractor objects in o, similar to the algorithm of @xcite which aims to maximise the discriminatory power of a description while minimising superfluous information. it appears that the speaker must model how well an object fits a description compared to other objects (typicality judgement) and how well a description fits an object (category judgement). further, it is apparent that humans will reason recursively about possible intentions of speakers and possible interpretations of listeners @xcite , making these judgements also necessary for listeners. let us consider a more concrete example. suppose we have a scenario as in figure 1 where a bowl, b, is on a table and there is one cube, c red , in it and one cube, c blue , next to and not touching it. it seems plausible that humans are more likely to categorise the configuration (c blue , b) with the preposition 'near' than the configuration (c red , b) even though when comparing the configurations humans may agree that c red is more 'near' the bowl than c blue . suppose a speaker gives an utterance 'the cube near the bowl' in order to refer to c blue which an agent must interpret. as may be expected, semantic models, e.g. @xcite , are likely to assign a better score for 'near' to (c red , b) than (c blue , b). if the system has a crude strategy for interpretation which simply selects the configuration with the highest semantic score, then such a system would erroneously select c red . many systems with more sophisticated pragmatic strategies, e.g. @xcite , have been developed which aim to take into account and reason with the possible utterances available to the speaker. in this case, such a system may correctly select c blue if it recognises that other better utterances would have been available to the speaker if the intended referent was c red . 'the cube in the bowl' would be a clear example of such an utterance which seems to clearly identify c red over c blue . however, supposing that our hypothesis is correct, we contend that this marked distinction is not simply a matter of typicality and the fact that it would be unusual to categorise (c red , b) with 'near' is more salient than any distinction in typicality between the two configurations. as c red is not 'in' the bowl in an ideal sense we can imagine that a semantic model based simply on the physical relationships between the objects may provide a more marked distinction between the configurations for 'near' than for 'in'. in this case, 'the cube in the bowl' wouldn't necessarily seem like a better utterance to identify c red than 'the cube near the bowl'. moreover, even if typicality on its own works well for rec, understanding and modelling the differences between categorisation and typicality is important for producing natural utterances in reg. for instance, suppose the speaker creates an utterance where r is more typical for the concept, c say, in d than any of the distractor objects. if categorisation is not aligned with typicality, such a strategy may produce unusual utterances where, though r is typical for c, it is uncommon to categorise r with c. such unnatural utterances may trigger unwanted conversational implicatures and be a source of confusion. for example, the utter-ance 'the ball on the bowl' in the context of figure 1 would be an unusual way to describe the ball as the preposition 'in' is often used with objects such as bowls. from this unconventional usage of 'on' a listener may imply that for some reason 'in' was not suitable e.g. if the speaker is actually referring to another unseen ball. the issue of producing natural utterances has been recognised by others in the field @xcite and is an important challenge to overcome if we are to develop more sophisticated reg systems. with regards to existing models of spatial language, it is generally assumed that the underlying semantics of categorisation and typicality are essentially the same. for example, in the pragr mechanism proposed in @xcite a pragmatic strategy is presented which aims to maximise both the acceptability and discriminatory power of a description. acceptability is calculated using similarity to a prototype based on physical relationships while the discriminatory power is calculated considering the acceptability of the description for the referent compared to other distractor objects. ## data collection in order to investigate typicality measures and categorisation judgements in detail, we conducted a study which is described below. collected data, details of the framework and results of the analysis can be found in the leeds research data repository. 3 the latest version of the data collection environment and code for analysis can be found in the github repository. foot_3 ## results in this section we use the collected data to test the hypothesis and conclude that we cannot verify the hypothesis. we propose that this is because objectspecific features are in fact salient in both category and typicality judgements and provide examples to support this. ## discussion regardless of the hypothesis, these results highlight the importance of including object-specific features in semantic models of spatial language. with the possible exception of @xcite , these types of features are rarely included in semantic models and many systems are developed in block-world type environments, e.g. @xcite @xcite , where these types of features are not needed. one approach to improving semantic models may be to incorporate information from knowledge bases such as conceptnet @xcite or @xcite . for example, from conceptnet one can determine that lids are used for covering and that bowls are containers. another approach is to leverage affordance detection systems, e.g. @xcite , which use information from the scene to predict object affordances. in (rodrigues et al., 2020) some object-specific features, e.g. whether or not the ground is a type of container, are taken to distinguish separate polysemes. 7 by leveraging previous work on modelling the polysemy of spatial prepositions @xcite , it may be possible to incorporate object-specific features into a semantic model by using these features to distinguish polysemes. in order to carry this out, further 7 a word is said to exhibit polysemy where the word has multiple related senses. each of these senses is called a polyseme.