Norm Participation Grounds Language (2022)

Abstract: The striking recent advances in eliciting seemingly meaningful language behaviour from language-only machine learning models have only made more apparent, through the surfacing of clear limitations, the need to go beyond the language-only mode and to ground these models “in the world”. Proposals for doing so vary in the details, but what unites them is that the solution is sought in the addition of non-linguistic data types such as images or video streams, while largely keeping the mode of learning constant. I propose a different, and more wide-ranging conception of how grounding should be understood: What grounds language is its normative nature. There are standards for doing things right, these standards are public and authoritative, while at the same time acceptance of authority can and must be disputed and negotiated, in interactions in which only bearers of normative status can rightfully participate. What grounds language, then, is the determined use that language users make of it, and what it is grounded in is the community of language users. I sketch this idea, and draw some conclusions for work on computational modelling of meaningful language use.

## introduction you fire up the lastest language model-a machine learning model induced by guessing words in a very large body of text-and "ask" it: "how would you describe the colour green?". after a short pause, the following text appears: "the colour green is refreshing and vibrant. it is often associated with nature and growth.". you: "how would you describe the colour red?" -"the colour red is exciting and full of energy. it is often associated with passion and power.". not bad. you hold two objects up in front of the computer screen: "which of these is green?". the reply suddenly is less than satisfying (and also old news in this conversation): "the colour green is refreshing and vibrant. it is often associated with nature and growth." 1 this of course is a profoundly unfair test. the model has no connection to you other than through what you type, and so can't observe what "these" refers to, and moreover, it never had access to anything other than language data. the question then arises what the consequences of this limitation are. is it just that those models (unsurprisingly) can't tell one visually presented object from another, or is there something fundamentally off about their grasp of language? in an influential recent paper, @xcite argue that indeed something is off, in that a model that only has access to form cannot learn to connect that form to meaning, which they tie to communicative intention, which in turn can be about the world. they also suggest that perhaps giving access to more than linguistic form might make it possible to learn this connection. in this their argument meets with more mainstream views that don't deny meaning status to language-only (or "internet world scope") models, but see them as deficient until augmented with additional forms of data @xcite . here, i will argue that just connecting language with non-language data still misses fundamental properties of language use, along two dimensions. first, the connection between world states and language is only one among several types of connections that must be gotten right (the others being intra-language connections, and connections between language and actions). secondly, the connections themselves need to be understood as normative ones, which again has two kinds of consequences: they effect commitments and entitlements, but they also are non-necessary and their applicability must be argued for (and can be argued against). just getting things right occasionally, or even very often, is not enough. the getting it right must come from an orientation towards the relevant standards of getting things right. this "orientation towards" shows in the ability to appeal to these standards when challenged, either directly or in the repair of understanding problems, and it forms the difference between what i will call "norm conformance" (which is what ai models are trained for), and "norm participation" (which is what what grounds language use). in short, there are interactive capabilities that need to underwrite meaningful language use, rendering agents that do not have them deficient language users. as i will argue, this has consequences for the use of nlp systems that can falsely appear as having these capabilities, and it also opens up interesting research directions. let us start by looking at simple observational statements, and use this to draw a schematic picture of meaning making in language use. ## "there is a tiger." leaving aside what these news should do to your friend, let's think a bit about what i, an overhearer, am justified to think may have been done to you to make you say that, and how that allows me to assign meaning to what you said. so, why did you say that? there are many possible types of answers to this question, a central class among which (namely those that not also insinuate malicious intent to deceive on your side) will mention in some form the state of affairs of there being a tiger, and your stating this as a fact. that is, there is an assumed connection between your expression e or more generally your action a (of uttering e) and a state of affairs c. you said "there is a tiger" in part because there is a tiger. but let's say you were wrong, and it was just tibby, my oversized tabby cat which can look, at least for a split second and when she is very hungry, like a (still very small though) tiger. we can't say anymore that you said "there is a tiger" because there was a tiger, as there was none. but we can fix the description by giving you a-potentially misguided-inner life: you said "there is a tiger", because you believed there to be a tiger, and you believed there to be a tiger, because you misperceived tibby as one. the chain now goes from c, the state of affairs (which in the modified example does not hold, i.e. turns out not to be a fact), to b, the belief, to a, the action. this chain can also be used to reconstruct understanding. 2 how can i get from observing a to forming my own belief about c? to reverse this chain, i need to see a as representative of a type of action a, and i need to know something about this type's connection to a type of belief c , and its connection to a type of states of affairs c-and i need to assume that you expected your addressees to know this and to be able to use this knowledge to reason back to the best explanation. 3 let us write out these connections in the form "if c, then ___ a", where the underscore shall work as a placeholder for a predicate describing the force of the connection, to be explored presently. if we take the step of seeing the forming of beliefs as an action as well, then this schema covers both parts of the chain described above: "if you are looking at a tiger (and your eyes are open, and you are sighted, etc.), then you ___ form the belief that there is a tiger", and "if you hold the belief that there is a tiger and you want to inform me of it, then you ___ say (something to the effect of) 'there is a tiger' ". looked at from a different perspective, your saying "there is a tiger" has committed you to believing that there is a tiger (insofar as that this is the best explanation for why you said that), and to having a good justification for that belief, where the best justification would be there indeed being a tiger (and you having the right kind of epistemic standing). having this belief further commits you to having other beliefs as well, such as "there is a mammal" or "there is a four-legged animal", "there is a catlike creature", "there is a living entity", etc.; these are just consequences (material inferences, to be precise) that we can see as being contained in having this belief, or, in other words, as contributing to individuating this belief as the one that it is. to collect what we have before we move on: this analysis assumes that there are connections between ways the world is and beliefs about it, between beliefs and other beliefs, and between beliefs (and other mental states, such as intentions) and actions. (we leave aside here whether in an ultimate analysis, these beliefs could not be explained away as dispositions to act in a proscribed way.) these connections can figure both in explanations of why you do something (you do a, because you are in c) and in abductions about states (as you did a, the state most likely is c). what is open is the exact nature of these connections, which is what we will turn to next. foot_4 ## norm conformance and norm participation our task now is to further specify the "if c, then ___ a" schemas. we want to achieve that they can figure in reasoning about why a speaker said what they said, and, equally importantly, can be offered by the speakers themselves as reasons for why they said what they said. as we will see, these are related, but separate aims: things can happen for a reason in different ways. to make the discussion more concrete, let us instantiate the c and a in this schema, as follows: could this conditional feature in reasoning about the behaviour of a human speaker? we would probably hesitate to allow such a description, wanting to qualify the antecedent with something like "and you want to inform your interlocutor about what you see, using the english language", for otherwise there are many ways in which you could react to the stimulus. also, there is still the question of how to fill the placeholder, and it seems that it should indeed be filled somehow, as a conditional of the form "if c, then you say e" seems too strong as description of human linguistic behaviour; even wanting to do something does not unconditionally lead to doing it. before we come to that, however, we can observe that something like (1), without any qualification about wanting to inform, is not a bad description of what the training set and learning objective of an image captioning model (e.g., @xcite al. (2012); vinyals et al. (2015) ) realises: to the extent that the model works (as measured by accuracy, or some other metric that measures agreement with a reference), it conforms to the norm described by (1). to the extent that this type of description fails to characterise the human language use situation, these models remain ungrounded. what (1) misses, however, is that these regularities, these norms, can feature in self-explanations, and exert a stronger force on language users, which, i propose, is better expressed by making it an element of the norm: one ought to behave in this way, given that the conditions are met; and, in reverse, one is committed to them being met, if one behaved in this way. this opens up two possible points of contention in the application of such a norm: first, do the conditions indeed hold, that is, can it be applied? second, is it even a norm, the authority of which i should accept? ("says who?" as a possible reply.) these are issues that can be, and not rarely are, raised in interaction (not in the artificial situations created by the language use of function-type models such as the aforementioned caption models). to distinguish this kind of actively following norms from just picking up regularities, i will use the label norm participation for it. before we turn to the ways that this participation process plays out in interactive language use, let us unpack this proposal a bit more. filling the placeholder and bringing in the intermediate belief state turns (1) into the following: to anticipate the discussion in the next section, the idea behind stressing the normative force of the connection is to explain why there is a pressure to correct disagreements, even if communicative success may have already been reached. in a very real sense, if you don't seem to be following these norms, then to me it seems that there is something wrong with you, at least as a participant in my system of norms; or, potentially, there is something wrong with my system. let us start with a simple disagreement. imagine you had pronounced tiger as in german (/"ti:g@/); i can recognise which norm you were aiming for, but can point out to you that the correct form contains /"taigä/, which one ought to use. or let us assume that i think that what is present is a leopard rather than a tiger; this allows me to spot that there is a deficiency in your belief/belief norm (and your perceptual one), at least compared to how i have it, which i can address by saying something like "they look similar, but have a different coat: tigers have stripes, while leopards have spots". (as we will discuss presently, i cannot force you to take this on; i can just try to make my claim of authority plausible to you.) note that this limits the possible misunderstandings-"this is not a tiger, it's a gazelle" is already odd; "this is not a tiger, it's a refrigerator" is far too odd, as the belief revision it indicates is too extreme to be plausible. lastly, an analysis of this form could also go some ways towards explaining why word uses can be so contentious, even if communicative success is not at issue: each use makes the implicit claim that this is how one ought to talk, and that it makes the right kinds of distinctions, a claim that addressees may want to disagree with. an analysis of slurs and linguistic interventions @xcite along these lines might be possible, but is left for future work here. again, let us take stock before we move on. i have argued that the right way to connect antecedent and consequent in constructs like (1) is to make direct appeal to their normative status: it is not just that if c is the case, one normally or conventionally does a, rather one ought to do this, and does something wrong or at least something inviting correction when one does not do it. doing things of these type then commits one in certain ways, and makes one suffer the consequences if these ways turn out to be not warranted. the analysis further has brought out a distinction between (mere) norm conformance, which is acting in accordance with a set of norms (for example, as they were realised in a data set of labelled examples) and norm participation, which involves treating the norms as possible reasons for acting, which can be offered, requested, and challenged. the interactive processes in which this is done and which justify the label "participation" will be our topic next. ## norm participation as interactive process and achievement the idea of the approach sketched here is that the question of which norms hold and how they are to be applied is never fully settled, and can become the overt topic of a conversation. that is, the fact that the connection is via an appeal to what one ought to do has practical consequences, which i will briefly trace in three related domains: language acquisition, conversational grounding, and conceptual disputes. more specifically, it shows in what in the field of conversational analysis is called repair, and is rightly assigned a central place in the study of conversation @xcite @xcite . 5 first language acquisition children start out without knowledge of the norms of the language community in which they were born. hence, they need to rely on the competent speakers around them to initiate them into these norms. the way they do this is by making attempts and observing reactions, which quite frequently involve repair. for example, @xcite found that about 50% of attempts by small infants (in their first verbal phase, from 1 to 1.5 years old) resulted in repair. in the light of the schema proposed here, we can understand this as attempts at using a norm, being recognised as such, and then getting demonstrated how the act ought to be performed. as the examples collected by @xcite show, this process can target both the form (that is, schemata of the type of (2-c)) as well as conceptual ones (as in (2-a) and (2-b)); indeed, these levels might often be addressed simultaneously. we can take away from this short review that an orientation towards shared norms seems to play a role already in the acquisition of language abilities. conversational grounding according to h. @xcite well-known proposal, it is a constant task in conversation to ensure that common ground is reached, sufficiently for the purposes at hand. we can recognise the stages of this process (presenting & identifying behaviour and signal; signalling & recognising propositional state; proposing & considering joint project) in the schema in (2). more directly related are the consequences of successful conversational grounding, as discussed by @xcite under the label "conceptual pacts". in the analysis proposed here, these can be understood as "local" norms that are not yet generalised, that is, ways the participants in an interaction mutually have come to think they ought to act with each other. 6 meaning disputes the status of these norms as reasons for acting shows most clearly in those rarer cases where they need to be overtly discussed. very occasionally, this can even become positive law: in (nix v. hedden, 149 u.s. 304, 1893), the us supreme court judged that for the purposes of taxation, tomatos are vegetables, despite biologically better fitting under the label fruit. in our framework, this can be understood as an adjudication between a norm that better fits to one type of belief/belief system (tomatos as fruit, for biological reasons) vs. one that better accords to actual usage (tomatos as vegetables, for similarity in properties to other vegetables). further examples are discussed by @xcite , and more recently, under the label word meaning negotiation, by @xcite and @xcite , who also provide the beginnings of a formalisation of the dialogue moves that structure this process. we can take from this very brief review that what is called norm participation here makes up a substantial amount of overt conversational moves, and is something that participants in verbal interactions actively engage in. if the story sketched above is on the right track, it provides a way to understand some ethical issues in the use of nlp systems. 7 consumers of computer speech acts will assume that, just like with human speakers, something like the chain in (2) is in place in a captioning system for example, even if in reality there is a more direct and simpler link between visual input and language. a disagreement with a labelling decision or apparent category will need to find an addressee, which the system cannot provide. organisations deploying such systems will need to take the responsibility for the "commitments" made by the system, as the system cannot do so -as it cannot "suffer the consequences". secondly, in the framework sketched above, as discussed, every use of language implicitly contains the claim "this is how one does this"; again, on the principle that the system provider will need to pick up "commitments" made by the system, this is something that seems to argue against the deployment of language generation systems that are wont to reproduce undesirable material @xcite . as a final example along these lines, consider the application of question answering. in the discussion above, i briefly mentioned the condition of needing to possess the right kind of epistemic standing to form beliefs (discussed in more detail by @xcite ). this epistemic standing can be "inherited" in knowledge through testimony @xcite . current search engines indirectly honour these mechanisms, by framing their job only as surfacing source material that provides its own reputational claims towards such epistemic standing. recent attempts at treating large language models as knowledge bases for question answering (surveyed by alkhamissi et al. ( 2022 )), however, break these links without providing others, which renders the status of their replies problematic (a similar point is made by @xcite and @xcite ). with these caveats in mind, some potentially productive lines of work can also be motivated from within the framework explored here. a language generating system that is able to maintain a coherent system of norms as described here, can use them to offer self-explanations, and can react to corrections, would go some way towards more grounded, and hence more meaningful, language use. components of this are already being explored separately. @xcite show that it is possible to explicate implicit commonsense knowledge from large language models (corresponding to the middle step (2-b)); @xcite show that a neuro-symbolic system can keep track of corrections to "beliefs" extracted from such models. it seems that combining these approaches in an interactive fashion, adding moves such as discussed by @xcite , would at least go some ways towards systems with more understandable meaning norms. ## related work the inspiration from the work of sellars for the ideas explored here has already been mentioned. beyond the work cited above, the role that giving and asking for reasons plays has been noted by @xcite and explanded upon by @xcite . 8 the varieties of rule following of course are an important topos from @xcite ) (see @xcite ; @xcite ), as is the necessarily public nature of judgements on the applicability of norms (on this see also hegel ( 1807 )). the notion of "orienting towards" is central in the field of conversation analysis. 9 on the computational side, schlangen (2016) makes some related points, although not yet under the normative framework explored here. @xcite made a similar point, and much more carefully (but also more restricted in scope). the forming of conceptual pacts is investigated with modern computational means by @xcite . work that could be enlisted for going towards norm participating has already been cited in the previous section. ## conclusions in this paper i have sketched a view of language as the purposeful use of norms for acting (where acting includes the forming of beliefs), where these norms can serve as reasons, can be negotiated, challenged, modified, and locally formed. i have speculated about the consequences of such a view on computational modelling of language use. no one could mistake this offering here for more than a sketch. to develop this into a fuller proposal, an enormous amount of work remains to be done. how exactly language lends itself to figure in such norms, and how these are composed (note that all examples used full sentences) is an open question (and compositionality is notoriously a problem for conceptual role semantics @xcite ), to mention just one technical challenge. nevertheless, what i hope to have offered is a potentially productive way to think about how language is grounded, not just in some link to perceptual information, but in the collective uses made of it, which are actively constructed and maintained to be collectively useful. it is my hope that this more interactive perspective on symbol grounding can be informative for computational work on simulating language use.