In Search of Meaning and Its Representations for Computational Linguistics (2022)

Abstract: In this paper we examine different meaning representations that are commonly used in different natural language applications today and discuss their limits, both in terms of the aspects of the natural language meaning they are modelling and in terms of the aspects of the application for which they are used.

## introduction a crucial component to produce a "successful" nlp system is sufficiently expressive representations of meaning. we consider a sufficiently expressive meaning representation to be one that allows a system's output to be considered acceptable to native speakers given the task. in this paper we present several features of meaning and discuss how different methods of deriving meaning representations capture these features. this list is by no means exhaustive. it might be viewed as a first attempt to discuss ways of establishing a general methodology for evaluating meaning representations and characterising what kinds of applications they might be useful for. ## formal meaning representations the rigour of the work on semantics by richard montague @xcite inspired early work on computational semantics (perhaps the earliest was @xcite . two high-points of the literature on computational semantics based on montague are @xcite , using logic programming, and van eijck and unger (2010), using functional programming. montague's semantic techniques have also played an important role in semantic treatments using combinatory categorial grammar @xcite . one problem with montague's treatment of semantics was that it was limited to the level of the sentence. it could not, for example, deal with crosssentence anaphora such as a dog i barked. it i was upset by the intruder. this, among several other things, led to the development of discourse representation theory (drt, @xcite and other variants of dynamic semantics such as @xcite and @xcite . here "dynamic" is meant in the sense of treating semantic content as context change potential in order, among other things, to be able to pass referents from one sentence to a subsequent sentence in the discourse. this is a much less radical notion of dynamic interpretation than we discuss in section 4, where the meaning associated with a word or phrase may change as a dialogue progresses. drt has played an important role in computational semantics from early work on the verbmobil project @xcite to work by johan bos and others on the groningen meaning bank foot_0 and the parallel meaning bank foot_1 . one of the cornerstones of montague's approach is compositionality, the ability to compute the meaning of phrases on the basis of the meanings of their immediate sub-constituents. another central feature in the montague tradition is the ability to derive conclusions based on logical inference, including logical inferences based on the semantics of logical constants such as and, not and logical quantifiers, and the ability to characterise additional axioms or "meaning postulates". defeasible reasoning has been added to this kind of framework (e.g., @xcite and systems have been connected to theorem provers and model builders @xcite . the variants of dynamic semantics discussed above gave us the ability to treat discourse phenomena. that is, phenomena occurring in texts or utterances of more than a single sentence, including cases of discourse anaphora. underspecified meaning representations are single representations which cover several meanings in cases where there is systematic ambiguity. while there is some work on underspecification of meaning in the theoretical literature @xcite , the most interest has been devoted to it in computational work based on formal semantics (such as @xcite @xcite . model theory deals with representing the relationship between language and the world, in computational terms to database queries @xcite . what we have sketched above might be called the classical canon of formal semantics as it relates to computational semantics. one of the features lacking in the classical canon includes dialogue. the notion that language is actually used in interaction between agents engaging in communication (and therefore going beyond the notion of discourse in texts discussed earlier) came quite late to formal semantics though there is now a significant body of theoretical work. notions of dialogue semantics covering plan-based approaches to dialogue @xcite , questions under discussion @xcite and communicative grounding @xcite became central in the literature on formal approaches to dialogue. this gave rise to the information state update approach to dialogue @xcite . ttr (a theory of types with records, @xcite has played an important role in this. similarity of meaning is another feature. in addition to meaning relations such as entailment there is a notion of words, phrases and sentences having similar meanings in various respects. in a formal meaning representation this can be represented, for example, by the use of record types in ttr. yet another feature is robust non-logical inference which is represented, for example, in work on textual entailment now commonly referred to as natural language inference (nli). this is hard to square with the logic-based inference discussed above. rather than representing something that follows logically, it corresponds to what conclusions people might draw from a given utterance or text. it is often reliant on background knowledge and is to a large extent defeasible. the work on topoi by @xcite and probabilistic ttr @xcite is suggestive of a computational approach to this. finally, while model theory purports to relate language and the world it tells us little about how we relate our perception of the world and action in the world to the meaning of words and phrases which is known as perceptual grounding. such issues become important, for example, if we want to put natural language on board a robot. this has become central to theories such as ttr @xcite @xcite and dynamic syntax @xcite . there is important formal work on multimodal nature of communication, for example @xcite . above we have mentioned examples of formal approaches which attempt to incorporate features which are not present in the classical canon. an alternative strategy is to try to incorporate features from the classical canon in non-formal approaches (e.g. @xcite or to combine aspects of non-formal and formal approaches in a single framework (e.g. @xcite . ## distributional meaning representations meaning as a function of its usage can be traced back to @xcite , but were popularised by @xcite . the idea at its core is that the meaning of a word is given by its context. @xcite primarily speaks about meaning in relation to the world and real world activities while @xcite speaks about language in relation to language. the second notion of meaning is the basis for distributional semantics. the notion that meaning in language can be found based on language context is related to the observation that if two words occur in the same context, their meaning is likely related. the two predominant approaches to constructing distributional meaning representations today is to use machine learning to construct distributed and contextualised word representations @xcite @xcite . in these approaches, the meaning of a word is encoded as a dense vector of real valued numbers. the values of the vector are obtained by training a neural network to perform some task, using a (possibly annotated) corpus. the task then helps guide the neural network to produce meaningful representations. distributed word representations focus on building static representations of words given a corpus. popular techniques for obtaining these representations are bow (bag-of-words) or sgns (skip gram with negative sampling), popularised by @xcite . the main trick to bow and sgns is to construct a training schema such that given a random meaning representation for the word x, the representation is transformed so it can be used to identify the word in a context 3 , or can be used to to identify the context of the word. the bow or sgns meaning representations can then be used as a component in another system. contextualised representations on the other hand build dynamic word representations, that is, a single word will have different vector representations in different contexts. these representations are typically informed by the output from a language model. thus, to really exploit contextual representations effectively a sentence is needed when extracting the meaning representation. this is in contrast to the bow and sgns representations which are fixed after being constructed. with distributed representations we may also reason analogically about words and combinations of concepts, e.g. "russia" + "river" = "volga" @xcite . that is, we may construct complex meaning by combining simpler parts. by combining the representation for "russia" and "river" we obtain some vector z which contains information about the contexts of both "russia" and "river". by querying the vector space for words with a similar representation to that of z we find other words with similar context. the success of distributed meaning representations, both static and contextualised, can in part be attributed to the ability of a model to predict similarity between units of language. because meaning is defined as the context in which words occurs, two vector representations can be compared and their similarity measured. @xcite . this similarity can be explored in terms of words @xcite and in term of sentences @xcite . the ability to model similarity allows models to discover relationships between units of language. it allows models to transfer knowledge between languages. for example, unsupervised word translation can be done by aligning monolingual vector spaces. @xcite . transformer models @xcite have also enabled zero-shot and transfer learning, e.g. by learning english word representations and evaluat-3 context here is typically a n-gram containing x. however, a word or a sentence may mean different things depending also on a larger context. for example a sentence in different domains will express different meanings even if the words are exactly the same. this presents a problem for distributed representations, as our observation of a word or sentence in the real world includes additional context from what we have recorded in the data. however, the effects of different domains may be counteracted by domain adaptation techniques @xcite . distributed representations enjoy success across a wide variety of nlp tasks. however, a consequence of automatically learning from a corpus results in some inherent shortcomings. a corpus is a snapshot of a subset of language and only captures language as it was used then and there. this means that the resulting meaning representations do not inherently capture language change (though they can used to study it, see section 4). additionally, the meaning representations are generally created from observing language in a corpora, not from language use in the world. a consequence of this is that distributional meaning representations don't capture the state-of-affairs in the world, i.e. the context in which the language was used. in practical terms this means that for tasks that depend on the state-of-affairs in the world, such as robot control, dialogue or image captioning, a system must incorporate this information somehow which is further explored in the remaining sections. ## dynamic meaning representations to see how meaning is context dependent in (at least) two different ways we can make the distinction between meaning potential and situated meaning @xcite . the situated meaning of a word is its disambiguated and contextually enriched interpretation in a particular context of use. meaning potential (or lexical meaning) is the system of affordances @xcite ) that a word offers for sense-making in a given context. in this conception, situated meaning is context dependent by construction, but we also claim that the meaning potential of a word depends on context of a certain kind. in particular, it depends on what is common ground @xcite between a speaker and their audience. at a linguistics conference, a speaker might use words like token or modalitywords that would mean something completely different (or nothing at all) at a family dinner. the conference speaker expects to be understood based on their and their audience's joint membership in the computational linguistics community, where they (rightly or wrongly) consider certain specialised meanings to be common ground. the communities that serve as a basis for semantic common ground can be as broad as speakers of spanish (grounding the "standard" spanish lexicon), or as small as a nuclear family or close group of friends (grounding specialised meanings particular to that group of people) @xcite . recent work in nlp has demonstrated the value of modelling context, including sentential (section 3) and multimodal context (section 5) for representing situated meanings. the dynamic representations given by language models like bert depend on the local context in which the word appears, but don't the context of the community or individual speakers involved. little work has been done in nlp to explicitly incorporate social context, which provides the basis for semantic common ground. recent work has shown that neural language models can be used to detect and analyse variation and change in post-hoc way @xcite . this suggests that explicitly modelling social context may be a fruitful way forward. in the following, we identify three kinds of social context that might be accounted for with dynamic meaning representations variation as demonstrated in the conference example, lexical meaning is community dependent. this doesn't necessarily mean that every nlp application needs to mimic the human ability to tailor our semantic representations to the different communities we belong to, but some applications may serve a broader set of users by doing so @xcite . consider, for example, an application that serves both the general public and experts in some domain. even where variation is not explicitly modelled, it is an important factor to consider on a meta level. in practice, nlp models typically target the most prestigious, hegemonic dialect of a given language, due in part to biases in what train-ing data is easily available on the internet @xcite . this results in applications that favour users who are more comfortable with the dominant language variety. furthermore, many applications assume a single variety of a given language, when in fact the training data of the models they rely on is rather specific. the standard english bert model, for example, is trained on a corpus of unpublished romance novels and encyclopedia articles, but is applied as if it represents english written large. alignment semantic common ground is not only based on joint community membership-it can also be built up between particular agents through interaction. experiments have shown that pairs of speakers develop shorter lexicalised referring expressions when they need to repeatedly identify a referent @xcite . additions or modifications to existing common ground can take place implicitly (through semantic accommodation) or meaning accommodation @xcite or explicitly, as in word meaning negotiation @xcite . there is some hope for developing models that dynamically update their meaning representations based on interaction with other agents. @xcite suggest an inventory of semantic update functions that could be applied to formal meaning representations based on the results of an explicit word meaning negotiation. dynamic interpretation theory (bunt, 2000) offers a way of representing meaning as change to the conversational context, including social context, and has been incorporated in the implementation of several dialogue managers @xcite . on the distributional side, one-or few-shot learning may eventually allow models to generalise from a small number of novel uses by drawing on existing structure in the lexicon @xcite . one question that remains unexplored in both these cases is which updates to local (dialogue or partnerspecific) semantic ground should be propagated to the agent's representation of the communal common ground (and to which community). this naturally bring up the issue of community-level semantic change. change how words change in meaning has long been an object of study for historical linguists (e.g., @xcite . historical change may not seem like a particularly important thing for nlp applications to model. after all, we can accommodate for changes over decades or cen-turies by simply retraining models with more current data, but significant semantic shift can also take place over a much shorter timeline, especially in smaller speech communities @xcite . the issue of semantic change also intersects with that of variation, since coinages and shifts in meaning that take place in one community can cause the lexical common ground to diverge from another community. conversely, variants in one community may come to be adopted by another (possibly broader) community. the recent widespread use of distributional semantics to study semantic change suggests that distributional representations are capable of capturing change. foot_2 diachronic distributional representations have been used to study semantic change on both a historic/language level (e.g., @xcite and on a shortterm/community level @xcite @xcite . while social context is not often taken into account in meaning representations, ongoing research on semantic variation and change suggests that such dynamic representations are possible as extensions of the formal and distributional paradigms. ## grounded meaning representations the meaning of words is not merely in our head. it is grounded in our surroundings and tied to our understanding of the world @xcite , particularly through visual perception @xcite . mapping language and vision to get multi-modal meaning representations imposes an important challenge for many real-world nlp applications, e.g. conversational agents. such agents typically learn by finding statistical relations and often lack causal reasoning about the world @xcite and common-sense knowledge @xcite . this section describes how different modalities are typically integrated to get a meaning representation for language-andvision (l&v) tasks and what is still missing in the respective information fusion techniques. historically, modelling of situated language has been influenced by ideas from language technology, computer vision and robotics, where a combination of top-down rule-based language systems was connected with bayesian models or other kinds of classifiers of action and perception @xcite @xcite . in these approaches, most of the focus was on how to ground words or phrases in representations of perception and action through classification. another reason for this hybrid approach has also been that such models are partially interpretable. therefore, they have been a preferred choice in critical robotic applications where security is an issue. the compositionality of semantic representations in these systems is ensured by using semantic grammars, while perceptual representations such as slam maps @xcite or detected visual features (lowe, 1999) provide a model for interpreting linguistic semantic representations. deep learning, where linguistic and perceptual features are learned in an interdependent manner rather than engineered, has proven to be greatly helpful for the task of image captioning @xcite @xcite and referring expression generation @xcite . a more in-depth analysis of how meaning is represented in these models is required. @xcite show that a neural language model can learn compositionality by grounding an element in the spatial phrase in some perceptual representation. in terms of methodology for understanding what type of meaning is captured by the model, attention @xcite has been successfully used. @xcite have shown that co-attending to image and question leads to a better understanding of the regions and words the model is focused on the most. @xcite demonstrate that attention can struggle with fusing multi-modal information into a single meaning representation based on the human evaluation of generated image paragraphs. this is because the nature of visual and linguistic features and the model's structure significantly impact what representations can be learned when using an attention mechanism. examining attention shows that attention can correctly attend to objects, but once it is tasked to generate relations (such as prepositional spatial relations and verbs), attention visually disappears as these relations are non-identifiable in the visual features utilised by the model. this leads several researchers to include specifically geometric information in image captioning models @xcite . on the other hand, it has also been shown that a lot of meaning can be extracted solely from word dis-tributions. choi (2020) demonstrates how linguistic descriptions encode common-sense knowledge which can be applied to several tasks while @xcite ; @xcite demonstrate that word distributions are an important part of the semantics of spatial relations. interactive set-ups such as visual question answering (vqa) @xcite or visual dialogue @xcite make first attempts in modelling multi-modal meaning in multi-turn interaction. however, such set-ups are asymmetric in terms of each interlocutor's roles, which leads to homogeneous question-answer pairs with rigid word meaning. conversational games have been proposed as set-ups in which the meaning of utterances is agreed upon in a collaborative setting @xcite . these settings allow for modelling meaning coordination of grounded perceptual classifiers @xcite and phenomena such as clarification requests. several corpora of perceptual dialogue exist where conversational partners need to leverage dialogue and visual information to achieve mutual understanding of a scene, for example meetup! @xcite , photobook @xcite and cups @xcite . examining l&v models and representations they learn points to several significant and interesting challenges. the first relates to the structure of both datasets and models. many corpora contain prototypical scenes where the model can primarily optimise on the information from the language model to generate an answer without even looking at the image @xcite . secondly, information captured by a language model is more compact and expressive than patterns of visual and geometric features. thirdly, commonsense and visual information are not enough @xcite @xcite : we also rely on mental simulation of the scene's physics to estimate, for example, from the appearance and position of a person's body that they are making a jump on their skateboard rather than they are falling over a fire hydrant. such representations are necessary for modelling embodied agents @xcite @xcite . fourthly, adding more modalities and representations puts new requirements on inference procedures and more sophisticated models of attention @xcite that weighs to what degree such features are relevant in a particular con-text. in recent years we have seen work along these lines implemented with a transformer architecture @xcite @xcite . however, the interpretability of how individual parts (self-attentions) of large-scale models process information from different modalities is still an open question @xcite . ## meaning expressed with our body meanings can result in bodily reactions and, conversely, they can be expressed with our bodies, for example non-verbal vocalisations, gaze and gestures. emotions meanings perceived from the environment can change our emotional states and be expressed in bodily reactions: evaluating events as intrinsically unpleasant may result in gaze aversion, pupillary constriction and some of the other components @xcite . on the other hand, our emotional states can be expressed and the expressions can be adjusted by emotional components, such as mood @xcite . over the last years appraisal theories became the leading theories of emotions (for overview, see @xcite . these theories posit that emotion arises from a person's interpretation of their relationship with the environment or appraisal. the key idea behind cognitive theories is that emotions not only reflect physical states of the agents but also emotions are judgements, depending on the current state of the affairs (depending on a person, significance/urgency of the event etc.). in our view, linguistic events can as well enter the calculation of appraisal on the level of information-state of the agent which can be modelled by formal theories. for instance, following @xcite we can distinguish emotions as either free-floating or requiring an object such as a linguistic entity, entity in the environment or a part of agent's information-state (e.g., obstruction of the agent's goal can lead to anger or irritation, and, vice versa, agent's sadness can lead to the search for a new plan). several attempts implement emotional appraisal in text and speech (e.g., alm, 2012), and within the agent models (e.g., @xcite . non-verbal vocalisations non-verbal vocalisations, such as laughter, are ubiquitous in our everyday interactions. in the british national corpus laughter is a quite frequent signal regardless of gender and age-the spoken dialogue part of the british national corpus contains approximately one laughter event every 14 utterances. in the switchboard dialogue act corpus non-verbally vocalised dialogue acts (whole utterances marked as non-verbal) constitute 1.7% of all dialogue acts and laughter tokens make up 0.5% of all the tokens that occur in the corpus. despite a distinct bodily reaction (laughter causes tensions and relaxations of our bodies), it is believed that we laugh in a very different sense from sneezing or coughing @xcite . many scholars agree that laughter is not involuntary but we laugh for a reason, about something and that laughter perfomrs a social function @xcite . it is associated with senses of closeness and affiliation, establishing social bonding and smoothing away discomfort. for example, tickling not only requires the presence of the other but also it is more likely if subjects have close relationships @xcite . therefore, the meaning of laughter ought to be represented so that an artificial agent can understand it and react to it accordingly @xcite . @xcite presents a function-based taxonomy of laughter, distinguishing functions such as indication of pleasant incongruity or smoothing the discomfort in conversation. @xcite propose a formal account of laughter within the information-state of dialogue participants which includes scaling up to non-verbal social signals such as smiling, sighing, eye rolling and frowning. gaze gaze has many functions. it can dictate attention, intentions, and serves to give communicative cues in interaction @xcite . gaze following can infer objects that people are looking at. while we scan a visual scene, our brain stores fixation sequences in memory and reactivates them when visualising the scene later in the absence of any perceptual information @xcite . scan-path theory illustrations indicate that meaning representations on scanned areas depend on the semantics of sentences @xcite . semantic eye fixations supports the view of mental imagery that is flexible and creative. being grounded in previous experiences, by selecting a past episode we are able to generalise the past information to novel images that share features @xcite . spatial representations associated with different semantic categories launch eye movements during retrieval @xcite . for dialogue participants gaze patterns represent resources to track their stances. interlocutors engage in mutual gaze while producing agreeing assessments @xcite . gaze shifts follow sequentially a problematic stance and are followed by a divergent stance by the person who produced the gaze shift. gaze patterns are not meaningful per se but acquire interpretation within their linguistic and interactional contexts. eye movement patterns, eeg signals and brain imaging are some of the techniques that have been used to augment traditional qualitative text-based features. temporal course and flexibility of the speaker's eye gaze can be used to disambiguate referring expressions in spontaneous dialogue. eyetracking data from reading experiments provide structured information with fine-grained temporal resolution which closely follows sequential structure of speech and is related to the cognitive workload of speech processing @xcite . deep convolutional neural networks have been used to classify text to gaze using eye movements. their performance has improved when human readers were tackling semantic challenges @xcite . for multimodal and multi-party interaction in both social and referential scenarios, @xcite calls for categorical representation of gaze patterns. gestures gestures are hand and body movements that help to convey information @xcite . the observational, experimental, behavioural and neuro-cognitive evidence indicates that language and gestures are linked both during comprehension and production @xcite . speech and gestures are semantically and temporally coordinated and therefore involved in co-production of meaning. gestures convey meaning through iconicity and spacial proximity providing information that is not necessarily expressed in speech. while shaping of gestures is related to conceptual and semantic aspects of the accompanying speech, gestures cannot be unambiguously interpreted by naïve listeners @xcite . however, @xcite showed that the semantic relationship between representational gestures vs their lexical affiliates and language is evaluated similarly. mentions of referents for the first time in a discourse are often accompanied by gestures. for example, debreslioska and gullberg (2020) report that the "entity" gesture accompanies referents expressed by indefinite nominals. as referents are introduced in clauses, inferable referents referred to by definite nominals are identified by the contrasting "action" gestures. head movements are produced to give feedback @xcite and it is possible to identify a specific pattern for a specific movement and that movements can be easily measured and their extent can be quantified @xcite . fixing gesture functions, integrating different modalities and determining their composite meanings is challenging. for artificial agents multimodal output planning is crucial and timing must be explicitly represented. @xcite takes a qualitative formal approach from a type-theoretic perspective, representing iconic gestures in ttr and linking them with linguistic predicates. (pustejovsky and krishnaswamy, 2020) take a hybrid approach linking qualitative representations in voxml with machine learning classification. ## conclusions we surveyed formal, distributional, interactive, multi-modal and body-related representations of meaning used in computational linguistics. they are able to deal with compositionality, underspecification, similarity of meaning, inference and provide an interpretation of expressions but in very different ways, capturing very different kinds of meaning. these aspects can be broadly categorised into (i) aspects that are related to the construction of linguistic forms and (ii) aspects concerned with the interpretations and understanding the world and human activity in it. current mainstream computational linguistics is a practical field which is not working toward a uniform model of human language but focuses on several sub-tasks which, although related, are frequently considered in isolation. for example, natural language understanding and natural language generation frequently use entirely different approaches and representations even when the linguistic context is the same, for example texts or image captions. solutions are provided given the practical goals and limitations of each task. secondly, the solutions are also limited by what linguistic information can be feasibly collected for this task and by our understanding of human language and behaviour (or its lack-of) as witnessed by ongoing work in linguistics and psychology. if our goal is to translate documents or answer general fact-based questions then a reasonable performance can be achieved even if the system is able to ground representations only indirectly in linguistic contexts over situations rather than situations themselves. however, for a situated robot semantic grounding in texts, although relevant, is not enough as it has to connect language with its environment that it accesses with its sensors and actuators. for example, word embeddings for left and right will tell us that they are similar relations but also that they have slightly different selectional preferences for objects that they relate. humans rely on different aspects of meaning for different kinds of descriptions and contexts and it may be perfectly fine for our task-specific computational models to only use some dimensions and in an indirect way. what is wrong to claim, however, is that any of these models have reached humanlike intelligence. humans can re-evaluate linguistic descriptions against different dimensions of meaning and this is something that our systems are not capable of. work on the cognitive notion of attention informs us how aspects of meaning representations are selected to disambiguate under-specified linguistic utterances by balancing information from different modalities. a stronger connection between representations from different tasks is certainly desirable and important progress has been made for example in integration of formal grammars in situated agent systems or integration of vision and language representations learned by neural networks. however, the challenge remains precisely because "linguistic experiences" of our systems are limited by the narrow tasks and domains that they are specialised on. transferring models across contexts of language use and aligning their representations is by no means straightforward as there may be very little overlap. at the same time we also expect that models learn generalisations that apply across contexts. in line with this we suggest that future work should focus on developing benchmarks that test different representations in different contexts. for this we need datasets of instances requiring some type of linguistic inference where instances are labelled by the context type and the type(s) of modality required for successful inference. we hope that this paper points to some of the aspects of representations that need to be taken into account.