Incremental Graph-Based Semantics and Reasoning for Conversational AI (2021)

Abstract: The next generation of conversational AI systems need to: (1) process language incrementally, token-by-token to be more responsive and enable handling of conversational phenomena such as pauses, restarts and self-corrections; (2) reason incrementally allowing meaning to be established beyond what is said; (3) be transparent and controllable, allowing designers as well as the system itself to easily establish reasons for particular behaviour and tailor to particular user groups, or domains. In this short paper we present ongoing preliminary work combining Dynamic Syntax (DS) - an incremental, semantic grammar framework - with the Resource Description Framework (RDF). This paves the way for the creation of incremental semantic parsers that progressively output semantic RDF graphs as an utterance unfolds in real-time. We also outline how the parser can be integrated with an incremental reasoning engine through RDF. We argue that this DS-RDF hybrid satisfies the desiderata listed above, yielding semantic infrastructure that can be used to build responsive, real-time, interpretable Conversational AI that can be rapidly customised for specific user groups such as people with dementia.

## introduction humans process language in real-time (i.e. incrementally word by word) @xcite @xcite which gives rise to many conversational phenomena such as interruptions, disfluencies, restarts, corrections and split utterances @xcite @xcite . these phenomena support normal conversation @xcite , and are likely to be more common in specific groups such as people with dementia (pwd) who have to frequently restart and reformulate entire utterances (boschi figure 1 : a word-by-word ds-rdf parse of "jane drinks water quickly" @xcite . yet all commercial voice assistants, and most research systems, are turn-based @xcite , tending to ignore these phenomena altogether @xcite . dialogue also involves reasoning with ad-hoc, domain-specific rules @xcite learned from an early age @xcite . this allows participants to infer meaning beyond what is said on the surface of the conversation, and constitutes knowledge of a domain or topic of discourse. like language processing, reasoning also proceeds on an incremental basis allowing e.g. a hearer to predict what the speaker is going to say before they have said it @xcite . in conversational ai, this incremental reasoning capability becomes even more important for specific user groups such as pwd who often need assis-tance from a hearer to complete their own turn, and carry the conversation forward @xcite . in this short paper, we extend previous work on incremental semantic processing in dialogue by combining an inherently incremental grammar framework, dynamic syntax (ds, @xcite ), with the resource description framework (rdf, @xcite ) -dubbed ds-rdf foot_0 -paving the way for incremental semantic parsers that output rdf semantic graphs word by word as utterances unfold in dialogue (see fig. 1 ). such a parser can then act as a transparent and controllable language processing layer in incremental conversational ai foot_1 . while ds already enjoys extensive research applying it to computational dialogue processing (see @xcite ; @xcite among many others), ds-rdf has the important added benefit, through rdf, that it can hook seamlessly into the rich semantic resources that exist for rdf @xcite @xcite . further, as we outline below, it can also integrate easily with existing incremental rdf reasoning engines such as rdfox @xcite . ## dynamic syntax dynamic syntax (ds, @xcite @xcite is an actionbased grammar framework that directly captures the time-linear, incremental nature of language processing (i.e. both understanding and generation), on a word by word or token by token basis. it models the linear construction of semantic representations (i.e. interpretations) as progressively more linguistic input is parsed or generated. ds is idiosyncratic in that it does not recognise an independent level of syntactic representation over words: syntax on this view is sets of constraints on the incremental processing of semantic information in potentially multiple modalities (e.g. language and vision); and grammaticality is defined in purely procedural terms, i.e. as parsability in a context: the successful incremental construction of a semantic representation using all information given by the words in a string. the output of parsing any given string of words, or non-verbal tokens, is thus a semantic tree representing its predicate-argument structure -see fig. 2 . ds trees are always binary branching, with argument nodes conventionally on the right and functor nodes to the left. in ds's original form, tree nodes correspond to terms in the lambda calculus, decorated with labels expressing their semantic type (e.g. t y(e)) and fol formulae; and beta-reduction determines the type and formula at a mother node from those at its daughters (fig. 2 ). these trees can be partial, containing unsatisfied requirements potentially for any element (e.g. ?t y(e), a requirement for future development to t y(e)), and contain a pointer, ♢, labelling the node currently under development. actions in ds the parsing process in ds is defined in terms of conditional actions: procedural specifications for monotonic semantic tree growth. computational actions are languagegeneral structure-building principles which apply whenever their preconditions are met; and lexical actions are language-specific actions corresponding to and triggered by specific lexical tokens (words or gestures). all actions take the form of 'macros' to provide update operations on semantic trees, instantiated as if. . . then. . . else rules which yield semantically transparent structures when applied. fig. 2 is a simplified illustration of the parsing process for "john upset mary" where the application of computational actions is omitted for simplicity. for more detail on lexical specification in ds, see @xcite , pages 4-5. ## knowledge graphs and rdf knowledge graphs formally represent semantics by describing entities and the relations between them, usually represented in rdf (resource description framework); a data representation model for knowledge graphs @xcite . these rdf models are both human and machinereadable, enabling the collaborative development of interoperable resources and tools. one pertinent example, for our use case, is the use of ontologies as a schema layer. these formal descriptions of data structures are curated and maintained by communities of experts in the ontology's respective field (e.g. linguistics @xcite ). the wikibaselexeme data model @xcite aims to align with the lemon model @xcite (which provides rich linguistic grounding for ontologies and the syntax-semantics interface) and is a sibling project of many other huge collaborative projects like wikipedia, wikidata, and wikifunctions @xcite . the creators plan to create a language-independent version of wikipedia -encouraging the creation of structured linguistic wikifunctions. by using a common data model, we can make use of future wikifunctions as well as current resources. in order to tweak speech processing with all of this structured linguistic information, we need a point at which we can control the system's behaviour and tailor it for a specific user-group or domain. using rdfox @xcite , we have implemented an rdf reasoning layer to enable logical inference for deducing implicit knowledge from known explicit knowledge. we can add and modify logical rules at this layer, providing us with our required ability to control the processing of language. these rules are modelled with the datalog language @xcite and can operate efficiently incrementally @xcite , not reasoning across the entire graph every time a new token is uttered. we can therefore define new rules as we learn more about speech impairment, or speech within a particular domain, and deduce implicit knowledge as a user speaks. ## ds-rdf: combining dynamic syntax with rdf dynamic syntax was originally conceived @xcite with the epsilon calculus (an extension of fol) as the formalism in which semantic representations were couched (see e.g. the node decorations in fig. 2 ). however, as @xcite themselves note in chapter 19, what ds models is the real-time parsing process, i.e. the compositional dynamics of language processing in terms of the twin concepts of underspecification and subsequent update. ds is thus able to remain entirely agnostic about the choice of semantic representation. this generality has indeed been exploited in the past: purver et al. (2010, 2011); eshghi et al. (2012) used it to combine ds with type theory with records (ttr, cooper (2005); cooper and ginzburg (2015)) with record types and functions over these decorating tree nodes, allowing, among other things, the maximal semantics of partial, as well as complete, trees to be computed via type inference at every step (hough and purver, 2012). sadrzadeh et al. (2018); purver et al. ( 2021 ) later showed how ds can be combined with a wholly different kind of semantic representation: that of distributional, or vector-space semantics (vss, see @xcite for an overview) captured via tensors and vectors, thus enabling vss to be derived incrementally. to interface ds with rdf -or any other semantic formalism -there are two key operations that need to be defined over the target (rdf) representations: semantic composition for symbolic representations like rdf graphs here, this operation is standardly some version of the beta-reduction operation from the lambda calculus. here, we follow @xcite in taking the semantics of verbs (as well as nouns and adjectives) to be functions from rdf graphs to rdf graphs which conjoin ( ∧ ) their argument graph with their body -more on this below. the result is a graph that connects the argument graph with the body graph at an argument node designated as head foot_2 and illustrated as h -what follows is a simple example where the functional semantics of the intransitive verb run is applied to its subject rdf graph representing jane foot_3 : notice that the resulting graph has only 2 nodes, not 3, with the argument graph head node collapsing on the function body's node with the same uri, namely that designated as g.head. conjunction relative clauses, adverbials as well as some elliptical dialogue phenomena are modelled in ds via linked trees: these structures have the effect of temporarily shifting processing to the linked tree, whose root content is required to share a term with the node from which they link off (for details, see @xcite , sec. 1.2.3). when evaluated, the root content of the linked tree is conjoined with that of the node from which they had linked off; in fol this operation is simply logical conjunction; in ttr, it is the meet operation @xcite ; in rdf we similarly define it to be that of asymmetric merge of two graphs whereby nodes with the same uri -namely the shared term -collapse while the decorations/contents of this collapsed node come from the right hand side argument of the merge operation ( ∧ ). we illustrate with an example showing how the semantics of the sentence "jane runs fast" is computed by conjoining the content of the adverbial, 'fast' (on a ds linked tree not shown here) as a modifier of the 'running' action, is conjoined with the the content of the matrix clause, 'jane runs', with the shared term/node collapsing: two more auxiliary mechanisms are needed for complete integration: (a) inferring the maximal semantics of ds partial trees involves decorating the nodes not yet developed with underspecified rdf graphs of the right type -this can be done by excluding node annotations in these graphs -see e.g. @xcite and grammar induction @xcite ; in ttr, this is the inverse of the subtype (⊑) operation (supertype). in rdf, we can define this relation to be that of the subgraph with appropriate node subsumption operations. with the above operations defined, we now have the interface between ds and rdf spelled out, allowing ds trees (see fig. 2 ) to be decorated with rdf representations, and for incremental semantics to be derived in rdf. this allows us in turn to integrate rdf easily with dylan @xcite , the existing ds parser implementation. ds-rdf can therefore progressively enrich an rdf semantic graph as a sentence or utterance unfolds in time. we have illustrated this in fig. 1 , showing a parse of "jane drinks water quickly", but abstracting away from the underlying ds machinery for simplicity, only showing the rdf semantic graphs at each step. ## next steps we have formally defined the operations required to create ds-rdf: an incremental graph-based semantic parser that can be integrated with an incremental reasoning engine (rdfox). we are currently implementing ds-rdf and have example parses working with rdfox. our next step is to bootstrap a wide-coverage lexicon from existing resources and evaluate it on meaning banks (e.g. the groningen meaning bank @xcite ). we also plan to edit the output structure to align with the wikibaselexemes model, enabling seamless integration with wikifunctions in the future. finally, we plan to evaluate ds-rdf using domain specific dialogues and a corpus that we are currently collecting with people that have dementia.