A Generative Model for Joint Natural Language Understanding and Generation (2020)

Abstract: Natural language understanding (NLU) and natural language generation (NLG) are two fundamental and related tasks in building task-oriented dialogue systems with opposite objectives: NLU tackles the transformation from natural language to formal representations, whereas NLG does the reverse. A key to success in either task is parallel training data which is expensive to obtain at a large scale. In this work, we propose a generative model which couples NLU and NLG through a shared latent variable. This approach allows us to explore both spaces of natural language and formal representations, and facilitates information sharing through the latent space to eventually benefit NLU and NLG. Our model achieves state-of-the-art performance on two dialogue datasets with both flat and tree-structured formal representations. We also show that the model can be trained in a semi-supervised fashion by utilising unlabelled data to boost its performance.

## introduction natural language understanding (nlu) and natural language generation (nlg) are two fundamental tasks in building task-oriented dialogue systems. in a modern dialogue system, an nlu module first converts a user utterance, provided by an automatic speech recognition model, into a formal representation. the representation is then consumed by a downstream dialogue state tracker to update a belief state which represents an aggregated user goal. based on the current belief state, a policy network decides the formal representation of the system response. this is finally used by an nlg module to generate the system response @xcite . it can be observed that nlu and nlg have opposite goals: nlu aims to map natural language to formal representations, while nlg generates utterances from their semantics. in research literature, nlu and nlg are well-studied as separate problems. state-of-the-art nlu systems tackle the task as classification @xcite or as structured prediction or generation @xcite , depending on the formal representations which can be flat slot-value pairs @xcite , first-order logical form @xcite , or structured queries @xcite . on the other hand, approaches to nlg vary from pipelined approach subsuming content planning and surface realisation @xcite to more recent end-to-end sequence generation @xcite . however, the duality between nlu and nlg has been less explored. in fact, both tasks can be treated as a translation problem: nlu converts natural language to formal language while nlg does the reverse. both tasks require a substantial amount of utterance and representation pairs to succeed, and such data is costly to collect due to the complexity of annotation involved. although unannotated data for either natural language or formal representations can be easily obtained, it is less clear how they can be leveraged as the two languages stand in different space. in this paper, we propose a generative model for joint natural language understanding and generation (jug), which couples nlu and nlg with a latent variable representing the shared intent between natural language and formal representations. we aim to learn the association between two discrete spaces through a continuous latent variable which facilitates information sharing between two tasks. moreover, jug can be trained in a semi-supervised fashion, which enables us to explore each space of natural language and formal representations when unlabelled data is accessible. we examine our model on two dialogue datasets with different formal representations: the e2e dataset @xcite where the semantics are represented as a collection of slot-value pairs; and a more recent weather dataset @xcite where the formal representations are tree-structured. experimental results show that our model improves over standalone nlu/nlg models and existing methods on both tasks; and the performance can be further boosted by utilising unlabelled data. ## model our key assumption is that there exists an abstract latent variable z underlying a pair of utterance x and formal representation y. in our generative model, this abstract intent guides the standard conditional generation of either nlg or nlu (figure 1a ). meanwhile, z can be inferred from either utterance x, or formal representation y (figure 1b ). that means performing nlu requires us to infer the z from x, after which the formal representation y is generated conditioning on both z and x (figure 1c ), and vice-versa for nlg (figure 1d ). in the following, we will explain the model details, starting with nlg. ## optimisation we now describe how jug can be optimised with a pair of x and y ( §3.1), and also unpaired x or y ( §3.2). we specifically discuss the prior choice of jug objectives in §3.3. a combined objective can be thus derived for semi-supervised learning: a practical scenario when we have a small set of labelled data but abundant unlabelled ones ( §3.4). ## experiments we experiment on two dialogue datasets with different formal representations to test the generality of our model. the first dataset is e2e @xcite , which contains utterances annotated with flat slot-value pairs as their semantic representations. the second dataset is the recent weather dataset @xcite , where both utterances and semantics are represented in tree structures. examples of the two datasets are provided in tables 1 and 2. natural language "sousa offers british food in the low price range. it is family friendly with a 3 out of 5 star rating. you can find it near the sunshine vegetarian cafe." semantic representation @xmath0, @xmath1, @xmath2, @xmath3, @xmath4, @xmath5: an example in weather dataset. the natural language in original dataset (first row) is used for training to have a fair comparison with existing methods. the processed utterances (second row) is used in our semi-supervised setting. ## related work natural language understanding (nlu) refers to the general task of mapping natural language to formal representations. one line of research in the dialogue community aims at detecting slot-value pairs expressed in user utterances as a classification problem @xcite @xcite . another line of work focuses on converting single-turn user utterances to more structured meaning representa-tions as a semantic parsing task @xcite @xcite . in comparison, natural language generation (nlg) is scoped as the task of generating natural utterances from their formal representations. this is traditionally handled with a pipelined approach @xcite with content planning and surface realisation @xcite . more recently, nlg has been formulated as an end-to-end learning problem where text strings are generated with recurrent neural networks conditioning on the formal representation @xcite @xcite @xcite . there has been very recent work which does nlu and nlg jointly. both @xcite and @xcite explore the duality of semantic parsing and nlg. the former optimises two sequence-to-sequence models using dual information maximisation, while the latter introduces a dual learning framework for semantic parsing. @xcite proposes a learning framework for dual supervised learning @xcite where both nlu and nlg models are optimised towards a joint objective. their method brings benefits with annotated data in supervised learning, but does not allow semi-supervised learning with unlabelled data. in contrast to their work, we propose a generative model which couples nlu and nlg with a shared latent variable. we focus on exploring a coupled representation space between natural language and corresponding semantic annotations. as proved in experiments, the information sharing helps our model to leverage unlabelled data for semi-supervised learning, which eventually benefits both nlu and nlg. ## conclusion we proposed a generative model which couples natural language and formal representations via a shared latent variable. since the two space is coupled, we gain the luxury of exploiting each unpaired data source and transfer the acquired knowledge to the shared meaning space. this eventually benefits both nlu and nlg, especially in a lowresource scenario. the proposed model is also suitable for other translation tasks between two modalities. as a final remark, natural language is richer and more informal. nlu needs to handle ambiguous or erroneous user inputs. however, formal representations utilised by an nlg system are more precisely-defined. in future, we aim to refine our generative model to better emphasise this difference of the two tasks.