Towards Unsupervised Language Understanding and Generation by Joint Dual Learning (2020)

Abstract: In modular dialogue systems, natural language understanding (NLU) and natural language generation (NLG) are two critical components, where NLU extracts the semantics from the given texts and NLG is to construct corresponding natural language sentences based on the input semantic representations. However, the dual property between understanding and generation has been rarely explored. The prior work is the first attempt that utilized the duality between NLU and NLG to improve the performance via a dual supervised learning framework. However, the prior work still learned both components in a supervised manner; instead, this paper introduces a general learning framework to effectively exploit such duality, providing flexibility of incorporating both supervised and unsupervised learning algorithms to train language understanding and generation models in a joint fashion. The benchmark experiments demonstrate that the proposed approach is capable of boosting the performance of both NLU and NLG. The source code is available at: https://github.com/MiuLab/DuaLUG .

## introduction spoken dialogue systems that assist users to solve complex tasks such as booking a movie ticket have become an emerging research topic in artificial intelligence and natural language processing areas. with a well-designed dialogue system as an intelligent personal assistant, people can accomplish certain tasks more easily via natural language interactions. nowadays, there are several virtual intelligent assistants, such as apple's siri, google assistant, microsoft's cortana, and amazon's alexa. the recent advance of deep learning has inspired many applications of neural dialogue systems @xcite . a typical dialogue system pipeline can be divided into several components: a speech recognizer that transcribes a user's speech input into texts, a natural language understanding module (nlu) to classify the domain along with domain-specific intents and fill in a set of slots to form a semantic frame @xcite . a dialogue state tracking (dst) module predicts the current dialogue state according to the multi-turn conversations, then the dialogue policy determines the system action for the next turn given the current dialogue state @xcite . finally, the semantic frame indicating the policy is fed into a natural language generationt (nlg) module to construct a response utterance to the user @xcite . generally, nlu is to extract core semantic concepts from the given utterances, while nlg is to construct corresponding sentences based on the given semantic representations. however, the dual property between understanding and generation has been rarely investigated, @xcite first introduced the duality into the typical supervised learning schemes to train these two models. different from the prior work, this paper proposes a general learning framework leveraging the duality between understanding and generation, providing flexibility of incorporating not only supervised but also unsupervised learning algorithms to jointly train nlu and nlg modules. the contributions can be summarized as 3-fold: this paper focuses on modeling the duality between understanding and generation towards unsupervised learning of the two components, related work is summarized below. ## proposed framework in this section, we describe the problem formulation and the proposed learning framework, which is illustrated in figure 1 . ## experiments our models are trained on the official training set and verified on the official testing set of the e2e nlg challenge dataset @xcite . the data preprocessing includes trimming punctuation marks, lemmatization, and turning all words into lowercase. each possible slot-value pair is treated as an individual label and the total number of labels is 79. to evaluate the quality of the generated sequences regarding both precision and recall, for nlg, the evaluation metrics include bleu and rouge (1, 2, l) scores with multiple references, while f1 measure is reported for evaluating nlu. ## future work though theoretically sound and empirically validated, the formulation of the proposed framework depends on the characteristics of data. not every nlu dataset is suitable for being used as a nlg task, vice versa. moreover, though the proposed framework provides possibility of training two models in a fully unsupervised manner, it is found unstable and hard to optimize from our experiments. therefore, better dual learning algorithms and leveraging pretrained models and other learning techniques, such as adversarial learning, are worthy to explore for improving the proposed framework. we leave the potential exploration as the future work. ## conclusion this paper proposes a general learning framework leveraging the duality between language understanding and generation, providing the flexibility of incorporating supervised and unsupervised learning algorithms to jointly train two models. the proposed framework provides a potential method towards unsupervised learning of both language understanding and generation models by considering their data distribution. the experiments on the benchmark dataset demonstrate that the proposed approach is capable of boosting the performance of both nlu and nlg models, motivating the potential research directions in this area.