Dual Inference for Improving Language Understanding and Generation (2020)

Abstract: Natural language understanding (NLU) and Natural language generation (NLG) tasks hold a strong dual relationship, where NLU aims at predicting semantic labels based on natural language utterances and NLG does the opposite. The prior work mainly focused on exploiting the duality in model training in order to obtain the models with better performance. However, regarding the fast-growing scale of models in the current NLP area, sometimes we may have difficulty retraining whole NLU and NLG models. To better address the issue, this paper proposes to leverage the duality in the inference stage without the need of retraining. The experiments on three benchmark datasets demonstrate the effectiveness of the proposed method in both NLU and NLG, providing the great potential of practical usage.

## introduction various tasks, though different in their goals and formations, are usually not independent and yield diverse relationships between each other within each domain. it has been found that many tasks come with a dual form, where we could directly swap the input and the target of a task to formulate into another task. such structural duality emerges as one of the important relationship for further investigation, which has been utilized in many tasks including machine translation @xcite , speech recognition and synthesis @xcite , and so on. previous work first exploited the duality of the task pairs and proposed supervised @xcite and unsupervised (reinforcement learning) @xcite learning frameworks in machine translation. the recent studies magnified the importance of the duality by revealing exploitation of it could boost the learning for both tasks. natural language understanding (nlu) @xcite and natural language generation (nlg) @xcite are two major components in modular conversational systems, where nlu extracts core semantic concepts from the given utterances, and nlg constructs the associated sentences based on the given semantic representations. @xcite was the first attempt that leveraged the duality in dialogue modeling and employed the dual supervised learning framework for training nlu and nlg. furthermore, @xcite proposed a joint learning framework that can train two modules seamlessly towards the potential of unsupervised nlu and nlg. recently, @xcite proposed a semi-supervised framework to learn nlu with an auxiliary generation model for pseudo-labeling to make use of unlabeled data. despite the effectiveness showed by the prior work, they all focused on leveraging the duality in the training process to obtain powerful nlu and nlg models. however, there has been little investigation on how to leverage the dual relationship into the inference stage. considering the fast-growing scale of models in the current nlp area, such as bert @xcite and @xcite , retraining the whole models may be difficult. due to the constraint, this paper introduces a dual inference framework, which takes the advantage of existing models from two dual tasks without re-training @xcite , to perform inference for each individual task regarding the duality between nlu and nlg. the contributions can be summarized as 3-fold: ## proposed dual inference framework with the semantics space x and the natural language space y, given n data pairs {(x i , y i )} n @xmath0, the goal of nlg is to generate corresponding utterances based on given semantics. in other words, the task is to learn a mapping function f (x; θ @xmath1. in contrast, the goal of nlu is to capture the core meaning from utterances, finding a function g(y; θ @xmath2. note that in this paper, the nlu task has two parts: (1) intent prediction and (2) slot filling. hence, x is defined as a sequence of words (@xmath3, while semantics y can be divided into an intent y i and a sequence of slot tags y @xmath4, (@xmath5, y s )). considering that this paper focuses on the inference stage, diverse strategies can be applied to train these modules. here we conduct a typical strategy based on maximum likelihood estimation (mle) of the parameterized conditional distribution by the trainable parameters θ @xmath6. ## experiments to evaluate the proposed methods on a fair basis, we take two simple gru-based models for both nlu and nlg, and the details can be found in appendix d. for nlu, accuracy and f1 measure are reported for intent prediction and slot filling respectively, while for nlg, the evaluation metrics include bleu and rouge-(1, 2, l) scores with multiple references. the hyperparameters and other training settings are reported in appendix a. ## conclusion this paper introduces a dual inference framework for nlu and nlg, enabling us to leverage the duality between the tasks without re-training the large-scale models. the benchmark experiments demonstrate the effectiveness of the proposed dual inference approach for both nlu and nlg trained by different learning algorithms even without sophisticated parameter search on different datasets, showing the great potential of future usage.