LEATHER : A Framework for Learning to Generate Human-like Text in Dialogue (2022)

Abstract: Algorithms for text-generation in dialogue can be misguided. For example, in task-oriented settings, reinforcement learning that optimizes only task-success can lead to abysmal lexical diversity. We hypothesize this is due to poor theoretical understanding of the objectives in text-generation and their relation to the learning process (i.e., model training). To this end, we propose a new theoretical framework for learning to generate text in dialogue. Compared to existing theories of learning, our framework allows for analysis of the multi-faceted goals inherent to text-generation. We use our framework to develop theoretical guarantees for learners that adapt to unseen data. As an example, we apply our theory to study data-shift within a cooperative learning algorithm proposed for the GuessWhat?! visual dialogue game. From this insight, we propose a new algorithm, and empirically, we demonstrate our proposal improves both task-success and human-likeness of the generated text. Finally, we show statistics from our theory are empirically predictive of multiple qualities of the generated dialogue, suggesting our theory is useful for model-selection when human evaluations are not available.

## introduction generating coherent, human-like text for dialogue remains a challenge. yet, it is an inseparable component of open domain and task oriented dialogue systems like alexa and siri. undoubtedly, it is also a complex process to learn. generation based on classification (e.g., next-word prediction) over-emphasizes the likelihood of text, leading to bland qualities, which are not human-like @xcite . meanwhile, framing dialogue generation as a markov decision process is highly data-inefficient when compared to classification @xcite . further, without careful design of rewards, models can suffer from mode-collapse in dialogue, producing repetitive behaviors that are not human-like @xcite . even carefully designed rule-based systems are brittle in the presence of unforeseen data-shift. theoretical analyses of learning are imperative as they provide solutions to these issues. for example, traditional (pac) learning theory @xcite studies similar issues arising from computational algorithms for learning to classify. progress in our understanding has been impressive, ranging from comprehensive guarantees on data-efficiency @xcite to insights for algorithm-design when the learner is faced with data-shift @xcite @xcite . while traditional theory may be applicable to simple generation objectives like next-word prediction, it is unfortunately unable to model more diverse goals. that is to say, it is insufficient to study replication of the diverse qualities inherent to human dialogue. the goal of this paper is to provide a new theory for analyzing the multi-faceted objectives in computational learning of dialogue generation. in particular, we propose leather 1 based on existing theories of computational learning. we demonstrate the utility of leather with a focus on understanding data-shift in learning algorithms. we also show empirical results for a task-oriented visual dialogue game. in detail, we contribute as follows: 1. in section 3, we propose leather, our novel theory for computational learning of dialogue generation. we use the guesswhat?! visual dialogue game @xcite as an example to ground abstract terminology in practice. we conclude section 3 by applying our theory to analyze a cooperative learning algorithm for guesswhat?!. our theory unveils harmful shifts in data-distribution that occur during training. 2. in section 4, we use leather to study the general problem of data-shift in text-generation. we provide new theoretical study that characterizes 1 learning theory for human-like dialogue generation figure 1 : examples of human and generated dialogue with original cooperative learning algorithm cl @xcite and our learning algorithm motivated by our theory (leather). roughly, leather works by applying a series of tests to generated dialogue and comparing the test results across the human and generated dialogue. well-generated dialogue is expected to perform similarly to human dialogue on these tests. the example tests the % of relevant questions. compared to cl, leather asks more relevant questions and therefore behaves more human-like. aggregate empirical results in section 5 echo this trend. our framework is publicly available through experimental code and a python package. 2 2 related works theories of learning to generate text most widely, text-generation is framed as a classification problem, in which a model predicts the next word provided existing context (e.g., previous words). while common pac learning analyses do apply to classification, this theory only describes the learner's ability at the next-word prediction task. in some specific cases, instead, pac analysis has also been used to analyze high-level objectives and motivate conversational strategies @xcite , but this analysis is problem-dependent. in contrast, our work offers a general problem-independent formalism for studying high-level qualities of generated text. another frequent formalism comes from partially observable markov decision processes (pomdps) used to motivate reinforcement learn-2 github.com/anthonysicilia/leather-aacl2022 ing. for example, see @xcite . while pomdps remedy the issues of typical pac analysis by supporting implementation of high-level objectives, as we are aware, there are no empirically verified theoretical studies of learning under data-shift in pomdps. in contrast, we demonstrate leather admits such a theory of learning, using it to predict the human-likeness of generated text under data-shift (where pomdps fall short). theories of learning with data-shift early learning theoretic models of data-shift in classification and regression are due to @xcite and @xcite . while modern approaches are generally similar in spirit, new statistics incorporate increasing information about the learning algorithm @xcite @xcite . ultimately, such techniques tend to improve the predictive capabilities of a theory in practical application @xcite . diverse additional approaches to describing the impact of data-shift have also been proposed, for example, using integral probability metrics @xcite @xcite . unfortunately, existing works focus on classification and regression which, as discussed, are not directly applicable to dialogue generation. further, this theory does not easily extend to generation (see section 3.3). ultimately, using leather, our work derives a new statistic (energy) for predicting changes in model performance, which is applicable to dialogue generation. ## theory with examples in this section, we develop our new theoretical framework. to assist our exposition, we use the guesswhat?! visual dialogue game -a variant of the child's game i spy -as a running example. we first describe the game along with our modeling interests within the game. we continue with a description of our theory and then apply this theory to analyze an algorithm that learns to play the game. ## text-generation under data-shift motivated by the guesswhat?! example and algorithm cl, we continue in this section with a general study of domain adaptation for dialogue generation. we begin by proposing a new (general) adaptation bound for leather. we then apply this general bound to the guesswhat?! algorithm cl, motivating fruitful modifications through our analysis. ## conclusion this work presents leather, a theoretically motivated framework for learning to generate humanlike dialogue. the energy statistic, which is derived from this theory, is used to analyze and improve an algorithm for task-oriented dialogue generation. further, energy is empirically predictive of improvements in dialogue quality, measured by both automated and human evaluation. future work may involve more experiments to test the utility of leather in other dialogue settings. theoretically, we hope to study sample-complexity in leather, which is a hallmark of common pac theories.