On the Discrepancy between Density Estimation and Sequence Generation (2020)

Abstract: Many sequence-to-sequence generation tasks, including machine translation and text-to-speech, can be posed as estimating the density of the output y given the input x: p(y|x). Given this interpretation, it is natural to evaluate sequence-to-sequence models using conditional log-likelihood on a test set. However, the goal of sequence-to-sequence generation (or structured prediction) is to find the best output yˆ given an input x, and each task has its own downstream metric R that scores a model output by comparing against a set of references y*: R(yˆ, y* | x). While we hope that a model that excels in density estimation also performs well on the downstream metric, the exact correlation has not been studied for sequence generation tasks. In this paper, by comparing several density estimators on five machine translation tasks, we find that the correlation between rankings of models based on log-likelihood and BLEU varies significantly depending on the range of the model families being compared. First, log-likelihood is highly correlated with BLEU when we consider models within the same family (e.g. autoregressive models, or latent variable models with the same parameterization of the prior). However, we observe no correlation between rankings of models across different families: (1) among non-autoregressive latent variable models, a flexible prior distribution is better at density estimation but gives worse generation quality than a simple prior, and (2) autoregressive models offer the best translation performance overall, while latent variable models with a normalizing flow prior give the highest held-out log-likelihood across all datasets. Therefore, we recommend using a simple prior for the latent variable non-autoregressive model when fast generation speed is desired.

## introduction sequence-to-sequence generation tasks can be cast as conditional density estimation p(y|x) where x and y are input and output sequences. in this framework, density estimators are trained to maximize the conditional log-likelihood, and also evaluated using log-likelihood on a test set. however, many sequence generation tasks require finding the best output ŷ given an input x at test time, and the output is evaluated against a set of references y * on a task-specific metric: r(ŷ, y * |x). for example, machine translation systems are evaluated using bleu scores @xcite , image captioning systems use meteor @xcite and text-to-speech systems use mos (mean opinion scores). as density estimators are optimized on log-likelihood, we want models with higher held-out log-likelihoods to give better generation quality, but the correlation has not been well studied for sequence generation tasks. in this work, we investigate the correlation between rankings of density estimators based on (1) test log-likelihood and (2) the downstream metric for machine translation. on five language pairs from three machine translation datasets (wmt'14 en↔de, wmt'16 en↔ro, iwslt'16 @xmath0, we compare the held-out log-likelihood and bleu scores of several density estimators: (1) autoregressive models @xcite , (2) latent variable models with a non-autoregressive decoder and a simple (diagonal gaussian) prior @xcite , and (3) latent variable models with a non-autoregressive decoder and a flexible (normalizing flow) prior @xcite . we present two key observations. first, among models within the same family, we find that loglikelihood is strongly correlated with bleu. the correlation is almost perfect for autoregressive models and high for latent variable models with the same prior. between models of different families, however, log-likelihood and bleu are not correlated. latent variable models with a flow prior are in fact the best density estimators (even better than autoregressive models), but they give the worst generation quality. gaussian prior models offer comparable or better bleu scores, while autoregressive models give the best bleu scores overall. from these findings, we conclude that the correlation between log-likelihood and bleu scores varies significantly depending on the range of model families considered. second, we find that knowledge distillation drastically hurts density estimation performance across different models and datasets, but consistently improves translation quality of non-autoregressive models. for autoregressive models, distillation slightly hurts translation quality. among latentvariable models, iterative inference with a delta posterior @xcite significantly improves the translation quality of latent variable models with a gaussian prior, whereas the improvement is relatively small for the flow prior. overall, for fast generation, we recommend a latent variable nonautoregressive model using a simple prior (rather than a flexible one), knowledge distillation, and iterative inference. this is 5-7x faster than the autoregressive model at the expense of 2 bleu scores on average, and it improves upon latent variable models with a flexible prior across generation speed, bleu, and parameter count. ## background sequence-to-sequence generation is a supervised learning problem of generating an output sequence given an input sequence. for many such tasks, conditional density estimators have been very successful @xcite @xcite . to learn the distribution of an output sequence, it is crucial to give enough capacity to the model to be able to capture the dependencies among the output variables. we explore two ways to achieve this: (1) directly modeling the dependencies with an autoregressive factorization of the variables, and (2) letting latent variables capture the dependencies, so the distribution of the output sequence can be factorized given the latent variables and therefore more quickly be generated. we discuss both classes of density estimators in depth below. we denote the training set as a set of tuples {(x n , y n )} n @xmath1, . . . , x t } and @xmath2, . . . , y t } (where we drop the subscript n for notational simplicity). we use θ to denote the model parameters. ## problem definition on a sequence generation task, a conditional density estimator @xmath3, y n )} n @xmath4: once training converges, the model f is evaluated on the test set {(x m , y m )} m @xmath5: r(f ) = r {(x m , y m , ŷm )} m @xmath6, where ŷm = argmax y log p f (y|x m ). to perform model selection, we can rank a set of density estimators {f 1 , . . . , f k } based on either the held-out log-likelihood or the downstream metric. we measure the correlation between the rankings given by the log-likelihood l(f ) and the downstream metric r(f ). ## experimental setup on machine translation, we train several autoregressive models and latent variable models and analyze the correlation between their rankings based on log-likelihood and bleu. ## related work for sequence generation, the gap between loglikelihood and downstream metric has long been recognized. to address this discrepancy between density estimation and approximate inference (generation), there has largely been two lines of prior work: (1) structured perceptron training for conditional random fields @xcite @xcite and ( 2 ) empirical risk minimization with approximate inference @xcite @xcite @xcite @xcite . more recent work proposed to train neural sequence models directly on task-specific losses using reinforcement learning @xcite @xcite or adversarial training @xcite . despite such a plethora of work in bridging the gap between log-likelihood and the downstream task, the exact correlation between the two has not been established well. our work investigates the correlation for neural sequence models (autoregressive models and latent variable models) in machine translation. among autoregressive models for open-domain dialogue, a concurrent work @xcite found a strong correlation between perplexity and a human evaluation metric that awards sensibleness and specificity. this work confirms a part of our finding that log-likelihood is highly correlated with the downstream metric when we consider models within the same family. our work is inspired by recent work on latent variable models for non-autoregressive neural machine translation @xcite @xcite . specifically, we compare continuous latent variable models with a diagonal gaussian prior @xcite and a normalizing flow prior @xcite . we find that while having an expressive prior is beneficial for density estimation, a simple prior delivers better generation quality while being smaller and faster. ## conclusion in this work, we investigate the correlation between log-likelihood and the downstream evaluation metric for machine translation. we train several autoregressive models and latent variable models on five language pairs from three machine translation datasets (wmt'14 en↔de, wmt'16 en↔ro and iwslt'16 @xmath7, and find that the correlation between log-likelihood and bleu changes drastically depending on the range of model families being compared: among the models within the same family, log-likelihood is highly correlated with bleu. between models of different families, however, we observe no correlation: the flow prior model gives higher held-out log-likelihood but similar or worse bleu score than the gaussian prior model. furthermore, autoregressive models give the highest bleu scores overall but the latent variable model with a flow prior gives the highest test log-likelihoods on all datasets. in the future, we will investigate the factors behind this discrepancy. one possibility is the inherent difficulty of inference for latent variable models, which might be resolved by designing better inference algorithms. we will also explore if the discrepancy is mainly caused by the difference in the decoding distribution (autoregressive vs. factorized) or the training objective (maximum likelihood vs. elbo).