The goal of unconditional text generation is training a model with real sentences, to generate novel sentences which should be the same quality and diversity as the training data. However, when different metrics are used for comparing these methods , the contradictory conclusions are drawn. The difficulty is that both the sample diversity and the sample quality should be taken into account simultaneously , when a generative model is evaluated. To solve this issue , a novel metric of distributional discrepancy (DD) is designed to evaluate generators according to the discrepancy between the generated sentences and the real training sentences. But, a challenge is that it can't compute DD directly because the distribution of real sentences is unavailable. Thus, we propose a method to estimate DD by training a neural-network-based text classifier. For comparison, three existing metrics, Bilingual Evaluation Understudy (BLEU) verse self-BLEU, language model score verse reverse language model score, Fr'chet Embedding Distance (FED), together with the proposed DD, are used to evaluate two popular generative models of LSTM and GPT-2 on both syntactic and real data. Experimental results show DD is much better than the three existing metrics in ranking these generative models.
The purpose of unconditional text generation is to train a model with real sentences, then generate novel sentences of the same quality and diversity as the training data. However, when different metrics are used for comparing the methods of unconditional text generation, contradictory conclusions are drawn. The difficulty is that both the diversity and quality of the sample should be considered simultaneously when the models are evaluated. To solve this problem , a novel metric of distributional discrepancy (DD) is designed to evaluate generators based on the discrepancy between the generated and real training sentences. However, it cannot compute the DD directly because the distribution of real sentences is unavailable. Thus, we propose a method for estimating the DD by training a neural-network-based text classifier. For comparison, three existing metrics, bi-lingual evaluation understudy (BLEU) versus self-BLEU, language model score versus reverse language model score, and Fr\'{e with the proposed DD, are used to evaluate two popular generative models of long short-term memory and generative pretrained transformer 2 on both syntactic and real data. Experimental results show that DD is significantly better than the three existing metrics for ranking these generative models.
R
goal
purpose
R
training
to train
R
to
then
R
which should be
of
R
these methods , the
the methods of unconditional text generation,
R
sample diversity and the sample quality should be taken into account simultaneously , when a generative model is
diversity and quality of the sample should be considered simultaneously when the models are
R
issue
problem
R
according to
based on
R
sentences and the
and
R
But, a challenge is that it can't compute
However, it cannot compute the
R
to estimate
for estimating the
R
Bilingual Evaluation Understudy
bi-lingual evaluation understudy
R
verse
versus
R
verse
versus
R
Fr'chet Embedding Distance (FED), together
and Fr\'{e
R
LSTM and GPT-2
long short-term memory and generative pretrained transformer 2
R
DD is much
that DD is significantly
R
in
for