  The softmax operator is one of the most important functions in machine
learning models. When applying neural networks to multi-category
classification, the correlations among different categories are often ignored.
For example, in text generation, a language model makes a choice of each new
word based only on the former selection of its context. In this scenario, the
link statistics information of concurrent words based on a corpus (an analogy
of the natural way of expression) is also valuable in choosing the next word,
which can help to improve the sentence's fluency and smoothness. To fully
explore such important information, we propose a graph softmax function for
text generation. It is expected that the final classification result would be
dominated by both the language model and graphical text relationships among
words. We use a graph total variation term to regularize softmax so as to
incorporate the concurrent relationship into the language model. The total
variation of the generated words should be small locally. We apply the proposed
graph softmax to GPT2 for the text generation task. Experimental results
demonstrate that the proposed graph softmax achieves better BLEU and perplexity
than softmax. Human testers can also easily distinguish the text generated by
the graph softmax or softmax.
