  Despite considerable advancements with deep neural language models (LMs),
neural text generation still suffers from degeneration: generated text is
repetitive, generic, self-inconsistent, and lacking commonsense. The empirical
analyses on sentence-level attention patterns reveal that neural text
degeneration may be associated with insufficient learning of inductive biases
by the attention mechanism. Our findings motivate on-the-fly attention
modularization, a simple but effective method for injecting inductive biases
into attention computation during inference. The resulting text produced by the
language model with attention modularization can yield enhanced diversity and
commonsense reasoning while maintaining fluency and coherence.
