  Recently, pre-trained language models (PLMs) have dominated conditional text
generation tasks. Given the impressive performance and prevalence of the PLMs,
it is seemingly natural to assume that they could figure out what to attend to
in the input and what to include in the output via seq2seq learning without
more guidance than the training input/output pairs. However, a rigorous study
regarding the above assumption is still lacking. In this paper, we present a
systematic analysis of conditional generation to study whether current PLMs are
good enough for preserving important concepts in the input and to what extent
explicitly guiding generation with lexical constraints is beneficial. We
conduct extensive analytical experiments on a range of conditional generation
tasks and try to answer in what scenarios guiding generation with lexical
constraints works well and why. We then propose a framework for automatic
constraint extraction, denoising, and enforcement that is shown to perform
comparably or better than unconstrained generation. We hope that our findings
could serve as a reference when determining whether it is appropriate and
worthwhile to use explicit constraints for a specific task or
dataset.\footnote{Our code is available at
\url{https://github.com/morningmoni/LCGen-eval}.}
