Although coherence modeling has come a long way in developing novel models, their evaluation on downstream applications  has largely been neglected. With the advancements made by neural approaches in applications such as machine translation , text summarization and dialogue systems, the need for standard coherence evaluation is now more crucial than ever. In this paper, we propose to benchmark coherence models on a number of synthetic and downstream tasks. In particular, we evaluate well-known traditional and neural coherence models on  sentence ordering tasks, and also on three downstream applications including coherence evaluation for machine translation, summarization and next utterance prediction . We also show model produced rankings for pre-trained language model outputs as another use-case. Our results demonstrate a weak correlation between the model performances in the synthetic tasks and the downstream applications, motivating alternate evaluation methods for coherence models . This work has led us to create a leaderboard to foster further research in coherence modeling .
Although coherence modeling has come a long way in developing novel models, their evaluation on downstream applications for which they are purportedly developed has largely been neglected. With the advancements made by neural approaches in applications such as machine translation (MT), summarization and dialog systems, the need for coherence evaluation of these tasks is now more crucial than ever. However, coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. To investigate how representative the synthetic tasks are of downstream use cases, we conduct experiments on benchmarking well-known traditional and neural coherence models on synthetic sentence ordering tasks, and contrast this with their performance on three downstream applications : coherence evaluation for MT and summarization, and next utterance prediction in retrieval-based dialog. Our results demonstrate a weak correlation between the model performances in the synthetic tasks and the downstream applications, motivating alternate training and evaluation methods for coherence models .
A
for which they are purportedly developed
R
, text summarization and dialogue
(MT), summarization and dialog
R
standard coherence evaluation
coherence evaluation of these tasks
R
In this paper, we propose to benchmark coherence models on a number of synthetic and downstream tasks. In particular, we evaluate
However, coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. To investigate how representative the synthetic tasks are of downstream use cases, we conduct experiments on benchmarking
A
synthetic
R
also
contrast this with their performance
R
including
:
R
machine translation, summarization
MT and summarization,
R
. We also show model produced rankings for pre-trained language model outputs as another use-case.
in retrieval-based dialog.
R
motivating alternate
motivating alternate training and
D
. This work has led us to create a leaderboard to foster further research in coherence modeling