  Existing work on probing of pretrained language models (LMs) has
predominantly focused on sentence-level syntactic tasks. In this paper, we
introduce document-level discourse probing to evaluate the ability of
pretrained LMs to capture document-level relations. We experiment with 7
pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be
overall the best model at capturing discourse -- but only in its encoder, with
BERT performing surprisingly well as the baseline model. Across the different
models, there are substantial differences in which layers best capture
discourse information, and large disparities between models.
