Discourse Probing of Pretrained Language Models (2021)

Abstract: Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse — but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.

## introduction the remarkable development of pretrained language models @xcite @xcite has raised questions about what precise aspects of language these models do and do not capture. probing tasks offer a means to perform fine-grained analysis of the capabilities of such models, but most existing work has focused on sentence-level analysis such as syntax @xcite @xcite , entities/relations @xcite , and ontological knowledge @xcite . less is known about how well such models capture broader discourse in documents. rhetorical structure theory is a framework for capturing how sentences are connected and describing the overall structure of a document @xcite . a number of studies have used pretrained models to classify discourse markers @xcite and discourse relations @xcite , but few @xcite have systematically investigated the ability of pretrained models to model discourse structure. furthermore, existing work relating to discourse probing has typically focused exclusively model type #param #data objective bert enc 110m 16gb mlm+nsp roberta 110m 160gb mlm albert 12m 16gb mlm+sop electra 110m 16gb mlm+disc gpt-2 dec 117m 40gb lm bart enc+dec 121m 160gb dae t5 110m 750gb dae on the bert-base model, leaving open the question of how well these findings generalize to other models with different pretraining objectives, for different languages, and different model sizes. our research question in this paper is: how much discourse structure do layers of different pretrained language models capture, and do the findings generalize across languages? there are two contemporaneous related studies that have examined discourse modelling in pretrained language models. upadhye et al. (2020) analyzed how well two pretrained models capture referential biases of different classes of english verbs. @xcite applied the model of @xcite to parse imdb documents @xcite into discourse trees. using this (potentially noisy) data, probing tasks were conducted by mapping attention layers into single vectors of document-level rhetorical features. these features, however, are unlikely to capture all the intricacies of inter-sentential abstraction as their input is formed based on discourse relations foot_0 and aggregate statistics on the distribution of discourse units. figure 1 : illustration of the rst discourse probing tasks (tasks 4-6). to summarize, we introduce 7 discourse-related probing tasks, which we use to analyze 7 pretrained language models over 4 languages: english, mandarin chinese, german, and spanish. code and public-domain data associated with this research is available at https://github.com/fajri91/discourse_ probing . ## pretrained language models we outline the 7 pretrained models in ## probing tasks for discourse coherence we experiment with a total of seven probing tasks, as detailed below. tasks 4-6 are component tasks of discourse parsing based on rhetorical structure theory (rst; @xcite ). in an rst discourse tree, edus are typically clauses or sentences, and are hierarchically connected with discourse labels denoting: (1) @xmath0. satellite (s); foot_1 and (2) discourse relations (e.g. elaborate). an example of a binarized rst discourse tree is given in figure 1 . 1. next sentence prediction. similar to the next sentence prediction (nsp) objective in bert pretraining, but here we frame it as a 4-way classification task, with one positive and 3 negative candidates for the next sentence. the preceding context takes the form of between 2 and 8 sentences, but the candidates are always single sentences. ## experimental setup we summarize all data (sources, number of labels, and data split) in we use a learning rate 1e -3, warm-up of 10% of total steps, and the development set for early stopping in all experiments. all presented results are averaged over three runs. 4 ## results and analysis in figure 2 , we present the probing task performance on english for all models based on a representation generated from each of the 12 layers of the model. first, we observe that most performance fluctuates (non-monotonic) across layers except for some models in the nsp task and some albert results in the other probing tasks. we also found that most models except albert tend to have a very low standard deviation based on three runs with different random seeds. we discover that all models except t5 and early layers of bert and albert perform well over the nsp task, with @xmath1.8, implying it is a simple task. however, they all struggle at sentence ordering (topping out at ρ ∼ 0.4), suggesting that they are ineffective at modelling discourse over multiple sentences; this is borne out in figure 4 , where performance degrades as the number of sentences to re-order increases. interestingly, for discourse connectives, rst nuclearity, and rst relation prediction, the models produce similar patterns, even though the discourse connective data is derived from a different dataset and theoretically divorced from rst. bart outperforms most other models in layers 1-6 for these tasks (a similar observation is found for nsp and sentence ordering) with bert and albert struggling particularly in the earlier layers. for is in included in the appendix. 4 more details of the training configuration are given in the appendix. edu segmentation, roberta and again the first few layers of bart perform best. for the cloze story test, all models seem to improve as we go deeper into the layers, suggesting that high-level story understanding is captured deeper in the models. we summarize the overall performance by calculating the averaged normalized scores in the last plot in figure 2 . 5 roberta and bart appear to be the best overall models at capturing discourse information, but only in the encoder layers (the first 6 layers) for bart. we hypothesize that the bart decoder focuses on sequence generation, and as such is less adept at language understanding. this is supported by a similar trend for t5, also a denoising autoencoder. bert does surprisingly well (given that it's the baseline model), but mostly in the deeper layers (7-10), while electra performs best at the three last layers. in terms of the influence of training data, we see mixed results. bart and roberta are the two best models, and both are trained with more data than most models (an order of magnitude more; see lastly, we explore whether these trends hold if we use a larger model (bert-base vs. bert-large) and for different languages (again based on monolingual bert models for the respective languages). results are presented in figure 3 . for model size ("english (large)" vs. "english"), the overall pattern is remarkably similar, with a slight uplift in absolute results with the larger model. between the 4 different languages (english, chinese, german, and spanish), performance varies for all tasks except for nsp (e.g. edu segmentation appears to be easiest in chinese, and relation prediction is the hardest in german), but the shape of the lines is largely the same, indicating the optimal layers for a particular task are consistent across languages. ## conclusion we perform probing on 7 pretrained language models across 4 languages to investigate what discourse effects they capture. we find that bart's encoder and roberta perform best, while pure language models (gpt-2) struggle. interestingly, we see a consistent pattern across different languages and model sizes, suggesting that the trends we found are robust across these dimensions. spanish treebank. in proceedings of the 5th linguistic annotation workshop, pages 1-10. wietse de vries, andreas van cranenburgh, and malvina nissim. 2020. what's so special about bert's layers? a closer look at the nlp pipeline in monolingual and multilingual models. in findings of the association for computational linguistics: emnlp 2020, pages 4339-4350, online. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. vanessa wei feng and graeme hirst. 2014. a lineartime bottom-up discourse parser with constraints and post-editing. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: long papers), pages 511-521, baltimore, maryland. association for computational linguistics. john hewitt and christopher d. manning. 2019. a structural probe for finding syntax in word representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4129-4138, minneapolis, minnesota. association for computational linguistics. ganesh jawahar, benoît sagot, and djamé seddah. 2019. what does bert learn about the structure of language? in proceedings of the 57th annual meeting of the association for computational linguistics, pages 3651-3657, florence, italy. association for computational linguistics. fajri koto, jey han lau, and timothy baldwin. to appear. top-down discourse parsing via sequence labelling. in proceedings of the 16th conference of the european chapter of the association for computational linguistics. fajri koto, afshin rahimi, jey han lau, and timothy baldwin. 2020. indolem and indobert: a benchmark dataset and pre-trained language model for indonesian nlp. in proceedings of the 28th international conference on computational linguistics (coling 2020). zhenzhong lan, mingda chen, sebastian goodman, kevin gimpel, piyush sharma, and radu soricut. 2020. albert: a lite bert for self-supervised learning of language representations. in iclr 2020: eighth international conference on learning representations. mike lewis, yinhan liu, naman goyal, marjan ghazvininejad, abdelrahman mohamed, omer levy, veselin stoyanov, and luke zettlemoyer. 2020. bart: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7871-7880, online. association for computational linguistics. yancui li, wenhe feng, jing sun, fang kong, and guodong zhou. 2014. building chinese discourse corpus with connective-driven dependency tree structure. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 2105-2114, doha, qatar. association for computational linguistics. fei liu, trevor cohn, and timothy baldwin. 2018. narrative modeling with memory chains and semantic supervision. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 278-284, melbourne, australia. association for computational linguistics. yinhan liu, myle ott, naman goyal, jingfei du, mandar joshi, danqi chen, omer levy, mike lewis, luke zettlemoyer, and veselin stoyanov. 2019. roberta: a robustly optimized bert pretraining approach. arxiv preprint arxiv:1907.11692. andrew l. maas, raymond e. daly, peter t. pham, dan huang, andrew y. ng, and christopher potts. 2011. learning word vectors for sentiment analysis. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 142-150, portland, oregon, usa. association for computational linguistics. william c. mann and sandra a. thompson. 1986. assertions from discourse structure. in strategic computing -natural language workshop: proceedings of a workshop held at marina del rey, california, may 1-2, 1986. julian michael, jan a. botha, and ian tenney. 2020. asking without telling: exploring latent ontologies in contextual representations. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 6792-6812, online. association for computational linguistics. nasrin mostafazadeh, nathanael chambers, xiaodong he, devi parikh, dhruv batra, lucy vanderwende, pushmeet kohli, and james allen. 2016. a corpus and cloze evaluation for deeper understanding of commonsense stories. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 839-849, san diego, california. association for computational linguistics. shashi narayan, shay b. cohen, and mirella lapata. 2018. don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1797-1807, brussels, belgium. association for computational linguistics. allen nie, erin bennett, and noah goodman. 2019. dissent: learning sentence representations from explicit discourse relations. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4497-4510, florence, italy. association for computational linguistics. yannis papanikolaou, ian roberts, and andrea pierleoni. 2019. deep bidirectional transformers for relation extraction without supervision. in proceedings of the 2nd workshop on deep learning approaches for low-resource nlp (deeplo 2019), pages 67-75, hong kong, china. association for computational linguistics. alec radford, jeffrey wu, rewon child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. openai blog, 1(8):9. colin raffel, noam shazeer, adam roberts, katherine lee, sharan narang, michael matena, yanqi zhou, wei li, and peter j. liu. 2019. exploring the limits of transfer learning with a unified text-to-text transformer. journal of machine learning research, 21(140):1-67. rishi sharma, james allen, omid bakhshandeh, and nasrin mostafazadeh. 2018. tackling the story ending biases in the story cloze test. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 752-757, melbourne, australia. association for computational linguistics. wei shi and vera demberg. 2019. next sentence prediction helps implicit discourse relation classification within and across domains. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5790-5796, hong kong, china. association for computational linguistics. damien sileo, tim van de cruys, camille pradel, and philippe muller. 2019. mining discourse markers for unsupervised sentence representation learning. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3477-3486, minneapolis, minnesota. association for computational linguistics. shiva upadhye, leon bergen, and andrew kehler. 2020. predicting reference: what do language models learn about discourse models? in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 977-982, online. association for computational linguistics. elena voita, rico sennrich, and ivan titov. 2019. the bottom-up evolution of representations in the transformer: a study with machine translation and language modeling objectives. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 4396-4406, hong kong, china. association for computational linguistics. zining zhu, chuer pan, mohamed abdalla, and frank rudzicz. 2020. examining the rhetorical capacities of neural language models. in proceedings of the third blackboxnlp workshop on analyzing and interpreting neural networks for nlp, pages 16-32, online. association for computational linguistics.