Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models (2022)

Abstract: In this paper, we extend the line of BERTology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infer discourse information for arbitrarily long documents. In our experiments, we find that the captured discourse information is local and general, even across a collection of fine-tuning tasks. We compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information. Lastly, we individually explore self-attention matrices to analyze the information redundancy. We find that similar discourse information is consistently captured in the same heads.

## introduction transformer-based machine learning models are an integral part of many recent improvements in natural language processing (nlp). with their rise spearheaded by @xcite , the pre-training/fine-tuning paradigm has gradually replaced previous approaches based on architecture engineering, with transformer models such as bert @xcite , bart @xcite , roberta @xcite and others delivering state-of-the-art performance on a wide variety of tasks. besides their strong empirical results on most real-world problems, such as summarization @xcite , questionanswering @xcite and sentiment analysis @xcite , uncovering what kind of linguistic knowledge is captured by this new type of pretrained language models (plms) has become a prominent question by itself. as part of this line of research, called bertology @xcite , researchers explore the amount of linguistic understanding encapsulated in plms, exposed through either external probing tasks @xcite @xcite or unsupervised methods @xcite . previous work thereby either focuses on analyzing the syntactic structures (e.g., @xcite ; @xcite ), relations @xcite , ontologies @xcite or, to a more limited extend, discourse related behaviour @xcite @xcite . generally speaking, while most previous bertology works has focused on either sentence level phenomena or connections between adjacent sentences, large-scale semantic and pragmatic structures (oftentimes represented as discourse trees or graphs) have been less explored. these structures (e.g., discourse trees) play a fundamental role in expressing the intent of multi-sentential documents and, not surprisingly, have been shown to benefit many nlp tasks such as summarization @xcite , sentiment analysis @xcite @xcite and text classification @xcite . with multiple different theories for discourse proposed in the past, the rst discourse theory @xcite and the lexicalized discourse grammar @xcite (underlying pdtb @xcite ) have received most attention. while both theories propose tree-like structures, the pdtb framework postulates partial trees up to the between-sentence level, while rst-style discourse structures consist of a single rooted tree covering whole documents, comprising of: (1) the tree structure, combining clause-like sentence fragments (elementary discourse units, short: edus) into a discourse constituency tree, (2) nuclearity, assigning every tree-branch primary (nucleus) or peripheral (satellite) importance in a local context and (3) relations, defining the type of connection holding between siblings in the tree. given the importance of large-scale discourse structures, we extend the area of bertology research with novel insights regarding the amount of intrinsic discourse information captured in established plms. more specifically, we aim to better understand to what extend rst-style discourse information is stored as latent trees in encoder selfattention matrices foot_0 . while we focus on the rst formalism in this work, our presented methods are theory-agnostic and, hence, applicable to discourse structures in a broader sense, including other treebased theories, such as the lexicalized discourse grammar. our contributions in this paper are: ## related work at the base of our work are two of the most popular and frequently used plms: bert @xcite and bart @xcite . we choose these two popular approaches in our study due to their complementary nature (encoder-only vs. encoder-decoder) and based on previous work by @xcite and @xcite , showing the effectiveness of bert and bart models for discourse related tasks. our work is further related to the field of discourse parsing. with a rich history of traditional machine learning models (e.g., @xcite ; @xcite ; @xcite ; @xcite , inter alia), recent approaches slowly shifted to successfully incorporate a variety of plms into the process of discourse prediction, such as elmo embeddings @xcite , xlnet @xcite , bert @xcite , roberta @xcite and spanbert @xcite . despite these works showing the usefulness of plms for discourse parsing, all of them cast the task into a "local" problem, using only partial information through the shift-reduce framework @xcite , natural document breaks (e.g. paragraphs @xcite ) or by framing the task as an inter-edu sequence labelling problem on partial documents @xcite . however, we believe that the true benefit of discourse information emerges when complete documents are considered, leading us to propose a new approach to connect plms and discourse structures in a "global" manner, superseding the local proxy-tasks with a new methodology to explore arbitrarily long documents. aiming to better understand what information is captured in plms, the line of bertology research has recently emerged @xcite , with early work mostly focusing on the syntactic capacity of plms @xcite @xcite , in parts also exploring the internal workings of transformerbased models (e.g., self-attention matrices @xcite ). more recent work started to explore the alignment of plms with discourse information, encoding semantic and pragmatic knowledge. along those lines, @xcite present a parameterfree probing task for both, syntax and discourse. with their tree inference approach being computationally expensive and limited to the exploration of the outputs of the bert model, we significantly extend this line of research by exploring the internal self-attention matrices of plms with a more computationally feasible approach. more traditionally, zhu et al. (2020) use 24 hand-crafted rhetorical features to execute three different supervised probing tasks, showing promising performance of the bert model. similarly, pandia et al. (2021) aim to infer pragmatics through the prediction of discourse connectives by analyzing the model inputs and outputs and koto et al. (2021a) analyze discourse in seven plms through seven supervised probing tasks, finding that bart and bert contain most information related to discourse. in contrast to the approach taken by both zhu et al. (2020) and koto et al. (2021a) , we use an unsupervised methodology to test the amount of discourse information stored in plms (which can also conveniently be used to infer discourse structures for new and unseen documents) and extend the work by @xcite by taking a closer look at the internal workings of the selfattention component. looking at prior work analyzing the amount of discourse information in plms, structures are solely explored through the use of proxy tasks, such as connective prediction @xcite , relation classification (kurfalı and östling, 2021), and others @xcite . however, despite the difficulties of encoding arbitrarily long documents, we believe that to systematically explore the relationship between plms and discourse, considering complete documents is imperative. along these lines, recent work started to tackle the inherent input-length limitation of general transformer models through additional recurrence in the transformer-xl model @xcite , compression modules @xcite or sparse patterns (e.g., as in the reformer @xcite , bigbird @xcite , and longformer ( beltagy et al., 2020) models ). while all these approaches to extend the maximum document length of transformer-based models are important to create more globally inspired models, the document-length limitation is still practically and theoretically in place, with models being limited to a fixed number of pre-defined tokens the model can process. furthermore, with many proposed systems still based on more established plms (e.g., bert) and with no single dominant solution for the general problem of the input length-limitation yet, we believe that even with the restriction being actively tackled, an in-depth analysis of traditional plms with discourse is highly valuable to establish a solid understanding of the amount of semantic and pragmatic information captured. besides the described bertology work, we got encouraged to explore fine-tuned extensions of standard plms through previous work showing the benefit of discourse parsing for many downstream tasks, such as summarization @xcite , sentiment analysis @xcite @xcite and text classification @xcite . conversely, we recently showed promising results when inferring discourse structures from related downstream tasks, such as sentiment analysis @xcite and summarization @xcite . given this bidirectional synergy between discourse and the mentioned downstream tasks, we move beyond traditional experiments focusing on standard plms and additionally explore discourse structures of plms fine-tuned on a variety of auxiliary tasks. ## discourse extraction method with plms rather well analyzed according to their syntactic capabilities, large-scale discourse structures have been less explored. one reason for this is the input length constraint of transformer models. while this is generally not prohibitive for intra-sentence syntactic structures (e.g., presented in wu et al. ( 2020 )), it does heavily influence large-scale discourse structures, operating on complete (potentially long) documents. overcoming this limitation is non-trivial, since traditional transformer-based models only allow for fixed, short inputs. aiming to systematically explore the ability of plms to capture discourse, we investigate a novel way to effectively extract discourse structures from the self-attention component of the bert and bart models. we thereby extend our previously proposed tree-generation methodology @xcite to support the input length constraints of standard plms using a sliding-window approach in combination with matrix frequency normalization and an edu aggregation method. figure 1 visualizes the complete process on a small scale example with 3 edus and 7 sub-word embeddings. the tree generation procedure we previously proposed in @xcite explores a twostage approach to obtain discourse structures from a transformer model, by-passing the input-length constraint. using the intuition that the selfattention score between any two edus is an indicator of their semantic/pragmatic relatedness, influencing their distance in a projective discourse tree, they use the cky dynamic programming approach @xcite to generate constituency trees based on the internal selfattention of the transformer model. to generate dependency trees, we apply the same intuition used to infer discourse trees with the eisner algorithm @xcite . since we explore the discourse information captured in standard plms, we can't directly transfer our two-stage approach in @xcite , first encoding individual edus using bert and subsequently feeding the dense representations into a fixed-size transformer model. instead, we propose a new method to overcome the length-limitation of the transformer model foot_1 . the sliding-window approach is at the core of our new methodology to overcome the inputlength constraint. we first tokenize arbitrarily long documents with n edus @xmath0, ..., e n } into the respective sequence of m sub-word tokens @xmath1, ...t m } with n ≪ m, according to the plm tokenization method (wordpiece for bert, byte-pair-encoding for bart), as show at the top of figure 1 . using the sliding window approach, we subdivide the m sub-word tokens into sequences of maximum input length t max , defined by the plm (t @xmath2, t @xmath3. using a stride of 1, we generate (mt max ) + 1 sliding windows w , feed them into the plm, and extract the resulting t max ×t max partial square selfattention matrices (m p in figure 1 ) for a specific self-attention head foot_2 . the frequency normalization method allows us to combine the partially overlapping selfattention matrices m p into a single document-level matrix m d of size m×m. to this end, we combine multiple overlapping windows, generated due to the stride size of 1, by adding up the self-attention cells, while keeping track of the number of overlaps in a separate m × m frequency matrix m f . we then divide m d by the frequency matrix m f , to generate a frequency normalized self-attention matrix m a (see bottom of figure 1 ). the edu aggregation is the final processing step to obtain the document-level self-attention matrix. in this step, the m sub-word tokens @xmath4, ...t m } are aggregated back into n edus @xmath5, ..., e n } by computing the average bidirectional self-attention score between any two edus in m a . for example, in figure 1 , we aggregate the scores in cells m a [0:1, 5:6] to compute the final output of cell [0, 2] (purple matrix in figure 1 ) and m a [5:6, 0:1] to generate the value of cell @xcite . this way, we obtain the average bidirectional selfattention scores between edu 1 and edu 3 . we use the resulting n × n matrix as the input to the cky/eisner discourse tree generation methods. ## conclusions in this paper, we extend the line of bertology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned plms with large-scale discourse structures. we propose a novel approach to infer discourse information for arbitrarily long documents. in our experiments, we find that the captured discourse information is consitently local and general, even across a collection of fine-tuning tasks. we compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information. lastly, we individually explore self-attention matrices to analyze the information redundancy. we find that similar discourse information is consistently captured in the same heads. in the future, we intend to explore additional discourse inference strategies based on the insights we gained in this analysis. specifically, we want to explore more sophisticated methods to extract a single discourse tree from multiple self-attention matrices, rather than only the single best-performing head. further, we want to investigate the relationship between supervised discourse parsers and plm generated discourse trees and more long term, we plan to analyze plms with enhanced input-length limitations.