Augmenting BERT -style Models with Predictive Coding to Improve Discourse-level Representations (2021)

Abstract: Current language models are usually trained using a self-supervised scheme, where the main focus is learning representations at the word or sentence level. However, there has been limited progress in generating useful discourse-level representations. In this work, we propose to use ideas from predictive coding theory to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations. As a result, our proposed approach is able to predict future sentences using explicit top-down connections that operate at the intermediate layers of the network. By experimenting with benchmarks designed to evaluate discourse-related knowledge using pre-trained sentence representations, we demonstrate that our approach improves performance in 6 out of 11 tasks by excelling in discourse relationship detection.

## introduction pre-trained language models are among the leading methods to learn useful representations for textual data. several pre-training objectives have been proposed in recent years, such as causal language modeling @xcite , masked language modeling @xcite , and permutation language modeling @xcite . however, these approaches do not produce suitable representations at the discourse level @xcite . simultaneously, neuroscience studies have suggested that predictive coding (pc) plays an essential role in language development in humans @xcite . pc postulates that the brain is continually making predictions of incoming sensory stimuli @xcite @xcite , with word prediction being the main mechanism @xcite . however, recent studies speculate that the predictive process could occur within and across utter-ances, fostering discourse comprehension @xcite . in this work, we propose to extend bert-type models with recursive bottom-up and top-down computation based on pc theory. specifically, we incorporate top-down connections that, according to pc, convey predictions from upper to lower layers, which are contrasted with bottom-up representations to generate an error signal that is used to guide the optimization of the model. using this approach, we attempt to build feature representations that capture discourse-level relationships by continually predicting future sentences in a latent space. we evaluate our approach on discoeval @xcite and scidtb for discourse evaluation @xcite to assess whether the embeddings produced by our model capture discourse properties of sentences without finetuning. our model achieves competitive performance compared to baselines, especially in tasks that require to discover discourse-level relations. sbert @xcite uses a siamese structure to obtain semantically meaningful sentence embeddings, focusing on textual similarity tasks. convert @xcite uses a dual-encoder to improve sentence embeddings for response selection tasks. these models focus on obtaining better representations for specialized sen-tence pair tasks, so they are not comparable with our which intended to be general-purpose. more recently, slm @xcite proposes a sentence unshuffling approach for a fine understanding of the relations among the sentences at the discourse level. conpono @xcite considers a discourse-level objective to predict the surrounding sentences given an anchor text. this work is related to our approach; the key difference is that our model predicts future sentences sequentially using a top-down pathway. we consider conpono as our main baseline. ## ablation study in order to verify the influence of pc mechanism on the pre-training result, we carry out ablation experiments. we use our predalbert-b as the default model, which includes top-down connections and recurrence from layer 12 to layer 1. ablations involve removing top-down connections and the recurrence of certain layers. the first experiment uses the pc mechanism on half the layers, i.e., the gru and predictive layer are present from layer 12 to layer 6. this variation exceeds the default model by ∼0.03 in disco-eval and ∼0.14 in scidtb-de. the second experiment uses the pc mechanism only on the last layer of the transformer. it means that the combination of the gru and prediction layer is only present in layer 12. this reduces the performance by ∼0.91 in discoeval and ∼2.41 in scidtb-de. also, we conducted an additional experiment where we removed the top-down connections (w/o tdc) to the default model. this is equiv- our findings indicate that top-down pathway is beneficial for improving discourse representations of bert-type models. however, it is not clear in which layers it is crucial to have the pc mechanism. we hypothesize that this is related to the fact that the bert-style models encode syntactic and semantic features in different layers @xcite , so a specialized pc mechanism for syntax or semantics would be desirable. we left this study for future work. because our model excels at detecting discourse relations, in this section, we explore whether the resulting vectors actually represent the role of a sentence in its discursive context. to illustrate what predbert learns, we follow the methodology proposed by @xcite . we use labeled sentences with discourse relations as queries to retrieve the top 3 most similar sentences from an unlabeled corpus using cosine similarity. we obtained the queries from the mit discourse relations annotation guide 2 and the unlabeled sentences from the gutenberg dataset @xcite . we compute the representations as mentioned in section 4.2. this process allowed us to verify that similar vectors share the same or equivalent discourse relations. temporal relation: @xmath0. 1. the last five words took tuppence's fancy mightily, especially after a meagre breakfast and a supper of buns the night before. 2. i know a disinterested man when i see him. 3. he had about ten pounds when i found him. sentence 1 has a succession relation due to the use of the word after. sentence 3 shows a synchrony relation because it uses when as the query. sentence 2 does not have a temporal relation. 2 https://bit.ly/3z45ig2 comparison relation: @xmath1, but the overall market fell. 1. the stock markets of the world gambled upon its chances, and its bonds at one time were high in favor. 2. tommy's heart beat faster, but his casual pleasantness did not waver. 3. i guess i was just a mite hasty, but i've been feeling bad about this money question. sentence 1 matched stock and market words but does not contain a comparison relation. sentences 2 and 3 include a counter-expectation relation similar to the query sentence that uses the word but. contingency relation: @xmath2. 1. i did the labor of writing one address this year, and got thunder for my reward. 2. i don't believe in a law to prevent a man from getting rich; it would do more harm than good. 3. when i fixed a plan for an election in arkansas i did it in ignorance that your convention was doing the same work. all sentences contain semantically related words like pay/reward and poor/rich. sentences 1 and 2 include a cause relation explicit and implicit. this is related to the query that has pragmatic cause relation. sentence 3 shows a temporal relation. ## conclusions we introduce an approach based on pc theory, which extends bert-style models with recursive bottom-up and top-down computation along with a discourse representation objective. our models achieve competitive results in discourse analysis tasks, excelling in relations detection.