Using Discourse Connectives to Test Genre Bias in Masked Language Models (2024)

Abstract: This paper presents evidence for an effect of genre on the use of discourse connectives in argumentation. Drawing from discourse processing research on reasoning based structures, we use fill-mask computation to measure genre-induced expectations of argument realisation, and beta regression to model the probabilities of these realisations against a set of predictors. Contrasting fill-mask probabilities for the presence or absence of a discourse connective in baseline and finetuned language models reveals that genre introduces biases for the realisation of argument structure. These outcomes suggest that cross-domain discourse processing, but also argument mining, should take into account generalisations about specific features, such as connectives, and their probability related to the genre context.

## introduction argumentative structures in discourse, which comprise a claim and a supporting or attacking premise, exhibit significant variation in their realisation. notably, as argumentative coherence can be achieved by alternative signals @xcite , arguments vary in whether the claim and premise are linked by a discourse connective or not. for example, because in item 1a explicitly conveys a causal relation. in contrast, item 1b, where two sentences are separated by punctuation, leaves the relation implicit, with the claim indicated only by the deontic modal should. 2 (1) a. masking should be mandated because it keeps everyone safe. the explicit or more implicit realisation of argumentation is a challenge for an understanding of argumentative discourse. however, the processing of arguments is likely not random and should conform with general discourse processing principles. in particular, it can be assumed that, following the uniform information density (uid) hypothesis @xcite , relations within discourse, such as the one between a claim and a premise, are more likely to be expressed explicitly when they are unexpected, and more likely to be implicit when a relation can be anticipated @xcite . the factors that shape expectations in discourse are diverse. local cues within a phrase or sentence, such as the use of the connective because in item 1a, play a role. however, more global forces, such as the overall nature of the document, also drive expectations and, with that, information density @xcite . knowing that genres guide expectations and influence human discourse understanding on many levels of a text @xcite , we explore in this paper how genre creates a bias for the ways argument structures are realised. these structures are based on relations of subjective causality, a coherence relation that is particularly likely to be driven by contextual signals, including the genre (e.g. @xcite . for example, for a reader of a newspaper editorial these argumentative structures will be much more expected than for one of a novel or monograph. our study compares the predicted presence or absence of discourse connectives in arguments taken from new york times (nyt) editorials. due to the uid principle we hypothesise that, in genres with predictable argumentative structures, such as editorials, there is a lower likelihood of making a relation explicit with a connective. we also assume that an lm finetuned with data from such genres is likely to show a stronger effect of this tendency. to test our hypothesis, we compare baseline (non-finetuned) masked language models (mlms) with the corresponding finetuned models genre-adapted to editorials. the comparison of models enables us to disregard frequency effects. in this way, the approach allows to verify genre-induced expectations for argument realisation and produces insights which could in the future improve crossdomain discourse processing. ## methodology our method involves comparing explicit to nonexplicit realisations of the claim-premise relation in a set of arguments. we use different lms to quantify the acceptability of the presence of a discourse connective as probabilities of the masked-tokens. these can be seen as a placeholder for the realisations in mlms. the bidirectional architecture of these models enables the prediction of token probabilities based on both adus (claim and premise). this prediction is dependent on the training data of an lm. to adapt the model to a genre of argumentative texts, we finetuned the lm on additional nyt editorials which are not part of our annotated data, which enables us to compare finetuned with non-finetuned models. ## discussion the lms have shown a clear preference for the absence of a discourse connective, which overall confirms a characteristic of argument structures, i.e. their subjective causal relations, in line with the psycholinguistic background to our approach (h1). also, in line with our expectations, after finetuning, the lms both showed a decreased probability for an explicit connective compared to the baseline ones (h2). however, only bert, but not roberta, shows a clear, i.e., statistically significant, increase. we believe the fact that we find a significant finetuning effect for bert but not for roberta is a true effect of genre (h3): the baseline version of bert was trained on less argumentative texts (specifically, books and wikipedia only) compared to roberta, which also includes news and websites in its training data. the increase in the "argumentativeness" of genres from baseline to finetuned is thus higher for bert than for roberta, which has already seen many argumentative texts before having been finetuned. for roberta, then, the finetuning effect is less pronounced. this difference that we observed is in line with the assumption that genre does create a bias for the realisation of argument structure in discourse. several effects we have presented suggest that the approach covers the use of connectives and genre conditions, as far as they are identifiable for an lm, reasonably well. the fact that the absence of a connective is highly likely across all models (h1) is likely a frequency effect. however, we were able to disregard this effect by focusing on the comparison of baseline and finetuned versions (h3), since it applies to both equally. our modelling also showed that, in line with expectations, absence of a connective becomes overall more likely with increased sentence length (number of tokens) -a finding which suggests that length is not only a control variable, in the sense of reflecting the complexity of the pairing of claim and premise. the effect of length also suggests that there are other features relevant for the explicitness of an argument, and their presence will become more likely the longer an argument gets. for example, other markers known to typically connect an argument's second constituent are features at the sentence beginning, the so-called "theme zone", such as adjuncts or demonstrative expressions @xcite . in general, a clear effect of overall length of the discourse units confirms the relevance of information density for the use of connectives. our results also indicate that argument mining could profit from genre generalisations. so far, approaches are typically developed and trained using data either from one genre (e.g., persuasive essays in stab and gurevych, 2017) or mixed-genre corpora with no systematic cross-genre transfer (e.g., @xcite . in the former case, while high accuracy is often achieved within the same genre, the transfer to another genre usually weakens the results. in the latter case, the approach often achieves moderate accuracy across genres without excelling in any specific genre. this indicates a general oversight of genre as a systematic factor in current methodologies. however, genre generalisations are crucial for dealing with a potential problem of lms when dealing with ar-gumentative discourse: the genre-specific use of explicit marking may lead lms to learn only the markings used within the genre(s) available in the training data, thereby possibly overlooking or neglecting other patterns. ## conclusion and outlook in this study, we have presented a method for testing genre bias in lms, and we have shown that discourse expectations as driven by the genre have an impact on the explicit linking of adus by way of discourse connectives. we used two discourse processing principles -forewarning and uid -to account for a general preference for the absence of a connective. testing this preference in the form of fill-mask probabilities of our lms enabled us to identify an expected genre bias after finetuning. even if the computational approach piloted with this work is not without its limitations -being based on a very small data set and focusing solely on single-word connectives while excluding other discourse markers -it successfully quantifies the influence of genre on discoursestructure realisation. in that sense, the method can serve as a role model for investigating genre effects. however, most argumentative discourse will contain many other cues for realising argumentation, which aligns with the identified effect of argument length. extending the approach to multi-word connectives, to combinations of connectives and punctuation, or to more complex "alternative lexicalizations" that equally express coherence relations @xcite would therefore be a promising endeavour. in addition, from a computational standpoint, it would be beneficial to apply our approach to other lms, particularly considering that only bert, not roberta, incorporates next-sentence prediction. overall, this work shows that both crossdomain discourse processing and argument mining can benefit from genre generalisations. while recent work in argument mining has aimed at making lms less genre-dependent by way of using connectives @xcite , our approach highlights a method of revealing genre bias in the use of connectives and could thus be a template for future, more genredependent work. a more details on finetuning hyperparameter value mlm probability 0.15 batch size 32 learning rate 0.00002 weight decay 0.01