A Weakly-Supervised Learning Approach to the Identification of “Alternative Lexicalizations” in Shallow Discourse Parsing (2023)

Abstract: Recently, the identification of free connective phrases as signals for discourse relations has received new attention with the introduction of statistical models for their automatic extraction. The limited amount of annotations makes it still challenging to develop well-performing models. In our work, we want to overcome this limitation with semi-supervised learning from unlabeled news texts. We implement a self-supervised sequence labeling approach and filter its predictions by a second model trained to disambiguate signal candidates. With our novel model design, we report state-of-the-art results and in addition, achieve an average improvement of about 5% for both exactly and partially matched alternativelylexicalized discourse signals due to weak supervision.

## introduction understanding the underlying structure of a text is a fundamental problem in computational linguistics. in discourse analysis, shallow discourse parsing in particular, we aim to identify individual discourse relations within a text. thus we can gain information that helps in downstream tasks such as automatic summarization, machine translation, and document classification. the study of connecting phrases not only helps in understanding the way people connect their thoughts but also in the identification of discourse relations anchored by them. for our work, we use the third version of the penn discourse treebank (pdtb) @xcite that distinguishes between explicit relations (signaled by a closed set of discourse connectives, e.g because, and, if-then, and before) and alternative lexicalizations (signaled by connecting phrases other than discourse connectives, e.g. this means, for that reason, and it all adds up to). in total, the pdtb contains 25878 signaled relations, most of which belong to the group of explicit relations (94%). only 1638 connecting phrases build the group of free connective phrases, in the corpus referred to as alternative lexicalizations. while explicit relations are more commonly used to verbalize expansions and comparisons between text spans, alternative lexicalizations often point to lexically grounded causal relations. also, they potentially contain information, e.g. the phrase the most crucial reason for that gives also evidence about the reason's importance, which is useful for understanding the full discourse. in our work, we aim to overcome the problem of very limited training data available for free connective phrases and examine a weakly-supervised scenario for continuously improving a model through its own predictions. we regularize these predictions by re-ranking the extracted signals through a separate model trained to discriminate possible signal candidates into signals with or without discourse usage. summarized, our contributions are: we (i) present a novel architecture and provide state-of-the-art results for recognizing alternative lexicalizations in the recent version of the pdtb. further, we (ii) improve its performance of recognizing phrases by integrating unlabeled data into the training process using weak supervision. ## related work self-supervised learning @xcite , the most simple semi-supervised learning algorithm, extends its training data by adding new samples with confident predictions on different data. selftraining has been successfully applied on constituent parsing @xcite by incorporating a re-ranking strategy @xcite to improve parsing results and reduce the bias of the trained model. also, @xcite improved performance on part-of-speech tagging via sequence labeling. in recent work, nishida and matsumoto (2022) study the empirical effectiveness of bootstrapping annotations from out-of-domain data and show its positive impact for bert-based discourse dependency parsers. for candidates selection, they study criteria inspired by @xcite . @xcite approach semi-supervised learning for named entity recognition, a similar training problem (sequence labeling) as ours. they propose an additional model for estimating confidence (self-testing) and removing samples with low scores. @xcite first apply semisupervised learning to rst discourse parsing using multiple views on the data by incorporating various auxiliary tasks, such as pdtb discourse parsing. @xcite improved their argument extraction by jointly training three separate models so-called tri-training on additional news documents. recently, @xcite successfully bootstrapped rst sub-trees using a combination of simpler feature-based teachers to train a more complex neural student. the group of alternative lexicalized relations has been rarely studied. @xcite did initial work on the identification and analysis of alternative lexicalized relations in an older pdtb version. @xcite ; rysová and rysová (2015) distinguished two classes of alternative lexicalizations and developed a dictionary approach for more regular alternative lexicalized phrases. most recently, @xcite implemented the first automatic neural-based model for recognizing alternative lexicalizations on a sentence level using a binary sequence labeling approach. in this work, we build on their approach and adapt this model to the paragraph level, similar to the explicit connective model of @xcite . ## discussion and conclusions we developed a new paragraph-based architecture to extract alternatively-lexicalized discourse signals and presented state-of-the-art performance. initial experiments on incorporating non-annotated data showed a further increase in performance. size seems to matter for this learning too, as this principle often holds for deep learning models. although the gaps are rather small for up to 10,000 sampled documents, we think the distance for the largest set of documents is very clear. due to time and computation constraints, we could not identify an upper performance bound yet. we notice throughout our signal extraction experiments a confusion between alternative lexicalizations and explicit connectives. we assume the model to have problems clearly understanding their difference, as both kinds of phrases signal discourse relations. filtering the connecting phrases as we have done seems unavoidable. contrary to this, however, it seems worthwhile to soften the boundaries between these two categories and develop models that combine both types. this is not trivial due to the differences between both signal types (explicit signals are usually shorter recurring phrases with higher frequencies; altlex signals tend to be longer phrases with more variance).