Projecting Annotations for Discourse Relations: Connective Identification for Low-Resource Languages (2024)

Abstract: We present a pipeline for multi-lingual Shallow Discourse Parsing. The pipeline exploits Machine Translation and Word Alignment, by translating any incoming non-English input text into English, applying an English discourse parser, and projecting the found relations onto the original input text through word alignments. While the purpose of the pipeline is to provide rudimentary discourse relation annotations for low-resource languages, in order to get an idea of performance, we evaluate it on the sub-task of discourse connective identification for several languages for which gold data are available. We experiment with different setups of our modular pipeline architecture and analyze intermediate results. Our code is made available on GitHub.

## introduction uncovering coherence relations in texts, also referred to as discourse parsing, is a complex task. it is comparably difficult and time-consuming for humans to annotate such relations, and as a result, relatively little training data is available for machines to train a system on. most of this data is in english. although recent shared tasks @xcite @xcite have had a strong multilingual focus and included up to 13 different languages, there is still a large variety of languages that are seriously under-resourced when it comes to research on discourse and coherence. in this paper, we attempt to address this issue by presenting an end-to-end, multi-lingual discourse parser. our parser essentially consists of a processing pipeline that exploits machine translation, an english discourse parser, and a word aligner, to project discourse relation annotations onto any non-english input text, without a need for any languagespecific training data. the goal of our pipeline is to kick-start the annotation of discourse relations in languages for which little to no resources are avail-able, or to provide rudimentary discourse relation annotations for downstream applications where accuracy is not the main concern. to get an idea of performance, we experiment with various different configurations of our modular architecture and evaluate on the sub-task of connective identification. we compare our results against a lexicon-based baseline that needs no training data either, and a state-of-the-art connective identification system trained specifically on the language and domain. our pipeline mostly outperforms the lexicon-based baseline, by a factor of up to 2.7, and while a system specifically trained on the task outperforms our pipeline for all languages and corpora for which training data is available, we retain up to 81% of performance for some of those corpora. we analyze the (intermediate) results from different system configurations, in order to investigate which components of our processing pipeline are the most error-prone. we hope that our system proves to be a useful tool for researchers working on automated approaches to shallow discourse parsing for languages for which little to no gold data is available. the rest of this paper is organized as follows: section 2 discusses related work, focusing mainly on discourse parsing. section 3 explains our system architecture. section 4 presents the results, which are discussed in section 5. finally, section 6 sums up our main contributions and discusses directions for future work. ## related work in 2015 and 2016, two consecutive conll shared tasks @xcite caused a spark in interest in the discourse parsing task. the 2015 iteration worked with english only, the 2016 iteration was multi-lingual by adding chinese. both followed the shallow discourse parsing paradigm proposed by the penn discourse treebank (pdtb, @xcite ). this approach is often referred to as shallow discourse parsing since contrary to other discourse parsing frameworks such as rhetorical structure theory (rst, @xcite ) or segmented discourse representation theory (sdrt, @xcite ), it makes no commitment to overall text structure, and deals with coherence relations on a local level. pdtb parsing is often done in end-to-end fashion, with plain text as input and a list of discourse relations as output, where each relation consists of a relation type (explicit, implicit, alternative lexicalization or other), arguments and relation sense. since the introduction of a pipeline architecture by @xcite , many systems adopted this setup @xcite @xcite . the majority of systems work on english, with some systems focusing on chinese @xcite @xcite . beyond that, to the best of our knowledge, the only other supported language for end-to-end parsing is german @xcite . with a series of shared tasks, the discourse relation parsing and treebanking workshops (disrpt, @xcite ; @xcite ) strongly encouraged a multi-lingual approach and moreover, attempted to converge work on different parsing paradigms, by including corpora following the annotation guidelines from both the pdtb, rst and sdrt. the shared task setup moved away from an end-to-end approach, and system submissions @xcite @xcite focused on particular sub-tasks. our contribution aims to enable discourse parsing for an even larger variety of languages, without the need for any language-specific discourse annotations. we hope that this opens up research into discourse parsing for seriously under-resourced languages. we integrate the end-to-end pdtb parser from @xcite , but in principle, an end-to-end rst parser @xcite @xcite could be plugged in as well. the components we implemented for both machine translation and word alignment were mostly selected because of their user-friendly apis. however, our system architecture is modular by design, and systems focusing on particular, low-resource languages can easily be plugged in. a good example for machine translation is presented by @xcite , whereas good examples for word alignment are provided by @xcite ; @xcite . using annotation projection for (sub-tasks of) discourse parsing is not novel. @xcite use annotation projection from english to french on a parallel corpus (europarl) and improve f1 score for discourse connective identification in french by 15 points. sluyter-gäthje et al. (2020) employ machine translation in combination with word alignment, in order to create a german corpus automatically annotated for discourse relations. however, in contrast to @xcite , we include machine translation and thereby dynamically enable discourse parsing for any language. in contrast to sluyter-gäthje et al. (2020), we focus on the pipeline itself and make that available, instead of focusing on curating and publishing the output of the process (e.g., a corpus annotated for discourse relations in a particular language). ## pipeline architecture the following subsections explain the three different components of our pipeline to annotate any non-english text with discourse relations, following the pdtb framework. we use a modular setup, such that individual components can be swapped out for alternatives that perform better for particular languages or domains. the system architecture is illustrated in figure 1 . the rounded boxes on the right depict the individual modules. the listed components are the ones we implemented, but for every module, additional components can easily be integrated. for machine translation, using a custom model, trained specifically for a low-resource language (pair) can improve performance. for the discourse parsing module, relevant alternatives that work end-to-end can be integrated. for word alignment, tools that can be trained on or tuned for specific language pairs might return better results. see section 2 for some suggestions. as long as these components accept and return input/output in the same format, they can easily be interchanged. a detailed description of the modules and the components that we integrated into our pipeline is provided in the following subsections. ## results our pipeline is specifically targeted at low-resource languages for which no discourse relation annotations exist at all. however, without any gold data, we cannot get any idea of performance of our setup. so, in order to assess this across various languages and domains, we use the pdtb-style corpora featured in the 2023 disrpt shared task @xcite as gold data to evaluate our pipeline. the shared task includes two sub-tasks, one focusing on segmentation and another focusing on relation sense classification. for pdtb-style corpora, the segmentation task is essentially about identifying connectives. the sense classification task assumes gold annotations for connective and relation arguments. while our pipeline returns discourse relations, fully specified with a connective, arguments and a relation sense, we decided to evaluate only on the segmentation task, i.e., connective identification, for now, as there is significant room for error propagation in our pipeline and we first want to get an idea of performance on the most upstream and comparably simpler task. the segmentation sub-task for pdtb-style corpora includes five (non-english) languages (italian, portuguese, turkish, thai and chinese), distributed over seven different corpora. an overview is included in in the task setup, the participants were provided with a train, dev and test set. systems could therefore be trained and tuned for the relevant language using the train and dev sets. since we do not train our system in any way for a particular language or domain, we do not expect to match performance of the trained systems that participated in the task, but for comparison, we do include results for dis-cut @xcite , as this is the only system that submitted results for connective identification in the plain track, a setup that most resembles ours. we consider this the upper-bound of expected performance. to compare against a reasonable baseline that also does not require any pre-training and is aimed at low-resource/no-resource languages, we use a lexicon-based approach. this comprises simple pattern-matching using connective lexicons bundled on a dedicated platform 3 . lexicons for all evaluation languages except thai are available on this platform. similarly, because deepl does not support thai, the corresponding results are not included either. for all (other) corpora, we experiment with different configurations for the individual modules and their integrated components. we calculate precision, recall and f1 scores for all corpora, based on the *.test.conllu files from the shared task foot_3 . since our system needs no training data, we could in principle evaluate against all available data (including train and dev sets), but to make a direct comparison to discut's performance possible, we evaluate on the test sets only. the results are included in ## discussion as illustrated by the performance of discut, with f1 scores generally in the 80s to 90s, given the availability of training data, identifying connectives is relatively easy, at least when compared to other subtasks in discourse parsing. we included the lexiconand pattern-matching-based baseline, performing considerably worse, to indicate performance when no training data is used at all, since this much more resembles the targeted application scenario of our pipeline. the mid section of the following sections discuss the influence of different system configurations with regard to machine translation and word alignment. ## conclusion & future work we present a multi-lingual shallow discourse parsing pipeline that makes use of machine translation, an english discourse parser and word alignment to project annotations onto the original, non-english input text. we specifically aim to support lowresource scenarios and make rudimentary discourse parsing possible for languages without any available training data, since our pipeline needs no training data at all. our code is made available online. 5 we evaluate our approach on the sub-task of connective identification and compare different configurations of our pipeline to a lexicon-based baseline, and to a system specifically designed for the task and trained on in-language, in-domain data. our system outperforms the baseline in most cases, and for individual corpora improves f1 score by a factor of 2.7. we find that a trained system still performs considerably better, but for the best-scoring corpus, we retain 81% of the upper-bound f1 score. in our current architecture, translation is done sentence-by-sentence, so as to keep sentences aligned for better word alignment performance. we consider more context-aware translation @xcite the most important piece of future work. in addition, further investigation of error propagation, as well as the effect of domain transfer, are promising venues for future work. in this paper, we evaluate our approach on the sub-task of connective identification only. our pipeline returns fully specified relations (with a type, arguments and relation sense), and we leave it to future work to evaluate on more than just connective identification. relevant related work in this respect is represented by @xcite , who work on implicit relation classification without exploiting any (language-specific) training data, and we consider it an important next step to experiment with zero-shot transfer @xcite for other sub-tasks of discourse parsing. our system architecture is modular by design, with relatively common exchange formats (pharaoh for word alignments, pdtb-style json for discourse relations) across modules, and where individual components fine-tuned to a particular language are available, these can easily plugged in. furthermore, our current architecture includes only a pdtb parser and another possible extension is the integration of rst parsers.