A Side-by-side Comparison of Transformers for Implicit Discourse Relation Classification (2023)

Abstract: Though discourse parsing can help multiple NLP fields, there has been no wide language model search done on implicit discourse relation classification. This hinders researchers from fully utilizing public-available models in discourse analysis. This work is a straightforward, fine-tuned discourse performance comparison of 7 pre-trained language models. We use PDTB-3, a popular discourse relation annotated dataset. Through our model search, we raise SOTA to 0.671 ACC and obtain novel observations. Some are contrary to what has been reported before (Shi and Demberg, 2019b), that sentence-level pre-training objectives (NSP, SBO, SOP) generally fail to produce the best-performing model for implicit discourse relation classification. Counterintuitively, similar-sized PLMs with MLM and full attention led to better performance. Our code is publicly released.

## introduction an utterance has multiple dimensions of meaning. discourse relation classification identifies one such dimension: the coherence relation between clauses or sentences arising from low-level textual cues @xcite . this makes the task important to several nlp fields, including multi-party dialogue analysis @xcite , social media postings analysis @xcite , and student literary writing analysis @xcite . a discourse relation is often marked with explicit connectives such as but, because, and. ## background the pre-train and fine-tune paradigm have been led by the remarkable downstream task performances of pre-trained language models @xcite . for several nlp tasks, a pre-trained language model could have likely done a fine job at learning syntax, semantics, and world knowledge -given enough data and model size @xcite . a pre-trained language model's competence in discourse was questionable until @xcite proposed that bert's pre-training objective can benefit implicit discourse relation classification. however, @xcite hints that bert is not the language model best suited to the task. implicit discourse relation classification is an active area of research @xcite @xcite @xcite @xcite @xcite @xcite . however, there has been no widerange model study on implicit discourse relation classification, limiting a researcher's scope of model choice. this issue is further complicated by the fact that discourse task performances do not always correlate with popular semantics-based natural language understanding (nlu) scores, such as glue @xcite . thus, it is difficult to predict which language model can perform well without a dedicated empirical exploration. with the a version update to penn discourse tree bank (pdtb-3) @xcite and the correspondingly updated evaluation method @xcite , we fine-tune seven language models to implicit discourse relation classification. the chosen language models are: roberta large @xcite , albert large @xcite , bigbird-roberta large @xcite , bart large @xcite , longformer large @xcite , spanbert large @xcite , deberta large @xcite . these models are selected with diversity in mind, especially in terms of input sequence length, attention type, and pre-train objectives. these models follow the popular transformer architecture @xcite , and we will not review each model in detail. a brief comparison is shown in ## conclusion researchers often build or modify a neural network to improve task performance. while such effort is essential, this paper shows that sota can also be raised through extensive search and application of existing resources. through a side-by-side comparison of seven plms, we also make handy observations on pre-training objectives, long-document modifications, and full-sentence setups. though some might consider these phenomena rather expected, nothing is scientifically conclusive until an analysis is performed at an adequate scale. we hope that our report helps researchers working towards discourse understanding, and we continue to discuss the missing details in the appendices.