CVAE -based Re-anchoring for Implicit Discourse Relation Classification (2021)

Abstract: Training implicit discourse relation classifiers suffers from data sparsity. Variational AutoEncoder (VAE) appears to be the proper solution. It is because ideally VAE is capable of generating inexhaustible varying samples, and this facilitates selective data augmentation. However, our experiments show that coupling VAE with the RoBERTa-based classifier results in severe performance degradation. We ascribe the unusual phenomenon to erroneous sampling that would happen when VAE pursued variations. To overcome the problem, we develop a re-anchoring strategy, where Conditional VAE (CVAE) is used for estimating the risk of erroneous sampling, and meanwhile migrating the anchor to reduce the risk. The test results on PDTB v2.0 illustrate that, compared to the RoBERTa-based baseline, re-anchoring yields substantial improvements. Besides, we observe that re-anchoring can cooperate with other auxiliary strategies (transfer learning and interactive attention mechanism) to further improve the baseline, obtaining the F-scores of about 55%, 63%, 80% and 44% for the four main relation types (Comparison, Contingency, Expansion, Temporality) in the binary classification (Yes/No) scenario.

## introduction implicit discourse relation classification is a task of determining relationships between arguments without connectives. we provide an example in appendix a. due to the omission of connectives @xcite , classifying relations heavily relies on recognizable representations of arguments. learning richer and diverse linguistic phenomena from a large number of samples (relation-aware argument pairs) helps to enhance encoding, producing more recognizable representations @xcite . however, there is lack of labeled data for learning. to overcome the bottleneck, previous studies have explored two classes of methods. some of them conducted data expansion using pdtb explicit samples @xcite @xcite and parallel corpora (wu * corresponding author et @xcite @xcite . others dug deeper into the existing data (instead of expanding it) to squeeze out additional salient features, where implicit connectives are speculated and annotated, and predicting them by machine is used as a supplementary task in a multi-task learning architecture @xcite . in this paper, we attempt to enhance representation learning without using any external resources or artificially-created implicit connectives. we couple vae @xcite with roberta @xcite and mlp to build a three-stage encoder community (figure 1 ). it is inspired by the ability of vae in generating variants (section 2), and more importantly, the variants cover a wider range of linguistic phenomena. specially, we utilize vae to generate numerous variants for initial representations of arguments, and use them to challenge both roberta and mlp (section 3). ideally, this helps to make the encoder community generalize well. however, the use of vae is proven ineffective in our experiments. it performs much worse than the less-sophisticated model that simply couples roberta with mlp. data analysis shows that the main drawback is caused by erroneous sampling. the errors occur when vae tends to produce quite unusual variants (section 4). to address the issue, we propose a re-anchoring strategy (section 5) to migrate potential variants from high-risk sampling areas to low-risk. instead of vae, cvae (sohn strongly-associated sampling area weakly-associated sampling area strongly-associated sampling area weakly-associated sampling area strongly-associated sampling area weakly-associated sampling area sample space anchor x (mean) ( m s e ) et al., 2015) is used for re-anchoring. we experiment on pdtb v2.0 @xcite . experimental results (section 6) show that re-anchoring yields significant performance advantages. more importantly, it is proven that the cooperation between re-anchoring and other auxiliary strategies (transfer learning and interactive attention mechanism) yields further improvements. ## variational autoencoder (vae) vae is an encoder which produces the hidden variable @xmath0, ...,z n } (z @xmath1, ...,x n } (x @xmath2. the variable is obtained by random sampling, from the finite sampling area where all samples are distributed with the posterior probability p(z|x). during training, vae plays an adversarial game with a decoder as below. vae originally prefers to sample the variables zs that are different from x. therefore, it computes p(z|x) to approximate less-associated probability distributions with x. however, the decoder tends to completely reconstruct x using zs. therefore, it requests vae to compromise, sampling similar zs by approximating strongly-associated distributions with x. as a result, vae learns to generate various different-butsimilar representations zs for x. all in all, vae regards the input representation x as an anchor in the sample space, and estimates the sampling area around the anchor where, as shown in graph (a) in figure 2 , some samples are of strongly-associated distributions, others weakly-associated. in our experiments we set p(z|x) to gaussian distribution function g(z|x, µ, σ 2 ), where µ denotes the geometric mean, while σ 2 the square deviation. in this case, the samples which are distributionally similar to µ will be more easily sampled and operationalized as zs. in addition, we combine bilstm @xcite with cnn @xcite to build vae, which serves to predict µ and σ 2 conditioned on the input x. the decoder is a bilstm unit. ## three-stage encoder community as shown in figure 1 , we carry out a three-stage encoding process for arguments arg 1 and arg2. first, the arguments are fed into roberta with the standard input format: [cls]arg1[sep]arg2 @xcite . the cls embedding (cls for short) output by roberta is used as the initial representation. it contains the self-attentive information of both arguments. at the second stage, cls is input into vae. using cls as the anchor (@xmath3, vae estimates the sampling area, and conducts random sampling in the area to produce the hidden variable z. at the final stage, mlp is utilized to encode z, computing the final representation ẑ of the arguments. conditioned on ẑ, the fully-connected layer with softmax normalization predicts the implicit relation of the arguments. due to the addition of vae in the middle, the mlp encoder that operates at the subsequent stage will encounter protean representations zs of a single pair of arguments, during all the training epochs (one z per epoch). ideally, this should produce the effect of data augmentation, and thus strengthens the representation learning of the mlp encoder. however, the fact remains that vae results in performance degradation during the test (section 6). ## erroneous sampling the primary drawback of using vae for data augmentation is erroneous sampling. it is caused by the following two reasons: we suggest that erroneous sampling is most probably a common phenomenon in the study on pdtb v2.0. in appendix b, we explain the cause of the phenomenon and provide a variety of examples. ## re-anchoring by conditonal vae to relieve erroneous sampling, we develop a reanchoring strategy to migrate the anchor away from the class boundary. conditional vae (cvae) is utilized for re-anchoring. it uses the relation types as the subsidiary conditions to constrain the encoding process of vae. assume r t denotes the relation type that is held by the argument pair t. it is represented by the embedding b t , which is obtained by random initialization and element-wise accumulation with the unit vector. on the basis, given cls t that is output by roberta for t, we combine b t with cls t (by element-with accumulation) to form the input xt of vae: @xmath4. this input appears as a new anchor migrating from the original position towards the relation-type embedding b t . conditioned on this new anchor, the sampling area will be re-estimated by vae in the region near b t of r t (see graph (c) in figure 2 ). from the perspective of the spatial position in the entire sample space, the re-estimated sampling area is pulled away from the class boundaries, more or less. this reduces the risk of erroneous sampling. nevertheless, cvae cannot be directly used during test because the relation type of every pair of arguments are unavailable at the moment (viz., it is an object needs to be predicted during test instead of being used as the prior knowledge). we made a detour, driving roberta to learn re-anchoring. assume r t and rt denotes the relation of the argument pair t and other relations, respectively. both of their embeddings b t and bt are obtained by random initialization. though, to distinguish between them, b t is combined with a unit vector. ## experimentation 6.1 dataset, evaluation and settings when the encoder community is trained for relation classification, the relation embeddings are kept unchanged. roberta is finetuned, though this is independent of self-supervised learning of vae. vae is trained separately, with the goal of reconstructing the input well. we shut down the training course when the observable development performance is going to be steady. during test taking, the relation embeddings are disable. ## conclusion we develop a three-stage encoder community for implicit relation recognition. vae is used for data augmentation. in particular, we propose a cvaebased re-anchoring strategy to solve the problem of erroneous sampling. experimental results show that our method yields substantial improvements. data analysis demonstrates that a pdtb argument may be accompanied with two different arguments, bringing inconsistent relations. in the future, we will develop a context-aware adversarial model to selectively assign attention weights to the central argument, conditioned on both companions.