Contrapositive Local Class Inference (2021)

Abstract: Certain types of classification problems may be performed at multiple levels of granularity; for example, we might want to know the sentiment polarity of a document or a sentence, or a phrase. Often, the prediction at a greater-context (e.g., sentences or paragraphs) may be informative for a more localized prediction at a smaller semantic unit (e.g., words or phrases). However, directly inferring the most salient local features from the global prediction may overlook the semantics of this relationship. This work argues that inference along the contraposition relationship of the local prediction and the corresponding global prediction makes an inference framework that is more accurate and robust to noise. We show how this contraposition framework can be implemented as a transfer function that rewrites a greater-context from one class to another and demonstrate how an appropriate transfer function can be trained from a noisy user-generated corpus. The experimental results validate our insight that the proposed contrapositive framework outperforms the alternative approaches on resource-constrained problem domains.

## introduction many nlp applications analyze a piece of text at multiple levels. for example, a product review might be analyzed for whether it is informative overall; its paragraphs might be analyzed for whether they are relevant to certain aspect of the product; or its words might be analyzed for whether they express intense emotions. these classification tasks of varying scopes of context are often related. for instance, one might posit that a review containing many words carrying extreme emotions might not be very informative. in some cases, the same class prediction task (e.g., sentiment polarity) might be asked at both the global and local levels. classification at the more global scope depends on clues gleaned at the local level; however, classification at the local scope is often more difficult because the roles of the words and phrases vary depending on the broader context in which they are used. therefore, a straightforward approach, such as lexicon lookup, which require domainspecific dictionaries, may not correctly identify the intended usage. while direct supervision may better take the context into account, it relies on the availability of the training corpus containing class labels for each local scope. this poses a significant bottleneck for new applications for which these resources are not widely available. to address this problem, we exploit the relationship between the global and local classification problems and propose a training framework to infer local predictions from the corresponding global label, which may be easier to obtain. for example, suppose someone reported a social media comment as inappropriate (global classification), if we find out which word(s) in the comment are contributed the most to the user's decision to report the comment, we already found the inappropriate words (local classification) and confirms whether the sentence is correctly reported or not. in prior work, researchers observed that local predictions might serve as rationales @xcite @xcite for the global problem. for example, if a review is classified as positive, there must be some words within that review that are also positive such that they serve as the rationale for the overall prediction. thus, if rationales can be identified for the global classification task, their local labels would be inferred to be the same as the global label. we refer to this mode of local prediction as direct inference. however, prior work also suggested that even modern global classifiers are still sensitive to noise, so such direct inference may fail to identify the most semantically relevant rationales. for ex-ample, irrelevant words or punctuation marks are more influential to the decision of neural text classifiers than verbs or other semantically related textual units @xcite ; or the presence of snow in an image is the main feature to distinguish huskies from wolves rather than the features related to the animal themselves @xcite . in this work, we argue that this semantic relationship can be made more robust by enforcing the contrapositive constraint between the local prediction and its corresponding global prediction. that is, suppose we know that an instance is globally predicted to belong to some class a and that some local portion l contributed the most to that prediction (the rationale); if l is replaced so as to negate its semantic contribution, then the global label should also change (e.g. now belongs to some class b). we propose that, if, and only if, the contrapositive constraint is satisfied should the local prediction be inferred from the global problem. modeled after style transfer and controlled text generation methods, we propose to implement the contrapositive inference scheme as a transfer function for global class transference. we first cast the problem as a rewriting exercise: rewrite the original global instance, which belong to some class a, so that it becomes more likely to belong to class b; then, those local textual parts that were changed in the rewriting are likely to be the heavy contributor to class a so we may infer the same class prediction for them. to train an appropriate transfer function, however, we need a corpus of training examples for the global prediction. for some problem domains, these resources are already available. for low-resource problem domains, where such annotated corpora are not available, we show how domain-inspired textual data augmentation can facilitate the training of the transfer function. the proposed prediction framework is evaluated on several problem domains: sentiment analysis (as a resource-rich problem domain), and semantic pleonasm detection and specificity detection (as two resource-constrained domains). results validate our insights about inferring local class labels from their corresponding global prediction labels and our proposed approach significantly outperform the alternative methods. we also demonstrate the robustness of the contrapositive local prediction to the noisy data and show that an appropriate transfer functions can be trained from corpora generated from heuristic-driven data augmentation schemes. ## inference by contraposition for many local prediction tasks, there is a corresponding, often easier to learn, global prediction task. global prediction is relatively easier, in part, because it is attempting to classify a greatercontext, which is semantically more distinctive than a localized smaller-context. moreover, when a classification task could be performed at multiple levels of granularity (e.g., paragraphs and sentences), there is a much higher chance of having training corpora with label annotation at a larger text span than a smaller text span, which makes the more global version of the task more feasible. the semantic purport of a greater-context is deriving from the co-occurrence of smaller semantic units. this implies that there is a semantic relation between the local and global predictions that could be used to infer the harder-to-learn local prediction from the corresponding easier-to-learn global prediction. it must be noted that prediction inference from the global prediction is only feasible for significantly contributing local features. we define the direct inference of local prediction from the global prediction as follows: @xmath0: if the global prediction for a greater-context be some class a, there exists a smaller local portion that significantly influenced the global prediction in the first place so the local prediction for it could infer the same class label as the global prediction. however, it is reported that the classifiers might sometimes fail to learn the most semantically related features and make predictions based on just salient ones @xcite @xcite . we believe adding an extra constraint to the inference, while keeping the relationship between the local and corresponding global prediction intact, can improve the identification of the semantically related local features, therefore, we introduce the contrapositive inference of local prediction from the global prediction as follows: in the next section, we describe how this inference scheme could be applied to the discrete predictions in nlp problems. ## transfer function model since our approach is based on the difference between original and generated greater-contexts, extensive or random modification of the original text might result in meaningless differences, so the key requirements of an ideal transfer function for our approach are: any reasonable transfer function that satisfies these requirements may make a suitable model to serve as the core of our proposed approach. as an implementational choice, we adapt the model proposed by @xcite and augmented it with an extra regularization (i.e., conciseness loss, given in equation 8 ) to explicitly control for req. 1. our transfer function model incorporates an autoencoder and a discriminator for the global prediction. the autoencoder is initially trained to learn a latent space (z) from the input greater-context instances (c) by minimizing the reconstruction loss, given in equation ( 6 ), where θ e and θ g are the encoder and decoder parameters, respectively. since the latent space z is learning independent of the class labels (q e (z|c)), it could be used to generate instances of both classes. therefore, the decoder is trained to learn to generate a greatercontext ĉ, given a desired class label (p(ĉ|z, y)). now, if we ask the encoder to generate a rewriting alternative for an input greater-context with a flipped class label (¬@xmath1, then the generated output would likely be a modified version of the input with some small changes that make it belong to the opposite class (req. 2). however, in order to learn class transference, the decoder also requires a signal to determine how likely the generated output belongs to the desired class. to provide that signal, we incorporate a class discriminator in our transfer function model, which is optimized as given in equation 7 , where θ d is the discriminator parameter. for simplicity, we only train the discriminator on the input examples (c) and use that to predict the label for generated examples (ĉ). by optimizing for the reconstruction loss, the autoencoder will learn to generate an output similar to the input, which, in part, may satisfy req. 1. we still want to directly enforce the class transference using minimal changes, which intuitively means the generated transferred version of the input greater-context that belongs to some class b should be as similar as possible to the input greatercontext that originally belongs to some class a. therefore, we also optimize for a conciseness loss (l con ) as given in equation 8, where c is the original input greater-context and ĉ is the transferred version of it to the other class. it must be noted that, sometimes, the original input and the transferred output might not have the same dimension. for these cases, we slice the bigger vector as the dimension of the smaller vector. @xcite ] (8) equation ( 9 for some initial training runs, we train the discriminator and autoencoder independently (λ @xmath2; the discriminator will learn to distinguish between classes, and the autoencoder will learn a latent space that could be used to regenerate instances of both classes. at this step, we do not enforce the minimal changes as well (λ @xmath3. after that, for a few more epochs, we jointly train the autoencoder and discriminator (λ @xmath4. for more details about the hyperparameters and the training process please see appendix a. it must be noted that to avoid the fading gradient phenomenon when training on discrete space, the model feeds with gumbel-softmax @xcite representation over the words to provide a continuous approximation and a stronger signal for optimizing the decoder. ## experiments this work argues that inference by contraposition allows for the identification of semantically relevant local features and thus a more accurate local prediction inference when compared to methods based on direct inference. in order to evaluate this hypothesis, we setup an experiment for local prediction inference at word-level from a large training corpus of naturally-occurring and manually-labeled sentences in the sentiment analysis domain (section 4.1). we also argue that the contrapositive inference scheme can be more robustly trained on usergenerated and weakly-labeled data compared to the alternative direct inference methods. in order to validate this aspect of the contrapositive local inference, we conducted a few experiments for local prediction inference from augmented training corpora of the global prediction in the sentiment analysis, pleonasm detection, and specificity detection problem domains (section 4.2). in these set of experiments, we compare two implementation of each inference schemes (a simpler and a more complex models), as follows: direct:atn. for the global prediction, we use a bilstm classifier and wrap the weights at each time-step into an attention weight using a multiplicative attention layer @xcite . the local feature with the highest attention weight would receive the same class prediction as the global class label. to train our transfer function, we use the 105k naturally-occurring polarity-labeled sentences (positive and negative) of the ypd that contain 30 or fewer words (refer to as ypd-train). as a benchmark dataset to evaluate the local inference performance of the models, we manually annotated some held-out sentences of ypd with wordlevel sentiment labels. we followed the same annotation scheme proposed by @xcite and created a unigram pool from all the words of the test set; then randomly picked a word and annotated it as very positive, positive, neutral, negative, or very negative. finally, filtering for sentences that contains a very positive or very negative words, we collected a set of 860 held-out sentences with wordlevel labels as our in-house benchmark dataset (refer to as ypd-test). ## discussion results from our studies suggest that enforcing the contraposition constraint between the local prediction and its corresponding global prediction can reveal their semantic relationship more robustly, and thus, lead to a a more accurate local class prediction inference, compared to direct inference alternatives. in the direct inference schemes, there are many cases that semantically irrelevant words, such as "punctuation marks", are the heavy contributors to the global prediction. these observations and the performance of the direct:atn baseline confirm the prior work's suggestions that attention weight may not correlate with the semantics of the prediction task. in addition, the direct:rationale baseline operates through finding a smaller local text span (rationale) that can replace the greater-context in the global prediction. intuitively, rationales with shorter text span (e.g., words or phrases) are likely to provide a noisy and unstable solution for the global problem, as we observed in our experiments. the contrapositive:loo baseline adopts a simple approach to apply a relaxed version of the contraposition constraint, by removing the local rationale from the global problem's context. as ob-served, a global context, after removing a small text span from it, would still be large and semantically expressive enough for a reliable global prediction. since our proposed contrapositive:tf approach operates by replacing a local rationale of a global context with another (adversarial) rationale, the span size of the greater-context remains relatively intact, which makes the prediction inference more robust. the transfer function also operates by disentangling the semantics from the surface representation and applying local contrapositive semantic perturbation to the global context, which makes it more robust to noise and irrelevant local features. for example, a punctuation mark might influence a class prediction for a sentence, however, replacing it with another punctuation mark will not make the sentence belong to another class, so it is not a semantically contributing word. furthermore, evaluation results suggest that the performance impact of training a transfer function on weakly-labeled user-generated data is not significant, so data augmentation may facilitate the training of the transfer function and make our approach applicable to a variety of resource-constrained nlp problem domains. however, in our experiments, we observed that the quality of augmented examples is a key-factor for training an appropriate model. we found that both global and local classification performance declined significantly when we trained models on the corpora that are augmented with simple heuristics, which generate obvious examples for different classes. this should be noted that our proposed contrapositive approach is only applicable to prediction tasks that can be performed in multiple levels of granularity, where the local prediction has a corresponding global prediction task. in addition, applying it to more general cases, such as "multi-class" classification, might pose some challenges. ## conclusion this paper presents a local contrapositive inference scheme that is only informed by corresponding global class predictions at a greater contexts. the inference scheme is implemented as a transfer function that learns to transform a context from one class to another. training such a transfer function, requires a large corpus of labeled sentences, which may not be available for many low-resource problem domains. our work demonstrates the robustness of the proposed approach when coupled with appropriate data augmentation methods and its applicability as a solution to resource-constrained local prediction tasks.