Ranking-Constrained Learning with Rationales for Text Classification (2022)

Abstract: We propose a novel approach that jointly utilizes the labels and elicited rationales for text classification to speed up the training of deep learning models with limited training data. We define and optimize a ranking-constrained loss function that combines cross-entropy loss with ranking losses as rationale constraints. We evaluate our proposed rationale-augmented learning approach on three human-annotated datasets, and show that our approach provides significant improvements over classification approaches that do not utilize rationales as well as other state-of-the-art rationale-augmented baselines.

## introduction text classification has been used for numerous applications including sentiment analysis (hemmatian and sohrabi, 2019), information retrieval @xcite , and language identification @xcite . when presented with a large number of labeled documents, common text classification models demonstrate impressive results. in practical settings, however, labeled data is often scarce. labeling documents is a tedious task that requires time and effort, thus curating a large labeled corpus can be expensive and even unrealistic. there is a wide range of use cases for businesses and industry that require curating a labeled dataset for the current task before the need to move on to the next task arises. for example, consider legal case document classification where documents need to be labeled as relevant/not-relevant to the current case at hand. the next legal case requires labeling the documents as relevant/not-relevant for that particular case, and so on. similarly, several fast-response tasks such as immediate analysis of news and social media posts for a breaking news, for a recently released product, for a policy announcement, etc., require fast curation of a small and yet informative labeled dataset. ## experimental setup in this section, we describe the three datasets, several baselines, and the experimental settings. ## results we first present results comparing lwr-rc with the baselines, and then discuss the effects of the two ranking-constrained losses on the performance of lwr-rc. ## limitations and future work we presented experimental results for binary classification tasks in this paper. to the best of our knowledge, prior learning-with-rationales frameworks also focused on binary classification tasks in their experiments. extending the framework to multi-class settings is a promising future direction. such an extension would require adapting the loss functions to multi-class settings and creating multi-class classification datasets with rationales. extending the framework to multi-label settings where a document can be assigned more than one label, however, is more challenging, both for formulating the problem as well as annotating the datasets with rationales, because rationales need to be assigned to their respective labels, which might be more than one in a single document. ## conclusions we presented a novel approach to incorporate rationales as ranking-constraints into the training of classification models with cross-entropy loss. the proposed approach is general enough that it can be used for simple models, such as logistic regression with one-hot encoding of documents, as well as deep learning models combined with text embeddings. we conducted empirical evaluations comparing the proposed approach to several baselines and observed that the proposed approach outperformed the baselines in most settings, and was comparable to them at the remaining settings.