Two-Step Classification using Recasted Data for Low Resource Settings (2020)

Abstract: An NLP model’s ability to reason should be independent of language. Previous works utilize Natural Language Inference (NLI) to understand the reasoning ability of models, mostly focusing on high resource languages like English. To address scarcity of data in low-resource languages such as Hindi, we use data recasting to create NLI datasets for four existing text classification datasets. Through experiments, we show that our recasted dataset is devoid of statistical irregularities and spurious patterns. We further study the consistency in predictions of the textual entailment models and propose a consistency regulariser to remove pairwise-inconsistencies in predictions. We propose a novel two-step classification method which uses textual-entailment predictions for classification task. We further improve the performance by using a joint-objective for classification and textual entailment. We therefore highlight the benefits of data recasting and improvements on classification performance using our approach with supporting experimental results.

## introduction textual entailment (te) is the task of determining if a hypothesis sentence can be inferred from a given context sentence. figure 1 shows examples of context-hypothesis pairs for te. previous works @xcite @xcite investigated several semantic approaches for te and demonstrated how they can be used to evaluate inference-related tasks such as ques- researchers have curated many resources foot_2 and benchmark datasets for te in english @xcite @xcite . however, to our knowledge, there is only one te dataset (xnli) in hindi, which was created by translating english data @xcite and another in hindi-english code-switched setting @xcite . hindi is the language with the fourth most native speakers in the world 4 . despite its wide prevalence, hindi is still considered a low-resource language by nlp practitioners because there are a rather limited number of publicly available annotated datasets. developing models that can accurately process text from low-resource languages, such as hindi, is critical for the proliferation and broader adoption of nlp technologies. creating a high-quality labeled corpus for te in hindi through crowd-sourcing could be challenging. in this paper, we employ a recasting technique from @xcite to convert four publicly available text classification datasets in hindi and pose them as te problems. in this recasting process, we build template hypotheses for each class in the label taxonomy. then, we pair the original anno-tated sentence with each of the template hypotheses to create te samples. unlike xnli, our dataset is based on the original hindi text and is not translated. furthermore, the multiple annotation artefacts @xcite present in the original classification data are leveled out for the textual entailment task on the recasted data due to label balance 5 . we evaluated state-of-the-art language models @xcite performance on the recasted te data. we then combine the predictions of related pairs (same premise) from te task to predict the classification labels of the original data (premise sentence), a twostep classification. we observed that a better te performance on the recasted data leads to higher accuracy on the followed classification task. we also observed that te models can make inconsistent predictions across samples derived from the same context sentence. driven by these observations, we propose two improvements to te and classification modeling. first, we introduce a regularisation constraint based on the work of @xcite that enforces consistency across pairs of training samples, thus correcting inconsistent predictions. second, we propose a joint objective for training te and classification simultaneously. our results demonstrate that the regularization constraint and joint training helps improve the performance of both the te models and the followed classification task. though our work demonstrates the use of recasting and modeling improvements for te in hindi, we expect these techniques can be applied to other low-resource languages and other semantic phenomenon beyond textual classification. following are the main contributions of this work: ## related work in this section, we list some of the related works in the field of nli as well as challenges encountered in low-resource settings. ## recasting classification datasets one of the main challenges for te evaluation for low-resource languages is the lack of labeled data. in this work, we employ recasting to convert annotated classification datasets in hindi to labeled te samples. as in @xcite , we selected four different datasets for recasting thus introducing linguistic diversity in the resulting te dataset. product review -the first dataset (pr) contains 5,417 samples of online user reviews in hindi for different products @xcite . these samples were annotated into one of the following four sentiment classes: positive, negative, neutral, and conflict. for recasting the samples in this dataset, we first built 8 hypothesis templates: 2 per class label. for each label, we create one positive and one negative hypothesis which roughly translate to: 'this product @xmath0. given a sample from the pr dataset, we treat it as the context sentence and combine with the 8 hypotheses sentences to create nli samples. if @xmath1, then the nli sample is marked as 'entailed'. likewise, if @xmath2, then the nli sample is also marked as 'entailed'. for the remaining cases, the sample is marked as 'non-entailed'. this process is summarized with an example in figure 1. for more detailed recasting illustration, see appendix section a.1 figure 5 . bhaav -the second dataset bhaav (bh ) @xcite ) contains 20,304 sentences from hindi short stories annotated for one of the following five emotion categories: joy, anger, suspense, sad, and neutral. we used a similar process as pr to recast bh using the following templates to create the hypothesis: 'it is a matter of @xmath3. ## methodology our objective in this paper is not only to use recasting to create a nli dataset in low-resource settings but also to understand how different models are effective in both te and classification task. furthermore, we also discuss our novel two-step classification technique with joint objective and regularization constraints. ## experiments most of the sentence embedding models have been designed and evaluated to perform well on english language. the experiments in this work are motivated to answer the following questions for a low-resource language, hindi: baselines -for evaluating our approach, we use the following baselines: infersent @xcite , sent2vec @xcite , bag-of-words (bow) and xlm-roberta @xcite which is state-of-the-art for multilingual language modelling. also, we evaluate a hypothesis-only analogue for each one of them as well. for experiments with recasted data, we use embeddings of context-hypothesis pair for baselines whereas for the hypothesis-only @xcite ) models, we only use embeddings of the hypothesis sentence, keeping it blind to the context. datasets. such biases often lead to high performance over nli tasks without completely comprehending the semantic reasonings in data and language. when the accuracy of a hypothesis-only model is much lower than the baseline and closer to random (50%), it exhibits that learning is not boosted due to statistical irregularities in data such as word count, unigram/bi-gram pattern or any other spurious pattern (artefacts). we achieve this using our approach since recasting ensures label balance for the augmentations of each class label for every sentence and its tokens. ## conclusion in this work, we share the first recasted nli dataset in a low-resource language hindi, and show how a large-scale nli data can be developed for low-resource languages without undergoing costly and time taking human annotations. we perform te experiments and introduce a consistency regulariser to avoid pairwise-inconsistent te predictions. furthermore, we propose a two-step classification approach with a joint training objective. our results with the joint objective shows significant improvement in performance. as a future work, we aim to analyse the proposed methodology which is language independent on other low-resource languages. we also aim to use more generalisable templates for linguistic diversity in recating data. it would be interesting to analyse how extending textual entailment knowledge especially the consistency regularization constraint affect other downstream nlp tasks apart from textual classification, not only in terms of the performance, but also in enhancing the model interpretability.