Towards Explainable Multi-Label Text Classification: A Multi-Task Rationalisation Framework for Identifying Indicators of Forced Labour (2024)

Abstract: The importance of rationales, or natural language explanations, lies in their capacity to bridge the gap between machine predictions and human understanding, by providing human-readable insights into why a text classifier makes specific decisions. This paper presents a novel multi-task rationalisation approach tailored to enhancing the explainability of multi-label text classifiers to identify indicators of forced labour. Our framework integrates a rationale extraction task with the classification objective and allows the inclusion of human explanations during training. We conduct extensive experiments using transformer-based models on a dataset consisting of 2,800 news articles, each annotated with labels and human-generated explanations. Our findings reveal a statistically significant difference between the best-performing architecture leveraging human rationales during training and variants using only labels. Specifically, the supervised model demonstrates a 10% improvement in predictive performance measured by the weighted F1 score, a 15% increase in the agreement between human and machine-generated rationales, and a 4% improvement in the generated rationales’ comprehensiveness. These results hold promising implications for addressing complex human rights issues with greater transparency and accountability using advanced NLP techniques.

## introduction multi-label text classification is a fundamental task in natural language processing (nlp) with wideranging applications, including document categorisation, sentiment analysis and content recommendation @xcite . even though deep learning models have achieved state-of-the-art performance for text classification in the last two decades, their black-box nature and the lack of understanding of why they assign specific labels to a text limits their application scope in high-stake domains @xcite . rationalisation models attempt to explain the outcome of a text classification model by providing a natural language explanation (rationale) @xcite . it has been observed that rationales are more understandable and easier to use than other explainability methods since they are verbalised in human-comprehensible natural language @xcite . recent evidence suggests that generating human-readable justifications for a model's predictions could empower users to grasp the reasoning behind a classifier's decisions, facilitating trust, accountability and the development of user-centric applications @xcite . rationales for explainable text classification can be categorised into extractive and abstractive rationales (figure 1 ). extractive rationales are a subset of the input text that supports a model's predictions, while abstractive rationales are explanations that are not constrained to be grounded in the input text @xcite . previous research has established that a multitask learning approach in training a rationalisation model can enhance the model's accuracy and generate more coherent and relevant explanations @xcite . when a rationalisation model is trained to classify items and explain its predictions, it learns to perform both tasks simultaneously, leveraging shared information to improve its predictions and rationales @xcite . recent evidence suggests that domain experts can play a pivotal role in this process by providing concise textual snippets (human rationales) that encapsu-late the reasons behind each classification decision @xcite . we propose a novel rationalisation framework to explain the outcome of a multi-label text classifier through extractive rationalisation. our framework uses multi-task learning to independently produce rationales at a label level and allows the alternative of including human rationales during training as an extra supervision signal. we employ our framework to identify indicators of forced labour, as defined by the international labour organization @xcite , for a rationale-annotated corpus of news articles @xcite . we hope our framework can help researchers and practitioners (e.g., social scientists or policymakers) in using supervised learning models to detect modern slavery with a more systematic approach. in summary, the main contributions of this paper are: (i) we present a novel rationalisation framework to generate extractive rationales at a label level in a multi-label setting using a multi-task learning approach and including human explanations during training; (ii) we provide results demonstrating that including human explanations during training can boost predictive performance and explainability of our rationalisation model for identifying indicators of forced labour. ## related work most research on extractive rationalisation has been carried out using an encoder-decoder foot_0 architecture @xcite . the encoder enc(x) serves as a tagging model, where each word in the input sequence x receives a binary tag indicating whether it is included in the rationale z. the decoder dec(x, z) then uses only the rationales and maps them to the target vector @xcite . @xcite pioneered the idea of using a multi-task learning approach modelling rationales as binary latent variables. they proposed jointly training the encoder and decoder to minimise a cost function composed of the classification loss and sparsity-inducing regularisation to keep the rationales short and coherent. considering that minimising the expected cost is challenging since it involves summing up all possible choices of rationales in the input sequence, they suggested training the architecture using reinforce-based optimisation @xcite . reinforce works by sampling rationales from the encoder and training the model to generate explanations using reinforcement learning. as a result, the model is rewarded for producing rationales that align with the desiderata defined in the cost function @xcite . using this multi-task learning approach, researchers have studied extractive rationalisation methods for binary and multi-class text classification @xcite . while some authors have kept using the latent binary variables and sparsity-inducing regularisation to encourage the model to select a limited set of words as rationales while suppressing irrelevant information @xcite , others have decided to transform the binary latent variables into continuous and differentiable variables. reparametrisation enables smoother optimisation without using the reinforce algorithm and allows for fine-tuning the length of rationales @xcite . even though research on learning with rationales has established over the last fifteen years that incorporating human rationales during training can improve classification performance, it is only in the past four years that studies have started looking into using human rationales to enhance the quality of the generated explanations @xcite . researchers have adapted the original implementation by @xcite , incorporating human rationales during training by modifying the model's cost function by adding components to force the generated rationales to be similar to the human explanations @xcite @xcite . our rationalisation approach draws inspiration from the work of @xcite and @xcite around using multi-task learning to enhance predictive performance and explainability when training the encoder-decoder architecture. following work by @xcite and @xcite , we also explore using human explanations during training as an extra supervision signal and check whether it significantly impacts the results of our framework. however, our work extends theirs by focussing on independently producing rationales tailored to each predicted label using pre-trained language models. the encoder processes the input sequences (x) to extract the rationales ( ẑ) at a label level. the rationales are then input to the decoder to predict the target labels ( ŷ ). the encoder and decoder are trained jointly via reinforcebased optimisation using a loss function composed by the classification loss (y, ŷ ), a rationale regularisation ( ẑ), and an additional rationale prediction loss (z, ẑ) in the case of the supervised rationale extraction variant. ## explainable text classification framework in this section, we detail our framework for explainable text classification based on a multi-task learning implementation of the encoder-decoder architecture @xcite to produce rationales at a label level for a multi-label setting. the encoder is the module responsible for identifying the rationales within the input sequence at a label level, and the decoder is tasked with predicting labels based on the generated rationales @xcite . it is important to note that our framework allows human rationales to be included as an extra supervision signal during training. throughout our paper, we refer to the architecture using target labels and human rationales during training as 'supervised rationale extraction' and refer to the implementation using only target labels as 'unsupervised rationale extraction'. figure 2 describes our framework in terms of its input data, encoder, decoder and loss function. input data the input data for our framework is composed of input sequences (x), target labels (y ) and, optionally, human rationales (z). target labels are encoded as c-dimensional vectors using onehot-encoding @xcite , where c is the number of classes. as mentioned before, the human rationales are snippets of the input sequence that support labelling decisions at the label level. the human rationales for each input sequence are subsequently post-processed and represented in a c × l matrix format, where l is the maximum sequence length associated with the tokenisation applied over the input sequence @xcite . each row corresponds to a rationale for a specific label, and it is filled with binary tags that indicate whether the token was selected to be part of the human explanation or not. we refer to appendix a for a detailed input data example. encoder drawing inspiration from the encoderdecoder architecture proposed by @xcite and @xcite , we employ a pre-trained language model such as bert @xcite , to induce contextualised representation of tokens. the encoder generates a scalar, denoting the probability of selecting that token as part of the rationale, for each bert hidden state using a set of c linear and bernoulli layers @xcite . it is important to note that each linear and bernoulli layer works independently to produce rationales at a label level for our multi-label classifier. decoder as a decoder, we use a second pretrained language model followed by a classifica-tion layer. the classification layer comprises a dropout @xcite , a linear @xcite and a sigmoid layers @xcite . in our implementation, the decoder accepts the tokens in the input sequence tagged as rationales by the encoder and independently predicts each label. ## experimental design we conduct experiments using various pre-trained language models to compare and contrast the results of the unsupervised and supervised rationale extraction architectures on our dataset. foot_1 ## results and discussion results from the anovas, performed separately for each performance and explainability indicator using a significance level of 0.05, suggest a statistically significant difference in the architectures' performance in all metrics. we replicate the experiments described by 1 are the high sufficiency scores for all architectures, regardless of whether they exploit human rationales during training and the transformer model they are based on. the sufficiency scores of 0.9 or above on average, indicate that the generated rationales provide enough information to justify the classification outcome @xcite @xcite . however, there is room for improvement regarding the plausibility and comprehensiveness of the machine-generated explanations. low plausibility signifies that the extracted rationales differ from the snippets the domain experts picked, potentially undermining the model's trust in a real-world setting @xcite . additionally, low comprehensiveness suggests that the rationales fail to encompass the essential information, including more information than necessary, potentially reducing the explanation's effectiveness @xcite . the implementation based on the xlnet model performed the best among the unsupervised architectures in all metrics except for the sufficiency. data from previous research suggests that permutation-based training of this model, which captures bidirectional context efficiently, might enhance our architecture's predictive performance and rationales' quality @xcite . regarding the supervised rationalisation models, data in similar to the results for the unsupervised rationalisation models, the architecture based on the xlnet model performed best among the supervised variants. results show a 10%, 15%, and 4% improvement in f1 score, plausibility and comprehensiveness, compared to the unsupervised architecture based on the same model. it is worth noting that these results are significant at a @xmath0.05 level. figure 3 presents examples of some "good" and "bad" rationales extracted using the supervised rationalisation architecture based on the xlnet model. on average, machine-generated rationales are 37% shorter than human rationales. "good" rationales are often a subset of the human rationales containing the most relevant information regarding the predicted forced labour indicator. in contrast, "bad" machine-generated rationales are either too short or have no intersection with the humanprovided explanations. in both cases, the iou scores are very low, directly affecting the generated rationales' plausibility @xcite . the results of our rationalisation approach may vary among labels due to differences in the number of news articles per indicator and the intrinsic complexity associated with each label @xcite @xcite . labels with fewer examples (e.g., "retention of identity documents") or inherently complex criteria (e.g., "intimidation and threats") may exhibit more significant variability in rationalisation performance compared to labels with readily available training data (e.g., "abusive working and living conditions") and more distinctive language characteristics (e.g., "debt bondage") @xcite . we observe that our rationalisation approach exploiting rationale-level supervision often improves the predictive performance and rationales' quality, as in prior work @xcite @xcite . nevertheless, there is a disparity in the results among labels where rationales for less-represented forced labour indicators tend to have low predictive performance and shallow agreement with human-provided rationales. ## conclusions forced labour is the most common type of modern slavery, affecting an estimated 27.6 million people worldwide. explainable text classification can aid stakeholders, such as ngos, police forces, and policy-makers, in understanding, addressing, and preventing the spread of forced labour by empowering them with actionable insights @xcite . in this work, we presented a novel multi-task rationalisation framework to extract rationales at a label level in a multi-label setting that allows the inclusion of human explanations during training. our experiments showed that using human rationales as an extra supervision signal can improve the classification performance of our model while enhancing the quality of the generated explanations. whilst the small number of cases limits the results of our framework in some of the forced labour indicators, it offers valuable insights into cases of "abusive working and living conditions" and "restriction of movement", among other indicators. in the future, we will focus on improving the framework's performance for less-represented labels.