MARE : Multi-Aspect Rationale Extractor on Unsupervised Rationale Extraction (2024)

Abstract: Unsupervised rationale extraction aims to extract text snippets to support model predictions without explicit rationale annotation.Researchers have made many efforts to solve this task. Previous works often encode each aspect independently, which may limit their ability to capture meaningful internal correlations between aspects. While there has been significant work on mitigating spurious correlations, our approach focuses on leveraging the beneficial internal correlations to improve multi-aspect rationale extraction. In this paper, we propose a Multi-Aspect Rationale Extractor (MARE) to explain and predict multiple aspects simultaneously. Concretely, we propose a Multi-Aspect Multi-Head Attention (MAMHA) mechanism based on hard deletion to encode multiple text chunks simultaneously. Furthermore, multiple special tokens are prepended in front of the text with each corresponding to one certain aspect. Finally, multi-task training is deployed to reduce the training overhead. Experimental results on two unsupervised rationale extraction benchmarks show that MARE achieves state-of-the-art performance. Ablation studies further demonstrate the effectiveness of our method. Our codes have been available at https://github.com/CSU-NLP-Group/MARE.

## introduction deep learning text classification systems have achieved remarkable performance in recent years @xcite . however, their black-box nature has been widely criticized. finding a sufficient approach to open the black box is urgent and significant. unsupervised rationale extraction @xcite ) is an explanation approach that aims to extract text snippets from input text to support model predictions without explicit rationale annotation. previous researchers @xcite have made many efforts to improve the rationalization performance of their models. however, example appearance: positive aroma: positive palate: positive text: thanks to bman1113vr for sharing this bottle . pours a murky orangish-brown color with a white head . the aroma is tart lemons . the flavor is tart lemons with some oak-aged character . the beer finishes very dry . medium mouthfeel and medium carbonation . as shown in figure 1a , existing rationale extraction models are uni-aspect encoding models, which can only predict and interpret one aspect of the text at a time. in real-world scenarios, one text often contains multiple aspects of an object. to address these problems, we propose the multi-aspect rationale extractor (mare). as shown in figure 1b , mare can encode all aspects simultaneously by prepending multiple special tokens to the input text, each corresponding to a specific aspect. this approach enables multi-aspect encoding in one model. furthermore, mare introduces a multi-aspect multi-head attention (mamha) mechanism for collaborative encoding across aspects. this mechanism allows the model to capture interactions and dependencies between different aspects, leading to more accurate predictions and rationales. finally, inspired by multi-task learning, mare iteratively accesses training data for different aspects, reducing the overall training cost. we validate the effectiveness of mare on two unsupervised rationale extraction benchmarks: beeradvocate @xcite and hotel review @xcite . results show that mare outperforms existing state-of-the-art methods across multiple evaluation metrics. ablation studies further demonstrate the effectiveness of mare. our main contributions are as follows: ## related work the rationalization framework, known as rnp @xcite , assumes that any unselected input has no contribution to the prediction and achieves remarkable performance on this task. however, rnp still has many weaknesses. various approaches have been proposed to improve rnp in different dimensions. ## problem definition existing uni-aspect encoding models extract rationales z i from the input x and predict the label y i for the i-th aspect. formally, they can be expressed as p (y i , z i | x; θ i ), where θ i represents the parameters of the model for the i-th aspect. to obtain the rationales and predictions for all k aspects, k independently trained models are required: however, this approach is time-consuming and computationally expensive. to address this issue, we propose a multiaspect rationale extraction task, where the rationales and predictions for all aspects can be generated simultaneously. this can be formalized as p (y 1 , z 1 , • • • , y k , z k | x; θ), where θ represents the parameters of the multi-aspect rationale extraction model. by utilizing a single model to extract rationales and make predictions for all aspects concurrently, we aim to improve the efficiency and reduce computational costs compared. ## method this paper proposes a multi-aspect rationale extractor (mare), which can simultaneously predict and interpret multiple aspects of text. as shown in the left part of figure 3 , mare is based on an encoder-based pre-trained language model and achieves multi-aspect collaborative encoding through a multi-aspect multi-head attention (mamha) mechanism. additionally, mare employs multi-task training during the training process, significantly reducing the training cost. ## analysis in this section, we delve into a more comprehensive analysis of our methodology. in section 6.1, we present a series of case studies derived from the hotel review dataset to exemplify the practical applications of our approach. in section 6.2, we conduct an ablation study to substantiate the efficacy of our method by incrementally removing its constituent elements. ## conclusion this paper proposed a multi-aspect rationale extractor to solve the limitations of traditional uniaspect encoding models. mare can collaboratively predict and interpret multiple aspects of text simultaneously. additionally, mare incorporated multi-task training, sequentially training on data from each aspect, thereby significantly reducing training costs. extensive experimental results on two unsupervised rationale extraction datasets have shown that the rationalization performance of mare is superior to all previous models. ablation studies further demonstrated the effectiveness of our method.