Main Predicate and Their Arguments as Explanation Signals For Intent Classification (2025)

Abstract: Intent classification is crucial for conversational agents (chatbots), and deep learning models perform well in this area. However, little research has been done on the explainability of intent classification due to the absence of suitable benchmark data. Human annotation of explanation signals in text samples is time-consuming and costly. However, from inspection of data on intent classification, we see that, more often than not, the main verb denotes the action, and the direct object indicates the domain of conversation, serving as explanation signals for intent. This observation enables us to hypothesize that the main predicate in the text utterances, along with the arguments of the main predicate, can serve as explanation signals. Leveraging this, we introduce a new technique to automatically augment text samples from intent classification datasets with word-level explanations. We mark main predicates (primarily verbs) and their arguments (dependency relations) as explanation signals in benchmark intent classification datasets ATIS and SNIPS, creating a unique 21k-instance dataset for explainability. Further, we experiment with deep learning and language models. We observe that models that work well for classification do not perform well in explainability metrics like plausibility and faithfulness. We also observe that guiding models to focus on explanation signals from our dataset during training improves the plausibility Token F1 score by 3-4%, improving the model’s reasoning.

## introduction intent classification is widely used in real-life chatbots like alexa, siri, and other conversational ai tools. it is used to grasp the nuanced aspects of user expressions. it involves (i) comprehending the intentions behind the expressions to guide the agent's actions and (ii) identifying slots that indicate crucial entities necessary for executing the actions. intent identification is studied as a multiclass classification problem by @xcite . however, it is observed that jointly solving it with slot filling improves the performance @xcite . while deep learning models perform well in intent classification for specific datasets @xcite , they face challenges in generalizing across diverse domains and fine-grained intents, particularly for classes with scarce data @xcite . in scenarios with numerous conversation domains and fine-grained intent classes, acquiring significant data for every class becomes challenging. additionally, the presence of shared words among fine-grained classes with subtle variations in meaning adds complexity to the task. in the text find me an airline with a meal facility, the intent is atis-airline as the focus is on the airline, not the meal. while meal is a key entity, it doesn't define intent. conversely, in what are my meal options on airlines from boston to denver, the intent is atis-meal as the user asks about meals, not the airline. however, the deep learning model learns from correlations and frequently used words. hence, it misclassifies the later utterance as atis-airline due to the abundance of that class in the training data and the higher frequency of the term airline. this leads to a low overall f1 score and low accuracy for classes with fewer samples ( ## dataset we use two intent classification benchmark datasets, atis @xcite , and snips @xcite . atis consists of a set of text utterances and their corresponding intent labels. it has 4478, 500, and 893 text utterances with gold intent in train, development, and test set, respectively. it consists of 21 intents about the airline travel domain. the second dataset, snips @xcite , consists of 13,084 training utterances; the validation and test sets consist of 700 utterances each, with 100 queries per intent. it has seven different intent classes, each belonging to a different domain. text utterances are from domains like restaurants, movies, music, etc. we select these datasets with a specific purpose. atis has fine-grained intents, and some classes have a low training and test set size (refer to we mark word-level explanation signals in both the atis and snips datasets. hence, we contribute around 21k samples with explanation signals. from atis, we filter out the utterances with more than one intent, and we also filter classes in the test set but not in the train set. the average length of silver annotated explanation for the snips dataset is 3.4 words, while it is 4.2 for the atis dataset. ## silver annotated explanations human annotation of word-level explanations for intent classification is difficult. deciding whether original slots in the dataset should be added as an explanation signal is subjective and generally requires multiple human annotators. for example, text utterance, show me the flights from pittsburgh to los angeles on thursday has gold intent of atisflight. in this example, pittsburgh and los angeles are slots with slot-types source city and destination city. these entities are essential to execute appropriate further actions. however, it does not explain the intent because changing the city names to other cities does not affect the intent of the utterance. generally, multiple humans annotate explanation signals, which is costly and timeconsuming. hence, we propose a method to introduce word-level explanation signals automatically to existing intent classification datasets consisting of text utterance and gold intent. we define our dataset as follows. n is the number of samples in the dataset, and each text sample x i has m i tokens denoted by (w 1 , w 2 , w 3 , .....w m i ). we add word-level explanation signals e denoted by (e 1 , e 2 , e 3 , ......e m i ) where each e @xmath0, 1}. y i is a gold intent label for every x i , directly adopted from the original intent classification datasets. we add these word-level explanation signals for both atis and snips datasets. ## analysis we analyze the reasoning of models listed in although the model focuses on the most frequent terms a lot, it understands the role of the main predicate well. as a result, the explanation of 70-80% text utterances consists of the main predicate in words with top 5 attribution scores in all experiments. in the snips dataset, all the models perform very well but struggle with explainability. for example, the class searchscreeningevent model should focus on the words "movies," "play" and "theater." however, it focuses on terms like names of the movies or actors. due to abundant samples, deep learning models perform well on the snips dataset but might not generalize well across multiple domains in scarce data scenarios. atis and snips. hence, we contribute a first-ofits-kind benchmark dataset for explainability in intent classification consisting of 21k samples. we validate the quality of silver standard annotated explanations by detailed human evaluation. this approach can be adopted across diverse domains and intents. we evaluated several models on this dataset and found that models performing well on intent classification do not perform well in plausibility and faithfulness in explainability metrics. however, ingesting feature attribution priors improves the model's reasoning and overall performance. in future work, one can attempt to improve the model's performance under scarce data scenarios by training it on explanation signals. it will be interesting to apply the concept of main predicate and arguments to other languages and build a multilingual benchmark dataset for explainability in intent classification.