An Information-Theoretic Approach to Analyze NLP Classification Tasks (2024)

Abstract: Understanding the contribution of the inputs on the output is useful across many tasks. This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks. Natural language processing (NLP) tasks take either a single or multiple text elements to predict an output variable. Each text element has two components: the semantic meaning and a linguistic realization. Multiple-choice reading comprehension (MCRC) and sentiment classification (SC) are selected to showcase the framework. For MCRC, it is found that the relative context influence on the output reduces on more challenging datasets. In particular, more challenging contexts allows greater variation in the question complexity. Hence, test creators need to carefully consider the choice of the context when designing multiple-choice questions for assessment. For SC, it is found the semantic meaning of the input dominates compared to its linguistic realization when determining the sentiment. The framework is made available at: https://github.com/WangLuran/nlp-element-influence.

## introduction natural language processing (nlp) requires machines to understand language to perform a specific task (chowdhary and chowdhary, 2020). nlp tasks take a single (e.g. summarization @xcite , sentiment classification @xcite , machine translation @xcite ) or multiple (e.g. reading comprehension @xcite , question generation @xcite )) text elements at the input and return a specific output. each input text element can further be partitioned into its semantic content and the linguistic realization. semantic refers to the inherent meaning while the linguistic realization is the specific wording to present the meaning. there are several possible linguistic realizations for any semantic content. therefore, for all nlp tasks, the output variable has contributions from at least two components: the semantic meaning of the element and the specific linguistic realization. here, element refers to a specific input that is formed of exactly two components. we analyze the relative sensitivity of the output variable to each of the input elements as well as in terms of the breakdown between the elemental semantic content and its corresponding linguistic realization. a theoretical information-theoretic approach is applied to find the shared information content between each input component and the output variable. here, the information-theoretic approach is framed for nlp classification tasks where the set of input components influence the output probability distribution over a discrete set of classes. we select multiple-choice reading comprehension (mcrc) and sentiment classification (sc) as case studies for the analysis. mcrc requires the correct answer option to be selected based on several input elements: the context paragraph, the question and the set of answer options. multiple-choice (mc) assessments are a widely employed method for evaluating the competencies of candidates across diverse settings and tasks on a global scale @xcite @xcite . given their consequential impact on realworld decisions, the selection of appropriate mc questions tailored to specific scenarios is important for content creators. consequently, there is a need to comprehend the underlying factors that contribute to the complexity of these assessments. complexity of an mc question is best modelled by the distribution over the answer options by human test takers. therefore, by understanding the influence of each input element on the output distribution, content creators can be better informed to what extent the complexity of an mc question can be controlled from changing each of the input ele-ments. moreover, analyzing the contribution of the semantic content vs the linguistic realization on the output human distribution informs the impact of the specific word choice in the element on the question complexity. however, it is not scalable to measure the variation in the output human distribution with variation in each of the input elements. @xcite demonstrated that the output distribution of automated systems is aligned (with minimal re-shaping parameters) to the human distribution. therefore, the information-theoretic framework is applied to the output probability distribution by an automated comprehension system. sc is a common nlp classification task where the dominant sentiment class must be selected from a discrete set of sentiments based on a block of input text. this is an example of a single input text element nlp task. the information-theoretic approach is applied here to understand the role of the semantic content and the linguistic realization on the output distribution over the sentiment classes for popular datasets. it is interesting to analyze sc as ideally the sentiment of a text block should be based on only its semantic meaning. here, we determine whether this ideal is held in practice. ## related work features or variables are separate properties that are input to tabular machine learning models to predict a target variable @xcite . feature importance is an active area of research @xcite where the influence of each feature on the output is determined. the ability to determine which features are most important is useful across many verticals e.g. computer assisted medical diagnosis @xcite , weather forecasting @xcite , fraud detection @xcite and customer churn prediction @xcite . similarly, we explore the importance of different aspects (can be interpreted as features) at the input including individual elements and the semantic vs linguistic components for nlp text classification tasks. typically, the structured nature of tabular data allows common feature selection algorithms to be applied including lasso @xcite , marginal screening @xcite , orthogonal matching pursuit @xcite and decision tree based @xcite . due to the relatively unstructured nature of text data (compared to tabular data), we propose an information-theoretic approach to identify the most influential inputs. @xcite find a weak correlation between question difficulty and context readability for mcrc. additionally, @xcite consider the impact on mcrc datasets when input elements are omitted. we propose instead an automated information-theoretic framework for this analysis. finally, @xcite apply an information-theoretic approach for prompt engineering. our approach can instead be generalized to any nlp classification task. ## theory here, we describe the generalized framework to analyze the influence of different elements in nlp text classification tasks: 1. the individual influence of each input element on the output class distribution; 2. the contribution of the semantic content vs its linguistic realization component for a given element. let an nlp task consist of a set of elements, {x 1 , . . . , x n } = x and the output, y, such that: let x denote the random variables of each the corresponding instances x. similarly, let y be the random variable for an instance of the output, y. to measure the influence of input x on output y, a good metric is the mutual information @xcite which measures how the output changes due to variation in the input. thus we can define i(y ; x) a measure of the total input influence. similarly we can define the influence from an individual element, x j , i(y ; x j ) and it should obey: for each element x j , its influence is always determined by two components: x (s) j , the semantic information and a relating linguistic realization method which turns an abstract meaning into natural language. thus, we can calculate the semantic influence as i(y ; x (s) j ) and the linguistic influence implicitly i(y ; x j |x (s) j ). they should satisfy: in practice for an element, x j , its semantic content is too abstract to be available. instead we get access to one of its realization rj which is considered to be generated from its unobserved semantic content, x (s) j . a set of possible realizations of this semantic element, r (i) , are additionally where each member of this set is, r with these settings, the mutual information is calculated as follows. the total influence is: we can also get the element influence as : it can be decomposed as the semantic influence: the relative contribution of an element to the total influence and of the semantic component for an element can respectively be expressed as: ## conclusions this work describes an information-theoretic framework for text classification tasks. the framework determines the influence of each input element on the output. additionally, each element is partitioned into its semantic and linguistic components. mcrc and sc are considered as case study tasks for analysis. for mcrc, it is found that both the context and question elements play influential roles on the output distribution. it is further established that selection of more challenging contexts permits greater variation (in terms of complexity) of questions on the context. simpler contexts limit the range of the complexity to only easy questions. hence, content creators need to carefully consider the choice of the context when designing mc questions to cater to a range of difficulty levels. in sc, the linguistic realization of the input has a measurable impact on the output. hence, the text wording cannot be neglected when deducing the sentiment. for both tasks, higher the readability of a specific linguistic realization, greater the probability of the true class in the output distribution. this research is partially funded by the epsrc (the engineering and physical sciences research council) doctoral training partnership (dtp) phd studentship and supported by cambridge university press & assessment (cup&a), a department of the chancellor, masters, and scholars of the university of cambridge.