Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques (2024)

Abstract: Ordinal Classification (OC) is a widely encountered challenge in Natural Language Processing (NLP), with applications in various domains such as sentiment analysis, rating prediction, and more. Previous approaches to tackle OC have primarily focused on modifying existing or creating novel loss functions that explicitly account for the ordinal nature of labels. However, with the advent of Pre-trained Language Models (PLMs), it became possible to tackle ordinality through the implicit semantics of the labels as well. This paper provides a comprehensive theoretical and empirical examination of both these approaches. Furthermore, we also offer strategic recommendations regarding the most effective approach to adopt based on specific settings.

## introduction ordinal classification (oc) is a key task in natural language processing (nlp), with many applications that require ordering or ranking in the output such as sentiment analysis @xcite , rating prediction (liu, 2020), age group classification @xcite , etc. in each of these examples, the output categories have a natural order, making these tasks suited to ordinal classification. broadly speaking, oc task can be tackled using one of the following two approaches -explicit vs implicit. the classical explicit approach relies on tweaking the loss function based on an explicit notion of distance between labels and penalising based on the degree of misclassification @xcite . alternatively, the second implicit approach we propose is based on more recent advancements in language modeling. this approach organically engages the semantics of the labels, thereby harnessing their * these authors contributed equally to this work inherent characteristics for the classification task. thus, while the former represents a time-tested approach, the latter offers a new way to tackle oc, bringing the richness of semantic information into the classification process. our paper comprehensively studies the different techniques that fall under both these strategies. the classical approach relies on tweaking cross entropy (ce) which is a commonly used loss function for nominal classification (nc) (where the assumption is that the classes are mutually exclusive and have no inherent order or relationship to each other) @xcite @xcite . the performance in nc task is usually measured in terms of accuracy-based metrics such f1 scores. while ce is optimal for nc, since it treats all misclassifications as the same, it is sub-optimal for oc. there are several approaches/tweaks proposed to extend ce loss for oc, such as the ones proposed by @xcite ; @xcite , which adds a penalty based on the absolute difference in the class rankings. these loss functions are designed such that the more the distance between prediction and ground truth, the more the penalty. the performance of these oc tasks is measured in terms of mean squared error (mse), mean absolute error (mae) and off-by-1 (ob1) accuracy @xcite . our work fills a crucial gap in the current understanding of these losses used in oc. prior to our research, a unifying analysis of these different loss based approaches was conspicuously absent, making it challenging to holistically compare and contrast their characteristics and performance. in an attempt to address this lacuna, we evaluate these loss functions for four desirable properties, namely, proper scoring rule, unimodality, convexity and ordinality. we also propose a loss function which exhibits these desirable properties while demonstrating that several of these loss functions can be interpreted as specialized versions of the proposed generalized loss function. pretrained language models (plms), which include both encoder models (like bert @xcite ) and decoder models (like gpt @xcite ), offer a robust mechanism to learn the semantics or representations of words. these models are trained on vast amounts of text data, during which they develop an understanding of the context in which words are used. encoder models like bert learn to predict a word based on its surrounding context, thereby creating a rich and nuanced understanding of word semantics. on the other hand, decoder models like gpt generate a sequence of words, learning to predict the next word in a sentence, hence understanding semantics in a left-to-right context manner. this learning process enables these models to develop high-dimensional vector representations (embeddings) that capture the underlying semantics of words. interestingly, these semantically rich embeddings can be utilized to implicitly factor in the ordinality of labels in oc tasks. the numeric vector representation of words inherently carries semantic relationships that can mirror ordinal relationships. for instance, the embeddings of words like "good", "neutral", and "bad" reflect their comparative semantics in the vector space. when used for oc tasks, such as sentiment analysis or rating prediction, these learned embeddings can provide a more natural and effective way to understand and encode the ordinality of labels. the model does not just see these labels as distinct classes, but as points on some latent space, thus allowing a more nuanced approach to such classification tasks. in our study, we make the following contributions to advance the understanding and application of oc in nlp: the rest of the paper is organized as follows: in §2, we discuss the various explicit approaches i.e. review the loss functions and discuss their theoretical properties. using this as motivation, we also propose a new hybrid loss function and study it's properties. in §3, we discuss the encoder models and show how oc can be approached from an entailment-style modeling perspective. in §4, we discuss the decoder models and how oc can be approached from a next word prediction task perspective. figure 2 depicts the explicit as well as implicit approaches. in §5, we empirically compare these three methods and give our recommendations. ## explicit approach: loss-functions and analysis let {(x i , y i )} n @xmath0; where y @xmath1, . . . , k} where k is the number of classes. the output of the classifier is denoted by φ θ (x i ) = @xmath2, . . . , p i k ) which is a probability distribution over the k classes. let i(y i ) be the one-hot encoding of y i . the classifier is trained by optimizing the parameters θ such that 1 n n @xmath3, i(y i )) reaches a minimum. in the rest of the paper, for ease of understanding, we omit the indexing with figure 2: [best viewed in color] (1) explicit approach: we explicitly impose ordinality in the predictions by designing losses like mll, oll, etc. (2) implicit approach: in this entailment technique, ordinality is implicitly enforced by passing the label semantic information in the input itself. (3) here we use a decoder lm to auto-regressively predict the label in natural language format given the input sentence. (notation: p lme: encoder plm, p lmd: decoder plm) respect to i, i.e. remove the 1 n and n @xmath4, wherever it is evident from the loss expression. in the next subsection, we give a few desirable theoretical properties of the loss function l in the context of oc and then follow it up by discussing some of the widely used loss functions. ## implicit approach: generative models decoder-based text generative models have seen notable advancements in recent years, facilitating the production of coherent and contextually relevant text. the development of models like the gpt series @xcite has led to state-of-the-art results in text generation and summarization benchmarks. one of the primary objectives of this paper is to investigate whether these models demonstrate ordinal behavior by accurately capturing the inherent order or ranking of elements in the generated text. formally, in the context of oc, for a given a textual input x i which comprises the following words (w i 1 , w i 2 , . . . , w in ) and its corresponding ground truth label l i , we append the input and label as (w i 1 , w i 2 , . . . , w in , l i ). next, the parameters θ of generative model are finetuned s.t. during inference for i th input x i , the generative model predicts the word w i n+1 from the vocabulary v which maximizes the conditional probability: @xmath5 further, it is possible that the predicted label may not be in the set of ground truth labels i.e. for an i th input x i the label li / ∈ l where l is the set of k distinct labels (see appendix 9). this is the notorious hallucination problem associated with generative models. to mitigate this issue during inference, we compare the conditional log-probabilities of each the k labels , given the text segment x i . the class corresponding to the highest probability is proposed as the generated label. that is, given that θ * represents the learned parameters of lm, the label selection during inference will be s.t. here (w i 1 , w i 2 , . . . , w in ) are the words in input x i and (l 1 , l 2 , . . . , l k ) are the labels in the set l. for fair comparison with other explicit and implicit approaches, we use gpt2-small as our base model for experiments to maintain similar number of model parameters (with bert-base). similar to the entailment approach ( §3), we also experiment with informative and un-informative verbalisers here. motivated by recent advancements and the accessibility of open-source large language models (llms), and to demonstrate the true potential of the generative approach, we also experiment with llama-7b @xcite , a decoder-based llm with 7 billion parameters. pre-trained on trillions of tokens using publicly available data, it 5395 100 % dataset metric ce mll snli f1 0.821 (0.00) 0.832 (0.00) mse 0.264 (0.00) 0.257 (0.01) mae 0.208 (0.02) 0.205 (0.01) ob1 0.972 (0.01) 0.974 (0.00) sst5 f1 0.357 (0.01) 0.378 (0.02) mse 1.197 (0.00) 1.125 (0.01) mae 0.768 (0.02) 0.742 (0.01) ob1 0.852 (0.00) 0.863 (0.00) amazon reviews f1 0.543 (0.02) 0.544 (0.00) mse 0.904 (0.04) 0.819 (0.01) mae 0.581 (0.01) 0.568 (0.00) ob1 0.903 (0.00) 0.915 (0.00) achieves state-of-the-art performance, surpassing its larger predecessors like gpt-3 (175b) on the majority of benchmarks. note that our approach is different from gpt2forsquenceclassification 1 where the last embedding of the last token is used for classification, which is similar to encoder-model (like bert) style classification. instead we train it for a language modelling task to generate within a fixed set of tokens i.e. the set of labels. ## results and discussion for the explicit loss-based approaches, the experimental results presented in when considering implicit approaches, we observe that the entailment (ent) approach performs on par with the mll loss across all three datasets, without any explicit ordinality enforcing mechanism. it even outperforms ce, mll, and gpt in almost all data settings for sst-5. this could be attributed to the fact that sst-5 has a significantly lower number of samples (∼12k) compared to snli and ar (∼200k). thus, the ent approach performs better in few-shot scenarios. another interesting observation is that for snli and ar, although the performances of ent and mll are similar, the standard deviation numbers are consistently lower in the case of ent, making it a reliable and relatively more stable approach to use in low-data settings. this supports our claimed hypothesis that the model is able to leverage natural language label descriptions to inherently enforce ordinality in its predictions. to examine the impact of incorporating informative verbalisers, we also fine-tuned using uninformative ones, replacing the label descriptions our observations, depicted in figure 3 , reveal intriguing findings. when the training data volume is low, the effect of label semantics becomes clearly visible. in the sst-5 dataset (with training data volume ∼10k), the influence of informative labels is evident across all settings (refer to middle section in figure 3 ). similarly, in the ar and snli datasets (with training data volume ∼250k), the impact of informative labels is noticeable only in the 25% or 10% data settings (refer to top and bottom sections in figure 3 ). in other cases, the distinction is less clear due to the sheer volume of training data; even un-informative labels yield decent performance, possibly leveraging some spurious correlations, thus rendering the semantics not so effective. this finding also aligns with the hypothesis proposed by @xcite , indicating that prompt-based models may not leverage label semantics as expected, with the dependency being influenced by the model and dataset size to some extent. on another note, for ent approach, the inference time is scaled by ∼o(k) (k: total labels). but with more carefully engineered verbalisers, similar or even better performance could be achieved even with smaller base models @xcite making it a suitable candidate for ordinal problems. in our generative (gen) approach, we initially employed the gpt2-small model for experiments, as it has a comparable number of fine-tunable para-meters to the encoder counterpart used in ent & explicit loss approaches. despite its usual underperformance compared to mll and ent, which aligns with the general observation that gpt-2 typically fares worse than bert-base on most classification tasks @xcite , we demonstrate its ability to recognize label order by contrasting informative versus un-informative verbalisers, akin to the ent approach. similar trends are observed with the entailment technique, particularly the significant failure of the un-informative verbaliser variant on the sst-5 dataset. across other datasets, performance in terms of f1-score is comparable to the informative variant, except for 10% case in snli and ar (low-data setting). notably, for ordinal metrics like mae and mse, we observe high variance in ar and sst-5 datasets (figure 3 ), underscoring the importance of employing informative label verbalizers for stable learning. two common findings emerge from the informative vs un-informative ablations in both ent and gen experiments: (a) the hypothesis posited in @xcite holds true for ordinal metrics as well as nominal metrics. (b) label semantics become significant only in low-data settings. the purpose of presenting gpt-2 results was to illustrate the true potential of generative approaches, through llms, as introduced in §4. we leverage the llama-adapter technique @xcite which introduces only 1.2m tunable parameters over the base model by prepending a set of learnable adaption verbalisers to the word tokens at higher transformer layers and show that it still outperforms all explicit & implicit approaches presented above (refer ## conclusion this paper presents an unified analysis of explicit and implicit strategies for addressing oc. it is the first study to thoroughly examine and compare these approaches from both theoretical and empirical standpoints. our analysis (summarized in