Why Generate When You Can Discriminate? A Novel Technique for Text Classification using Language Models (2024)

Abstract: In this paper, we propose a novel two-step technique for text classification using autoregressive Language Models (LM). In the first step, a set of perplexity and log-likelihood based numeric features are elicited from an LM for a text instance to be classified. Then, in the second step, a classifier based on these features is trained to predict the final label. The classifier used is usually a simple machine learning classifier like Support Vector Machine (SVM) or Logistic Regression (LR) and it is trained using a small set of training examples. We believe, our technique presents a whole new way of exploiting the available training instances, in addition to the existing ways like fine-tuning LMs or in-context learning. Our approach stands out by eliminating the need for parameter updates in LMs, as required in fine-tuning, and does not impose limitations on the number of training examples faced while building prompts for in-context learning. We evaluate our technique across 5 different datasets and compare with multiple competent baselines.

## introduction in recent years, the autoregressive or causal language models (lm) such as gpt-3 @xcite and gpt-neo @xcite have been successful in a variety of natural language processing tasks such as summarization, machine translation, question answering, etc. recently, there have been attempts to use such lms for text classification @xcite @xcite in a zero-shot or few-shot manner. in this paper, we propose a novel way of using moderatesized (#@xmath0.7b) and open-source autoregressive language models for text classification. the central idea is that generating new text using lms is not absolutely essential for text classification as is the case for other tasks such as summarization or machine translation, because the final goal is simply to discriminate among a finite set of class labels. there are several challenges in using moderatesized lms like gpt-neo-2.7b for text classification in both zero-shot as well as few-shot settings: very large lms like gpt-3 may not face these challenges, but their usage through api entails sharing the data to be classified and this may not be desirable for private and confidential data. hence, in this paper, we focus on only moderate-sized lms such as gpt-neo-2.7b which can be deployed inhouse with very limited hardware. to overcome the above-mentioned challenges for such lms, we propose a novel two-step technique for text classification. in the first step, for any text x to be classified, we elicit a set of feature values from the lm based on perplexity and log-likelihood of prompt: this is an overall sentiment classifier for movie reviews. classify the overall sentiment of the input as positive or negative. input: if this movie were a book, it would be a pageturner, you can't wait to see what happens next. sentiment: the movie is a mess. prompt: this is an overall sentiment classifier for movie reviews. a review with positive sentiment finds the movie to be great, good, encouraging, brilliant, excellent, accurate, realistic, engaging, funny, or exciting. a review with negative sentiment finds the movie to be terrible, bad, unrealistic, frustrating, boring, forgettable, predictable, thoughtless, appalling, or incomprehensible. classify the overall sentiment of the input as positive or negative. input: together, tok and o orchestrate a buoyant, darkly funny dance of death. sentiment: tok and o are a couple of misfits who... the generated text is shown in blue and italics. the key phrases proposed in our approach are similar to the verbalizers used in techniques such as pattern exploiting training (pet) @xcite and knowledgeable prompt-tuning (kpt) @xcite . however, these techniques are designed to work with encoder-only models like bert @xcite or roberta @xcite whereas our technique is designed to work with decoder-only (causal) language models like gpt-2. a major limitation of techniques such as pet and kpt is that only single token verbalizers can be used for describing class labels. on the other hand, the key phrases used in our technique can be multi-word and hence overcome this major limitation. this is especially useful in real-life examples where multi-word key phrases are necessary, e.g., fixed assets (used in our experiments with financial audit reports in section 6). here, neither the individual words fixed and assets capture the complete underlying meaning nor a list of single token verbalizers (e.g., land, machinery) is sufficient enough. on the contrary, as our technique harnesses causal (decoder-only) models, it allows both single-word as well as multi-word key phrases. moreover, techniques such as pet involve fine-tuning of the underlying model whereas our technique does not require such fine-tuning. to summarize, the key contributions of this paper are as follows: ## perplexity and log-likelihood perplexity is used as a metric to evaluate language models @xcite . intuitively, a better model of a text is the one which assigns a higher probability to a word that actually occurs. in this paper, we propose to use perplexity for a different purpose -judging plausibility of a text fragment using an autoregressive lm and comparing multiple such text fragments to decide which one is the most plausible. here, by plausibility of a text, we mean that it is seemingly more reasonable or probable. a similar idea was explored by @xcite for detecting misinformation. consider a text fragment @xmath1, w 2 , • • • , w n ] which consists of n tokens. the perplexity of x as computed by an lm m is as follows: the conditional perplexity of a text fragment x given another text @xmath2, c 2 , • • • , c m ] as its prefix, can be computed as: similarly, log-likelihood and conditional loglikelihood for any text x are computed as follows: overall, lower the perplexity of x (or higher the log-likelihood of x), better is its plausibility. ## text classification the task of text classification is to assign one or more applicable class labels from a pre-defined set of labels l to a piece of text x. there have been several attempts to use autoregressive lms for text classification where a response is generated from an lm by providing the text to be classified as part of a prompt. we hypothesize that there is no need to generate new text using an lm for text classification as we only need to discriminate among a finite set of class labels. hence, rather than asking an lm to generate some new text, it is enough to simply compare plausibility of a set of text fragments (label-specific augmentations as shown in ## step 2: learning a classifier this step is needed only in case of a supervised setting where labelled training instances are available. in the above zero-shot classification rule (zs p p l /zs ll ), a very simple function which maps the feature values to a class label is used, i.e., simply considering minimum or maximum over certain feature values. on the other hand, if training instances are available, a more complex function which maps these feature values to a class label can be learned. as one of the ways to learn such a function, in this step, we simply learn a supervised machine learning classifier using the feature values obtained for the training instances. this classifier can then be used to predict class labels for new unseen instances. we explored multiple lightweight classifiers and observed logistic regression (lr) and support vector machines (svm) to be the best performing in both multi-class and multi-label (one-vs-all) settings. ## related work while lms enhance performance across various nlp tasks, prior research has revealed several challenges when applying them to text classification, such as designing appropriate prompts in zero-shot setting, limited input prompt length when using in-context learning, and costly as well as timeconsuming fine-tuning. given these constraints, there is a line of research which explores novel ways using moderate-sized lms for text classification. one of the recent prominent work in this area is by @xcite . they introduce "noisy channel" as well as "direct" methods which compute conditional probability of the input text given the label or vice versa, for few-shot text classification through in-context learning and prompt tuning. our proposed technique resembles their approach to some extent in computing conditional perplexity, but there are several key differences -(i) computing multiple features using domain knowledge based key phrases, (ii) no limitation on number of training examples, and (iii) learning a classifier based on these features. another relevant work for our technique is by @xcite wherein the authors propose to calibrate output probabilities of an lm through prior adaptation to perform text classification tasks. they propose two variations of their approachunsupervised (ucpa) where no labelled data is needed and semi-unsupervised (sucpa) where some training examples (600) are used for prior adaptation. both @xcite and estienne (2023) are most relevant for our technique in the sense that they only use moderate-sized lms such as gpt2-xl and hence we consider both of these as important baselines. a recent approach by @xcite presents an innovative approach by integrating the general language understanding of llms with task-specific data in the form of clues and reasoning from labeled datasets, providing an effective solution. another work by @xcite focuses on a method for building a text classifier from an llm all within a black box paradigm, without direct access to internal model parameters. yang and liu (2022) introduces a robust prefix-tuning framework, enhancing robustness while maintaining efficiency, particularly in the context of text classification. this is achieved by leveraging language model activation and batch-level prefix tuning. @xcite presented an interesting technique where a causal lm generates classconditioned texts guided by prompts, which are used as the training data for fine-tuning an encoderonly model. we believe that auto-generating new training instances is reasonable for simpler text classification problems like sst2 but not for trec (section 5.1) which is a more challenging text classification problem. in trec, because the text to be classified is a question and the expected label is its answer type, it is not trivial to come up with a answer type based prompt which can generate suitable questions as expected in the technique by @xcite . our cht-bert baseline (section 5.2) is similar where labelled instances are used for fine-tuning the encoder model. in fact, this baseline is more competitive than @xcite given it uses gold-standard labelled instances instead of auto-generated instances. ## analysis of financial audit reports financial audit is a complex process used by organizations to assure the stakeholders about the quality and trustworthiness of the governance @xcite . one important outcome of an audit is the audit report, wherein the auditor declares the financial statements of a company are free from material misstatement, are fair and accurate and are presented in accordance with the relevant accounting standards. a good comprehensive audit report is an important indicator of a good audit. audit monitoring bodies such as the chartered accountants (ca) society of india have issued guidelines on the contents of audit reports wherein they describe a set of audit aspects which the auditor should touch upon and describe. the problem of verifying whether an audit report has covered these audit aspects, can be modelled as a multi-class multi-label text classification problem where each sentence in the report can be labelled with zero or more audit aspects. we have identified a set of 15 audit aspects from standard auditing checklist (icai, 2017) and companies (auditor's report) order, 2020 (caro) (icai, 2020), such as payables, inventory, and fixed assets (see ## conclusions and future work we proposed a novel two-step technique for text classification using moderate-sized (#@xmath3.7b) autoregressive language models (lm). in the first step, for a text instance to be classified, a set of perplexity and log-likelihood based features are obtained from an lm. a light-weight classifier (svm or lr) is trained in the second step to predict the final label. our technique presents a new way of exploiting the available labelled instances, in addition to the existing ways such as fine-tuning lms or in-context learning. it neither needs any parameter updates in lms as in fine-tuning nor it is restricted by the number of training examples to be provided in the prompt for in-context learning. the key advantages of our technique are its explainability through most suitable key phrases and its applicability in resource poor environments. we demonstrate effectiveness of the proposed technique by comparing it with multiple baselines in the context of two lms (gpt-neo-2.7b and gpt2-xl) on five different datasets. in future, we plan to extend this work by -(i) automatically discovering optimal set of key phrases and connector sentences, (ii) learning a function which exploits the inter-dependence between multiple features in a better way, (iii) exploring an ensemble where features from multiple lms are combined, and (iv) evaluating the generated explanations quantitatively through a user study.