Analyzing Text Representations by Measuring Task Alignment (2023)

Abstract: Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.

## introduction recent advances in text classification have shown that representations based on pre-trained language models are key, especially in few-shot learning scenarios @xcite . it is natural to ask: what makes a representation good for text classification in this setting? is the representation good due to intrinsic geometric properties of the space or because it is well aligned with the classification task? the goal of this paper is to answer this question to better understand the reason behind the performance gains obtained with pre-trained representations. our hypothesis is that representations better aligned with class labels will yield improved performance in few-shot learning scenarios. the intuition is simple: in this setting, the limited number of labeled samples will only provide a sparse coverage of the input domain. however, if the representation space is properly aligned with the class structure, even a small sample can be representative. to illustrate this, take any classification task. suppose we perform clustering on a given representation space that results in a few pure clusters (with all samples belonging to the same class). then, any training set that 'hits' all the clusters can be representative. notice that there is a trade-off between the number of clusters and their purity. a well-aligned representation is one for which we can obtain a clustering with a small number of highly pure clusters. based on this, we propose a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity: task hierarchical alignment score (thas). to test our hypothesis that task alignment is key we conduct experiments on several text classification datasets comparing different representations. our results show that there is a clear correlation between the thas of a representation and its classification performance under the few-shot learning scenario, validating our hypothesis and showing that task alignment can explain performance. in contrast, our empirical study shows that intrinsic geometric properties measured by classical clustering quality metrics fail to explain representation performance in the few-shot learning scenario. our study suggests an answer to our main question: a good efficient representation (i.e. one that enables few-shot learning) is a representation that induces a good alignment between latent input structure and class structure. our main contributions are: 1) we develop a score based on hierarchical clustering ( §2) that measures the extent to which a representation space is aligned with a given class structure and 2) we conduct an empirical study using several textual classification datasets ( §3) that validates the hypothesis that the best representations are those with a latent input structure that is well aligned with the class structure. ## task hierarchical alignment score we now present the task hierarchical alignment score (thas) designed to measure the alignment between a textual representation and the class label for a given task. the idea is quite simple, in a good representation space, points that are close to each other should have a higher probability of belonging to the same class. therefore, we could perform clustering of the points and obtain high purity clusters, where most points belong to the same class. we assume that we are given: a dataset @xmath0, y i )} n @xmath1.g., a sentiment classification label) and a representation function r : @xmath2.g., a sparse bag-of-words). our goal is to compute a metric τ (s, r) that takes some labeled domain data and a representation function and computes a real value score. fig. 1 illustrates the steps involved in computing thas. there are three main steps: 1) hierarchical clustering, 2) computing clustering partition alignments, and 3) computing the aggregate metric. in the first step, we compute the representation of each point and build a data dendrogram using hierarchical clustering. the data dendrogram is built by merging clusters, progressively unfolding the latent structure of the input space. traversing the tree, for each level we get a partition of the training points into k clusters. in step 2, for each partition, we measure its alignment with the class label distribution producing an alignment curve as a function of k. finally, we report the area under this curve. algorithm 1 summarizes the whole procedure. implementation details and performance information can be found in a.1. ## related work representation choice has recently gained significant attention from the active learning (al) community @xcite @xcite . some work has attempted to quantify what representation is best when training the initial model for al, which is usually referred to as the cold start problem @xcite . the importance of word embeddings has been also studied in the context of highly imbalanced data scenarios @xcite @xcite . most research conducted by the al community on textual representations has focused on determining which representations lead to higher performance for a given task. however, our paper aims to investigate why a certain representation performs better in the few-shot scenario. our work, focused on examining properties of various textual representations, is closely related to recent research on evaluating the general capabilities of word embeddings. many studies are interested in testing the behavior of such models using probing tasks that signal different linguistic skills @xcite @xcite @xcite . others have targeted the capacity of word embeddings to transfer linguistic content @xcite . looking at approaches that analyze the properties of representations directly, without intermediate probes, @xcite developed a correlation method to compare representations during consecutive pre-training stages. analyzing the geometric properties of contextual embeddings is also an active line of work @xcite @xcite . while these previous works focus on analyzing representation properties independently, without considering a specific task, our study investigates the relationship between representations and task labels. we conduct a comparison between this relationship and the unsupervised analysis of representation properties. our work falls in line with broader research on the relationship between task and representation. @xcite proposed a method to measure the alignment between documents and labels in a given representation space using a data complexity measure developed in the learning-theory community. in the computer vision area, @xcite introduced a loss metric and investigated the entanglement of classes in the representation space during the learning process. @xcite proposed a heuristic to approximate the version space of classifiers using hierarchical clustering, highlighting how representations induce the separability of class labels, thereby simplifying the classification task. in contrast, our work specifically examines the few-shot performance and emphasizes the importance of unbalanced scenarios. we find that in these more realistic situations, the choice of representation plays a critical role, paving the way for advanced strategies in active learning. ## conclusion in this paper, we asked the question: what underlying property characterizes a good representation in a few-shot learning setting? we hypothesized that good representations are those in which the structure of the input space is well aligned with the label distribution. we proposed a metric to measure such alignment: thas. to test our hypothesis, we conducted experiments on several textual classification datasets, covering different classification tasks and label distributions (i.e. both balanced and unbalanced). we compared a range of word embedding representations as well as a baseline sparse representation. our results showed that when labeled data is scarce the best-performing representations are those where the input space is well aligned with the labels. furthermore, we showed that the performance of a representation can not be explained by looking at classical measures of clustering quality. the main insight provided in this work could be leveraged to design new strategies in active learning. the fact that good representations induce clusters of high purity at different granularities creates opportunities for wiser exploration of the representation space in an active manner. similar to the work of @xcite , we could employ the data dendrogram to guide this exploration.