D irect P robe: Studying Representations without Classifiers (2021)

Abstract: Understanding how linguistic structure is encoded in contextualized embedding could help explain their impressive performance across NLP. Existing approaches for probing them usually call for training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. In this work, we argue that doing so can be unreliable because different representations may need different classifiers. We develop a heuristic, DirectProbe, that directly studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DirectProbe can shine lights on how an embedding space represents labels and also anticipate the classifier performance for the representation.

## introduction distributed representations of words (e.g., @xcite have propelled the state-of-the-art across nlp to new heights. recently, there is much interest in probing these opaque representations to understand the information they bear (e.g., @xcite @xcite . the most commonly used strategy calls for training classifiers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy (e.g. @xcite @xcite @xcite . using these classifiers, criteria such as accuracy or model complexity are used to evaluate the representation quality for the task (e.g. @xcite @xcite . such classifier-based probes are undoubtedly useful to estimate a representation's quality for a task. however, their ability to reveal the information in a representation is occluded by numerous factors, such as the choice of the optimizer and the initialization used to train the classifiers. for example, in our experiments using the task of preposition supersense prediction @xcite , we found that the accuracies across different training runs of the same classifier can vary by as much as ∼ 8%! (detailed results can be found in appendix f.) indeed, the very choice of a classifier influences our estimate of the quality of a representation. for example, one representation may achieve the best classification accuracy with a linear model, whereas another may demand a multi-layer perceptron for its non-linear decision boundaries. of course, enumerating every possible classifier for a task is untenable. a common compromise involves using linear classifiers to probe representations @xcite , but doing so may mischaracterize representations that need non-linear separators. some work recognizes this problem @xcite and proposes to report probing results for at least logistic regression and a multi-layer perceptron @xcite , or to compare the learning curves between multiple controls @xcite . however, the success of these methods still depends on the choices of classifiers. in this paper, we pose the question: can we evaluate the quality of a representation for an nlp task directly without relying on classifiers as a proxy? our approach is driven by a characterization of not one, but all decision boundaries in a representation that are consistent with a training set for a task. this set of consistent (or approximately consistent) classifiers constitutes the version space for the task @xcite , and includes both simple (e.g., linear) and complex (e.g., non-linear) classifiers for the task. however, perfectly characterizing the version space for a problem presents compu-tational challenges. to develop an approximation, we note that any decision boundary partitions the underlying feature space into contiguous regions associated with labels. we present a heuristic approach called directprobe, which builds upon hierarchical clustering to identify such regions for a given task and embedding. the resulting partitions allow us to directly probe the embeddings via their geometric properties. for example, distances between these regions correlate with the difficulty of learning with the representation: larger distances between regions of different labels indicates that there are more consistent separators between them, and imply easier learning, and better generalization of classifiers. further, by assigning test points to their closest partitions, we have a parameter-free classifier as a side effect, which can help benchmark representations without committing to a specific family of classifiers (e.g., linear) as probes. our experiments study five different nlp tasks that involve syntactic and semantic phenomena. we show that our approach allows us to ascertain, without training a classifier, (a) if a representation admits a linear separator for a dataset, (b) how different layers of bert differ in their representations for a task, (c) which labels for a task are more confusable, (d) the expected performance of the best classifier for the task, and (e) the impact of fine-tuning. in summary, the contributions of this work are: ## representations and learning in this section, we will first briefly review the relationship between representations and model learning. then, we will introduce the notion of ϵ-version spaces to characterize representation quality. figure 1 : an illustration of the total work in a learning problem, visualized as the distance traveled from the input bar to the output bar. different representations e i and different classifiers h i provide different effort. ## approximating an ϵ-version space although the notion of an ϵ-version space is well defined, finding it for a given representation and task is impossible because it involves enumerating all possible classifiers. in this section, we will describe a heuristic to approximate the ϵ-version space. we call this approach as directprobe. ## representations, tasks and classifiers before looking at the analysis offered by the partitions obtained via directprobe in §5, let us first enumerate the english nlp tasks and representations we will encounter. ## experiments and analysis directprobe helps partition an embedding space for a task, and thus characterize its ϵ-version space. here, we will see that these clusters do indeed characterize various linguistic properties of the representations we consider. ## related work and discussion in addition to the classifier based probes described in the rest of the paper, a complementary line of work focuses on probing the representations using a behavior-based methodology. controlled test sets @xcite are designed and errors are analyzed to reverseengineer what information can be encoded by the model (e.g., @xcite @xcite . another line of work probes the space by "opening up" the representation space or the model (e.g., @xcite . there are some efforts to inspect the space from a geometric perspective (e.g., @xcite . our work extends this line of work to connect the geometric structure of embedding space with classifier performance without actually training a classifier. recent work @xcite @xcite probe representations from an information theoretic perspective. these efforts still need a probability distribution p(y|x) from a trained classifier. in §5.3, we use clusters to predict labels. in the same vein, the conditional probability p(y|x) can be obtained by treating the negative distances between the test point x and all clusters as predicted scores and normalizing via softmax. our formalization can fit into the information theoretic analysis and yet avoid training a classifier. ## conclusion in this work, we ask the question: what makes a representation good for a task? we answer it by developing directprobe, a heuristic approach builds upon hierarchical clustering to approximate the ϵ-version space. via experiments with several contextualized embeddings and linguistic tasks, we showed that directprobe can help us understand the geometry of the embedding space and ascertain when a representation can successfully be employed for a task.