Zero-Shot Data Maps. Efficient Dataset Cartography Without Model Training (2023)

Abstract: Data Maps (Swayamdipta, et al. 2020) have emerged as a powerful tool for diagnosing large annotated datasets. Given a model fitted on a dataset, these maps show each data instance from the dataset in a 2-dimensional space defined by a) the model’s confidence in the true class and b) the variability of this confidence. In previous work, confidence and variability are usually computed using training dynamics, which requires the fitting of a strong model to the dataset. In this paper, we introduce a novel approach: Zero-Shot Data Maps based on fast bi-encoder networks. For each data point, confidence on the true label and variability are computed over the members of an ensemble of zero-shot models constructed with different — but semantically equivalent — label descriptions, i.e., textual representations of each class in a given label space. We conduct a comparative analysis of maps compiled using traditional training dynamics and our proposed zero-shot models across various datasets. Our findings reveal that Zero-Shot Data Maps generally match those produced by the traditional method while delivering up to a 14x speedup. The code is available [here](https://github.com/symanto-research/zeroshot-cartography).

## introduction practices like annotation crowdsourcing and largescale web scraping have enabled nlp researchers to create large annotated corpora @xcite . concurrently, this growth in data volume has led to an increased prevalence of label noise and annotation artifacts within datasets @xcite . further complicating matters, the widening gap between dataset annotators and endusers -researchers and practitioners who use the datasets to train models -often transforms these datasets into black boxes @xcite . this division of labour, while economically advantageous in terms of time and resources, hinders thorough comprehension of the data. dataset cartography using training dynamics @xcite offers a partial solution to this problem as it can help interpreting annotated corpora with respect to a model. specifically, data maps reveal three loose groups of instances within an annotated dataset: easy-to-learn, ambiguous and hard-to-learn instances; the latter often correspond to annotation errors. training new models based on samples from these regions can lead to better models with improved generalization. while cartography based on training dynamics can generally compile high quality maps, it does so at the cost of training a large model on the target dataset. in this work, we introduce a novel, fast method for drawing data maps using zero-shot models. this approach capitalizes on the sensitivity of zeroshot models to label description framing, offering a more efficient and resource-friendly alternative to the original training dynamics method. our approach allows for the generation of data maps without the requirement of extensive computational resources for the training of a robust model on a large dataset. when evaluated on data selection and error detection tasks, our experimental results show that zero-shot map coordinates are as useful as coordinates obtained through training dynamic maps, while requiring a fraction of the time to compute. zero-shot data maps speed up the analysis of large datasets, removing the need for model training; they are a handy tool for industry practitioners managing daily unique datasets. research questions in this study, we aim to answer the following research questions (rq): contributions we introduce a novel method for compiling data maps up to 14x faster. we compare our method against training dynamics-based maps. furthermore, we show that zero-shot maps can be used for automatic annotation error detection. finally, we will provide an easy-to-use implementation of our method for enabling nlp practitioners to inspect new datasets. ## data maps with zero-shot models in this section, we first explain how our chosen zero-shot architecture works and why we chose it over other options (section 2.1). we then proceed to illustrate the sensitivity of zero-shot models to label descriptions (section 2.2). finally, we describe how we can use this feature to build data maps with zero-shot models, thus avoiding model training (section 2.3). ## experimental setup we first conduct a visual evaluation of the quality of our zero-shot data maps, comparing them with the maps obtained through training dynamics. next, we follow the original data selection experiments and train a set of models on 8 sub-samples of three datasets. finally, we evaluate zero-shot maps as error detectors and compare run time and resource usage against training dynamics maps. ## related work recently, several lines of research have presented findings related to the concept of dataset cartography. @xcite demonstrated how weak models, as opposed to strong ones, recreate similar data maps by leveraging annotation artifacts and biases. on the subject of instance difficulty, @xcite proposed v-usable information, a generalization of shannon information @xcite , which provides an aggregate measure of data difficulty in relation to a model; the pointwise v -information, an extension of v, has been shown to correlate strongly with data map coordinates, and offers the additional benefit of explaining why an instance can be challenging for a model. data maps have been recently applied to a variety of tasks such as syntactic parsing @xcite , hate speech detection @xcite , and visual question answering @xcite . furthermore, data maps have been shown to be an effective data selection tool for active learning @xcite . the idea of using semantic similarity for zero-shot classification can be traced back to @xcite . @xcite introduced the concept of dataless classi-fication by using label descriptions encoded with shallow features for zero-and few-shot classification. recently, @xcite have shown how modern approaches using dense bi-encoders can outperform more resource-intensive cross-encoder models. for a comprehensive review of automatic annotation error detection, one can refer to @xcite . ## conclusion in this paper, we have introduced the concept of zero-shot data maps, a new method for understanding annotated datasets. our evaluation on various datasets demonstrates the effectiveness of zero-shot data maps in providing a precise visual representation of labelled datasets. our zero-shot data maps can match the precision of maps produced with training dynamics, while providing the added benefit of being available instantly. however, training dynamic maps can be drawn for a broader array of tasks, making zero-shot maps as a valuable complement, rather than a direct replacement. furthermore, our approach proves to be an effective tool for spotting potential annotation errors: the confidence of ensembled zero-shot models acts as a reliable indicator for identifying incorrect gold labels. the quality of zero-shot maps largely depends on carefully crafted label descriptions and their ensemble combination. an intriguing direction for future research lies not only in improving the generation of these descriptions, potentially through automated methods, but also in experimenting with different zero-shot architectures to approach more task types than bi-encoders allow. the combination of our method with active learning and automated error detection algorithms could pave the way for more efficient dataset curation and improvement processes. we believe that our approach to create data maps can help researchers and practitioners improve data quality and ultimately build better models.