info V erse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information (2023)

Abstract: The success of NLP systems often relies on the availability of large, high-quality datasets. However, not all samples in these datasets are equally valuable for learning, as some may be redundant or noisy. Several methods for characterizing datasets based on model-driven meta-information (e.g., model’s confidence) have been developed, but the relationship and complementary effects of these methods have received less attention. In this paper, we introduce infoVerse, a universal framework for dataset characterization, which provides a new feature space that effectively captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. infoVerse reveals distinctive regions of the dataset that are not apparent in the original semantic space, hence guiding users (or models) in identifying which samples to focus on for exploration, assessment, or annotation. Additionally, we propose a novel sampling method on infoVerse to select a set of data points that maximizes informativeness. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines in all applications. Our code and demo are publicly available.

## introduction the construction of large datasets is one of the essential ingredients for success in various nlp tasks @xcite . however, not all data points are equally important to learn from; many datasets often contain low-quality samples, e.g., incorrect labels @xcite or annotation artifacts @xcite . thus, data characterization (roth and @xcite , a technique for transforming raw data into useful information for a target task, has a huge potential to improve the model's performance by trimming the problematic samples @xcite or providing better practices for effective data collection, e.g., active learning @xcite and adversarial annotation @xcite . however, data characterization via human assessment is highly limited due to the huge cost of dealing with a large dataset and the vagueness of the assessment itself. to this end, several model-driven meta-information foot_0 have been investigated; for example, the model's confidence is a standard meta-information widely used in active learning @xcite . @xcite recently show that the training dynamics of the model's prediction can indicate the relative importance of training samples. various types of metainformation are continuously proposed from different intuitions @xcite , but their relationship and potential beyond relying on individual one have yet to be explored. hence, this work answers the following two research questions: (1) is there a (hidden) complementary effect between various meta-information for better data characterization, and (2) is the combined metainformation useful for real-world applications? in this paper, we introduce infoverse: a universal framework for better dataset characterization by incorporating multiple aspects of data characteristics. to be specific, infoverse combines various types of meta-information which offer the different aspects of data characteristics (e.g., how difficult the sample is to learn, how certain multiple models are, and how likely the sample is). consequently, we can extract richer information about data informativeness from their complementary effect, and infoverse could guide users (or models) in what samples to focus on for the exploration, assessment, or annotation. to extend the advantages of infoverse into real-world problems, we further propose a novel sampling method suitable for in-foverse based on determinantal point processes (dpp), which is known to be effective for finding a diverse and high-quality set of samples @xcite . it enables us to select data points that maximize the information at a set level rather than a sample level on the multidimensional space of infoverse. in detail, we first construct infoverse based on the diverse meta-information, which could be broadly classified into four different categories in section 3. the complementary effect from the multiple meta-information in infoverse helps reveal distinct regions in the dataset, such as hard-to-predict and mis-labeled ones, which are not observable in the original semantic feature space (section 4). in section 5, we empirically show that our framework has consistently outperformed the strong baselines in various data-centric applications, like data prun-ing @xcite , active learning @xcite , and data annotation @xcite , although it is not specifically designed for those problems. this result opens up the potential of infoverse to other data-centric applications, unlike the application-specified approaches. our results show that a dataset could be distinctively characterized when many different but complementary dimensions are considered together. we believe that our infoverse framework could evolve continuously with the development of new meta-information and hence serve as an effective platform for better characterization of datasets and construction of high-quality datasets. ## infoverse: universal framework for multi-aspect data characterization in this section, we present infoverse, a universal framework for better data characterization. our high-level idea is extracting the complementary effect between various meta-information, as they are oriented from the different aspects of data characteristics. in section 3.1, we briefly introduce the used meta-information to construct infoverse. in section 3.2, we present a novel sampling method to extend the advantages of infoverse for solving real-world problems. we remark that our framework can be easily extended with a new metainformation and not limited to specific ones, while the fixed ones are used for the experiments. ## dataset characterization via infoverse in this section, we demonstrate how infoverse could help analyze a given dataset via better data characterization. specifically, figure 4 presents infoverse on qnli dataset foot_1 along with other representative feature spaces for data characterization: classifier's embedding (at final layer before linear head) and data map @xcite . as the classifier's embedding and infoverse are high dimensional space, we project them to 2d space via t-sne (van der @xcite for visualization. first, one can observe that infoverse maps the samples into distinguishable regions based on their characters; for example, samples with high variability are further mapped to some different regions. to be specific, in figure 4 , they have high variability in a common while having a difference in other measures: the regions with dotted boxes have relatively high ensemble entropy and mc entropy, respectively, which cannot be distinguishable in the data map, showing the advantage of infoverse in dataset characterization. this benefit of infoverse for dataset characterization is more clear when we focus on the incorrectly predicted samples (black squares/circles in figure 4 ). as shown in the left of figure 4 , it is hard to find their characteristics on the classifier's embedding as they are scattered over different regions. data map @xcite ) maps these samples to regions with low confidence (hard-to-learn) or high variability (ambiguous), but not perfectly as these regions also include correctly predicted samples. in contrast, infoverse successfully characterizes these incorrectly predicted samples and maps them into three distinct regions with a different distribution of meta-information: 1) hard-anddisagreed, 2) easy-to-mistake, and 3) ambiguous. as shown in the right of figure 4 , both hard-anddisagreed and ambiguous regions have high ensemble uncertainty, but hard-and-disagreed region has relatively low confidence and variability which means that it is also hard to learn. it might imply its incorrect prediction due to intrinsic difficulty as one can verify in the examples in figure 4 . in contrast, easy-to-mistake region has much lower uncertainty than other incorrectly predicted regions, which indicates the prediction is certainly wrong. it might indicate that the mistakes happened during annotation even though they are easy ones to annotate correctly. more results of data characterization on other datasets with infoverse are presented in appendix d. ## infoverse for real-world applications in this section, we demonstrate the advantages of infoverse on three real-world problems: 1) data pruning @xcite , 2) active learning @xcite , and 3) data annotation @xcite . these problems commonly require characterizing and quantifying data to determine which samples to select, but with dif-ferent goals; hence, various methods have been explored separately. in contrast, we will demonstrate the potential of infoverse as a universal framework to deal with such data-centric problems. ## conclusion we propose a new framework, infoverse to characterize the dataset in various aspects of data informativeness. to be specific, infoverse utilizes various types of meta-information which offer different aspects of data characteristics. the combination of diverse meta-information helps detect distinct regions of dataset characteristics, which are not observable in the previous feature spaces. in addition, we further propose a novel sampling method to select data points that maximize the information at a set level rather than a sample level on the multidimensional space of infoverse. we empirically demonstrate the benefit of infoverse on three applications: data pruning, active learning, and data annotation. infoverse with the proposed subset selection method shows consistent improvement over the strong baselines of each problem. we believe our framework will emerge with the growth of data-centric approaches and contribute to a better understanding of the dataset and improvement of the dataset's quality.