D ata L ab: A Platform for Data Analysis and Intervention (2022)

Abstract: Despite data’s crucial role in machine learning, most existing tools and research tend to focus on systems on top of existing data rather than how to interpret and manipulate data. In this paper, we propose DataLab, a unified data-oriented platform that not only allows users to interactively analyze the characteristics of data but also provides a standardized interface so that many data processing operations can be provided within a unified interface. Additionally, in view of the ongoing surge in the proliferation of datasets, DataLab has features for dataset recommendation and global vision analysis that help researchers form a better view of the data ecosystem. So far, DataLab covers 1,300 datasets and 3,583 of its transformed version, where 313 datasets support different types of analysis (e.g., with respect to gender bias) with the help of 119M samples annotated by 318 feature functions. DataLab is under active development and will be supported going forward. We have released a web platform, web API, Python SDK, and PyPI published package, which hopefully, can meet the diverse needs of researchers.

## related work toolkits for nlp pipelines there are a wealth of toolkits that support the processing of various nlp tasks, making it easier to build a composable nlp workflow. typical examples are nltk @xcite , nlpcurate @xcite , stanford corenlp @xcite , allennlp @xcite , spacy @xcite , gluonnlp @xcite , forte @xcite , huggingface @xcite . in contrast to these toolkits, datalab focuses on data analysis, bias diagnostics, and standardization of data-related operations. moreover, besides providing the sdk, datalab also provides a web-based interactive platform, featuring hundreds of datasets and millions of additional annotations w.r.t. diverse features. kyd @xcite also provides a web platform for data analysis but it mainly focuses on image data. explainaboard @xcite ) presents an analysis platform while it focuses on system diagnostics. ## datalab in this section we detail four major varieties of functionality provided by datalab. ## huggingface.tokenizers aggregating aggregation operations are used to compute corpus-level statistics such as tf-idf @xcite , label distribution. currently, datalab supports both generic aggregation operations applicable to any task and some customized ones for four nlp tasks (classification, summarization, extractive question answering and natural language inference). prompting prompt-based learning @xcite has received considerable attention, as better utilization of pretrained language models benefits many nlp tasks. in practice, what makes a good prompt is a challenging question. we define the prompt schema as shown in fig. 3 . the elements we included in a prompt cover diverse aspects including its features (e.g. length, shape, etc.), metadata (e.g. unique identifier, language, etc.), attributes (e.g. template, answers, etc ## case study we perform three case studies to show the utility of datalab and put more in the appendix. ## implications and roadmap datalab was born from our two visions (1) it is essential to standardize both the format of data and the interface of data-centric operations. in the future, we will extend datalab more broadly in terms of following perspectives: (1) index more data with different domains such as scientific and medical ones, and different modalities, such as image, video. (2) refine the current data typology over time, (3) add more built-in feature functions (e.g., labeling function @xcite ). ( 4 ) introduce more effective data management methods into datalab, such as fair. 19 searchers can gain a deeper and broader understanding of the characteristics of the datasets, developers can more easily access the datasets and manipulate the data samples, and analysts can see some social insights from the datasets. during the whole data analysis process, we tried to make it as transparent as possible, and the results of the analysis were well-grounded on sufficient evidence so that users could more reliably use it. additionally, users are encouraged to report a case where the annotation results are not precise. for those private datasets, we introduce three processing strategies for users: (1) users could flexibly use datalab sdk to deal with their data offline. (2) we introduce an account system foot_7 for users to set up a private space for their datasets. (3) we also open-source the code of datalab web platform foot_8 so that users can host this web service locally. as no feature analysis method will ever be perfectly reliable, 22 to alleviate this issue, so far, datalab sdk provides several built-in feature functions for users to choose from and also make it customized by users. we are now working on benchmarking different feature functions in different domains of data so that users can have a better idea when choosing. currently, datalab only supports public datasets. in addition, knowing more about the characteristics of the test sets might make overfitting easier for model training. one possible approach is through multi-dataset evaluation, i.e., a good system should achieve good results across a series of different datasets. datalab can privilege some datasets and data sources over others. to alleviate this problem, we introduce the dataset finder foot_10 functionality, where users can retrieve the dataset either from datalab and external resources like paperswithcode datasets foot_11 based on their requirement. additionally, we will continue to keep the data corpus updated. for example, we recently added datasets from acl2022. although datalab categorizes datasets into common tasks and paradigms, it still retains the original information provided by the creators of each dataset. for example, the text to be classified in imdb is named as "review" while in ag_news is named as "sentence". we introduce an intermediate variable foot_12 "text_column" to store the datasetdependent naming information. we appreciate any suggestions you have on how to make datalab better. your issues are highly welcome, foot_13 and we will actively update.