D ata F inder: Scientific Dataset Recommendation from Natural Language Descriptions (2023)

Abstract: Modern machine learning relies on datasets to develop and validate research ideas. Given the growth of publicly available data, finding the right dataset to use is increasingly difficult. Any research question imposes explicit and implicit constraints on how well a given dataset will enable researchers to answer this question, such as dataset size, modality, and domain. We operationalize the task of recommending datasets given a short natural language description of a research idea, to help people find relevant datasets for their needs. Dataset recommendation poses unique challenges as an information retrieval problem; datasets are hard to directly index for search and there are no corpora readily available for this task. To facilitate this task, we build the DataFinder Dataset which consists of a larger automatically-constructed training set (17.5K queries) and a smaller expert-annotated evaluation set (392 queries). Using this data, we compare various information retrieval algorithms on our test set and present a superior bi-encoder retriever for text-based dataset recommendation. This system, trained on the DataFinder Dataset, finds more relevant search results than existing third-party dataset search engines. To encourage progress on dataset recommendation, we release our dataset and models to the public.

## introduction innovation in modern machine learning (ml) depends on datasets. the revolution of neural network models in computer vision @xcite was enabled by the imagenet large scale visual recognition challenge @xcite . similarly, data-driven models for syntactic parsing saw rapid development after adopting the penn treebank @xcite . figure 1 : queries for dataset recommendation impose constraints on the type of dataset desired. keyword queries make these constraints explicit, while fullsentence queries impose implicit constraints. ground truth relevant datasets for this query are colored in blue. with the growth of research in ml and artificial intelligence (ai), there are hundreds of datasets published every year (shown in figure 2 ). knowing which to use for a given research idea can be difficult @xcite . to illustrate, consider a real query from a graduate student who says, "i want to use adversarial learning to perform domain adaptation for semantic segmentation of images." they have implicitly issued two requirements: they need a dataset for semantic segmentation of images, and they want datasets that include diverse visual domains. a researcher may intuitively select popular, generic semantic segmentation datasets like coco @xcite or ade20k @xcite , but these are insufficient to cover the query's requirement of supporting domain adaptation. how can we infer the intent of the researcher and make appropriate recommendations? to study this problem, we operationalize the task of "dataset recommendation": given a fullsentence description or keywords describing a research topic, recommend datasets to support research on this topic ( §2). a concrete example is shown in figure 1 . this task was introduced by @xcite , who framed it as text classification. in contrast, we naturally treat this task as retrieval @xcite , where the search collection is a set of datasets represented textually with dataset descriptions 3 , structured metadata, and published "citances" -references from published papers that use each dataset @xcite . this framework allows us to measure performance with rigorous ranking metrics such as mean reciprocal rank @xcite . to strengthen evaluation, we build a dataset, the datafinder dataset, to measure how well we can recommend datasets for a given description ( §3). as a proxy for real-world queries for our dataset recommendation engine, we construct queries from paper abstracts to simulate researchers' historical information needs. we then identify the datasets used in a given paper, either through manual annotations (for our small test set) or using heuristic matching (for our large training set). to our knowledge, this is the first expert-annotated corpus for dataset recommendation, and we believe this can serve as a challenging testbed for researchers interested in representing and searching complex data. we evaluate three existing ranking algorithms on our dataset and task formation, as a step towards solving this task: bm25 @xcite , nearest neighbor retrieval, and dense retrieval with neural bi-encoders @xcite . bm-25 is a standard baseline for text search, nearest neighbor retrieval lets us measure the degree to which this task requires generalization to new queries, and bi-encoders are among the most effective search models used today @xcite . compared with third-party keywordcentric dataset search engines, a bi-encoder model trained on datafinder is far more effective at finding relevant datasets. we show that finetuning the 3 from www.paperswithcode.com bi-encoder on our training set is crucial for good performance. however, we observe that this model is as effective when trained and tested on keyphrase queries as on full-sentence queries, suggesting that there is room for improvement in automatically understanding full-sentence queries. ## dataset recommendation task we establish a new task for automatically recommending relevant datasets given a description of a data-driven system. given a query q and a set of datasets d, retrieve the most relevant subset r ⊂ d one could use to test the idea described in q. figure 1 illustrates this with a real query written by a graduate student. the query q can take two forms: either a keyword query (the predominant interface for dataset search today @xcite ) or a fullsentence description. textual descriptions offer a more flexible input to the recommendation system, with the ability to implicitly specify constraints based on what a researcher wants to study, without needing to carefully construct keywords a priori. evaluation metrics our task framing naturally leads to evaluation by information retrieval metrics that estimate search relevance. in our experiments, we use four common metrics included in the trec_eval package, foot_0 a standard evaluation tool used in the ir community: ## the datafinder dataset to support this task, we construct a dataset called the datafinder dataset consisting of (q, r) pairs extracted from published english-language scientific proceedings, where each q is either a fullsentence description or a keyword query. we collect a large training set through an automated method (for scalability), and we collect a smaller test set using real users' annotations (for reliable and realistic model evaluation). in both cases, our data collection contains two primary steps: (1) collecting search queries q that a user would use to describe their dataset needs, and (2) identifying relevant datasets r that match the query. our final training and test sets contain 17495 and 392 queries, respectively. figure 3 summarizes our data collection approach. we explain the details below and provide further discussion of the limitations of our dataset in the limitations section. we will release our data under a permissive cc-by license. ## experimental setup on datafinder how do popular methods perform on our new task and new dataset? how does our new paradigm differ from existing commercial search engines? in this section, we describe a set of standard methods which we benchmark, and we consider which thirdparty search engines to use for comparison. ## related work most work on scientific dataset recommendation uses traditional search methods, including termbased keyword search and tag search @xcite @xcite @xcite . in 2019, google research launched dataset search @xcite , offering access to over 2 million public datasets. our work considers the subset of datasets from their search corpus that have been posted on papers with code. some work has explored other forms of dataset recommendation. ben ellefi et al. (2016) study using "source datasets" as a search query, while @xcite use a set of related research papers as the user's query. @xcite are the only prior work we are aware of that explores natural language queries for dataset recommendation. they model this task as classification, while we operationalize it as open-domain retrieval. their dataset uses abstracts and citation contexts to simulate queries, while we use realistic short queries (with an expert-annotated test set). ## conclusion we study the task of dataset recommendation from natural language queries. our dataset supports search by either full-sentence or keyword queries, but we find that neural search algorithms trained for traditional keyword search are competitive with the same architectures trained for our proposed fullsentence search. an exciting future direction will be to make better use of natural language queries. we release our datasets along with our ranking systems to the public. we hope to spur the community to work on this task or on other tasks that can leverage the summaries, keyphrases, and relevance judgment annotations in our dataset.