An Empirical Investigation of Implicit and Explicit Knowledge-Enhanced Methods for Ad Hoc Dataset Retrieval (2023)

Abstract: Ad hoc dataset retrieval has become an important way of finding data on the Web, where the underlying problem is how to measure the relevance of a dataset to a query. State-of-the-art solutions for this task are still lexical methods, which cannot capture semantic similarity. Semantics-aware knowledge-enhanced retrieval methods, which achieved promising results on other tasks, have yet to be systematically studied on this specialized task. To fill the gap, in this paper, we present an empirical investigation of the task where we implement and evaluate, on two test collections, a set of implicit and explicit knowledge-enhancement retrieval methods in various settings. Our results reveal the unique features of the task and suggest an interpolation of different kinds of methods as the current best practice.

## introduction tens of millions of datasets have been published on the web @xcite , providing government data, scientific data, etc. accordingly, ad hoc dataset retrieval is becoming an important specialized information retrieval task @xcite , aiming at finding datasets that are relevant to a user's query, which is ad hoc since the number of possible queries is huge. due to the magnitude and heterogeneity of dataset contents, existing solutions such as google dataset search @xcite rely on the retrieval of dataset metadata provided by data publishers for describing their datasets, as illustrated in motivation. the metadata of a dataset resembles a structured multi-field document, and current implementations of ad hoc dataset retrieval are mainly adapted from conventional document retrieval methods like bm25 or fsdm (chapman * corresponing author @xcite . it is well-known that such lexical retrieval methods cannot identify semantic matches and hence fail to find datasets that are lexically disjointed with but semantically relevant to the query. the problem can be alleviated by incorporating knowledge into retrieval-either implicit knowledge encoded in a pre-trained language model (plm) or explicit knowledge stored in an encyclopedic knowledge base. for example, to identify the semantic connection between "protein" in the query and "nutrient" in the dataset in rq1. despite a few preliminary plm-based implementations @xcite , they were finetuned on a small training set. researches on ad hoc dataset retrieval are still in their infancy, and existing test collections provide less than 300 queries for training @xcite . the performance of plm-based methods fine-tuned in such an in-domain setting may not be generalizable to practical settings. will implicit knowledge-enhanced retrieval methods remain effective in ad hoc dataset retrieval in an out-of-domain setting? rq2. the high-quality encyclopedic knowledge bases available today, like wikipedia and wikidata, can be used to annotate query and dataset metadata so that their semantic similarity can be measured to enhance lexical retrieval. more importantly, this is unsupervised and not limited by the availability of training data. however, such methods are so far under-studied and their effectiveness remains unknown. will explicit knowledge-enhanced retrieval methods be effective in ad hoc dataset retrieval? rq3. lexical matching, implicit knowledge, and explicit knowledge-they have the potential to capture different signals in retrieval. while each of them alone may not exhibit superb performance, their appropriate combination may produce better results, e.g., by an interpolation of their retrieval scores. will interpolated methods be more effective in ad hoc dataset retrieval? our work and contribution. to answer the above questions, we conducted a systematic investigation of implicit and explicit knowledge-enhanced methods for ad hoc dataset retrieval on two test collections @xcite . for implicit knowledge, we evaluated five plm-based methods in both in-domain and out-of-domain settings. for explicit knowledge, we designed and evaluated methods based on entity similarity computed over two knowledge bases. we also explored different interpolation strategies. as the first empirical investigation of this kind, our work fills the gap and our results will provide practical guidelines for researchers and developers working with ad hoc dataset retrieval, or even information retrieval in general. it helps establish an empirical basis that will facilitate future studies on this trending information retrieval task. code: https://github.com/nju-websoft/ahdrknowledgeenhanced paper structure. we will discuss related work in section 2, describe the evaluated methods in section 3, present our experimental setup and results in section 4 and section 5, respectively, and finally conclude the paper in section 6. ## methods we divide knowledge-enhanced methods for ad hoc dataset retrieval into two types: using implicit knowledge and using explicit knowledge. employing implicit knowledge embodied in plms to enhance retrieval has been widely used, which we will briefly review in section 3.1. in section 3.2 we will present a method for employing explicit knowledge in a knowledge base to enhance retrieval. problem statement. given a query q, the main task in ad hoc dataset retrieval is to compute the relevance of each dataset d to q denoted by rel(d, q), so that a ranked list of datasets can be returned. ## experimental results each of the following three subsections reported experimental results to answer one of the three research questions raised in section 1. ## conclusion and future work we summarize our empirical findings to answer the three research questions raised in section 1. rq1: will implicit knowledge-enhanced retrieval methods remain effective in ad hoc dataset retrieval in an out-of-domain setting? according to the results presented in section 5.1, only monot5 remained effective on its own, whereas other plm-based methods could not consistently bring improvements. however, interpolated with bm25 scores, most of these methods effectively generalized to this new task as shown by the results in section 5.3.1. it demonstrates the necessity of combining dense and sparse retrieval for this task. rq2: will explicit knowledge-enhanced retrieval methods be effective in ad hoc dataset retrieval? according to the results presented in section 5.2 and section 5.3.1, reranking with explicit knowledge could be beneficial to this task only when performing interpolation with bm25 scores, and significant improvements were consistently observed only for the configuration using wikipedia knowledge with tagme for entity linking. it suggests that the incorporation of explicit knowledge into this task should be carefully designed, and calls for more effective implementations in the future. rq3: will interpolated methods be more effective in ad hoc dataset retrieval? according to the results presented in section 5.3.1 and section 5.3.2, not only interpolation with bm25 scores was helpful, but also implicit and explicit knowledge exhibited complementarity. a combination of lexical matching, implicit knowledge, and explicit knowledge consistently achieved the best performance in our experiments, representing the current best practice for solving this task. hopefully our empirical findings and conclusions should provide useful guidelines for the community to research and practice ad hoc dataset retrieval. in future work, we will continue to explore and address the unique challenges of this important task. regarding implicit knowledge-enhanced retrieval, we are interested in constructing a large test collection to support in-domain supervision of more generalizable plm-based retrieval methods. we also plan to apply plm-based methods to not only dataset metadata but also dataset contents which are large and heterogeneous, thus posing great challenges. a related trending research direction is to explore the effectiveness of large language models in ad hoc dataset retrieval. regarding explicit knowledge-enhanced retrieval, since the accuracy of entity linking observed in our experiments was not satisfactory and might distort the results, an idea is to extend existing test collections with manually annotated entities, which would also provide a useful resource for entity linking research.