Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval (2024)

Abstract: Despite the recent advancements in information retrieval (IR), zero-shot IR remains a significant challenge, especially when dealing with new domains, languages, and newly-released use cases that lack historical query traffic from existing users. For such cases, it is common to use query augmentations followed by fine-tuning pre-trained models on the document data paired with synthetic queries. In this work, we propose a novel Universal Document Linking (UDL) algorithm, which links similar documents to enhance synthetic query generation across multiple datasets with different characteristics. UDL leverages entropy for the choice of similarity models and named entity recognition (NER) for the link decision of documents using similarity scores. Our empirical studies demonstrate the effectiveness and universality of the UDL across diverse datasets and IR models, surpassing state-of-the-art methods in zero-shot cases. The developed code for reproducibility is included in https://github.com/eoduself/UDL

## introduction in information retrieval (ir), zero-shot learning is an essential problem that emerges when dealing with a new language or domain with little to no availability of the associated queries. traditional ir methods primarily utilized sparse retrieval, while recent methods revolve around dense retrieval (dr), demonstrating the promising result @xcite . yet, using pre-trained dr directly on zero-shot cases results in substantial performance degradation, requiring dedicated finetuning @xcite . one strategy for fine-tuning without relying on query traffic involves expanding the queries based on existing queries or documents with rule-based methods or language models (lms) to obtain additional context in unseen domains @xcite @xcite . rm3 @xcite and axiomat-icqe @xcite are classical ways to expand the queries with additional relevant terms while the recent studies indicate that large lms (llms) can produce sophisticated synthetic data @xcite , often resulting in better transfer learning than human-curated datasets @xcite . while llms like gemini @xcite generate superb synthetic queries for fine-tuning, devising a cost-effective way for ir remains challenging without additional recipes like dimensionality reduction @xcite . to address the limitations of document-to-query generation, we propose a novel algorithm called universal document linking (udl), which offers an intuitive yet effective solution for zero-shot. ## motivation figure 1 illustrates the overall flow of fine-tuning a retrieval model in zero-shot scenario, where actual queries do not exist during fine-tuning. instead, we use documents to generate synthetic queries, which aids the ir model in learning the distribution of the unseen domain @xcite . according to hwang et al. (2023a) and our initial findings ( measure the cosine-similarity between candidate documents using a model from a 2. if cosine-@xmath1. in the first step, denoted as a, the appropriate similarity model is selected for each dataset. we explore term frequency-inverse document frequency (tf-idf) and pre-trained lm to derive document embeddings. notably, tf-idf considers lexical similarity, which is valuable for identifying unique features (e.g., disease like covid), while pre-trained lm provides semantic similarity, aiding in contextual understanding. to determine the suitable similarity model, we initially compute tf-idf scores for all documents, followed by calculating d m based on the shannon entropy of terms using tf-idf. entropy values greater than 1 (i.e., numerator in d m ) describe high uncertainty since random variables have approximately uniform distribution in multiple classes. this concept is extended to the term entropy (equation ( 1 )) where we calculate the entropy for each term across documents. to accommodate the d m for the massive documents, we introduce the γ value where articles and relatively common terms are mostly distributed in entropy greater 1 as expected (see after defining the similarity model, we proceed to determine the criteria in step b for deciding whether candidate documents should be linked. each dataset contains varying levels of domainspecific terminology, which must be taken into account during document linking. to address this, we initially translated non-english documents into english using google translator 2 to handle multilingual cases. after removing special characters, we compute d t based on the number of keywords extracted from ner models that are pre-trained on general (n g ) and specialized documents (n s ) while considering the vocabulary size of each ner for unbiased comparison. note that a large size of vocabulary can have a higher chance of capturing broad keywords. the entity coverage is detailed in based on this analysis, a higher value of d t indicates that a dataset is more similar to a group of general documents, enabling the linking of diverse documents without concerns of domain-specific jargon, resulting in a lower score (i.e., δ). conversely, a lower d t value suggests that a dataset consists of specialized documents, which benefits from linking similar documents that share domain-specific jargon, resulting in higher scores (i.e., 1δ). thus, general and specialized documents are considered opposites. in section 4, we tested the udl across multiple datasets from different domains (e.g., qa, scientific documents) to show its applicability without requiring a specific ner for each domain. this was confirmed with the selected ners but our udl could be readily extendable to any other ner. ## results and discussions experimental setup the details of the experimental setup are covered in appendix a, where we empirically set two hyperparameters in udl as γ=0.7 and δ=0.4, and reported the averaged ndcg@k (n@k) and recall@k (r@k), along with the standard deviation (sd). for reproducibility, the training framework is covered in appendix b, and the code is included in the supplementary material. steps of fine-tuning are as follows: (1) classifying linked and unlinked documents based on udl, taking into account the order of the linked ones. (2) feeding them as the inputs to the models and generating the synthetic queries with the same process as the original approach (e.g., model or prompt-based generations). (3) fine-tuning the ir models based on generated queries. research questions we aim to address four research questions (rqs): rq1. what is the most suitable query augmentation method in zero-shot ir? rq2. how does udl enhance zero-shot ir? rq3. how well does udl generalize? rq4. is udl competitive with state-of-the-art (sota)? main results furthermore, we ablated the document merging mechanism of udl by generating the synthetic queries from each document individually and mapping them to documents found by the linking procedure (mapping+qgen in lastly, we investigated the influence of decisions in udl separately. we compared the results between fixed similarity models (i.e., tf-idf or lm+qgen) and flexible ones (i.e., udl+qgen) where the latter excels. also, we tested the results by fixing the similarity scores (i.e., fixed score (0.4) or fixed score (0.6)+qgen) and lm where flexible scores from udl enhances the performance. therefore, our evolved approach with flexible choices of the similarity models and scores promises the results. hyperparameters choice figure 3 shows the grid search for udl's hyperparameters using nf-corpus, scifact, and arguana yielding γ=0.7 and δ=0.4 as most optimal. (see tables 14 and 15 for detailed results). we also checked the quality between synthetic queries and the offered train queries in used datasets. detail of logic is shown in algorithm 2 where 93% of synthetic queries generated from linked documents in udl have sufficient quality as the train set to map the relevant documents. does udl generalize? ## conclusions we propose a novel udl to mitigate the limitations of conventional fine-tuning of ir models in zeroshot. udl uses entropy and ner to tailor a linking method for each dataset with diverse tasks. our comprehensive experiments show the effectiveness of udl across various datasets and models.