Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps (2022)

Abstract: IR models using a pretrained language model significantly outperform lexical approaches like BM25. In particular, SPLADE, which encodes texts to sparse vectors, is an effective model for practical use because it shows robustness to out-of-domain datasets. However, SPLADE still struggles with exact matching of low-frequency words in training data. In addition, domain shifts in vocabulary and word frequencies deteriorate the IR performance of SPLADE. Because supervision data are scarce in the target domain, addressing the domain shifts without supervision data is necessary. This paper proposes an unsupervised domain adaptation method by filling vocabulary and word-frequency gaps. First, we expand a vocabulary and execute continual pretraining with a masked language model on a corpus of the target domain. Then, we multiply SPLADE-encoded sparse vectors by inverse document frequency weights to consider the importance of documents with low-frequency words. We conducted experiments using our method on datasets with a large vocabulary gap from a source domain. We show that our method outperforms the present state-of-the-art domain adaptation method. In addition, our method achieves state-of-the-art results, combined with BM25.

## introduction information retrieval (ir) systems are widely used nowadays. most of them are based on lexical approaches like bm25 @xcite . because lexical approaches are based on bag-of-words (bow), they suffer from vocabulary mismatch, where different words express the same notion. recently, ir models with a pretrained masked language model (mlm), such as bert @xcite have overcome this problem and outperformed bm25 @xcite @xcite . in particular, splade @xcite ) is an effective model for practical use. splade addresses vocabulary mismatch by expanding queries and documents through an mlm. concretely, splade encodes texts to sparse vectors using the logits of the mlm for each token of the texts. as a result, each element of these vectors corresponds to a word in the vocabulary of the mlm. in addition, the nonzero elements other than tokens appearing in the texts can be considered as query and document expansion. because the encoded vectors are sparse, splade can realize a fast search by utilizing inverted indexes and outperforms bm25, even when splade is applied to out-of-domain datasets from a source domain of training data. however, splade still struggles with the exact matching of low-frequency words in the training data @xcite . this problem is amplified for out-of-domain datasets. in addition, @xcite discussed that large domain shifts in vocabulary and word frequencies deteriorate the performance of vector-based ir models. furthermore, preparing massive supervision data for every dataset is impractical due to annotation costs. thus, a method to address the domain shifts without supervision data is necessary. unsupervised domain adaptation (uda) is an approach to overcome domain shift without supervision data. however, as discussed in section 7, generated pseudo labeling (gpl) @xcite , a state-of-the-art uda method using generated queries, cannot solve the problem of lowfrequency words on some datasets. in this paper, we propose a uda method that fills the vocabulary and word-frequency gap between the source and target domains. specifically, we use adalm @xcite , which is a domain adaptation method for an mlm through vocabulary expansion @xcite and continual pre-training @xcite on a domain-specific corpus. we expect adalm to realize more accurate query and document expansions in the target domain. furthermore, because splade struggles with exact matching of lowfrequency words in the training data, we weight such words by multiplying each element of the splade-encoded sparse vectors by inverse document frequency (idf) weights. we call our method combination of adalm and idf (cai). we apply cai to splade and conducted experiments with it on five ir datasets from biomedical and science domains in the beir benchmark @xcite . we used these datasets because the five datasets have the largest vocabulary gap in beir from ms @xcite , the source dataset. the experimental results show that splade with cai outperforms splade with gpl and achieves state-the-art results on average across all datasets by adding scores of bm25. finally, to confirm whether cai can address the problem of exact matching words of low-frequency words in training data, we analyzed the weights of exact matching words, following the approach of @xcite . our analysis confirms that splade with cai addresses the problem of the exact matching, whereas splade with gpl cannot. our contributions can be summarized as follow: ## method this paper proposes the uda method to tackle the domain shifts in vocabulary and word frequency. an outline of our method is illustrated in figure 1 . our method consists of three parts: (1) executing adalm for domain adaptation of an mlm, (2) training splade with supervised data, and (3) weighting sparse vectors encoded by splade with idf when searching. our proposal parts are (1) and (3). we first introduce splade as preliminary. ## experimental setup we confirm the effectiveness of our proposed cai through experimental results. first, we introduce baselines. they show us how effective cai is. second, we explain ir datasets and domain corpora. the last subsection gives details of the implementation. 1 ## results this section compares our method with baselines. we first show the results of our approach and other ir methods. next, we present the results of cai and gpl as a comparison of uda methods. ## ablation with adalm for confirming assumption we conducted ablation studies for adalm to confirm the assumptions presented in section 3.2. precisely, we considered the case of only applying vocabulary expansion or continual pretraining. in addition, we also used models trained on a domain-specific corpus from scratch. by comparing adalm with continual pretraining, we confirm whether ir performance is further enhanced by expanding the vocabulary. in addition, we observe the effect of vocabulary size based on the results achieved by vocabulary expansion and the scratch models. as scratch models, we used pub-medbert 11 @xcite for the biomedical domain and scibert 12 @xcite for the science domain. by contrast, the scratch model of the biomedical domain failed to learn splade. thus, scratch models on the domain-specific corpus may not learn splade, and adalm seems a more stable method than the scratch models. we further analyzed the effect of vocabulary size on adalm in section e. the result suggests that adalm with a larger vocabulary size tended to perform better in ndcg@10 for splade. ## analysis for weight of words this section shows whether our method can solve the problem of exact matching of low-frequency words. @xcite analyzed ir models with an mlm, using robertson spärck jones (rsj) weight @xcite . rsj weight measures how a token can distinguish relevant from non-relevant documents in an ir corpus. this weight also indicates an ideal weight in terms of lexical matching. we denote rsj weight as rsj t,q for a tokenized query @xmath0. to infer the rsj weight of the ir models, @xcite replaced relevant documents with top-k documents retrieved by the model. we express the inferred rsj weight as rsj t,q . we set @xmath1, following the authors. we give the formal expression of rsj t,q and rsj t,q in appendix b. following @xcite , we take the difference between rsj t,q and rsj t,q , i.e., as an indicator to measure the gap between the ideal rsj weight and rsj weight of the models. if ∆rsj t,@xmath2, an ir model overestimates the weights of tokens. conversely, if ∆rsj t,@xmath3, an ir model underestimates the weight of the tokens. for analysis, we also divide all tokens into highrsj and lowrsj at the 75-percentile. furthermore, we split all tokens into groups of high-idf and lowidf at the median idf of all tokens in queries. this analysis is conducted on the nf-corpus. the tokenizer used is the analyzer of pyfigure 2 shows the ∆rsj t,q of splade, splade with gpl, splade with cai, and hybrid splade with cai. first, splade with cai underestimates the rsj weight less than splade in the groups of highrsj and highidf. in addition, hybrid-splade with cai is closer to ∆rsj t,@xmath4. this result suggests that our approach solves the problem of term matching for rare words. by contrast, splade with gpl shows lower ∆rsj t,q than splade. gpl seems to accelerate the problem of term matching for lowfrequency words. as a result, gpl may lead to lower ir performance than splade, as shown in ## case study finally, we confirm the case where splade with cai improves the ir performance by matching important and rare words, i.e., highrsj and highidf words. ## conclusion this paper presented an effective unsupervised domain adaptation method, cai. we showed that the combination of splade with cai and the lexical approach gave a state-of-the-art performance on datasets with a large vocabulary and wordfrequency gap. in addition, cai outperformed gpl and was robust enough to show high accuracy even when bow representations were used for query expression. finally, our analysis showed that splade with cai addressed the problem of the exact matching of low-frequency words in training data. we believe that cai works on smaller mlms by distilling adalm because @xcite showed that a distilled adalm achieved higher performance than bert on nlp tasks @xcite showed that the results of splade initialized with distilbert-base 14 was competitive on ms marco with other ir models initialized with bert. integrating sparse and dense retrieval is a promising way to enhance ir performance further on out-of-domain datasets. future work will integrate them to reveal the factors contributing to ir. macedo maia, siegfried handschuh, andré freitas, brian davis, ross mcdermott, manel zarrouk, and alexandra balahur. 2018. www'18 open challenge: financial opinion mining and question answering. in companion proceedings of the the web conference 2018, www '18, page 1941-1942. international world wide web conferences steering committee. tri nguyen, mir rosenberg, xia song, jianfeng gao, saurabh tiwary, rangan majumder, and li deng. 2016. ms marco: a human generated machine reading comprehension dataset. in proceedings of the workshop on cognitive computation: integrating neural and symbolic approaches 2016 co-located with the 30th annual conference on neural information processing systems (nips 2016), barcelona, zhilin yang, peng qi, saizheng zhang, yoshua bengio, william cohen, ruslan salakhutdinov, and christopher d. manning. 2018. hotpotqa: a dataset for diverse, explainable multi-hop question answering. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369-2380, brussels, belgium. association for computational linguistics. yunzhi yao, shaohan huang, wenhui wang, li dong, and furu wei. 2021. adapt-and-distill: developing small, fast and effective pretrained language models for domains. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 460-470, online. association for computational linguistics.