HIL : Hybrid Isotropy Learning for Zero-shot Performance in Dense retrieval (2024)

Abstract: Advancements in dense retrieval models have brought ColBERT to prominence in Information Retrieval (IR) with its advanced interaction techniques.However, ColBERT is reported to frequently underperform in zero-shot scenarios, where traditional techniques such as BM25 still exceed it.Addressing this, we propose to balance representation isotropy and anisotropy for zero-shot model performance, based on our observations that isotropy can enhance cosine similarity computations and anisotropy may aid in generalizing to unseen data.Striking a balance between these isotropic and anisotropic qualities stands as a critical objective to refine model efficacy.Based on this, we present ours, a Hybrid Isotropy Learning (HIL) architecture that integrates isotropic and anisotropic representations.Our experiments with the BEIR benchmark show that our model significantly outperforms the baseline ColBERT model, highlighting the importance of harmonized isotropy in improving zero-shot retrieval performance.

## introduction recent advancements in information retrieval (ir) have seen a shift from classical techniques like bm25 to more advanced models that rely on dense vector representations. colbert @xcite ) is one such model utilizing multivector representations to capture token-level interactions, for often exceeding performance of single-vector dense models like dpr @xcite . despite its strengths, colbert lags behind bm25 and other methods in zero-shot tasks @xcite , revealing a weakness in its representational strategy. to improve representation quality from an isotropy perspective, methods can be categorized into approaches that promote isotropy and those that induce anisotropy. specifically, approaches such as cosreg @xcite promote isotropy, ## proposed method we introduce interiso measuring isotropy in section 3.1, and then discuss ensemble model using interiso in section 3.2. we then propose colbert-hil mitigating the limitations of ensemble in section 3.3. ## conclusion addressing the challenge of applying prior insights on isotropy to ir, we introduce interiso, a metric designed to quantify isotropy between queries and passages in dr settings. leveraging interiso, we propose hil architecture that seamlessly integrates isotropic and anisotropic representations. our experiments with beir benchmark demonstrate that colbert-hil significantly outperforms the baselines, colbert and bm25. it indicates the pivotal role of harmonized isotropy in enhancing zero-shot retrieval performance.