C o RAG : Collaborative Retrieval-Augmented Generation (2025)

Abstract: Retrieval-Augmented Generation (RAG) models excel in knowledge-intensive tasks, especially under few-shot learning constraints. We introduce CoRAG, a framework extending RAG to collaborative settings, where clients jointly train a shared model using a collaborative passage store. To evaluate CoRAG, we introduce CRAB, a benchmark for collaborative homogeneous open-domain question answering. Our experiments demonstrate that CoRAG consistently outperforms both parametric collaborative learning methods and locally trained RAG models in low-resource scenarios. Further analysis reveals the critical importance of relevant passages within the shared store, the surprising benefits of incorporating irrelevant passages, and the potential for hard negatives to negatively impact performance. This introduces a novel consideration in collaborative RAG: the trade-off between leveraging a collectively enriched knowledge base and the potential risk of incorporating detrimental passages from other clients. Our findings underscore the viability of CoRAG, while also highlighting key design challenges and promising avenues for future research.

## introduction retrieval-augmented generation (rag) models @xcite @xcite , which incorporate large external datastores of text passages, have shown promise in knowledge-intensive and few-shot tasks. however, their exploration has mainly focused on centralized settings where a single entity controls both the model and the datastore. the potential of rag within a collaborative learning framework, where multiple clients jointly train a shared model without directly exchanging their labeled data @xcite , but potentially building 1 code is available at https://github.com/ aashiqmuhamed/corag a shared passage store, remains largely unexplored. consider competing businesses in the same industry, each possessing expensive to acquire (labeled) data on customer behavior. directly sharing these data would be strategically disadvantageous, yet they could collaborate to build a shared passage store of relatively inexpensive (unlabeled) market research documents and economic analyses. this allows them to collectively train a more effective rag model for market prediction without revealing their valuable labeled data. this approach, particularly in low-resource settings enables them to train a more effective model than any single client could achieve independently. this work introduces corag, a framework for collaborative rag that enables multiple clients to jointly train a shared model using a collaborative passage store, while allowing them to use their local passage stores during inference. corag introduces unique challenges stemming from the dynamics of constructing and utilizing this shared store. the composition of this knowledge base, particularly the balance of relevant, irrelevant, and hard-negative passages, significantly impacts the model's performance and generalization capabilities. our experiments reveal that relevant passages are crucial for model generalization, while hard negatives can be detrimental, and, surprisingly, irrelevant passages can even be beneficial. this introduces a fundamental tension in corag: clients must balance the advantages of a richer, shared knowledge base with the risk of incorporating potentially detrimental passages from others. to explore these dynamics, we introduce crab, a homogeneous open-domain question answering benchmark. using crab, we empirically demonstrate that a carefully curated collaborative store, rich in relevant passages and minimizing hard negatives, significantly improves model performance compared to parametric collaborative learning methods and local rag training. our contributions include: we investigate how the composition of the collaborative store (relevant, irrelevant, and hard-negative passages) affects model generalization and client participation incentives. our analysis uncovers a fundamental tension: clients must weigh the benefits of accessing an enriched collaborative store against the risk of incorporating potentially detrimental passages from other clients. ## corag framework rag models @xcite enhance parametric lms by incorporating external knowledge in the form of a passage store. given an input x (e.g., a question), a rag model retrieves relevant documents z from the passage store and uses them to generate an output y (e.g., an answer). the model estimates the probability of generating y given x, denoted as p rag (y|x), by marginalizing over the top k retrieved documents: corag (algorithm 1) combines collaborative learning with rag models, enabling clients to jointly train a shared model while leveraging a collaboratively constructed passage store. this is particularly advantageous in low-resource settings, where individual clients may have limited local data. by pooling their knowledge through a shared passage store, clients gain access to a broader and more diverse knowledge base, facilitating improved learning and generalization. corag operates in three phases: during pretraining, each retriever and reader are pretrained on a large, shared dataset d pre using self-supervised objectives to enable general language understanding. in the collaborative learning phase, clients collaboratively finetune the pretrained retriever and reader on their local training datasets {d train,i } m @xmath0, constructed through algorithm 1 collaborative retrieval-augmented generation require: m clients, pretraining data dpre, train question answer pairs per client {dtrain,i} m @xmath1, collaborative train passage store itrain, test passage stores {itest,i} m @xmath2, test queries {qi} m @xmath3: responses {oi} m @xmath4: pretrain retriever r and reader g using dpre collaborative training: for each round do for each client i do ri, gi ← r, g ▷ init with global model pi ← r(dtrain,i, itrain) ▷ retrieve passages update local ri, gi using pi and dtrain,i in addition to the reader and retriever, corag employs the collaborative passage store i train , a collection of text passages contributed by all participating clients. separate passage stores are used for training and testing, with their composition (relevant, irrelevant, and hard-negative passages) significantly influencing both model performance and client incentives for contributing high-quality passages, as we will explore further. ## conclusion and future work this work introduces corag, a framework extending rag to collaborative learning, enabling clients to jointly train a shared model and collaboratively construct a passage store. our experiments on crab, a collaborative qa benchmark, demonstrate the significant performance advantage of corag in few-shot settings. we analyze the impact of passage store composition on performance, highlighting the importance of relevant and, surprisingly, irrelevant passages, while showing the detrimental effects of hard negatives. future work includes evaluating corag on heterogeneous client distributions, and designing robust incentive mechanisms.