BS hared RAG : Backbone Shared Retrieval-Augmented Generation for the E -commerce Domain (2024)

Abstract: Retrieval Augmented Generation (RAG) system is important in domains such as e-commerce, which has many long-tail entities and frequently updated information. Most existing works adopt separate modules for retrieval and generation, which may be suboptimal since the retrieval task and the generation task cannot benefit from each other to improve performance. We propose a novel Backbone Shared RAG framework (BSharedRAG). It first uses a domain-specific corpus to continually pre-train a base model as a domain-specific backbone model and then trains two plug-and-play Low-Rank Adaptation (LoRA) modules based on the shared backbone to minimize retrieval and generation losses respectively. Experimental results indicate that our proposed BSharedRAG outperforms baseline models by 5% and 13% in Hit@3 upon two datasets in retrieval evaluation and by 23% in terms of BLEU-3 in generation evaluation. Our codes, models, and dataset are available at https://bsharedrag.github.io.

## introduction large language models (llms) have shown impressive performance in reasoning and remembering extensive knowledge @xcite @xcite @xcite @xcite . however, when facing domain-specific questions, especially the e-commerce domain that has many long-tail entities and frequently updated information, llms often have issues such as hallucinations @xcite @xcite . retrieval-augmented generation * corresponding authors. c1 (dataset): how to build a high-quality rag dataset with an informative knowledge base and correct question-answer (qa) pairs? for important domains like e-commerce, the existing knowledge bases are usually short and noisy, as most of them are from user-generated product reviews. the qa pairs are also of various qualities @xcite and may contain subjective or conflicting information because the answers are provided by users. c2 (framework): how to design a suitable rag framework for domain adaptation? as shown in figure 1 (a), existing works typically adopt separate modules for retrieval and generation @xcite @xcite @xcite , for example, using bert-like @xcite embedding models for retrieving documents and an llm for generating answers. in this design, retrieval and generation operate independently, hindering their potential to achieve optimal performance. it is often challenging for the retriever to leverage the continuous pre-training of the generator, while the generator's performance may be constrained by retrieved documents that, although similar to the query, are less effective in addressing the questions (section 5.2.3). although we can build the retriever and generator both on a fully shared llm and optimize it with multi-task learning as shown in figure 1 (b), it can suffer from negative transfer between tasks, i.e., performance decrease due to the potential conflicts in the retrieval task and the generation task. moreover, balancing the retrieval loss and the generation loss is non-trivial and may lead to an effort-taking hyperparameter search. to address the two challenges, we first propose the worthbuying dataset with 735k high-quality documents, 50k question-document-answer (qda) tuples, and human annotated test data of relevant documents for 1k questions and 500 qa pairs (c1). the knowledge base in our dataset comes from professional users, reducing conflicts and errors, and is more informative, with 1.1k words per document rather than a few dozen words as in existing e-commerce knowledge bases. we also annotate high-quality qa pairs with gpt-4 @xcite and manually review the test set. we then propose a specifically designed bsharedrag framework that effectively adapts rag to the e-commerce domain (c2). as shown in figure 1 (c) we apply two task-specific modules to independently minimize the retrieval loss and generation loss, which avoids effort-taking loss balancing. with the shared backbone that benefits from domain-specific continual pre-training, the retriever and the generator can benefit from each other to improve both retrieval and generation performance. our contributions are summarized as follows: ## method in this section, we introduce our bsharedrag for scenarios that require extensive domain knowledge, such as e-commerce. ## worthbuying dataset existing e-commerce datasets are mainly constructed from user-generated product reviews, which are brief and noisy, resulting in limited assistance to llms in generating rich content @xcite . to solve this issue, we build a worthybuying dataset, which includes an accurate knowledge base and highquality qda tuples for the e-commerce domain. ## conclusion in this paper, we construct a high-quality ecommerce dataset called worthbuying, which contains an e-commerce knowledge base and qa pairs for training and evaluating retrieval and generation