CPRM : A LLM -based Continual Pre-training Framework for Relevance Modeling in Commercial Search (2025)

Abstract: Relevance modeling between queries and items stands as a pivotal component in commercial search engines, directly affecting the user experience. Given the remarkable achievements of large language models (LLMs) in various natural language processing (NLP) tasks, LLM-based relevance modeling is gradually being adopted within industrial search systems. Nevertheless, foundational LLMs lack domain-specific knowledge and do not fully exploit the potential of in-context learning. Furthermore, structured item text remains underutilized, and there is a shortage in the supply of corresponding queries and background knowledge. We thereby propose CPRM (Continual Pre-training for Relevance Modeling), a framework designed for the continual pre-training of LLMs to address these issues. Our CPRM framework includes three modules: 1) employing both queries and multi-field item to jointly pre-train for enhancing domain knowledge, 2) applying in-context pre-training, a novel approach where LLMs are pre-trained on a sequence of related queries or items, and 3) conducting reading comprehension on items to produce associated domain knowledge and background information (e.g., generating summaries and corresponding queries) to further strengthen LLMs. Results on offline experiments and online A/B testing demonstrate that our model achieves convincing performance compared to strong baselines.

## introduction relevance modeling is designed to evaluate the correlation between queries and items, an essential component of commercial search engines and crucial for the user experience. mini-app service search is a common search application scenario. unlike traditional e-commerce searches that only provide product search functions, mini-app services encompasses numerous scenarios such * corresponding author. as livelihoods, government affairs, transportation, healthcare and dining. moreover, the item consists of structured data with multiple fields; for instance, a hospital mini-app might include fields like title, keywords, category and description. considering the diverse scenes and the complexity of structured data across multiple fields, conducting relevance modeling within the such search scenario poses a significant challenge. the current relevance model in commercial search systems is a semantic matching model, leveraging llms combined with domain-annotated data through supervised fine-tuning (sft) methods. these llms like @xcite , glm @xcite , llama @xcite , qwen @xcite , having more extensive parameters and utilizing a massive corpora of texts during training compared to previous pretrained models such as bert @xcite , roberta @xcite and xlnet @xcite , demonstrate superior performance in semantic matching tasks. despite the great success of llms, there still remain certain limitations in their application to relevance modeling. firstly, llms are pre-trained on a broad range of data sources @xcite @xcite , which do not afford special attention to particular domains @xcite @xcite , resulting in a lack of domain-specific knowledge. besides, queries tend to be colloquial and present in short-text form, whereas items are typically expressed in a more formal long-text form, leading to a "semantic gap" @xcite @xcite between their representations. secondly, the pre-training phase of llms is "task-agnostic" @xcite , which impedes direct connection with downstream tasks and precludes the possibility of in-context pretraining enhancements tailored for these tasks @xcite . finally, the item tends to be highly structured and difficult to leverage, which prevents llms from fully realizing their potential with such data. to address the above problems, we investigate a continual pre-training approach of llms for relevance modeling, cprm for short. initially, we introduce a pre-training technique using pairs of queries and multi-field item as inputs. this method enables the llms to explicitly model the semantic representations between queries and items, thus bridging the semantic gap between them. subsequently, we collect sets of semantically similar queries and items based on user click logs, then further refine these sets through semantic modeling to filter out semantically irrelevant cases. following that, we reorder these refined sets according to semantic similarity and ultimately construct incontext pre-training instances via prompting techniques. utilizing this approach to data construction, llms are able to make better predictions within such contexts during the training process, which benefits the efficient learning current domain knowledge for llms. lastly, we employ a larger parameter llm (teacher llm) to conduct reading comprehension on structured item data, facilitating the generation of relevant domain knowledge for pre-training, which can be considered as a secondary development and exploitation of the item. more specifically, we leverage teacher llm to summarize and paraphrase item to produce fluent domain knowledge, while also guiding teacher llm to produce background information related to the items. additionally, we prompt teacher llm to create diverse queries and provide further reasons for their generation. in summary, the contributions of this paper are as follows: relevance modeling in search corresponds to the semantic matching task in nlp. with the advancement of neural network and pre-trained models, deep semantic matching models have become mainstream. deep semantic matching models are categorized into two types: representation-based @xcite @xcite and interaction-based methods @xcite @xcite . the former focuses on learning low-dimensional representations, while the latter emphasizes capturing the interactions between inputs. the representation-based model with independently encoded inputs struggles to capture complex correlations, whereas interaction-based methods that concatenate the two inputs for semantic computation can alleviate this issue. in recent years, pre-trained models like bert @xcite has show its superiority on natural language understanding (nlu) tasks. consequently, both representation-based and interaction-based methods have begun leveraging the capabilities of these pre-trained models for semantic modeling. most recently, llms like gpt-3 (brown et al., 2020), glm @xcite , llama @xcite , qwen @xcite pre-trained extensive volumes of data with numerous parameters have garnered significant performance in language understanding, generation and reasoning tasks. compared to traditional pre-trained models like bert @xcite , llms possess significant advantages in both the scale of pretraining data and the quantity of model parameters, leading to their evident superiority in performance across a variety of downstream tasks. recent research work @xcite @xcite indicates that combining llms with downstream applications presents sig-nificant potential, llms can achieve competitive or even superior results compared to traditional supervised methods on information retrieval benchmarks. some research leverage llms for relevance modeling in search engines, adopting approaches such as behavior-augmented @xcite or robust learning @xcite to improve the capability of relevance modeling. our work mainly focuses on enhancing llms from the perspective of continual pre-training for relevance modeling. llms are pre-trained on a wide variety of data sources @xcite @xcite ) without pay more attention on specific domains, resulting in a lack of domain knowledge. on the other hand, the pretraining phase of llms is task-agnostic (brown et al., 2020), making it difficult to direct connect with downstream tasks. this also means we can't easily customize the pre-training process to better fit those tasks @xcite . previous work of injecting domain knowledge involves continued training of pre-trained models on domain-specific data @xcite , as well as incorporating knowledge graphs @xcite or selectively masking important information @xcite @xcite . another line of research aims to enhance the pretraining for downstream tasks, which simple concatenate relevant documents together for in-context pre-training @xcite @xcite . however, these approaches assume that downstream tasks contain only a single domain representation, neglecting the possibility of there being multiple or more. for instance, in commercial search relevance tasks, queries and items belong to two distinct domains with substantial differences. our research work involves injecting domain knowledge and conducting in-context pre-training simultaneously, while being able to establish a connection between the two domains. ## problem formulation given a query q and an item i, llm-based relevance modeling in search engines aims to predict the relevance degree between them. essentially, referring to pet (schick and schütze, 2021), we first design the prompt p(q, i), and llm determines which verbalizer v (e.g., "no" or "yes") is the most likely candidate for "[mask]" conditioned on the likelihood m(v|p(q, i)). the above process is defined as follows: p(q, i) = is [q] and [i] related? [mask] (1) @xmath0, i)), for @xmath1, yes}, (2) where relevance label @xmath2, 1} can be associated with a verbalizer (e.g., "no" or "yes") from the vocabulary of llms to represent "irrelevant" and "relevant" between q and i respectively. to enable the adaptation of general llms to the relevance modeling task, sft operation is selected for training with the cross-entropy loss function. consequently, the relevance degree could be given from llms for subsequent applications in search scenarios. ## methodology in this section, we present the details of the cprm framework, including domain knowledge enhancement (dke), in-context pre-training (icp) and reading comprehension distillation (rcd). ## online a/b testing we deploy the proposed model on the online search platform to provide search services for mini-apps, and conduct a two-week online a/b testing with 5% proportion of the experiment traffic. the experimental results show that, compared to the baseline system (glm-2b), our cprm method yields a statistically significant increase of 0.32% in valid pvctr foot_1 at a 95% confidence level. human evaluation indicates a 0.75% reduction in badcase@10 metric and a 4.71% decrease in the error filtering rate foot_2 . the model has now been serving search functions to mini-apps for over nine months. these results suggest that our proposed method can effectively enhance relevance models' performance in real-world search systems. ## conclusion in this paper, we have investigated cprm framework, a continued pre-training approach of llms tailored to relevance modeling tasks, which comprises three methods: dke, icp and rcd. both offline experiments and online a/b testing results demonstrate that our proposed method boosts the search relevance of llms effectively. our model has been successfully deployed online search platform.