A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products (2022)

Abstract: We present a biomedical knowledge enhanced pre-trained language model for medicinal product vertical search. Following ELECTRA’s replaced token detection (RTD) pre-training, we leverage biomedical entity masking (EM) strategy to learn better contextual word representations. Furthermore, we propose a novel pre-training task, product attribute prediction (PAP), to inject product knowledge into the pre-trained language model efficiently by leveraging medicinal product databases directly. By sharing the parameters of PAP’s transformer encoder with that of RTD’s main transformer, these two pre-training tasks are jointly learned. Experiments demonstrate the effectiveness of PAP task for pre-trained language model on medicinal product vertical search scenario, which includes query-title relevance, query intent classification, and named entity recognition in query.

## introduction pre-trained language models (plms) have significantly improved the performance of various natural language processing (nlp) tasks in the recent years. it is now a common practice to adapt the pretrain-then-finetune approach in nlp. plms such as bert @xcite and elec-tra @xcite capturing word meaning through self-supervised learning from large corpus have shown a significant improvement on various text mining tasks. in the biomedical domain, many bert variants such as biobert @xcite , scibert @xcite , pubmedbert @xcite and biomedbert @xcite , follow either continual pre-training or pretraining from scratch approach using domain specific corpora to further improve the model performance. our motivation is to apply powerful pre-trained language models on medicinal product vertical * corresponding author search engine to solve query understanding and query title relevance tasks. most of the models trained on either the general corpus or the medical literature corpus lack medicinal product knowledge. for the product vertical search scenario, medicinal product information is usually stored in structured relational database tables, and traditional pretrained language models focus on natural language text in the form of sentences without considering the semantic relationship modeling of structured text in the product information tables. from the perspective of users' search habits, in addition to searching directly for the medicine name, users also often search for disease, symptoms and other important proudct attribute words to find the medicine. therefore, we propose a novel pre-training task called product attribute prediction (pap). although we could carefully craft a medicinal product knowledge graph from product databases and then try to inject the extracted explicit knowledge graph into the pre-trained language model, we argue that the procedure of building product knowledge graph is laborious but avoidable by training the language model directly on product structural information. in this paper, we propose a novel elec-tra @xcite based biomedical knowledge enhanced pre-trained language model. it consists of two pre-training tasks: replaced token detection and product attribute prediction. our approach is inspired by electra and transe @xcite methods but distinguish itself in two prominent ways. firstly, we use entity masking strategy for biomedical text instead of only masking random tokens. text spans of biomedical named entities are masked dynamically before each training iteration. we let the model predict whether these terms are replaced to incorporate biomedical domain knowledge. secondly, we utilize medicinal product textual information instead of node identifiers in product knowledge graphs, to further bring rich medicinal product knowledge into the pre-trained language model. the triples (product title, attribute name, attribute values), which can be easily drawn from medicinal product databases, are all encoded and then used in the contrastive loss of pap pre-training task to capture product knowledge. our main contributions can be summarized as follows: 1) we augment electra's replaced token detection pre-training task by leveraging biomedical entities masking (em) to learn better contextual word representation; 2) we propose a novel pre-training task, product attribute prediction (pap), which can inject medicinal product knowledge into the pre-trained language model by exploiting medicinal product databases directly. the proposed pre-training task is also applicable to vertical search scenarios for products in general, not limited to medicinal products; and 3) we have demonstrated the effectiveness of pap pre-training task for plms in medicinal product vertical search scenario. foot_0 2 related work ## method to augment the pre-trained language model with biomedical domain knowledge, we follow elec-tra's replaced token detection (rtd) pre-training task on general domain and biomedical specific corpora, and further introduce dynamic biomedical entity word masking to learn better contextual word representations. although it is possible to use other pre-training tasks such as the masked token prediction in bert, we chose the elec-tra's pre-training task due to the sample efficiency of the rtd task and the comparable performance of electra-base and bert-large models @xcite . for better application to the vertical search scenario of medicinal product search, it is desirable that the pre-trained language model contains product-related knowledge. considering that important product attributes such as drug names, diseases and symptoms are common search terms and the structured information of products also contains these terms, so we propose the product attribute prediction pre-training task to model the semantic relationships of product attributes. as shown in figure 1 , our approach consists of two pre-training tasks: replaced token detection and product attribute prediction. ## experiments and results to demonstrate the effectiveness of the pap pretraining task, we train the pre-trained language dataset #sen #tok model from scratch, first on the general corpus and biomedical domain corpus shown in ## discussion as mentioned in section 3, other self-supervised learning tasks for language models could be learned together with the pap pre-training task. we only explore the combination of electra's rtd and pap for joint training. due to limited computational resources, we could not push the model to larger sizes. these may prevent the full potential of pap pre-training task from being unleashed. for vertical search applications, pap pretraining task is also applicable to other product searches. for example, based on movie structured information such as movie title, genre and story type, it is feasible to use pap task to model the semantic relationships between these movie attributes. in the study of product knowledge en-hanced language models, comparing the way pap uses product structured information with the graph embedding approach based on product knowledge graphs may be an interesting research problem for the future. ## conclusion in this article, we propose a biomedical knowledge enhanced pre-trained language model for medicinal product vertical search. we improve electra's replaced token detection pre-training task with biomedical entity masking (em). then we present a novel pre-training task, product attribute prediction (pap), to incorporate medicinal product knowledge into the plm. we train electra+em and electra+em+pap two biomedical knowledge enhanced pre-trained language models to demonstrate the effectiveness of pap pre-training task for medicinal product vertical search. our work may shed some light on combining the powerful pretrained language models with product knowledge for vertical search scenarios.