Contrastive Pre-training for Personalized Expert Finding (2023)

Abstract: Expert finding could help route questions to potential suitable users to answer in Community Question Answering (CQA) platforms. Hence it is essential to learn accurate representations of experts and questions according to the question text articles. Recently the pre-training and fine-tuning paradigms are powerful for natural language understanding, which has the potential for better question modeling and expert finding. Inspired by this, we propose a CQA-domain Contrastive Pre-training framework for Expert Finding, named CPEF, which could learn more comprehensive question representations. Specifically, considering that there is semantic complementation between question titles and bodies, during the domain pre-training phase, we propose a title-body contrastive learning task to enhance question representations, which directly treats the question title and the corresponding body as positive samples of each other, instead of designing extra data-augmentation strategies. Furthermore, a personalized tuning network is proposed to inject the personalized preferences of different experts during the fine-tuning phase. Extensive experimental results on six real-world datasets demonstrate that our method could achieve superior performance for expert finding.

## introduction nowadays, community question answering (cqa) platforms such as stackexchange 1 , baiduzhidao 2 have been popular and extremely attracted millions of users @xcite , which can raise their questions or post answers for questions they are interested in or good at. due to the large participation, there are too many questions to wait for answers @xcite . in order to alleviate that, it needs to recommend suitable experts to provide satisfactory answers @xcite for these questions. expert finding aims to effectively route questions to a suitable expert based on her/his historically answered questions, which has attracted considerable attention recently. based on the powerful neural network @xcite , many approaches @xcite @xcite have been shown effective to improve the performance of expert finding. the core idea is to first model the question semantically, and then learn the expert preference based on his/her historically answered questions. for example, peng et al. @xcite proposed pmef equipped with a multi-view question modeling paradigm aiming to model questions more comprehensively and then capture expert features. hence, the capacity of the designed question modeling paradigm directly affects the expert finding performance. recently, pre-trained language models (plms) have achieved great success in natural language processing (nlp) due to their strong ability in text modeling @xcite . different from traditional models that are usually directly trained with labeled data in specific tasks, plms are usually pre-trained on a large unlabeled corpus via self-supervision to encode universal semantic information. however, existing plms are usually pre-trained on general corpus such as bookcorpus and wikipedia, which have some gaps with the cqa domain and the expert finding task. directly fine-tuning the existing plms with the expert finding task may be sub-optimal for cqa question understanding. besides, such an architecture does not take into consideration the expert's personalized characteristics, which could impact expert modeling and question-expert matching (i.e., expert finding). at present, the exploration of the pre-training paradigm in expert finding is pre-liminary. in this paper, we aim to design a novel expert finding framework, which could learn more comprehensive question and expert representations with domain pre-training and personalized finetuning paradigm. for pre-training question representations, in addition to the domain pre-training masked language model (mlm) task, we design a contrastive task to capture the inherent semantic information from the unlabeled cqa question corpus. different from the vanilla contrastive learning paradigm with data augmentation progress, we directly consider the question title and the question body as mutually augmented perspectives. in this way, the model could omit the tedious process of data augmentation while incorporating the two relevant types of information (i.e., question title and body) for question semantic modeling. for fine-tuning phase, we transfer the pre-trained question representations to model expert representations and the downstream expert finding task. considering different experts would have different tastes in the same question, hence we design a personalized tuning network, which could effectively inject personalized preference (i.e., expert id) to learn more personalized expert representations for better question-expert matching. in summary, the contributions of our method are: ## related work in this section, we briefly review the related work in two aspects, namely expert finding and pretraining for recommendation. expert finding has received much attention from both research and industry community @xcite @xcite @xcite and it could help find capable experts for providing answers to questions. early studies employed feature-engineering @xcite or topic-modeling technology @xcite to model questions and experts, then measured similarities between them for routing questions to suitable experts. subsequently, with the development of deep learning @xcite , recent methods employed neural networks to learn expert and question representations, then matched them @xcite @xcite . for example, peng et al. @xcite designed a multi-view learning paradigm to learning question features, which could improve expert finding. different from most existing methods learning question representations that rely on supervised data, we design a question title-body contrastive pre-training paradigm for modeling question. in this way, the model could capture more comprehensive question semantic representations, which are the basis of expert modeling and question-expert matching. furthermore, expert finding aims to recommend suitable experts for answering target questions, which is similar to recommender system. hence, we will introduce some recommendation researches with pre-training @xcite @xcite in this part. for example, cheng et al. @xcite proposed clue, which designed contrastive pre-training tasks to optimize transferable sequence-level user representations for recommendation. compared with existing recommendation works under the pre-training paradigm, the expert finding has different characteristics, e.g., the target question is usually cold-started and id-based recommendation may not be feasible. it is noted that the very recent method ex-pertbert @xcite designs a preliminary expert-specific pre-training framework, which has achieved the best performance. different from that, our approach focuses on more comprehensive question semantic representations via integrating the title-body contrastive learning task during pre-training. in addition, we design a personalized tuning network to tune the model and model expert according to different expert preferences. ## problem definition expert finding in cqa websites is to predict the most suitable expert to answer questions. suppose that there is a target question q t and a candidate expert set @xmath0, • • • , e m } respectively, where m is the number of experts. given a candidate expert e @xmath1, she/he is associated with a set of her/his historical answered questions, which can be denoted as q @xmath2, • • • , q n } where n is the number of historical questions. in addition to the title, each question contains the body information, which is a detailed description of the question; however, most existing methods ignore the question bodies. in fact, the question body could be a complementation of the question title. hence, in this paper we would explore how to utilize the title and body together effectively. ## proposed method in this section, we will introduce the training procedure of our method cpef, which is composed of the following stages. ## experiments in this section, we first introduce the experiment settings and then present the results and analysis. ## conclusion in this paper, we propose a domain contrastive pretraining expert finding model to learn more effective representations of questions and experts. the core of our method is that we design a contrastive framework between question bodies and titles to integrate into the pre-training phrase with the domain cqa corpus. in this way, our method could obtain precise features of questions as well as experts. in addition, the personalized tuning network could further enhance expert representations. extensive experiments on several cqa datasets validate the effectiveness of our approach, which could outperform many competitive baseline methods.