E xpert PLM : Pre-training Expert Representation for Expert Finding (2022)

Abstract: Expert Finding is an important task in Community Question Answering (CQA) platforms, which could help route questions to potential users to answer. The key is to learn representations of experts based on their historical answered questions accurately. In this paper, inspired by the strong text understanding ability of Pretrained Language modelings (PLMs), we propose a pre-training and fine-tuning expert finding framework. The core is that we design an expert-level pre-training paradigm, that effectively integrates expert interest and expertise simultaneously. Specifically different from the typical corpus-level pre-training, we treat each expert as the basic pre-training unit including all the historical answered question titles of the expert, which could fully indicate the expert interests for questions. Besides, we integrate the vote score information along with each answer of the expert into the pre-training phrase to model the expert ability explicitly. Finally, we propose a novel reputation-augmented Masked Language Model (MLM) pre-training strategy to capture the expert reputation information. In this way, our method could learn expert representation comprehensively, which then will be adopted and fine-tuned in the down-streaming expert-finding task. Extensive experimental results on six real-world CQA datasets demonstrate the effectiveness of our method.

## introduction community question answering (cqa) websites have become a popular platform, which can help people share their knowledge in the form of questions and answers. some large portals such as stack exchange 1 have extremely attracted millions of users @xcite , which can raise their questions or post answers for questions they 1 https://stackexchange.com qiyao peng and hongtao liu are equal contribution. ## related works in this section, we briefly review some related works about pre-training for nlp, pre-training for rs, and expert finding. ## problem definition in this section, we formulate the problem of expert finding in cqa websites. suppose that there is a target question q t and a candidate expert set the question is represented by a question title, which consists of a sequence of words. the primary objective of the expert finding is to predict the most suitable expert for answering the target question. note that the expert who provides the "accepted answer" for the question will be regarded as the ground truth. it is noted that one question only have one "accepted answer". ## proposed method in this section, we will introduce our method ex-pertplm in detail. the expert pre-training language model is demonstrated in figure 2 . in the pre-training stage, we pre-train the model based on the concatenated expert historical answered questions (i.e., one input line is one expert all historical question titles) from different cqa domains for capturing expert interest. for indicating the expert different abilities to answer different questions, we integrate the vote score embedding with the corresponding question input embedding. furthermore, we design a reputation-augmented mlm pre-training task for capturing the expert overall expertise and cqa language knowledge. in the fine-tuning stage, as shown in figure 3 , via conducting the supervised expert finding task on expert and question representations generated by the pretrained weight, we can obtain an improved expert finding model for a specific domain. ## conclusion in this paper, we propose expertplm, a pretraining language model for the expert finding task in cqa. the core of our method is that we design an expert-specific pre-training framework based on a masked language model, towards precisely modeling experts (i.e., interest and expertise) based on the historical answered questions and vote scores. meanwhile, the pre-trained language model could capture the cqa language knowledge, which is beneficial to the downstream task. we conduct detailed experiments on real world cqa datasets, and the results fully validate the effectiveness of our proposed pretraining method, in the future, we would like to explore a larger scale comprehensive expert pre-training model and extend the pretrained model to more downstream tasks.