Effective distributed representations for academic expert search (2020)

Abstract: Expert search aims to find and rank experts based on a user’s query. In academia, retrieving experts is an efficient way to navigate through a large amount of academic knowledge. Here, we study how different distributed representations of academic papers (i.e. embeddings) impact academic expert retrieval. We use the Microsoft Academic Graph dataset and experiment with different configurations of a document-centric voting model for retrieval. In particular, we explore the impact of the use of contextualized embeddings on search performance. We also present results for paper embeddings that incorporate citation information through retrofitting. Additionally, experiments are conducted using different techniques for assigning author weights based on author order. We observe that using contextual embeddings produced by a transformer model trained for sentence similarity tasks produces the most effective paper representations for document-centric expert retrieval. However, retrofitting the paper embeddings and using elaborate author contribution weighting strategies did not improve retrieval performance.

## introduction to help navigate a large body of academic knowledge, it can be useful to identify expert individuals. identifying such individuals may be useful to find collaborators @xcite @xcite , to find paper reviewers @xcite , to find supervisors @xcite , or to investigate literature in a certain domain. this process of identifying experts given a particular topic is called expert finding @xcite , expertise retrieval @xcite , or expert search. expert search systems are information retrieval systems that can automatically rank candidate experts based on their expertise on a certain subject @xcite . in this study, we target the domain of retrieving academic experts based on papers they authored. given the central role of papers to defining expertise in this domain, we focus on document-centric expert search systems @xcite . these systems largely rely on statistical language modeling, topic modeling, or term frequency-based approaches to represent documents @xcite . surprisingly, given the rapid advances in the field of contextualized text embeddings @xcite , little work has been done in applying these approaches to document representation for this task. we hypothesize that considering single words, which is common in the bag-of-words and probabilistic termbased approaches, may significantly reduce the system's "understanding" of the underlying academic documents. to achieve a potentially deeper understanding of these papers, contextualized text embeddings could be used. thus, in this paper, we explore the impact of contextualized text embeddings on the performance of the expert search. specifically, we make the following contributions: additionally, all experiments are conducted using different techniques for assigning author weightings based on author order. overall, this paper provides evidence for the efficacy of contextualized embeddings for the task of academic expert search. note that this paper primarily focuses on investigating the performance of various contextualized embeddings and expert ranking aggregation methods within expert retrieval, and not on the entire retrieval process. therefore, some aspects of neural information retrieval systems such as query understanding, query expansion, or reranking are out of the scope of this study. source code for the methods and data processing used in this paper can be found at https://github. com/mabergerx/sdp500_expert_search . the processed data used by our methods is available at @xcite . the rest of this paper is organized as follows. we begin with a discussion of related work. afterwards, the data used in this study is described. this is followed by a description of the various embeddings used and our approach to author ranking. section 6 defines the evaluation and section 7 details its results. we, then, briefly describe a prototype implementation using these representations. finally, we discuss the limitations of the work, potential future work and conclude. ## related work in this section, we introduce the primary paradigm for expert search. we then discuss work on voting models, document representations, and the use of text embedding techniques within expert search. probabilistic models a driving force behind expertise retrieval research was the launch of the trec enterprise track in 2005 @xcite . this evaluation campaign led to the emergence of probabilistic models, in particular in the form of language models, as the primary paradigm for expertise retrieval. the core idea behind these approaches is to estimate a language model for each document and then rank the documents by the likelihood of the user query according to the language models @xcite . ## data description the microsoft academic graph (mag) @xcite was used at the primary data source. the data consists of over 200 million papers (titles & abstracts) as well as a variety of metadata. we accessed the november 2018 snapshot of the mag data through the open academic graph initiative foot_0 , in particular the oag v2 release. due to the very large size of the mag, we created a custom subset of the data that mainly consisted of computer science (cs) related papers. this domain allows us to interpret results better than other science domains. our approach in extracting computer science (cs) related papers was to take the 113.864 paper titles obtained from arxiv foot_1 -a widely used preprint server -and do an exhaustive title matching on the full mag dataset. this search resulted in 29.237 exact title matches, which corresponds to 26,6% of the arxiv data. this set provided us with a substantial initial seed of papers to extract more cs papers from the mag data. to allow retrofitting later in the process and create a larger dataset, we expanded this set with the references of all the 29.237 papers, which resulted in a set of 221.347 papers. these references were retrieved by accessing the references field of each of the 29.237 paper in the mag data. note that these references are not necessarily always complete: some cited articles may not be present in our data due to incompleteness of the source mag data. from these 221.347 papers, we then performed bounded stratified sampling for the authors to retrieve a subset of 5.000 authors who are representative of both highly-, medium-and less prolific author populations. the full sampling method is described in algorithm 5 in the algorithms appendix. this set of 5.000 authors served as a starting point for a second, final round of data retrieval. for these authors, we retrieved all their papers and references, resulting in a set of 127.716 papers, which included authors of the referenced papers. for all these new authors, we collected the metadata from the mag authors dataset and aggregated this information into a single final authors dataset. the reason for expanding the set of authors beyond the 5.000 sampled authors is that a larger pool of papers is beneficial for the retrieval due to the larger search space. for all titles and abstracts in our dataset, we performed data cleaning. specifically, (corpusspecific) stopwords were removed, redundant whitespace and unicode characters were both normalized. urls and e-mails were removed. ## paper embedding methodology in this section, we describe the paper embedding techniques employed and discuss our approach to embedding indexing and search, as well as retrofitting embeddings. ## author ranking via voting from the faiss index we can, given a query, retrieve top n similar papers. to produce a final author ranking, we adopt a voting model based approach. we can consider the retrieved paper results as the "expertise evidence" for the authors of these papers. a range of different voting approaches based on data fusion techniques has been proposed @xcite @xcite to produce an author ranking given the documents. each retrieved document d from the set of retrieved documents r(q) has an associated similarity score s(d, q) to it, with regard to the query q. we can then combine these document scores into aggregated author scores using the expcombsum (ecs) data fusion function @xcite : each author gets fullscore # authors for a document. this strategy assumes that each author contributed equally but does normalize the score by the number of authors. using these data fusion approaches, a fairness problem may occur: highly prolific authors, that may be associated with many documents (for instance, because they are the head of a lab) may receive an unfairly large number of votes, which does not necessarily indicate their expertise. candidate length normalization is proposed to deal with this unfairness, just as document length normalization is often performed in document retrieval systems @xcite . the use of a classical document normalization technique based on the divergence from randomness framework @xcite is proposed @xcite , and has the following formula: in practice, we discovered that because of the nature of our dataset, if we apply the above normalization technique, many authors from the long tail are retrieved, even when we use high α values. therefore, we experimented with introducing another term to the equation: β, which serves as a profile length "booster": while it does introduce bias and eases the normalization, in our case, it provided an extra parameter to tune and resulted in a better mix of welland lesser-known authors. ## evaluation methods to evaluate the different retrieval strategies, we developed a method which uses the field of work tags present in the mag dataset for the authors as a proxy for evaluating the relevance of an author. because we use a document-centric retrieval strategy which uses strictly only the embeddings of the paper's title and abstract, the field of work tags are not used in the retrieval process, allowing us to use these tags in the evaluation process. the query test set for our research was selected from the full distribution of author tags. the final query test set contained a hundred computer science related queries. the set is available in ## results in this section, first the overall quantitative evaluation results are discussed and then we zoom in on the performance of retrofitting and author contribution weighting. ## prototype implementation we implemented our model int a prototype, which consists of a rest api, where the users can, given a search query, look for n most relevant experts. each individual expert representation is contained within a json object which contains not only the author's name and mag id, but also the authors affiliation information (retrieved using grid foot_5 ), the list of papers which voted for the author and their corresponding document scores in relation to the query, and additional author information from wikidata @xcite . ## discussion & future work this study faced multiple challenges regarding data quality, data freshness, embedding strategy considerations, and the retrieval base on embeddings. in this section, we discuss these issues. data completeness and variability. the fieldof-work tags used in the evaluation are not always complete and are subject to the specific format used by the mag. in addition, for some authors, no tags are present in our snapshot of the mag data. this has an effect on the evaluation performance by introducing both false positives and -negatives into the relevancy determining process. overall, our system would profit significantly from a larger pool of papers and authors. ## conclusion in this study, we investigated different approaches for embedding academic papers and using the embeddings in an expert search task. overall, we found that transformer-based contextual text embeddings work well on the domain of academic papers. by using the sentence-bert model trained on nli and sns tasks, we outperformed the strong lsi baseline often employed in information retrieval systems on all ten evaluation metrics. we also outperformed the baseline strategies of average pooled bert and glove embeddings. employing a weighted embedding combination strategy to represent a paper can however be valu-able, as we found that using a separate embedding combination strategy outperformed the "default" merged strategy on nine out of ten metrics. we hypothesized that enriching the paper embeddings with citation information, in a process called retrofitting, could improve improve retrieval performance. our experiments did not confirm this hypothesis, as non-retrofitted embeddings performed better in our task on all but two evaluation metrics. finally, we employed various data fusion techniques to convert the top n retrieved papers given a query into a ranking of authors. a voting model was used, where each document served as evidence for the corresponding author's expertise. we investigated whether using author contribution weighting strategies within the voting process would improve expertise retrieval. we observed no performance gain over the "default" binary strategy. given the direction of this study, we think that the most suitable application areas for the methodology we proposed are reviewer finding, supervisor finding and investigating literature on a topic. the reasoning behind this is that finding a collaborator may require and involve more sophisticated information about the institution, availability and current field of interest of the found experts. the proposed three areas, however, allow for less specificity and can better benefit from the improved retrieval. while research in the field of expertise retrieval is not as active in the second half of the 2010s as it was in the first half, the area of text representations and retrieval has seen dramatic improvements. this study was an effort to apply these new techniques into the field of expertise retrieval and has shown that substantial improvements can be made over the existing retrieval algorithms. we hope that this study can contribute to a new research wave within the field of (academic) expertise retrieval. algorithm 1: paper embedding creation following the merge strategy. result: paper embeddings m e following the merge strategy