Expertise-seeking research studies how people search for expertise and choose whom to contact in the context of a specific task. An important outcome are models that identify factors that influence expert finding. Expertise retrieval addresses the same problem, expert finding, but from a system-centered perspective. The main focus has been on developing content-based algorithms similar to document search. These algorithms identify matching experts primarily on the basis of the textual content of documents with which experts are associated. Other factors, such as the ones identified by expertise-seeking models, are rarely taken into account. In this article, we extend content-based expert-finding approaches with contextual factors that have been found to influence human expert finding. We focus on a task of science communicators in a knowledge-intensive environment, the task of finding similar experts, given an example expert. Our approach combines expertise-seeking and retrieval research. First, we conduct a user study to identify contextual factors that may play a role in the studied task and environment. Then, we design expert retrieval models to capture these factors. We combine these with content-based retrieval models and evaluate them in a retrieval experiment. Our main finding is that while content-based features are the most important, human participants also take contextual factors into account, such as media experience and organizational structure. We develop two principled ways of modeling the identified factors and integrate them with contentbased retrieval models. Our experiments show that models combining content-based and contextual factors can significantly outperform existing content-based models. Received March 16, 2009; revised November 19, 2009; accepted November 19, 2009 1 This is an expanded and revised version of Hofmann, Balog, Bogers, & de Rijke, 2008 .  The increasing amount of information available is making the need to criticially assess information more important. The burden of credibility assessment and quality control is partly shifting onto individual information seekers, but the need for information intermediaries (e.g., experts) has not disappeared and is actually increasing in cases where the credibility of information has to meet high standards (Metzger, 2007) . Against this background, expert finding is a particularly relevant task: identifying and selecting individuals with specific expertise, for example, to help with a task or solve a problem. Expert finding has been addressed from different viewpoints, including expertise retrieval, which takes a mostly system-centered approach, and expertise seeking, which studies related human aspects. The goal of expertise retrieval is to support search for experts using information-retrieval technology. Following the experimental paradigm and evaluation framework established in the information-retrieval community, expertise retrieval has been addressed in world-wide evaluation efforts (Craswell, de Vries, & Soboroff, 2006) . Promising results have been achieved, particularly in the form of algorithms and test collections (Bailey, Craswell, Soboroff, & de Vries, 2007; Balog, 2008) . State-of-the-art retrieval algorithms model experts on the basis of the documents with which they are associated, and retrieve experts on a given topic using methods based on document retrieval, such as language modeling (Balog, Azzopardi, & de Rijke, 2009; Balog, Soboroff, et al., 2009) . In evaluations of these algorithms, user aspects have been abstracted away. While research into expertise retrieval has primarily focused on identifying good topical matches between needs for expertise and the content of documents associated with candidate experts, behavioral studies of human expertise seeking have found that there may be important additional factors that influence how people locate and select experts (Woudstra & Van den Hooff, 2008) ; such factors include accessibility, reliability, physical proximity, and up-todateness. We term these contextual factors to distinguish them from content-based factors that have been explored in previous work (discussed later). Context can be considered to encompass many dimensions, including factors relating to the organization, information seeker, information objects, work task, search task, and so on (cf. Cool & Spink, 2002; Ingwersen & Järvelin, 2005; Kelly, 2006) , but here we focus on one specific dimension: contextual factors related to the information objects, in our case-experts. Other elements may play a role, but their impact cannot be studied in our setup: task, organization, and information seeking system are fixed; individual differences between information seekers are left out of our consideration. Our aim in this article is to explore the integration of contextual factors into content-based retrieval algorithms for finding similar experts. We look at this problem in the setting of the public relations department of a university, where communication advisors employed by the university get requests for topical experts from the media. The specific problem we are addressing is: The top expert identified by a communication advisor in response to a request is not available because of meetings, vacations, sabbaticals, or other reasons. In this case, communication advisors have to recommend similar experts, and this is the setting for our expert finding task. Based on this task, we address three main research questions: • Which contextual factors influence (human) decisions when finding similar experts in the university setting we study? • How can such factors be integrated into content-based algorithms for finding similar experts? • Can integrating contextual factors with existing, contentbased approaches improve retrieval performance? To answer our research questions, we proceed as follows. Through a set of questionnaires completed by a university's communication advisors, we identify contextual factors that play a role in how similar experts are identified in this situation, and we construct a test dataset to evaluate retrieval performance. We evaluate both content-based approaches and approaches where we integrate contextual factors. The contribution of the article is threefold. For a specific expert-finding task, we succeed at identifying factors that humans use to select similar experts. We model several identified factors and integrate these with existing, content-based approaches to finding similar experts. We show that our new models can significantly outperform previous approaches. Our results demonstrate that it is possible to identify and model contextual factors in the studied task of finding similar experts, and we think that this may be the case for other retrieval tasks as well. The remainder of the article is organized as follows. We first provide background information on human expertise seeking and expertise retrieval. Next, we describe our approach, including the methods used for data collection, retrieval models and retrieval evaluation. Our results are then given and discussed. Lastly, we present conclusions and outline future work. Research on how to enable people to effectively share expertise can be traced back to at least the 1960s when studies in library and information science explored what sources of information knowledge workers such as researchers and engineers use (Menzel, 1960) . Subsequent work has identified complex information-seeking strategies relying on a variety of information sources, including human experts (Hertzum, 2000; Rosenberg, 1967) . From results of this research (and other influences) grew the realization that the expertise of employees is a major value of an organization and that effective sharing of knowledge can lead to material gains (Davenport & Prusak, 1998; Sproull & Kiesler, 1996; Wiig, 1997) . The field of knowledge management developed, with the goal of using knowledge within an organization as well as possible. One focus was on developing information systems that could support search for expertise. Initial approaches were mainly focused on how to unify disparate and dissimilar databases of the organization into a single data warehouse that could easily be mined (ECSCW'99 Workshop, 1999; Seid & Kobsa, 2000). Resulting tools rely on people to self-assess their skills against a predefined set of keywords, and often employ heuristics generated manually based on current working practice. Despite the achievements made so far, the question of how to provide effective access to expertise is far from solved, and continues to be addressed from different viewpoints. In human-centered research, which we term expertise seeking, one of the goals has been to develop descriptive and prescriptive models of how human information sources are used. Some of this work forms the basis for our study in that it helps us to identify the factors that play a role in finding similar experts in the setting we study. We present a selection of work relevant to this article in the next section. In system-centered work, for which we use the term expertise retrieval, one focus has been on the development of effective retrieval algorithms. This work forms the basis for the retrieval aspects of our study, including the content-based baseline retrieval algorithm, relevance assessment, and evaluation methodology. An overview of recent work in this area is given later. Of the human-centered research that we term expertise seeking, we are particularly interested in models of how people choose an expert. Most relevant are models that identify specific factors that may play a role. We need to be able to identify quite specific factors so that we will be able to model these and integrate them into a retrieval model. Several studies have identified factors that may play a role in decisions of what expert to contact or recommend. In a study of trust-related factors in expertise recommendation, Heath, Motta, and Petre (2006) found that experience and impartiality of the expert may play a role, and may additionally depend on a task's criticality and subjectivity. Borgatti and Cross (2003) showed that knowing about an expert's knowledge, valuing that knowledge, and being able to gain access to an expert's knowledge influence which experts searchers contact for help. Differences between job roles regarding the amount and motivation of expert search, as well as the type of tools used indicate a possible influence of work tasks . The use of social network information is expected to benefit expert search based on domain analysis (Terveen & McDonald, 2005) , and users are more likely to select expertise search results that include social network information (Shami, Ehrlich, & Millen, 2008) . Woudstra and Van den Hooff (2008) focused on factors related to quality and accessibility in source selection (i.e., the task of choosing which expert candidate to contact in a specific situation). Quality-related factors include reliability and up-to-dateness of the expert; accessibility includes physical proximity and cognitive effort expected when communicating with the expert. These factors are identified in a study of information-seeking tasks carried out at participants' workplaces. The study follows a think-aloud protocol, and the importance of individual factors is assessed through counts of how frequently they are mentioned when experts are evaluated. We list the factors identified by Woudstra and Van den Hooff (2008) in Table 1 . Quality-related factors (Factors 1-4) appear to be the most important while familiarity (Factor 10) also appears to play a role. We use the factors they identified as the basis for identifying contextual factors in our study. Further evidence of the usefulness of individual contextual factors (e.g., social network information) is provided by systems that apply expertise retrieval; however, because these systems are typically not directly evaluated in terms of retrieval performance, the contribution of individual factors cannot easily be assessed. Answer Garden 2 is a distributed help system that includes an expert-finding component (Ackerman & McDonald, 2000) . Besides topical matches, the system implements a number of heuristics found to be used in human expertise seeking, such as "staying local" (i.e., first asking members of the same group) or collaborators. This heuristic may be related to factors such as familiarity and accessibility. K-net is targeted at improving sharing of tacit knowledge by increasing awareness of others' knowledge (Shami, Yuan, Cosley, Xia, & Gay, 2007) . The system uses information on the social network, existing skills, and needed skills of a person, which are provided explicitly by the users. The SmallBlue system mines an organization's electronic communication to provide expert profiling and expertise retrieval (Ehrlich, Lin, & GriffithsFisher, 2007). Both textual content of messages and social network information (patterns of communication) are used. The system is evaluated in terms of its usability and utility. Finally, Liebregts and Bogers (2009) detailed the development and systematic evaluation of an expert system. The authors performed system-based, expert-based, and end-user evaluation and showed that combining diverse sources of expertise evidence improves search accuracy and speed. Expertise retrieval aims at developing algorithms that can support the search for expertise using information-retrieval technology. The topic has been addressed at the expertfinding task of the enterprise track, which ran from 2005 to 2008 at the annual Text REtrieval Conference (Bailey, Craswell, de Vries, & Soboroff, 2008; Balog, Soboroff, et al., 2009; Craswell, de Vries, & Soboroff, 2006; Soboroff, de Vries, & Crawell, 2007) . The specific task setup and evaluation of the expert-finding task at TREC have changed over time. The goal was to design a task that reflects real-life expert finding as well as possible while taking into account constraints regarding the available data and resources for relevance assessment. In the first 2 years, the task was based on the W3C collection, consisting of a crawl of the Web site w3c.org, the platform of the W3C, 2005). In 2005, the task was to predict membership in expert groups, given the name of the group as the topic of expertise. Ground truth data for this task was contained in the corpus. In 2006, topics were created and judged by task participants. Topics were similar to ad hoc topics. Systems had to return candidate experts and supporting documents, and expertise was judged by participants based on these supporting documents. In 2007, a new collection was developed based on the intranet pages of the Commonwealth Scientific and Industrial Research Organization (CSIRO; Bailey et al., 2007) . The goal of using this collection was to evaluate expertise retrieval and enterprise search in the context of a realistic work task-science communicators of CSIRO developed topics and provided relevance judgments. This setup is closest to the one used in this article. In 2008, the same collection was used, and topics were created and assessed by participants. Expertise retrieval experiments at TREC have exclusively focused on expertise finding (i.e., given a topic, find experts on the topic). Other tasks that have been explored are expert profiling (i.e., given a person, list the areas in which he or she is an expert) and finding similar experts (i.e., given a person, return similar experts). In addition to the TREC collections, the UvT collection was introduced by Balog, Bogers, Azzopardi, de Rijke, and van den Bosch (2007) . It represents a typical intranet of a large organization. In this setting, a relatively small amount of clean, multilingual data is available for a large number of experts. This collection is used in our study (discussed later). The task addressed in the current article is finding similar experts, and was first formulated and addressed in Balog and de Rijke (2009): An expert-finding task for which a small number of example experts is given, and the system's task is to return similar experts. Balog and de Rijke defined, compared, and evaluated four ways of representing experts: through their collaborations, through the documents with which they are associated, and through the terms with which they are associated (either as a set of discriminative terms or as a vector of term weights). Similarity between experts is computed using the Jaccard coefficient and cosine similarity. Later, we will use these methods and extend them with representations of experts based on self-provided profiles to form our baseline models. A challenge in content-based expertise retrieval is that systems need to go beyond document retrieval, as they are required to retrieve entities instead of documents. Evidence from documents is used to estimate associations between experts and documents or experts and topics (Balog et al., 2006; Balog, Azzopardi, & de Rijke, 2009 ). Few algorithms have been proposed that take factors beyond textual document content into account. Extending evidence of expertise beyond the documents directly associated with an expert candidate, Serdyukov, Rode, and Hiemstra (2008) introduced graphbased algorithms that propagate evidence of expertise via several steps of related documents and experts. This approach is similar to the search strategies observed in case studies of expertise seeking: People chain evidence of expertise, finding experts via documents and other experts (Hertzum, 2000) . Following a similar intuition, Karimzadehgan, White, and Richardson (2009) leveraged information from graphs representing an organizational hierarchy to enrich information on employees about which little information is known. They showed that people who are close in terms of the organizational hierarchy typically have similar expertise, and that this can be exploited to improve retrieval performance. Amitay et al. (2008), Balog and de Rijke (2009), Jiang, Han, and Lu (2008), and Serdyukov and Hiemstra (2008) explored the Web as a source of additional expertise evidence. Amitay et al. focused on evidence that can be obtained from Web 2.0 applications such as social bookmarking and blogs. Serdyukov and Hiemstra systematically explored different types of Web evidence considering local, regional, and global evidence, and sources from the general Web, such as specific types of documents, news, blogs, and books. Serdyukov and Hiemstra found that using more sources of evidence typically results in better rankings. Balog and de Rijke found that short text snippets returned by a Web search engine can be used effectively to generate expert profiles. In this section, we present the experiments we designed to address our research questions. We first describe the setting in which our experiments were conducted, followed by the datacollection method used to identify contextual factors and to create a test dataset used to perform expertise-retrieval experiments. Next, we describe our expertise-retrieval models: the content-based models that form our baseline, and the principles used to model and integrate contextual factors. Finally, we detail the measures used to evaluate our retrieval models and methods used for parameter estimation. The work task on which we focus is finding similar experts in the context of the public relations department of Tilburg University. The university employs 6 communication advisors, 1 responsible for the university as a whole and 1 advisor for each of the faculties of Economics and Business Administration, Law, Social and Behavioral Sciences, Humanities, and Theology (cf. Table 2 ). Typically, the communication advisors receive requests from the media for locating experts on specific topics. Such requests originate from, for example, newspapers and radio shows desiring quick, but informed, reactions to current events, or from magazine and newspaper publishers requiring more in-depth knowledge for producing special issues or articles about a certain broader theme. Locating the top expert for each request is not always trivial: The expert in question may not be available because of meetings, vacations, sabbaticals, or other reasons. In this case, the communication advisors have to recommend similar experts. This is the situation on which we focus in our article: What similar experts should be recommended if the top expert is The document collection we use for our experiments is the existing UvT Expert Collection 2 , which was developed for expert finding and expert profiling tasks . This collection is based on a crawl of an expert-finding system, WebWijs, in use at Tilburg University. WebWijs 3 is a publicly accessible database of university employees who are involved in research or teaching. This includes, for example, professors, technical and support staff, postdocs and other researchers, and graduate students. Each of the 1,168 experts in WebWijs has a page with contact information and, if made available by the expert, a research description and publications list. In addition, each expert can self-assess his or her expertise areas by selecting from a list of 1,491 knowledge areas, and is encouraged to suggest new knowledge areas that are added upon approval of the WebWijs editor. Knowledge areas are organized in a topical hierarchy, and each knowledge area has a separate page devoted to it that shows all experts associated with that area and, if available, a list of related areas. For our experiments, we extended the collection with topics and relevance ratings for the finding similar experts task. A further resource that we use for our experiments is the media list available at Tilburg University. This list is compiled annually by the university's Office of Public and External Affairs. This list ranks researchers by media appearances, with different media types having a different influence on the score. In this scheme, media hits receive between 1 and 6 points, with mentions in local newspapers receiving 1 point, and international television appearances receiving 6 points. We considered the media rankings of the 3 years (2005)(2006)(2007) available to us and collected the average and the total media score for each expert on these lists. Data collection served two purposes: (a) to gain insights into the factors that play a role when people decide on similar experts in the studied setting and (b) to construct a test set for 2 http://ilk.uvt.nl/uvt-expert-collection/ 3 http://www.uvt.nl/webwijs/ evaluation of our similar expert finding models using standard retrieval-evaluation metrics. All 6 communication advisors of Tilburg University participated in our study. Data were collected through a printed questionnaire that was filled out by participants in their normal work environment and returned by mail. This setup was chosen because it was deemed to require the least effort for the communication advisors, whose time available for participating in the study was limited. A copy of the questionnaire is provided in the Appendix. Note that the study was conducted in Dutch, the native language of the participants; but here, we provide an English translation. The questionnaire consisted of three parts: (a) background information, (b) relevance assessment, and (c) explicit rating of contextual factors. In the first part, participants were asked for background information about their job function, daily activities, and what information sources they usually consult in their daily activities. They also were asked how often they receive requests for experts, to provide examples of such requests, and to explain how these are typically addressed. The second part of the questionnaire focused on eliciting relevance judgments for the similar experts task and the factors influencing these decisions. To identify the reasons for participants' relevance decisions, we posed three follow-up questions for each assessed topic: "Why would you recommend this expert?" "Why did you rank experts in this way?" "Why did you assign the lowest score to this expert?" Questions were formulated as open questions to allow us to discover new factors. To compare frequencies of factor mentions to particpants' perceived importance of factors, the third part of the questionnaire asked participants to explicitly rate the overall influence of these factors on their recommendation decisions. We used a 4-point Likert-type scale and the factors identified in Woudstra and Van den Hooff (2008) (discussed earlier, see Table 1 ). Note that Part 3 of the questionnaire was added only once, at the end of the questionnaire (i.e., after all relevance judgments were made and open questions answered), so that assessors would not be biased by seeing the factors. As explained earlier, the second part of the three-part questionnaire used for data collection focused on obtaining relevance judgments for our retrieval test set. In our setting, the test set consists of a set of pairs (target expert, list of similar experts), so "test topics" are experts for whom similar experts need to be found. These test topics were developed as follows. For each communication advisor, we selected the 10 top-ranked employees from their faculty based on the media lists produced by the university's public relations department (discussed earlier). For 1 faculty, the media list contained only 6 employees, and 2 employees were members of two faculties. For the university-wide communication advisor, JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 5 DOI: 10.1002/asi the top-10 employees of the entire university were selected. 4 In total, 56 test topics were created; these included 12 duplicates, leaving us with 44 unique test topics. For each test topic, we obtained two types of relevance judgment from the communication advisors. First, we asked the (appropriate) advisor(s) to produce one or more similar experts, together with the reasons for the recommendations and the information sources used or would use to answer this request (cf. Appendix Questions II.1-3). Second (on a separate page of the questionnaire), we asked the appropriate advisor(s) to rate the similarity of a list of 10 systemrecommended experts as a substitute on a scale from 1 (least likely to recommend) to 10 (most likely to recommend; cf. Appendix Questions II.4-5). This list of 10 systemrecommended experts per test topic was pooled from three different runs, corresponding to the three topic-centric baseline runs (DOCS, TERMS, AREAS) described in the next section. Participants were then asked to justify their rating decisions (as described earlier). We chose to collect the two types of relevance judgments to identify any obvious candidates that the baseline runs were missing. The order of presenting the two questions was chosen to avoid biasing participants by the list of names. The expert relevance judgments were then constructed in the following way: The ratings supplied by the participants on the 10 experts listed in Appendix Question II.4 were used as the relevance judgments for each test topic. Similar experts who were only mentioned in response to Question II.1, but not in the top-10 list of Question II.4, received the maximum relevance judgment score of 10 (if they were mentioned in both questions, then the rank assigned in Question II.4 was used). Experts who were not rated or not employed by the university anymore were removed. For the 12 duplicate test topics, the ratings by the 2 communication advisors were averaged and rounded to produce a single set of relevance judgments for each topic. To estimate the reliability of the relevance assessments, we compare assessments on the 12 overlapping topics. Percentage agreement between annotators is 87% if we consider two classes: top-ranked experts (i.e., rated as "10") are considered relevant; all other ratings are considered to be not relevant. In this case, Cohen's κ is 0.674, indicating substantial agreement (cf. Landis & Koch, 1977) . In addition, in half of the cases, both annotators independently suggested the same expert (i.e., before seeing our suggestion list). This relatively high agreement may indicate that participants can easily identify a small number of similar experts. Agreement at finer granularity (i.e., when considering each rank as one class) is difficult to establish due to low overlap between rankings (Some candidates were not ranked when participants did not feel comfortable rating a candidate), but is generally lower than at the top rank. 4 We used the most recent version of the list that was available to us (Covering 2006; the elicitation effort took place in early 2008); this was done to ensure that the communication advisors would know the test topics and be able to suggest a similar expert. Baselines. As our baseline approach for finding similar experts, we consider content-based similarity only, leaving out any contextual factors. For computing content-based similarity, we represent employees through the content associated with them. We consider two sources: (a) documents associated with these experts and (b) the expertise areas that experts manually selected for their expertise profile in WebWijs. These two information sources reflect the two types of sources that are typically available in large organizations and that have previously been used for expert-finding experiments. The first, documents associated with an expert, can be obtained from e-mail or document-versioning systems, or associations can be inferred from people names mentioned in documents. This type of information has been the main focus of experiments in the TREC Enterprise task (Bailey et al., 2007) . The second information source is representative of data-warehousing systems where employees have to selfassess their skills (Seid & Kobsa, 2000). Similar information could be mined from publications where authors self-select topical areas from a taxonomy such as the ACM Computing Classification System. 5 From the two sources of information that we have available, we measure (content-based) similarity of two experts using the following three representations:  This is the set of knowledge areas manually selected by expert e from a finite set of predefined knowledge areas. Using the representations described previously, we construct the function sim T (e, f) ∈ [0, 1] that corresponds to the level of content-based similarity between experts e and f (Table 3) . For the set-based representations [D(e), K(e)], we compute the Jaccard coefficient. Similarity between vectors of term frequencies [ t(e)] is estimated using the cosine distance. The three methods for measuring similarity based on the representations listed earlier are referred to as DOCS, TERMS, and AREAS, respectively. Methods DOCS and TERMS are taken from Balog and de Rijke (2007) while AREAS is motivated by the data made available in WebWijs. As our similarity methods are based on two sources (i.e., documents and knowledge areas), we expect that combinations may lead to improvements over the performance of individual methods. The issue of retrieval run combinations has a long history, and many models have been proposed. We consider one particular choice, Fox and 5 cf. http://www.acm.org/about/class; similar taxonomies exist for most scientific disciplines.  (e, f) = |D(e) ∩ D(f)| |D(e) ∪ D(f)| TERMS documents vector: t(e) sim TERMS (e, f) = cos( t(e), t(f)) Shaw's (1994) combination combSUM rule, also known as linear combination. In this model, overall scores consist of the weighted sum of individual scores. In cases where weights sum to 1, this corresponds to the weighted average of individual scores. Optimal weights are estimated empirically (discussed later). Given an employee e, we compute overall contentbased similarity scores sim T (e, f ) (for candidates f ) as the weighted sum of individual content-based similarity scores sim T i : where w i denotes the weight and sim T i is the similarity score according to method T i as defined in Table 3 . On top of the content-based similarity score defined earlier, we model contextual factors that are found to influence human decisions on what similar expert to recommend. Our goal is to optimize the ranking of candidate experts by taking into account additional information from these factors. When we identify which contextual factors play a role in people's decisions on recommending experts, we do not know exactly how these factors influence the decision, and thus how to model them. Therefore, we explore several principled ways of modeling and combining these factors, each of which may reflect ways in which these factors work. We consider two ways of modeling these factors: (a) inputdependent and (b) input-independent. In the former case, a candidate f 's contextual score depends on features of the person e for whom similar experts are being sought. In the latter case, we compute a contextual score for a candidate expert f independent of the (input) person e. In both models, the contextual score of a candidate expert f is combined with the content-based similarity score sim T (e, f ) defined earlier. Input-dependent modeling of contextual factors. Modeling factors in an input-dependent way means that we model a factor as a similarity measure. This option is similar to the content-based baseline models previously discussed in the sense that similarity between experts is estimated using distance in some feature space. The features are designed to reflect the contextual factors that are found to play a role in recommending similar experts (discussed earlier and again in the Results section). For example, one could assume that two employees who are part of the same research group might be recommended as similar experts. A simple model could assign a similarity score of 1 if the two work in the same group, and a score of 0 if they work in different groups. More formally, given an expert e and a candidate f, we determine the input-dependent similarity score sim D i (e, f) between these experts in terms of the contextual factor C i . For factors with nominal values, we set the similarity to 1 if the values are equal, and to 0 if they differ: where C i (·) denotes the value for the contextual factor C i . In case factor C i is numeric, we determine the similarity score based on the absolute difference between the normalized values of C i : where C i min and C i max are the minimum and maximum values of factor C i , so that sim D i (e, f) ∈ [0, 1]. The final input-dependent contextual similarity score is the linear combination of content-based and contextual similarity scores: where w i,j denote the weights for individual content-based and contextual similarity methods, as determined through a series of experiments (dicussed later). Here, we model contextual factors independently of the expert e for whom we are seeking similar experts, meaning that we assume that candidates with certain characteristics are more likely to be recommended as a similar expert. For example, a person with a long publication record may be judged to be very reliable, which may increase the overall chance of this person being recommended as an expert. We assume independence of the individual contextual factors C i and put where w i denotes the weight of the content-based similarity method i and w j denotes the weight of candidate expert f 's score given the value of the contextual factor C i . Note that the assumption of independence is made for simplification, but may not always be realistic. We model score C i (f ) as the probability of f being recommended as an expert conditioned on C i : where E denotes the event that a candidate f is recommended as an expert, and P[C i (f)] denotes the probability of observing the value C i (f) for the specified contextual factor. are estimated from the frequencies observed in the data. For nominal contextual factors, obtaining the frequencies is straightforward. For numeric values, we first discretize factors using unsupervised discretization into 10 equal-sized bins and then obtain the counts per bin. Thus, we approximate the probability that a candidate f is recommended as an expert, given an observed value for contextual factor C i by counting the number of candidates in a specific interval that have been recommended as a similar expert, dividing this by the number of candidates in this interval. To avoid division by zero, we apply the commonly used Laplace estimator (i.e., we initialize all counts to 1). We use three metrics to evaluate the task of finding similar experts: Expert Coverage (ExCov), Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulated Gain (NDCG). ExCov is the percentage of target experts for which an algorithm was able to generate recommendations. Because of data sparseness, an expert-finding algorithm may not always be able to generate a list of similar experts (e.g., if the target expert did not select any expertise areas). In recommender systems evaluation, this is typically measured by coverage (Herlocker, Konstan, Terveen, & Riedl, 2004) . MRR is defined as the inverse of the rank of the first retrieved relevant result (in our case, "expert"). Since communication advisors are unlikely to recommend more than one alternative expert if the top expert is unavailable, achieving high accuracy in the top rank is paramount. Additionally, our experiments show that human judges achieve high agreement on who they would recommend as the most suitable similar expert (discussed previously). Therefore, we will use MRR as our primary measure of performance. NDCG is an IR measure that compares ranked lists and credits methods for their ability to retrieve highly relevant results at top ranks. We use NDCG in our evaluation because the questionnaire participants were asked to rate the recommended experts on a scale from 1 to 10. These ratings correspond to 10 degrees of relevance, which are then used as gain values. We calculate NDCG according to Järvelin and Kekäläinen (2002) , as implemented in trec_eval 8.1. 6 MRR and NDCG are computed for all experts, including those for which similarity methods resulted in empty lists of recommendations. In other words, "missing names" contribute a value of 0 to all evaluation measures. 7 This allows for a more meaningful comparison between methods, as all scores are calculated based on the same set of test topics. We evaluate our contextual retrieval models against a baseline consisting of the optimal combination of content-based retrieval models. Significance testing against this baseline is performed using a paired, two-tailed Student's t test. Throughout the article, we use to indicate runs that significantly outperform the baseline at the 0.05 level. Our goal is to optimize the parameter settings for each model for high MRR (our primary evaluation measure). As is customary when the amount of available training data is small, we optimize and evaluate on the same training set. Note that our goal is not to determine the generalizability of the optimal parameter settings to unseen datasets. Rather, we aim to determine the best possible performance of each model on the available data to determine the upper bound of each model on this data. We then compare these optimized models. For small numbers of parameters, we obtain optimal settings using parameter sweeps. In cases where the number of parameters to estimate is large (e.g., >5), this approach is impractical, as exhaustive search for optimal settings is exponential in the number of parameters. In those cases, we use a simple hill-climbing algorithm to approximate optimal settings in these situations. We randomly initialize the parameters, then vary each parameter between 0 and 1 with increments of 0.1. We select the value for which the target evaluation measure is maximized and then continue with the next parameter. The order in which parameter values are optimized is randomized, and we repeat the optimization process until the settings have converged. Because this algorithm is susceptible to local maxima, we repeat this process 200 times and select the weights that result in the best performance. We present our results in three subsections corresponding to our three research questions. First, we analyze the questionnaires we used to identify contextual factors that play a role in the studied setting. We recommend which of these factors should be integrated with content-based retrieval models to address the finding similar experts task, and develop models for these factors. Finally, we present the results of evaluating 6 The trec_eval program computes NDCG with the modification that the discount is always log 2 (rank + 1) so that rank 1 is not a special case. 7 This corresponds to running trec_eval with the switch −c. 4. Example statements, frequency distribution for statements and participants implicitly mentioning a factor, and explicit importance ratings (0 = no influence, 3 = strong influence) of factors mentioned. Factors marked with * were newly identified on the basis of the data. Participants Mdn Factor (with example statements) (N = 354) (N = 6) rating Topic of knowledge ( "academic record", "has little overlap with the required expertise", "is only in 44.5% 100% 3.0 one point similar to X's expertise", "topically, they are close", "works in the same area") *Organizational structure ("position within the faculty", "project leader of PROJECT", 24.4% 100% N/A "work for the same institute") Familiarity ("know her personally", "I don't know any of them") 17.3% 83% 3.0 *Media experience ("experience with the media", "one of them is not suitable for talking to the media") 5.5% 33% N/A Reliability ["least overlap and experience", "seniority in the area", "is a university professor (emeritus)"] 3.1% 33% 3.0 Availability ["good alternative for X and Y who don't work here any more", "he is an emeritus 2.4% 66% 2.5 (even though he still comes in once in a while)"] Perspective ("judicial instead of economic angle", "different academic Orientation") our models in a retrieval task and analyze the contribution of each factor to the obtained retrieval performance. In this section, we analyze the communication advisors' responses to the questionnaire. We give a short overview of the responses to Part 1 (background information), but then focus on Parts 2 and 3 (identifying contextual factors) of the questionnaire. The amount of expertise requests typically received and the time spent on answering these requests vary widely between study participants. Half of the participants receive requests one to several times per week. The other half reported receiving requests about once a month. Answering the requests typically takes between 5 and 15 min; however, 1 participant reported that answering complex requests can take up to several hours. Participants use a large variety of sources to keep up with current research within their department. All participants mentioned direct contact with colleagues as an information source. Other sources mentioned are press releases, Web sites of researchers, project descriptions, descriptions of research programs, and the WebWijs system. To identify contextual factors that play a role in expertise recommendations, we analyzed the responses to the open questions of Part 2 of the questionnaire, on why specific recommendation decisions were made. These responses were transcribed and analyzed through content analysis. First, the responses were split into statements expressing one reason each, resulting in 254 statements. These were coded independently by two of the authors. Coding was based on the coding scheme developed by Woudstra and Van den Hooff (2008) ; two additional factors, Organizational structure and Media experience were identified and added to the coding scheme (discussed later). Interannotator agreement was 77.5%, and the chance-corrected agreement Cohen's κ was 0.697, indicating substantial agreement (cf. Landis & Koch, 1977) . Conflicts were resolved through discussion. Table 4 gives an overview of the frequency distribution of the resulting factors and the median rating each factor received when participants were asked explicitly to rate these factors. Topic of knowledge was mentioned the most often and was mentioned by all participants. Thus, if we assume that the frequency with which a factor is mentioned relates to the importance of the factor, then the topic is the most important. Other frequently mentioned factors are Familiarity, and the newly identified factors Organizational structure and Media experience. Physical proximity and Saves time were not mentioned by any of the participants. Figure 1 allows for a more detailed comparison of factors resulting from coding open responses ("implicit ratings") versus the explicit ratings collected in Part 3 of the questionnaire. There is agreement over all participants and all measures that topic of knowledge is the most important factor, and Familiarity also appears important according to both measures. Factors that appear less important according to both measures are Cognitive effort, Saves time, Approachability, and Physical proximity. The frequencies of Organizational structure and Media experience cannot be compared to explicit ratings, as they were only discovered during the analysis stage. Some factors display large disagreements in importance according to implicit and explicit rating. The largest discrepancy is found in Up-to-dateness, which was consistently perceived as having a strong influence on expertise recommendations, but was hardly ever mentioned as a reason for a specific expertise decision. Similar differences exist between Reliability, Availability, and Contacts. Based on the survey results, we develop recommendations as to which contextual factors should be considered Because of the large differences between implicit and explicit ratings of these factors, results of evaluating these factors in a retrieval experiment may provide insight into the validity of the two methods used to elicit factors. Approachability, Cognitive effort, Physical proximity, and Saves time do not appear to play a major role in the studied environment and are not discussed further. Not all factors can be easily modeled. We discuss these aspects for each factor; factors that will be included in the follow-up experiments are marked with "+" and ones that will not be considered further are marked with "−." For the sake of simplicity, for each contextual factor addressed in this section, we demonstrate the implementation of that factor in one specific way, except for Reliability for which we consider both publication record and position. For the factors that are modeled, we detail their specific implementations C i (f). Recall that we are considering two types of models: the input-dependent model computes similarity scores between expert candidates and a given candidate based on the value of a contextual factor C i (f) (cf. Equations 3-5). The input-independent model estimates the probability of a candidate being recommended as a similar expert, again, based on the value of contextual factor C i (f) (cf. Equations 6 and 7). + Topic of knowledge is taken to correspond to the contentbased similarity measures presented earlier. This approach represents experts based on (a) the documents associated with them (DOCS and TERMS) and (b) the expertise areas manually selected by the experts themselves (AREAS). We assume that the manually selected labels are the most representative of a person's expertise, as they can be considered as ground truth labels. In addition, participants participating in our user study repeatedly reported using overlap in topic areas as found in the WebWijs system as a basis for their (topic-based) decisions on whom to recommend as a similar expert. We include DOCS and TERMS because these have been used to capture expert knowledge in previous work . + Organizational structure can be implemented by taking membership in workgroups or departments into account. In our setting, we have information about the organizational hierarchy down to the level of individual departments for the entire university and down to the project group level for one faculty. We can use this information to filter out experts from certain faculties or to compensate for data sparseness . In our current implementation, we only use the top level of the available organizational hierarchy and consider organizational structure as a nominal measure: where faculty is a string, for example "FGW" (Faculteit Geesteswetenschappen-Humanities), "FRW" (Faculteit Rechtsgeleerdheid-Legal studies), or "FEB" (Economie en Bedrijfswetenschappen-Economics and Business studies). A staff member may be a member of multiple faculties. Thus, in an input-dependent model, this factor expresses whether two staff members are part of the same faculty; in an input-independent model, it expresses how likely someone is to be recommended as an expert, given that they work in a specific faculty. − Familiarity could be implemented in settings where social network information is available, such as patterns of e-mail or other electronic communication (discussed earlier). In our setting, this type of information is currently not available. + Information on media experience can be obtained from the university's media list (discussed earlier). These media hit counts represent a quantification of media experience and can serve, for instance, as expert priors. We model the media experience of an expert as the sum of all media scores a candidate has accumulated: where media y (f) is the total media appearance score of expert f for year y. + Reliability can be modeled in various ways. For example, a long publication record, or the position within the organization can indicate that an expert is reliable. We have access to both through the data crawled from WebWijs . Because both sources of information are readily available, we develop two models for this factor. First, we use the publication record of academics to estimate the degree of reliability. In principle, a long publishing record grants that a person has valid and credible knowledge and competence. Reliability is then measured as the total number of publications by a candidate f : where pub y (f) is the number of publications of expert f for year y. A second possibility for assessing an expert's reliability is his or her position within the university or, more generally, the organization. For example, a professor may be more likely to be considered a reliable expert by a communication advisor than may a Ph.D. student. This factor is modeled as nominal. Thus: where position is a string; for example, "Professor," "Lecturer," '"PhD student." + Up-to-dateness can be modeled by assigning higher weights to more recent documents associated with an expert, such as recent publications. An ideal candidate not only has credible knowledge but this knowledge also is recent. To measure this, we again use the publication records of people, but here, more recent publications receive a higher weight: where y 0 is the current year, and pub y i (f) is the number of publications of expert f and year i. We model the decrease in the influence of older publications using an exponential weight function with base θ. This parameter can be tuned to adjust the rate at which the impact of publications decrease. In our experiments, we set θ = 0.7. − Perspective is often expressed as a different angle on the same topic, such as judicial instead of economic. This suggests that looking at the organizational structure is a way of preventing too divergent perspectives. Another way of modeling this factor could be to consider coauthorship, as collaborating researchers can be expected to have a similar perspective on a topic. Currently, we do not have robust ways of estimating this factor. − Availability cannot be modeled with the data currently available to us. This may be possible in systems designed to increase the effectiveness of social processes, such as awareness of coworkers' workload (Erickson & Kellogg, 2000) . + An expert's contacts could be modeled by systems that have access to social network information. As we do not have access to this type of data, we model this factor on the basis of coauthored articles and cotaught courses. We assume that the size of the collaboration network is important, so this is what we model. We consider only the number of coauthors and colecturers; that is, the number of people with which f has coauthored a publication or jointly taught a course. Formally: where coauth(f) is the number of distinct people with whom f has coauthored a document or colectured a course. In sum, we model six factors, two nominal (C org and C position ), and four numeric (C media , C publication , C uptodate , and C contacts ). Results of integrating these contextual factors with a content-based retrieval system are provided in the next section. This section contains the results of our models on the retrieval experiment described earlier. We present results for the content-based baseline models, and then for the models integrating contextual factors in both an input-dependent way and in an input-independent way. For each model, we show results for individual factors, and for the optimal combinations of factors. Table 5 shows the experimental results for our content-based runs: individual performance of the three individual similarity methods DOCS, TERMS, and AREAS (as listed in Table 3 ), and weighted combinations of these. We form two combinations: BL-MRR is a baseline combination of all content-based methods with weights optimized for MRR, and BL-NDCG is a baseline combination optimized for NDCG, as described previously. Later, we compare the contextual models against these two baselines. Of the three content-based similarity methods, AREAS performs best in terms of both MRR and NDCG. This method achieves an MRR of 0.4, which means that it identifies a correct expert on Rank 2 to 3, on average. The NDCG score is slightly higher than 50% of an optimal ordering. The relatively good performance of AREAS is expected, as this method makes use of the experts' self-provided profiles, which are expected to contain clean data that accurately JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 11 DOI: 10.1002/asi TABLE 5. Factor weights and retrieval results for content-based models. The optimal combinations of all three content-based models form the baselines for subsequent experiments (BL-MRR and BL-NDCG). Best scores are in bold. Weights are normalized to sum to 1. Content weights
REF showed that many of these factors can be modelled.
Contextual Factors for Finding Similar Experts
J. Assoc. Inf. Sci. Technol.
J. Assoc. Inf. Sci. Technol.
Computer Science