This paper describes and extensively evaluates a system for the automatic routing of submitted papers to reviewers and area committees, without the need for any human annotation from the reviewers or the program chair. Routing is based on a profile of previous writings obtainable on-line for the reviewer pool, a generally stable and reusable resource that requires no manual adaptation for new submission streams. The paper explores a wide set of variations and extensions on the core model, and achieves system accuracy approaching that of several human judges on the same task. Routing submitted papers, abstracts or grant proposals to qualified reviewers is a central task of the academic enterprise, and a remarkably difficult one. Typically it is conducted under significant time pressure in a conference reviewing cycle. As the number of submissions and size of the reviewer pool grows, it becomes increasingly difficult for a conference chair to be familiar with the different expertise of all members of the program committee. It is also difficult for one person to master the subtleties of fine subject area distinctions as topic diversity in a conference becomes large. For these reasons, conferences such as ACL (the Association for Computational Linguistics) often use a hierarchical program committee structure, where submitted papers are first routed to area committees, and then more specialized area chairs have the task of assigning papers to individual reviewers in the committee. However, in a diverse and multidisciplinary field such as natural language processing, it is often difficult to define clear cut committee descriptions and the program chair still must be cognizant of the detailed expertise of the area committee members in order to route atypical or multidisciplinary papers to committees with the most appropriate pool of reviewers. The low inter-rater consistency results shown in Table  12 indicate that humans find even area committee routing to be a difficult task. The following paper focuses on a range of automated solutions to this task of routing papers to their most appropriate area committee. It presents extensive empirical investigation and evaluation of a wide range of issues related to this task. Previous published research into the problem of automatic routing of conference paper submissions is surprisingly limited. Approaches to this task can be essentially broken down into four major strategies: The first strategy is keyword based. Authors are required to specify a list of topic/subtopic areas for their papers (often from a prespecified term list), and reviewers then complete a survey of their relative level of expertise on this list of topics/subtopics. This approach is followed by AAAI conference reviewing. It suffers from the problem that authors often have a difficult time selecting keywords to adequately describe their work. It works best in conferences that are very broad, and is least effective in more focused workshops where routing distinctions in subject area and paradigm are more subtle. The second strategy is to build a statistical profile of reviewers' expertise by eliciting relevance judgments on a set of abstract data. AAAI also requires its reviewers to rank (bid on) submitted abstracts, and there is currently unpublished work exploring the application of supervised routing to the ranked reviewer bids on AAAI submitted abstracts (Hirsh, personal communication). In groundbreaking work, Dumais and Nielsen (1992) developed a system for the routing of Hypertext'91 abstracts using latent semantic indexing (Deerwester et al., 1990) , trained from available text sources including a small set of reviewer-submitted abstracts, on-line books and ACM articles as a source for the termby-document matrix used in their singular value decompositions. Reviewers manually ranked their interest in all submitted abstracts, and best performance was achieved when reviewers were assigned twice their target number of abstracts and asked to choose their preferred half. One problem with the modeling of reviewer rankings/bids is that these may be based more on what the reviewer finds interesting rather than what 220
The most extensive work was done by Yarowksy et al. (1999) , who performed their experiments on the papers submitted to the ACL '99 conference REF .
Taking The Load Off The Conference Chairs - Towards A Digital Paper-Routing Assistant
Joint SIGDAT Conference On Empirical Methods In Natural Language Processing And Very Large Corpora
Computer Science