Where to Submit? Helping Researchers to Choose the Right Venue (2020)

Abstract: Whenever researchers write a paper, the same question occurs: “Where to submit?” In this work, we introduce WTS, an open and interpretable NLP system that recommends conferences and journals to researchers based on the title, abstract, and/or keywords of a given paper. We adapt the TextCNN architecture and automatically analyze its predictions using the Integrated Gradients method to highlight words and phrases that led to the recommendation of a scientific venue. We train and test our method on publications from the fields of artificial intelligence (AI) and medicine, both derived from the Semantic Scholar dataset. WTS achieves an Accuracy@5 of approximately 83% for AI papers and 95% in the field of medicine. It is open source and available for testing on https://wheretosubmit.ml .

## introduction when choosing a scientific conference or journal (in the following called venue) to submit a manuscript, researchers consider several factors. while factors such as the venue's impact, time, or location are important, the main factor is the manuscript's thematic fit to the conference. this can be ensured by inspecting the call for papers (e.g. this paper fits into emnlp as it is an "interesting application nugget") or by analyzing previously published papers at the given conference. given the growing number of conferences (e.g., the exponential growth of computer science publications indicates more and/or larger conferences foot_1 ), the second approach has become harder than ever, especially for novice researchers, or even senior researchers wanting to publish in a new domain. finding a thematically fitting venue for a manuscript therefore is a time consuming task. figure 1 : overview of wts. title, abstract, and keywords are processed separately by a convolution layer and max-over-time pooling. the output vectors are concatenated and fed through two fully connected layers that predict fitting venues. for the top five entries, we calculate the important words and phrases using the integrated gradient method @xcite . in this work, we try to simplify this process by introducing "where to submit" (wts), an nlp system based on a convolutional neural network that recommends academic venues given the title, abstract, and/or keywords of a planned publication. the system is trained on previously published manuscripts. to understand the system's choice of recommending a specific venue, wts analyzes the words and phrases that had the highest impact on a recommendation. a researcher can then use the list of recommended venues as a starting point to find the best fitting venue based on other factors such as the call for papers, rank, or deadline. our main contributions are: (1) we describe an effective method that can recommend scientific venues based on a paper's title, abstract, and keywords. ## related work there exist several online services that recommend venues based on the contents of a publication, but all of them are lacking in some ways: elsevier; journal guide; springer; wiley; enago; edanz; manuscript matcher; journal/author name estimator; sjfinder). especially in the fields of computer science and ai, most work is published on conferences @xcite , making this a severe drawback for ai researchers. ## task and methodology now we describe the task, our proposed method wts, and the baselines we compare it to. ## datasets for training and testing, we extract all publications from the semantic scholar dataset @xcite 3 that were published in the research fields artificial intelligence (ai) and medicine. 4 a publication is considered to be an ai paper if it was published in one of the scientific venues given in @xcite . we manually match them as closely as possible to the semantic scholar venues. this procedure leads to 77 distinct venues. we also add a class called "non-ai" consisting of 20 000 publications from other fields, to let the model learn the difference between ai and non-ai venues, resulting in 78 classes for this dataset. in the field of medicine, we only use publications from semantic scholar that originate from medline, a medical publication database. due to a high number of venues with few publications, we only consider the top 78 venues (the same number as for the ai dataset, making the performance metrics comparable), which account for about 10 % of the publications. in general, we only keep publications where no input information is missing. ## experimental setup given the training, validation, and test splits of both datasets described in section 4, we train all models on the training dataset and perform hyperparameter optimization on the validation data. we then report several metrics, detailed below, on the test dataset. hyperparameters for wts, we perform random search @xcite with 100 runs choosing the model with the best validation accuracy. since our method uses a cnn that can have different filter sizes in the convolution layer, we randomly sample up to three different filter sizes from {1, . . . , 9}. the number of filters per size are drawn from {5, . . . , 300}. also, we randomly choose the optimizer from {adam, sgd} @xcite , the learning rate from continuous values in [0.01, 0.1], and the dropout probability from [0.1, 0.5]. other parameters such as the lengths to which the three inputs are padded and the early stopping patience on the validation loss were set manually. metrics to compare our methods, we use three common ranking metrics: accuracy, accuracy@5, and mean reciprocal rank (mrr). accuracy@5 measures the fraction of test examples where the method correctly puts the venue in the top five ranks. mrr is defined as follows: given a test set of queries q for which the model returns a ranking of all items, @xmath0, where rank i is the position at which the target item is ranked by the model. always predicting the correct item at first position leads to a mrr of one, while bad models achieve a mrr closer to zero. ## results interestingly, compared to the medicine dataset, the method by iana et al. performs poorly on ai publications. we suspect this is due to the smaller size of the ai dataset and a higher skew in publication counts per venue (cf. ## explainability as a key part of wts is explainability, we do not only want to recommend venues to the user but also explain why a certain venue was recommended. we use the integrated gradients method introduced by @xcite implemented in the py-torch captum library foot_2 to find the most influential words and phrases for the top five recommendations of the network. the method varies the input by linearly transitioning in 50 steps from the actual embedding inputs to matrices filled with zeros. then, the gradients of the desired venue output with respect to the inputs are calculated for each of these steps. the gradients are averaged and multiplied by the initial input, giving positive or negative values to words and phrases that had positive or negative impact on the score of the desired venue. figure 2 visualizes an excerpt from wts's output for the well-known bert paper, which received the best paper award at naacl 2019 @xcite . it correctly ranks "naacl" first. integrated gradients correctly identifies "transformers" and "language understanding" as words that qualify this publication as an nlp paper. ## website in order to make our system available to the public, we release wts as a web service foot_3 where researchers can input their ai paper's information and receive recommendations for venues. the web service applies the trained cnn and explainability method and shows the top five predicted venues for the given paper along with a color-coded explanation (cf. section 7) and venue-related links. the accuracy@5 results described in section 6 indicate that most of the times, a fitting venue is displayed to the user. ## conclusion we have presented wts, an nlp system that recommends scientific venues based on the title, abstract, and/or keywords of a publication. wts is designed to provide an explanation as to why a certain venue was recommended, making it the first interpretable and open recommendation service for both, conferences and journals. we have shown that wts provides strong recommendations on publications from the areas of ai and medicine. future work may regard evaluation: while each publication only was published at one specific venue, it might also be suitable for multiple other venues, implying that our current scores are just lower bounds for the actual performance. improvements to the provided web service could be to make the list of venues sortable based on their deadline, impact, or other configurable factors.