Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers (2020)

Abstract: The question of the utility of the blind peer-review system is fundamental to scientific research. Some studies investigate exactly how “blind” the papers are in the double-blind review system by manually or automatically identifying the true authors, mainly suggesting the number of self-citations in the submitted manuscripts as the primary signal for identity. However, related work on the automated approaches are limited by the sizes of their datasets and the restricted experimental setup, thus they lack practical insights into the blind review process. In this work, we train models that identify the authors, their affiliations, and their nationalities through real-world, large-scale experiments on the Microsoft Academic Graph, including the cold start scenario. Our models are accurate; we identify at least one of authors, affiliations, and nationalities of held-out papers with 40.3%, 47.9% and 86.0% accuracy respectively, from the top-10 guesses of our models. However, through insights from the model, we demonstrate that these entities are identifiable with a small number of guesses primarily by using a combination of self-citations, social, and common citations. Moreover, our further analysis on the results leads to interesting findings, such as that prominent affiliations are easily identifiable (e.g. 93.8% of test papers written by Microsoft are identified with top-10 guesses). The experimental results show, against conventional belief, that the self-citations are no more informative than looking at the common citations, thus suggesting that removing self-citations is not sufficient for authors to maintain their anonymity.

## introduction scientific publications play an important role in dissemination of advances, and they are often reviewed and accepted by professionals in the domain before publication to maintain quality. in order to avoid unfairness due to identity, affiliation, and nationality biases, peer review systems have been studied extensively @xcite @xcite , including analysis of the opinions of venue editors @xcite and evaluation of review systems @xcite . it is widely believed that a possible solution for avoiding biases is to keep the author identity blind to the reviewers, called double-blind review, as opposed to only hiding the identity of the reviewers, as in single-blind review @xcite . since some personal information (e.g., author, affiliation and nationality) could implicitly affect the review results @xcite , these procedures are required to keep them anonymous in double-blind review, but this is not foolproof. for example, experienced reviewers could identify some of the authors in a submitted manuscript from the context. in addition, the citation list in the submitted manuscript can be useful in identifying them @xcite , but is indispensable as it plays an important role in the reviewing process to refer readers to related work and emphasize how the manuscript differs from the cited work. to investigate blindness in double-blind review systems, @xcite and @xcite train a classifier to predict the authors, and analyze the results. however, they focus primarily on the utility of self-citations in the submitted manuscripts as a key to identification @xcite @xcite , and do not take author's citation history beyond just self-citations into account. the experiment design in these studies is also limited: they use relatively small datasets, include papers only from a specific domain (e.g., physics @xcite , computer science @xcite or natural language processing @xcite ), and pre-select the set of papers and authors for evaluation @xcite . furthermore, they focus on author identification, whereas knowing affiliation and the nationality also introduces biases in the reviewing process @xcite . in this paper, we use the task of author identity, affiliation, and nationality predictions to analyze the extent to which citation patterns matter, evaluate our approach on large-scale datasets in many domains, and provide detailed insights into the ways in which identity is leaked. we describe the following contributions: 1. we propose approaches to identify the aspects of the citation patterns that enable us to guess the authors, affiliations, and nationalities accurately. to the best of our knowledge, this is the first study to do so. though related studies mainly suggest authors avoid self-citations for increasing anonymity of submitted papers, we show that overlap between the citations in the paper and the author's previous citations is an incredibly strong signal, even stronger than self-citations in some settings. 2. our empirical study is performed on (i) a realworld large-scale dataset with various fields of study (computer science, engineering, mathematics, and social science), (ii) study different relations between papers and authors, and (iii) two identification situations: "guess-at-leastone" and "cold start". for the former, we identify authors, affiliations and nationalities of the affiliations with 40.3%, 47.9% and 86.0% accuracy respectively, from the top-10 guesses. for the latter, we focus on papers whose authors are not "guessable", and find that the nationalities are still identifiable. 3. we perform further analysis on the results to answer some common questions on blind-review systems: "which authors are most identifiable in a paper?", "are prominent affiliations easier to identify?", and "are double-blind reviewed papers more anonymized than single-blind?". one of the interesting findings is that 93.8% of test papers written by a prominent company can be identified with top-10 guesses. the dataset used in this work is publicly available, and the complete source code for processing the data and running the experiments is also available. foot_0 ## related work here, we summarize related work, and describe their limitations in analyzing anonymity in the blind review systems. ## identification approach training and test datasets are independently prepared, and papers in the training dataset are older than those in the test dataset. we extract features from the training dataset to model each author's citation pattern, and the entity also can be affiliation or nationality depending on what we guess in the test papers. building entity models, we score each entity based on its extracted features for a test paper, and sort the scores for the paper to rank all the entities. we describe the detail of each process in the following sections. ## experimental setup we define some terms and variables used in the following sections, and then describe the mag dataset and how we develop benchmarks from it. ## experiments and results using various approaches explained above, we perform experiments in two different identification scenarios: "guess-at-least-on" and "cold start". through the first experiment, we show how anonymized a paper is in each of author, affiliation and nationality identifications. in the second experiment, we show that there remain identity leaks even when no authors in a paper are identifiable.