Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community (2021)

Abstract: Drawing causal conclusions from observational real-world data is a very much desired but a challenging task. In this paper we present mixed-method analyses to investigate causal influences of publication trends and behavior on the adoption, persistence and retirement of certain research foci – methodologies, materials, and tasks that are of interest to the computational linguistics (CL) community. Our key findings highlight evidence of the transition to rapidly emerging methodologies in the research community (e.g., adoption of bidirectional LSTMs influencing the retirement of LSTMs), the persistent engagement with trending tasks and techniques (e.g., deep learning, embeddings, generative, and language models), the effect of scientist location from outside the US e.g., China on propensity of researching languages beyond English, and the potential impact of funding for large-scale research programs. We anticipate this work to provide useful insights about publication trends and behavior and raise the awareness about the potential for causal inference in the computational linguistics and a broader scientific community.

## related work there are two complementary causal inference frameworks -structural causal models @xcite and treatment effect estimation approaches @xcite . existing approaches to learn the causal structure (aka causal discovery) broadly fall into two categories: constraint-based @xcite and score-based @xcite . recently, there have been an increased interest in causal inference on observational data @xcite , including text data, in the computational linguistics and computational social science communities @xcite . for example, recent work by @xcite estimated the effect of perceived author gender on the number of citations of the scientific articles and @xcite measured the effect that presence of a theorem in a paper had on the rate of the paper's acceptance. additional examples in the computational social science domain include: measuring the effect of alcohol mentions on twitter on college success @xcite ; estimating the effect of the "positivity" of the product reviews and recommendations on sales on amazon @xcite ; understanding factors effecting user performance on stackexhange, khan academy, and duolingo @xcite ; estimating the effect of censorship on subsequent censorship and posting rate on weibo @xcite and word use in the mental health community on users' transition to post in the suicide community on reddit @xcite ; or the effect of exercise on shifts of topical interest on twitter @xcite . moreover, @xcite presented the overview of causal approaches for computational social science problems focusing on the use of text to remove confounders from causal estimates, which was also used by @xcite . earlier work utilized matching methods to learn causal association between word features and class labels in document classification @xcite , and use text as treatments @xcite or covariates e.g., causal embeddings @xcite . several studies have leveraged the acl anthology dataset to analyze diversity in computer science research @xcite , and perform exploratory data analysis such as knowledge extraction and mining @xcite @xcite . however, unlike any other work, our approach focuses on leveraging complementary methods for causal inference -structural causal models and treatment effect estimation to discover and measure the effect of scientists' research focus on their productivity and publication behavior, specifically the emergence, retirement and persistence of computational linguistics methodologies, approaches and topics. ## data preprocessing our causal analysis relies on the publication records from the association of computational linguistic (acl) research community from 1986 through 2020. we collect the acl anthology dataset foot_0 @xcite with the bibtex provided with the accompanying abstracts. excluding all records that do not contain authors (e.g., bibtex entries for the workshop proceedings), we convert the bibtex representation into a data representation where each row represents each paper-author combination (i.e., for a paper (paperx) with three authors, there are three representative rows: paperx-author1, paperx-author2, and paperx-author3). then, we extract features that encode paper properties: the year it was published, whether the paper was published in a conference or journal, the number of authors, the number of pages, and word count in paper. we also compute gunning fog index @xcite -influenced by the number of words, sentences, and complex words. we then annotate each row with properties related to the author during the year the paper was published. as a proxy of the length of the author's research career in the computational linguistics community, we calculate the number of years since the author's first publication in the anthology. each author's location is represented as the location (country) of the institution the author is associated with in the metadata or full text. to measure productivity at varying granularities, we calculate the number of one's papers published in total, in the last year, and in the last five years. we then construct a dynamic network representation of the anthology using author-to-paper relationships for each calendar year, as encoded in the metadata. after projecting those relationships into the dynamic co-authorship network that reflects author to co-author connections by year, we calculate centrality and page rank network statistics over time to measure the influence of the author. these collaboration behavior features complement previously described author properties. we also added three features to encode the diversity in co-authorship. first, the number of all co-authors who published the papers with the author. second, the average number of papers co-authored per coauthor, which is computed as the total number of papers co-authored per co-author divided by the number of co-authors. the last is a likelihood that a co-author is an author on a paper, which is the second feature divided by the total number of the author's papers. this enables us to measure the diversity, or lack thereof, of collaborative relationships of each author, and encodes how collaboration behavior evolves over time. ## methodology we investigate the causal relationships between characteristics of individual publications for each researcher who authored at least two publications hosted in the acl anthology. we identify causal influences of publication outcomes related to the adoption, retirement, and maintenance of computational linguistic methodologies, approaches, or topics, as well as the productivity of authors using both causal discovery and pairwise causal effect estimation. ## results in this section, we present a series of key findings surfaced within the causal mechanisms discovered and treatment effects estimated focusing on contrastive analysis over time: whether publishing on a given research entity (methodology, task, material) influences continuing to publish in that area or with that methodology, how authors shift from existing to novel methodology over time, and evidence of external events (i.e., funding or large research programs) potentially impacts the adoption and maintenance of publication trends. ## discussion and conclusions our analyses show that publishing once on a specific task, application, or methodology has a causal influence that causes authors not to publish on the same approach in the following year e.g., robustness, interpretability etc. this is consistent across a significant number of the research methods, tasks, and application domains represented in our consolidated annotations. there are several potential drivers of this causal relationship. first, publishing in the cl community, particularly in recent years, is extremely competitive. acceptance rates are low and the number of submissions each year continue to growthere is a lot of competition for few spots. this 2014 2015 2016 2017 2018 could also be reflective of the churn in novelty and state-of-the-art technologies: as one technology (e.g., lstms) are replaced by a new sota methodology (e.g., bilstms, or transformers), this naturally leads to the retirement of the previous methods, which we also see reflected in the findings presented in subsection 4.2. as we drive the field forward, we stop publishing on older methods, materials, and tasks because of novelty incentives or requirements to be accepted in top-tier venues. limitations a limitation of our current causal analysis approach is the restriction to acl anthology publication records only. as this dataset comprises only acl venues, it does not guarantee inclusion of all possible publications from each author. for example, authors who publish in acl venues may also publish in iclr, neurips, etc. future work can address this limitation by augmenting the data set with supplementary collects from these venues (e.g., using google scholar). another avenue of future work is to incorporate funding information directly in the analyses. as we have shown, there is evidence that funding has causal relationships with publication outcomes and expertise evolution, and may act as a confounder for other relationships. future work will extract funding from full text pdfs (e.g., from the acknowledgements section) in order to adjust for the effects of funding as a confounder, which may also be an impactful treatment to analyze.