A First Step towards Measuring Interdisciplinary Engagement in Scientific Publications: A Case Study on NLP + CSS Research (2024)

Abstract: With the rise in the prevalence of cross-disciplinary research, there is a need to develop methods to characterize its practices. Current computational methods to evaluate interdisciplinary engagement—such as affiliation diversity, keywords, and citation patterns—are insufficient to model the degree of engagement between disciplines, as well as the way in which the complementary expertise of co-authors is harnessed. In this paper, we propose an automated framework to address some of these issues on a large scale. Our framework tracks interdisciplinary citations in scientific articles and models: 1) the section and position in which they appear, and 2) the argumentative role that they play in the writing. To showcase our framework, we perform a preliminary analysis of interdisciplinary engagement in published work at the intersection of natural language processing and computational social science in the last decade.

## introduction as scholarly disciplines have become increasingly specialized, there is a growing need to share knowledge across traditional interdisciplinary boundaries in order to address broader challenges @xcite . recognizing this trend, scholarly institutions have established interdisciplinary centers @xcite and funding agencies are incentivizing such collaborative efforts @xcite . however, it is unclear how this interdisciplinarity is reflected in the main outcome of science: scientific publications. existing measures of interdisciplinarity are overly simplistic and fail to capture the depth of cross-disciplinary integration in research @xcite . current metrics, such * work done before joining aws ai labs. as affiliation diversity, keywords, and citation patterns @xcite @xcite often overlook how co-authors' expertise is used, and keywords fail to truly reflect a paper's content. moreover, where and how a citation is made within papers remains largely unexplored. for example, citing papers from other fields in the opening paragraph of the introduction might signal higher interdisciplinarity than doing so in a footnote of the method section. in turn, using these references to ground findings and substantiate claims signals a deeper cross-disciplinary engagement than briefly discussing the connections between two bodies of work. to address this challenge, we propose an automated framework for characterizing and measuring the degree of meaningful cross-disciplinary engagement in scientific publications by explicitly modeling the structure (i.e., where do interdisciplinary references appear?) and content (i.e., how are these references used to make claims?) of scientific papers. to do this, we track all interdisciplinary citations in a given article. we consider a citation to be interdisciplinary if it belongs to a venue outside of the current paper discipline. then, for each sentence containing an interdisciplinary citation, we identify its argumentation type. to define argumentation types we build on the schema proposed by @xcite , which distinguishes between claims made about the author's own work, claims made about the background of the author's work, and factual statements that serve as evidence for or against a claim. details about the implementation of this framework are provided in sec. 3. as a case study, we apply our framework to research papers in the interdisciplinary field of natural language processing (nlp) and computational social science (css). in the past decade, greater volumes of text data and more accessible methods have caused a growth in popularity of this research area @xcite , including a dedicated workshop in leading nlp conferences * . however, like other interdisciplinary research, the synergy between nlp and css is often violated due to a misalignment between the research outlook, requirements, and expertise of the researchers in the corresponding fields. in recent work, mc-carthy and dore (2023) manually review a set of css papers that incorporate text analysis methods, and conclude that many of these contributions present what they call descriptive findings: papers that present descriptive catalogs of evidence derived from analyzing social data (e.g., word distributions found in tweets about mass shootings) but that fail to integrate these findings with relevant social science theory. they contrast these contributions with integrative findings, which seek to achieve synergistic methodology to meet the standards of both disciplines, furthering theory. similarly, @xcite note that available nlp methods often fail to meet the needs of social science research, where a limited ability to incorporate theory damages methods' validity. our work presents the first large-scale analysis of the way in which authors working on nlp+css have engaged with the literature in fields outside of computer science and linguistics over the past 10 years. we make the following contributions: 1) we propose a first step towards a general computational framework to analyze interdisciplinary engagement in scientific publications. 2) we construct a comprehensive dataset of computational social science articles published in nlp venues in the last 10 years. 3) we perform a large scale analysis of interdisciplinary engagement in nlp+css research, and show that while interest in nlp+css work is growing, there is a decreasing trend in the engagement with outside disciplines in the mainstream nlp conferences. we explore these trends in the context of the main topics of interest in the nlp+css community and how they have shifted over time. we also show that dedicated workshops like the nlp and computational social science workshop attract highly interdisciplinary contributions, fulfilling their mission of providing an outlet for this type of work. ## related work most previous studies at the intersection of nlp and the science of science have analyzed scientific publications by looking at their citation patterns. while language-centered approaches are scarce in science of science research, there have been some prior efforts in this direction. some notable examples are: studying cross-field jargon interpretation @xcite , the influence of articles in the scientific community @xcite @xcite , the evolution of scientific topics @xcite , and the prevalence of different research themes @xcite . in this paper, we look at when and how interdisciplinary citations are used in scientific articles. previous work looking at citation context in scientific discourse has modeled the sentiment towards cited articles @xcite , citation intent @xcite , purpose and influence (n. @xcite , and critical vs non-critical arguments @xcite . in our study, we focus on the location and argumentation role of interdisciplinary citations. our work is broadly related to the argumentation mining literature @xcite . we study argumentation in the context of scientific publications. while previous studies focus on identifying argumentative discourse units @xcite and their relations @xcite , we study how work coming from outside disciplines is used to make arguments in scientific articles. ## framework in this section, we describe our automated framework to model the content and structure of nlp+css papers. our framework is composed of three sub-tasks: 1) identifying papers that present css findings and contributions, 2) for each relevant paper, identifying all cross-disciplinary citations, and 3) for each cross-disciplinary citation, identifying the type of argument being made. ## experiments and analysis to validate our framework, we first perform an experimental evaluation of each component. then, we use the full framework to perform an extensive analysis of interdisiciplinary engagement for our full dataset of css+nlp papers. ## discussion and future work we emphasize that this paper represents only a first step toward our envisioned framework. going forward, we will expand our labelled dataset for greater coverage of additional tracks and workshops typical of nlp conferences. we aim to predict, with high confidence, what track a paper was submitted to, allowing for a more holistic analysis of the field of nlp and its relationship with out-ofdiscipline work. we will conduct both general and per-track analyses to identify additional trends and explore whether the trends observed in nlp+css hold true in other types of work. we are particularly interested in studying the lasting power of nlp work and how that has shifted under recent advancements in the field. we are also interested in expanding on @xcite to further investigate the differences in integrative and descriptive work and their engagement patterns with out-of-discipline scholarship in the field of nlp as a whole. we seek to provide examples of how integrative work may be carried out in the age of llms. in addition to expanding beyond nlp+css, we are also interested in improving on our argument analysis component. we want to experiment with alternative frameworks, as well as with modeling argumentative relations between claims. moreover, we will further investigate our hypothesis that the most interdisciplinary @xcite this paper presents a study of values, morals and gender bias in a dataset of folk tales spanning cultures. most experiments are designed to reinforce a relevant theoretical claim. @xcite this is a study of the relationship between personal traits and brand preference. the method is grounded in theory and seeks to investigate a debated topic in marketing research through a large scale study. @xcite this paper studies differences in word choices for significant others and indefinite people to examine shifts in societal attitudes toward sexuality and gender. motivation for their method and discussion of results relies heavily on theoretical background. least interdisciplinary @xcite this paper presents a method for using facial emotions to extract sentiment from sentence-image pairs. @xcite this paper studies enhancing limited personal usage information with the goal of improved search personalization. they mainly call on previous computational methods to motivate deign choices. @xcite this paper examines the effectiveness of identifying stance in social media posts given background knowledge about the topic. their proposed method is a variant of retrieval augmented chatgpt. ## conclusion we recognize a need to evaluate interdisciplinary research due to its growing popularity paired with a lack of sufficient methods for studying engagement between disciplines. in line with this, we propose a general scalable framework for tracking interdisciplinary citations within publications. our framework allows for tracking the section where interdisciplinary citations appear and the argumentative role they play within a publication. to showcase our framework, we performed a large scale analysis on the interdisciplinary engagement of research in the field of natural language processing and computational social science. to do this, we constructed a comprehensive dataset of nlp+css papers published in the nlp venues in the last decade. our analysis revealed several trends, including a growth in the number of nlp+css publications, compounded with a decrease in cross-disciplinary engagement in nlp+css papers coinciding with the advent of llms. these findings are in line with previous work highlighting the gaps between the two fields @xcite . however, we also find that dedicated workshops, such as the nlp and computational social science workshop, attract contributions that exhibit higher engagement with the social scientific literature.