Theory-Grounded Computational Text Analysis (2023)

Abstract: In this position paper, we argue that computational text analysis lacks and requires organizing principles. A broad space separates its two constituent disciplines—natural language processing and social science—which has to date been sidestepped rather than filled by applying increasingly complex computational models to problems in social science research. We contrast descriptive and integrative findings, and our review of approximately 60 papers on computational text analysis reveals that those from *ACL venues are typically descriptive. The lack of theory began at the area’s inception and has over the decades, grown more important and challenging. A return to theoretically grounded research questions will propel the area from both theoretical and methodological points of view.

## introduction computational text analysis methods-an umbrella combining natural language processing with social science-are in a honeymoon period @xcite . today's social scientist might reach for the tools of computer science for their speed, scale, granularity, and consistency; for instance, natural language processing offers "to analyze signals ranging from simple lexical cues to word clusters to choices of syntactic structure" @xcite . the numerical outputs tell a story that is simple, easy to make sense of, and in that regard comforting. conversely, today's computer scientist may see the problems of social science as answerable by objectivity and reductionism, eschewing interpretation for quantitative analysis. the conclusion of this reasoning, and the dominant stance in computational social science, is a reliance on machines alone to answer questions in the field, surrendering to their supposed objectivity * equal contribution. or impartiality. can a machine's output go beyond descriptive catalogs of evidence, accelerating understanding of processes and motivations? from our experience, computers are nowhere near supplanting humans in interpreting social science results. 1 an interdisciplinary inquiry must go farther than matching computational techniques to social science questions @xcite . it embraces synergistic methodology and connects the norms and standards of evidence from both. this means partnering computer science's preference for the structured, generalizable, and objective with the unstructured, critical, and contextual which the social sciences champion. this level of interdisciplinarity addresses the question raised by descriptive findings: so what? we see theory as the solution, empowering rather than shackling investigations. what this paper advocates is not one particular theory-certainly these are myriad, and "even subject matter which has been under intensive and prolonged study remains at the unsettled periphery of research" @xcite . instead, we expand on our prior work @xcite to clarify calls echoed for decades by computational and social science @xcite @xcite @xcite . underlying each, we find, is the urge to return to theory, which we espouse herein. ## description vs. integration we contrast descriptive findings and theoretical analysis. an example of a descriptive finding is that an apple falls, or that it falls faster when pushed than dropped, or even that it falls at a particular rate estimated with some standard error by a complex interpolation. a theoretical analysis of the same phenomenon, credited to newton, is that a fundamental force acts upon the apple, and that this same force governs the motion of the heavens. the theoretical analysis links the finding about the world critically to a broader body of knowledge and context. despite advances in causal inference in nlp, the descriptive is all that a machine can provide to the social sciences @xcite . certainly the methods of computational text analysis have advanced since the general inquirer @xcite and mosteller and wallace's statistical inference of text authorship @xcite . but methods are means, not ends. they uncover more descriptive findings in data: the rate of an apple's fall, the topics of refugees' tweets @xcite , the space given to marginalized groups in textbooks @xcite , or patterns of state censorship @xcite . the foils to descriptive findings are integrative findings @xcite , which offer causal explanations that enable future predictions-a theory, or as a 'model' in the sense of the standard model, rather than of a statistical model. integrative findings can either offer new theories or couch their explanations in existing theories-but the theory is essential either way. ## we don't integrate to contrast descriptive and integrative findings, we reviewed approximately 60 papers in computational text analysis published in *acl venues. in publication venues within the acl focus on methods over theory. descriptive @xcite the article presents new quantitative methods to measure semantic meaning in inferred topics. the authors emphasize the qualitative relevance of their findings as it validates the use of topics for corpus exploration and information retrieval. however, their working hypothesis and empirical findings are not connected to the extremely relevant field of communication theory. @xcite the article presents the first large-scale analysis of political content censorship in social media. the authors miss the opportunity to relate their hypothesis and findings to censorship theory, a natural theoretical context for the research, which would strengthen the relevance and generalizability of the findings. field et al. (2018) the article discusses media manipulation in russia in the context of agenda-setting and framing, the tools that russian state-owned (or heavily influenced) media outlets use to distract public attention from domestic economic politics. the authors implicitly refer to propaganda theory and autocratic theory throughout the article even though their findings are not discussed in relation to these theories. demszky et al. (2019) the article applies "a more comprhensive nlp framework to study linguistic aspects of polarization in social media". while the article implicitly refer to theories of social conformity and social conflict, the findings are not linked or discussed (either explicitly or implicitly) to the theoretical frameworks that the authors touch on in their §1. ## moving forward we are not denying the value of computational approaches to analyzing text. certainly, comput-ing can be an instrumental approach for modeling and understanding social complexity. this does not mean that other approaches, such as historical, ethnographic, or mathematical, become irrelevant. on the contrary, computational methods necessarily (whether awarely or not) rely on these earlier approaches to add value, in terms of improving our explanations and understanding @xcite . as we are a field that prioritizes methods, consider the seminal book on methods in science: @xcite taxonomizes scientific ways of knowing. its five broad categories are ethnography, historical narration, standard causal analysis, small-n comparison, and formal modeling. we in nlp myopically choose the third and fifth of these, ignoring the value of the others. but the broader point of methods of discovery is not methods. it is the research question. any methodology should be grounded in the question, not incremental tweaks and reviewers' comfort @xcite . this admits even qualitative or mixed-method approaches to text analysis. the role of humans in scientific inquiry is nothing new. using qualitative analysis to complement quantitative techniques has its roots in achen and snidal (1989)'s recommendation to use historical case studies as a complement to statistical research. ## conclusion we have explained the distinction between descriptive and theoretical findings as it pertains to computational text analysis. the bulk of work we found provided vast descriptive findings, often of high quality, but not giving back to questions of theory. we offer several suggestions on how to 'push the pendulum back' by prioritizing theory-building or theory-affirming research questions and accepting whichever methods are best suited toward answering it-not only the familiar and entrenched ones. we are not the first to advocate for a shift in the patterns of applying computational techniques to real-world problems. there is a steady drumbeat from voices in the field advocating careful approaches @xcite @xcite @xcite @xcite . what we see underlying all of thesethose writing against 'mathiness' and speculation, advocating for clear evaluation over anecdotes, criticizing textual researchers' dilution of conceptual standards, highlighting work that ties linguistic information into complex models-is an unspoken, perhaps unrealized, call for a return to theory. not only do we aver that incorporating theory is essential; but also, other fields have strengthened themselves when espousing organizing principles beyond those of their progenitors. behavioral economics is a success story here. it transcended the neat (but psychosocially stripped) mathematics it draws from to acknowledge deviations from rationality and blend economics with cognitive science @xcite @xcite . for scientific-not simply engineeringadvances to arise from the *acl community, authors and reviewers alike must resist the temptation toward incremental, 'safe' research and follow @xcite : "controversial papers are great; boring unobjectionable incremental papers are not." in reviewing new research, we should privilege not only work that presents new and unusual computational methods, but also interactions between computational and humanistic approaches to answering research questions. emnlp was founded because of reviewing biases at acl against groundbreaking methodological advances, and since then the two have homogenized; "emnlp reviewing is no longer much of a differentiator" @xcite . we found that theoretically grounded findings in text analysis are often published in non-*acl venues (