Causally Modeling the Linguistic and Social Factors that Predict Email Response (2025)

Abstract: Email is a vital conduit for human communication across businesses, organizations, and broader societal contexts. In this study, we aim to model the intents, expectations, and responsiveness in email exchanges. To this end, we release SIZZLER, a new dataset containing 1800 emails annotated with nuanced types of intents and expectations. We benchmark models ranging from feature-based logistic regression to zero-shot prompting of large language models. Leveraging the predictive model for intent, expectations, and 14 other features, we analyze 11.3M emails from GMANE to study how linguistic and social factors influence the conversational dynamics in email exchanges. Through our causal analysis, we find that the email response rates are influenced by social status, argumentation, and in certain limited contexts, the strength of social connection.

## introduction from greeting friends to discussing work, emails have become a vital component of human communications in modern society. a key intent behind many emails is to get a reply from the recipient. when making requests of other people, multiple theories have suggested specific factors that predict whether a request will be successful, e.g., its politeness @xcite . prior work in nlp has focused on testing the effects of specific factors like power and face-saving as predictors (e.g., @xcite . yet, these studies often lack the big picture of the environment in which a request is made to understand how multiple factors interact to influence successful requests. email represents an ideal setting for studying such requests due to social structure (revealed through communication) and the flexible content length. as a result, here, we propose a new study of requests by using to model the dynamics of email conversation, this study analyzes over 11m emails from the gmane corpus @xcite , a dataset of email chains on a public forum. we introduce a new dataset, sizzler, that captures the intent and reply-expectation of emails, and create a comprehensive list of social and linguistic factors for each email, such as the sender's position in the gmane network and the argument quality of their text. figure 1 illustrates our modeling framework. our contribution is threefold. first, we introduce a new dataset, sizzler, of emails annotated for intents and sender expectations by human experts. second, we propose a structured framework for modeling the mechanisms that potentially increase the likelihood of an email receiving a reply. we release these labels for 11m gmane emails, along with the associated computational features (e.g., argument quality). third, we conduct a causal analysis using propensity score matching to determine which of a comprehensive list of social network and linguistic factors influence email responses. we find that email response rates are generally more impacted by social status and argumentation than by the strength of social connection. all data and code are available on github. 1 ## the sizzler dataset we present the social information for analyzing and characterizing the likelihood of email replies (sizzler) dataset, which is based on the webis gmane email corpus 2019 @xcite . the gmane corpus is one of the largest email datasets available, comprising over 153 million emails from gmane.io. to analyze the effects of social network and linguistic factors on reply behavior in emails, sizzler includes not only each email's reply status and the pragmatic factors of interest, but also a number of covariates that influence email replies, such as the sender's communicative intent @xcite @xcite and expectation of receiving a reply @xcite ) (e.g., allowing us to account for confounders like: an email thanking someone for their help is less likely to receive a reply and more likely to be polite). to create this dataset, we first extract features related to email threads and networks from the gmane corpus and then develop models to capture the sender's ## classifying intents and expectations we use our labeled data to model the sender's communicative intent and reply expectation across the entire dataset, in a multi-label classification task. to understand why this task is important, consider an email simply saying "thank you!" where the sender probably not expecting a reply. this type of message would confound our analysis, because it is both less likely to receive a reply and has systematic differences in linguistic features (e.g., poor argumentation). the gmane dataset is not labeled with the sender's communicative intent or reply expectation, so we modeled these features. having access to these features at scale allows us to then match replied and unreplied messages on the sender's reply expectation and intent using propensity score matching, and analyze the effects of factors like network position, argumentation quality, ## factors influencing getting a response our analysis studies how the likelihood of receiving a reply is influenced by several factors that relate to social connection, social status, and use of argumentation to engage on gmane. this includes the style or tone of their writing and how the author is positioned in the social network. we propose 14 factors (eight linguistic, six network properties) to quantify different aspects of connection, status, and argumentation. we use propensity score matching to estimate how these 14 factors affect reply behavior, on a subset of emails that are matched to 4 we experimented with a two-shot setting for the single-choice task, but performance declined for some intent types, with the micro-averaged f1 score dropping from 0.53 to 0.47. adding two-shot examples made the model more likely to predict the positive label, decreasing precision while increasing recall. these results highlight llama3-8b-instruct's limited ability to judge email intent, and that roberta is superior both in performance and speed for constructing our dataset. ## discussion this research presents an integrated view of factors that are often considered separately in analyses of message reply behaviors: social connection, social status, and argumentation. social connection and social status play complementary roles in email reply behaviors, in ways that are heavily moderated by the conversational setting. for instance, attributes related to social status and connection that promote replies in initial emails (when a sender is addressing a broader audience) are unimportant or even anticorrelated to response rates in response emails. instead, network social status markers are the most predictive of replies in response emails. in addition to various network and linguistic markers of social status and connection, we also explored the role of argumentation strategy. for instance, affective argumentation consistently produces higher response rates than cognitive argumentation. this work makes three contributions. first, we offer a new dataset labeled with each post's email intentions and reply expectation. we used these labels as control variables in analyses on the dynamics of communication on gmane. second, we propose a methodological framework for causally modeling the effects of network and linguistic factors on reply behavior using propensity score matching. and third, we examine the role of linguistic and network factors in reply behavior, presenting a theoretically integrated view of social connection, social status, and argumentation factors in message reply behavior on online forums. ## conclusion despite the intention to elicit a response, some emails remain unanswered. in this work, we investigate why some emails don't get a response; we analyze more than 11 million emails from over 1.78 million users in the gmane dataset to identify linguistic and social determinants of receiving a reply. we model the email's intention and expectation of getting a reply with models trained on our new siz-zler dataset. we then conduct a causal analysis to assess the impact of social connection, status, and argumentation on the likelihood of receiving a response. we find email response rates are impacted, at different times and in different contexts, by these pragmatic factors. future work could investigate if the factors influencing email responses generalize across different platforms, languages, and cultural contexts. applying our framework to observe changes in email patterns over time could also reveal insights into the evolution of digital communication and social interaction norms.