Using Text-Based Causal Inference to Disentangle Factors Influencing Online Review Ratings (2025)

Abstract: Online reviews provide valuable insights into the perceived quality of facets of a product or service. While aspect-based sentiment analysis has focused on extracting these facets from reviews, there is less work understanding the impact of each aspect on overall perception. This is particularly challenging given correlations among aspects, making it difficult to isolate the effects of each. This paper introduces a methodology based on recent advances in text-based causal analysis, specifically CausalBERT, to disentangle the effect of each factor on overall review ratings. We enhance CausalBERT with three key improvements: temperature scaling for better calibrated treatment assignment estimates; hyperparameter optimization to reduce confound overadjustment; and interpretability methods to characterize discovered confounds. In this work, we treat the textual mentions in reviews as proxies for real-world attributes. We validate our approach on real and semi-synthetic data from over 600K reviews of U.S. K-12 schools. We find that the proposed enhancements result in more reliable estimates, and that perception of school administration and performance on benchmarks are significant drivers of overall school ratings.

## introduction understanding the influence of specific aspects mentioned in text reviews on the overall ratings of products or services is a complex yet important endeavor in many industries. for example, in education, how does feedback on academic performance or facility quality influence a school's overall ratings? precisely quantifying this influence can help businesses identify key areas for improvement. traditional approaches to this problem include aspect-based sentiment analysis, which extracts sentiments tied to predefined aspects @xcite , and exploratory analyses that measure the correlation between certain terms and overall rating @xcite . however, these methods generally fail to account for confounding variables, leading to biased results. for example, consider a school often praised for its academic performance, with reviews about its excellent programs and great teachers. traditional analysis might directly link these positive attributes to the school's high ratings. however, if these reviews also frequently mention extensive extracurricular opportunities or high parental involvement, it could imply that the school's high ratings reflect its socioeconomic advantages, not educational quality alone. overlooking such factors could lead to incorrect assessments of the true influence of educational quality on overall perceptions. our goal is to estimate the impact of aspects mentioned in reviews on overall ratings. to achieve this, we have developed a causal inference framework to control for confounding variables within text. our framework starts by identifying an aspect of interest from the text related to an entity, e.g., if the school reviews frequently praise facility quality. we then utilize the remaining text of the entity, excluding the aspect-related text, as covariates to analyze the overall rating for the entity. here, the textual content is treated as a proxy for real-world factors that may influence an entity's rating. this approach helps us control for confounders associated with the entity and isolate the treatment effect of the specific aspect on the overall rating. contribution we apply causalbert @xcite to estimate the effects of specific topics on overall review ratings, isolating genuine influences from other textual elements. we enhance causalbert by (i) integrating temperature scaling to calibrate propensity scores; (ii) optimizing a key hyperparameter that balances treatment and outcome prediction, reducing overadjustment for confounds; (iii) employing interpretability methods to characterize discovered confounds. we validate our approach using 600k u.s. k-12 school reviews from greatschools.org, finding that issues of administration personnel and academic performance are significant drivers of perceived school quality. ## methods we apply the potential outcomes framework @xcite , observing for each subject (school) i a tuple (x i , y i , t i ), where x @xmath0, y @xmath1, and t @xmath2, 1} is the treatment assignment (presence of topic in reviews). the potential outcomes y i (0) and y i (1) represent the outcomes under control (no treatment) and treatment scenarios, respectively. the outcome y i is defined as the goal is to estimate the average treatment effect (ate), τ , which quantifies the expected difference in outcomes due to the treatment: to estimate the ate, we consider several core assumptions and estimators: ignorability assumes that the treatment assignment t i is independent of the potential outcomes, a critical condition that allows the use of a naive unbiased estimator (τ unadjust ) directly: however, this assumption is often unrealistic when treatment assignment correlates with confounds. thus, we further assume conditional ignorability, which posits that the treatment assignment t i is independent of the potential outcomes given the covariates x i . if we denote e[y i |x @xmath3, t @xmath4, x) and e[y i |x @xmath5, t @xmath6, x), then the ate can be estimated by: here, q(t i , x i ) is the estimated response given treatment status and covariates. positivity assumes that every subject has non-zero probability of receiving the treatment (0 < p (t @xmath7. with the propensity score g(x) = p (@xmath8, let ĝ(x) denote the estimate of the true propensity score g(x). then the inverse probability weighting (ipw) estimator is: to mitigate the instability in ipw estimates due to extreme propensity scores, we also use augmented inverse propensity weighted (aipw) estimator @xcite : ## experiments we empirically examine the capabilities of causal-bert and the proposed enhancements, using both real and semi-synthetic review data, with a focus on the following questions. rq1: how does causal-bert performance vary with confounder strength? rq2: what effect does temperature scaling have on treatment effect estimation? rq3: how does hyperparameter α in the loss function influence overadjustment and how does its optimal value relate to treatment prediction accuracy? rq4: how effective are interpretability methods at surfacing the confounders discovered by causalbert? rq5: to what extent do educational aspects, such as 'bullying' and 'administration,' impact overall school ratings in real-world data? dataset we analyze 677,210 reviews from greatschools.org foot_0 , covering 83,795 public, private, and charter schools in the united states between 2002-2019. we investigate the impact of school-related topics such as 'bullying,' 'academic performance,' 'administration,' 'extracurricular activities,' and 'curriculum,' each defined by a keyword list established by prior work @xcite . for each topic, we first separate reviews into those that discuss the topic and those that do not. for outcome, we normalize review ratings by computing state-specific z-scores and average these scores for each school over the selected period. thus, outcome values are in standard units. for treatment assignment, we adopt different methods based on the nature of the topics discussed in the reviews. for neutral topics such as 'administration,' treatment is determined by sentiment about that topic. in this case, an entity is considered treated (@xmath9. schools with a mix of positive and negative reviews on administration are excluded from the analysis to maintain a clear treatment distinction. for the negative topic of 'bullying,' treatment is 1 if the topic is mentioned, 0 otherwise. in our framework, the 'bullying' task aims to assess how the presence of bullying affects the school's overall ratings. for other topics like 'administration,' we treat the sentiment of text referencing administration as a proxy for real-world administrative quality and measure how that factor's sentiment influences the school's overall rating. our causal pathway assumes multiple school attributes can affect overall ratings, so we isolate the effect of a specific aspect by controlling for other factors. for instance, to examine bullying, we separate it from other negative conditions-like poor administration-that might also lower ratings. we have between 3,900 and 13,300 schools for each topic; more detailed statistics (e.g., average total reviews per school) are provided in §a.1. semi-synthetic data setup first, we employ a semi-synthetic evaluation framework @xcite to evaluate causalbert's treatment effect estimation capabilities using the bullying topic. we simulate a binary confound c @xmath10, 2} by inserting text related to an academic challenges topic into certain reviews (see §a.2 for terms). schools in class 1 receive these injected sentences, while those in class 2 do not. to manipulate the ate, we vary the true ate by defining two outcome models based on treatment status: for class 1, outcomes are modeled as y ∼ n (u 2 , 0.3) when treatment @xmath11, 0.3) when @xmath12. class 2 maintains uniform effects with y ∼ n (u 2 , 0.3), indicating no treatment effect from textual confounding. here, u 2 is set at -0.3, and u 1 varies within {0.3, 0.4, 0.5}, creating corresponding true ate u values of {-0.3, -0.35, -0.4}. the confounder strength is controlled by adjusting the probability p, which defines the treatment assignment probabilities within each class. we vary p from 0.9 to 0.5, where for class 1: p (@xmath13, and inversely for class 2: p (@xmath14. in our semi-synthetic experiments, we construct data according to specific values for u and p by sampling schools to satisfy these constraints. thus, the only synthetic part of these experiments is the sampling procedure and the injection of confounding sentences. each experiment samples 5,000 instances from a population of 13,361. evaluation metrics we perform 5-fold crossvalidation for the semi-synthetic data experiments and bootstrap aggregation for real-world data analysis ( §a.7). estimators τq , τip w , and τaip w , and their calibrated versions, are used to estimate the ate, with the naive estimator τunadjust as a baseline. since we have different true causal treatment effect designs, we use the error ratio to evaluate the estimation results, defined as |τ estτ true |/τ true , where τest is the estimated treatment effect and τ true is the true treatment effect. we also report the accuracy of treatment prediction and the mean squared error (mse) of outcome prediction. ## related work aspect-based sentiment analysis analyzes sentiment toward specific topics @xcite @xcite , yet generally does not address the causal impact of these aspects on overall ratings, potentially conflating correlation with causation. there is growing research in causal inference with text @xcite @xcite . a common approach is to use nlp to identify confounds and adjust for them in estimation @xcite @xcite . other methods method adjust for confounders from text with supervised nlp. @xcite introduced causalbert, leveraging pre-trained bert models @xcite to derive "sufficient" embeddings that capture confounding properties within texts, optimizing predictions for both treatment and counterfactual outcomes. while they highlighted the potential for deep learning methods to improve estimation accuracy, they also outlined several future challenges: refining deep learning approaches to enhance estimation accuracy; developing visualization and sensitivity analysis tools to clarify the "black box" nature of embeddings; and expanding semi-synthetic simulations into a comprehensive benchmarking strategy. these challenges have inspired the work in this paper. ## conclusions in this study, we have extended causalbert to understand how factors mentioned in school reviews affect overall ratings. through semi-synthetic experiments, we verified the effectiveness of causal-bert in a controlled setting, which then guided our application to real-world data. our analysis indicates that temperature scaling and integrated gradients can refine causal estimates and enhance interpretability. analysis of u.s. k-12 school reviews found that educational aspects like 'administration' and 'academic performance' have significant influence on school ratings.