REFER : Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting (2025)

Abstract: Individuals express diverse opinions, a fair summary should represent these viewpoints comprehensively.Previous research on fairness in opinion summarisation using large language models (LLMs) relied on hyperparameter tuning or providing ground truth distributional information in prompts. However, these methods face practical limitations: end-users rarely modify default model parameters, and accurate distributional information is often unavailable. Building upon cognitive science research demonstrating that frequency-based representations reduce systematic biases in human statistical reasoning by making reference classes explicit and reducing cognitive load, this study investigates whether frequency framed prompting (REFER) can similarly enhance fairness in LLM opinion summarisation. Through systematic experimentation with different prompting frameworks, we adapted techniques known to improve human reasoning to elicit more effective information processing in language models compared to abstract probabilistic representations. Our results demonstrate that REFER enhances fairness in language models when summarising opinions. This effect is particularly pronounced in larger language models and using stronger reasoning instructions.

## introduction large language models (llms) have demonstrated exceptional capabilities in general language tasks @xcite @xcite @xcite . however, their training on uncurated datasets inadvertently incorporates societal biases, leading to the perpetuation of social stereotypes in both the models and their downstream applications @xcite @xcite @xcite @xcite , influencing how users process and interpret information @xcite @xcite . this challenge has motivated research into the cognitive mechanisms of llms, such as developing human-inspired debiasing strategies @xcite . previous research has demonstrated that human decision-making often relies on cognitive heuristics rather than factual analysis. when forming judgements, individuals tend to utilise readily accessible information instead of considering empirical evidence, base rates, and statistical distributions @xcite . while these evolved shortcuts generally serve us well when solving everyday questions, they can lead to systematic biases when applied to contexts requiring precise statistical reasoning @xcite . approaches to cognitive debiasing such as direct warnings about overconfidence, have demonstrated limited to no effectiveness @xcite . in contrast, facilitating the transition from abstract reasoning to frequency-based cognitive frameworks has been shown to improve inference accuracy @xcite @xcite . this is achieved by reframing abstract probability questions into explicit frequency-based terms (e.g., "how likely is this outcome?" becomes "out of 100 cases, how many would show this outcome?"). this approach improves systematic critical reasoning by providing a concrete numerical framework for analysis @xcite @xcite . when making predictions, humans often deviate from probability theory, overlooking diverse perspectives and weighting information toward their own biases @xcite @xcite . these cognitive limitations stem from how humans process statistical information: we struggle with abstract probability representations but excel when the same information is presented in concrete frequency formats @xcite . language models exhibit oversimplification patterns and biases similar to human cognitive heuristics @xcite . similar to how humans benefit from frequency-based representations over abstract probabilities, llms trained on human-generated text may have internalised comparable representational preferences @xcite . consequently, llms often produce summaries that inadequately represent the full spectrum of opinions @xcite @xcite , exhibiting the same selective attention patterns observed in human reasoning. current debiasing approaches focus primarily on hyperparameter optimisation and explicit distribution prompts @xcite . however, these methods face practical limitations: end-users rarely modify default parameters, and accurate distributional information is often unavailable. these constraints motivate exploring frequency-based debiasing strategies, which cognitive science research shows improve human statistical reasoning and could enhance llm opinion summarisation without requiring user configuration or distributional specifications. this approach remains unexplored for llm opinion summarisation. building on this potential, we hypothesise that frequency framing enhances llm fairness through analogous computational mechanisms to those observed in humans. specifically, frequency prompts should: (1) specify explicit reference classes by directing model attention to concrete distributional information rather than abstract concepts, (2) provide numerical anchoring that reduces the computational complexity of proportion estimation during text generation, and (3) enable sequential deliberation by activating statistical reasoning patterns learnt during training on frequency-formatted data. unlike probability statements, which require normalisation and comparison operations, frequency statements provide explicit numerical anchors that should guide attention allocation during summarisation. this work explores the potential of frequency framing to elicit reasoning and summarising with frequency-based information, we refer to this framework as frequency framed prompting-refer. our key contributions are: ## frequency framed prompting (refer) research demonstrates that frequency formats facilitate more accurate statistical reasoning and reduce bias compared to abstract probabilities @xcite . framing questions to elicit frequency-based responses rather than abstract probabilities has been shown to promote critical thinking in human reasoning @xcite @xcite . this improvement occurs through three key mechanisms: (1) specifying explicit reference classes through concrete denominators and comparison groups, (2) providing numerical anchoring that simplifies the representation of uncertainty, and ## conclusion this study introduces refer (frequency framed prompting), a cognitive science-inspired debiasing framework for opinion summarisation. drawing from research showing humans make more accurate statistical judgements with concrete frequencies rather than abstract probabilities, we demonstrate that frequency-based prompting enhances fairness in llm-generated summaries. our evaluation reveals that refer improves model fairness across multiple datasets, models, and metrics, particularly when combined with stronger reasoning instructions and in larger models with superior instruction-following capabilities. notably, refer improvements often exceed differences between base prompting frameworks, suggesting frequency-based reasoning plays a greater role than specific prompting strategies in addressing fairness. refer offers a lightweight, end-user accessible solution that works with both open-source and proprietary models without requiring retraining or hyperparameter tuning, addressing critical gaps in existing computationally demanding debiasing approaches.