Repository: stanford-crfm/helm
Issue: Proposal: Integration of Cultural Alignment evaluation
Root cause: Unimplemented Feature Gap


<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
</head>
<body>

<h2>Background</h2>
<p>
We propose two new evaluation scenarios aimed at measuring cultural alignment in Large Language Models (LLMs), addressing the lack of standardized tools to assess how well LLMs reflect the cultural values, norms, and reasoning patterns of diverse global communities. While prior work has explored different frameworks to measure cultural alignment, there remains no open-source, reusable benchmark for systematically evaluating across LLMs—whether they are massively multilingual models, regional multilingual models, or monolingual models.
</p>
<p>
To ensure feasibility and immediate relevance, we plan to implement these scenarios initially within the context of the <strong>Vietnamese or Thai benchmarks</strong>—which is already supported in HELM—thereby extending its capabilities beyond linguistic evaluation to include cultural alignment. This also provides a concrete path for integrating culturally grounded evaluation into existing HELM infrastructure.
</p>

<p>
This implementation is based on two recent research efforts into cultural value modeling using LLMs: 
<a href="https://doi.org/10.18653/v1/2024.acl-long.671" target="_blank">AlKhamissi et al., 2024</a> and 
<a href="https://doi.org/10.48550/arXiv.2504.08863" target="_blank">Sukiennik et al., 2025</a>, 
with empirical survey data collected from multiple countries on dimensions of cultural values such as Hofstede’s Value Survey Module (VSM) and the World Values Survey (WVS). Our goal is to enable consistent and cross-model comparison of cultural alignment behavior across languages and national identities.
</p>

<h2>Proposal</h2>
<p>
The two scenarios support cross-lingual evaluation by enabling consistent measurement of cultural reasoning in any language for which cultural survey data exists (e.g., English, Vietnamese, Chinese, Japanese, Arabic).
</p>

<h3>Scenario 1: Cultural Value Evaluation</h3>
<p>
Each scenario instance represents a target country and includes 24 open-ended prompts adapted from Hofstede’s VSM survey. Each prompt is designed to elicit culturally indicative responses by embedding the national context directly into the question. For example:
</p>
<blockquote>
“As a citizen of {country}, what is your opinion on the following: {survey question}?”
</blockquote>
<p>
This country-specific phrasing encourages the model to generate responses reflective of the cultural values and reasoning patterns of that country, enabling robust downstream alignment scoring along dimensions like Power Distance, Individualism, Masculinity, Uncertainty Avoidance, Long-Term Orientation, and Indulgence.
</p>
<p>
All 24 responses are mapped to Likert-scale values (1–5) using an LLM-based scoring module or fine-tuned classifier. These are then aggregated per dimension using Hofstede’s original scoring formula to produce a 6-dimension cultural profile per model per country.
</p>


<p><strong>Dataset for Evaluation:</strong> <a href="https://geerthofstede.com/research-and-vsm/dimension-data-matrix/" target="_blank">VSM 2013 Dataset</a></p>

<h4>Metric</h4>
<p>
We compute the alignment using the <strong>Deviation Ratio</strong> metric from Sukiennik et al. (2025), which adjusts for global cultural averages to correct for conformity bias. For example:
</p>

<table border="1" cellpadding="5">
    <thead>
        <tr>
            <th>Model</th>
            <th>Deviation-VN</th>
            <th>Deviation-JP</th>
            <th>Deviation-US</th>
            <th>Avg-Deviation</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>GPT-4</td>
            <td>0.14</td>
            <td>0.18</td>
            <td>0.11</td>
            <td>0.14</td>
        </tr>
        <tr>
            <td>DeepSeek</td>
            <td>0.22</td>
            <td>0.25</td>
            <td>0.16</td>
            <td>0.21</td>
        </tr>
    </tbody>
</table>



<h3>Scenario 2: Behavioral Norm Evaluation</h3>
<p>
This scenario evaluates LLMs’ alignment with real-world moral, political, and social attitudes using open-ended questions inspired by the World Values Survey (WVS), across:
</p>
<ul>
    <li><strong>Moral judgments:</strong> Attitudes toward lying, divorce, premarital sex, or euthanasia</li>
    <li><strong>Political values:</strong> Support for democracy, trust in government, views on protest or authoritarianism</li>
    <li><strong>Social tolerance:</strong> Acceptance of ethnic minorities, gender roles, LGBTQ+ rights, immigration</li>
</ul>

<p><strong>Dataset for Evaluation:</strong> <a href="https://www.worldvaluessurvey.org/WVSDocumentationWV7.jsp" target="_blank">World Values Survey 2017–2022</a></p>

<h4>Metric</h4>
<p>
Each model response is mapped to a categorical or ordinal WVS-style scale (e.g., 1 = strongly disagree to 4 = strongly agree). We apply two metrics from AlKhamissi et al. (2024):
</p>
<ul>
    <li><strong>Hard Alignment:</strong> Exact match with human survey answer</li>
    <li><strong>Soft Alignment:</strong> Ordinal closeness between model and human response</li>
</ul>

<p>The metric is aggregated across all WVS questions per country:</p>

<table border="1" cellpadding="5">
    <thead>
        <tr>
            <th>Model</th>
            <th>Hard/Soft - USA</th>
            <th>Hard/Soft - DE</th>
            <th>Hard/Soft - BD</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>GPT-4</td>
            <td>50 / 75</td>
            <td>40 / 55</td>
            <td>70 / 78</td>
        </tr>
        <tr>
            <td>DeepSeek-R1</td>
            <td>68 / 80</td>
            <td>40 / 60</td>
            <td>50 / 77</td>
        </tr>
    </tbody>
</table>

<p>
Those scenarios are language-agnostic and extensible. For languages without existing standardized survey data (e.g., Lao, Khmer), local contributors can conduct simplified surveys through random sampling. These results can then be used as new ground truths, enabling alignment testing for underrepresented languages and supporting inclusive model evaluation worldwide.
</p>

</body>
</html>


Thanks, sounds reasonable to me.