Validity of these indices was further explored through construct and narrative validity. The former examines how closely related indices were to relevant real-life outcomes, such as women’s labor force participation, willingness to work, and whether they have looked for work. Results show statistically significant positive relationships between liberal beliefs and expectations and all three outcomes using the revised indices, confirming that working women and their male counterparts were more likely to have more liberal perceptions about women working. Narrative analysis uses the open-ended narrative questions to perform checks using basic text analysis (e.g. if working women and men with working female counterparts were more likely to define their relationships as “not traditional”) and calculate aggregated “sentiment” scores for each response,<sup>4</sup> to understand how they correlated with the indices and a full index combining all four components. The mean aggregated sentiment scores across all the narrative responses are significantly correlated with both the full and the individual component indices, and also mean scores for a subset of narratives are correlated with relevant variables.

# _Test-Retest Reliability_

A short follow-up survey was conducted with a random subset of 1,000 of the original respondents, between 2-3 weeks after the quantitative survey to test the reliability of the social norms measures. Respondents were asked a subset of the social norms and personal belief questions across the different themes as shown in Table 4 below.

The comparison of responses across the two periods assesses the stability of social norms and personal beliefs over time. Table 4 shows the percentage of responses that were the same, the mean of the absolute difference, and the standard deviation of the difference. Items measuring personal beliefs were measured on a shorter Likert scale, and had the highest percentage matches, with an average match rate of 62%. Social empirical and normative questions were typically measured on a scale of 0-10, and had higher discrepancies between the two rounds. For example, the average absolute difference for questions on the longer scale for social empirical questions is 2.46 and for social normative questions is 2.73.

We also report the Pearson’s correlation statistic, and Cohen’s Kappa score in Table 4. The average Pearson’s correlation is 0.25, which indicates weak positive linear relationship between the variables from the two rounds (scores can vary from -1 to 1). The average for Cohen’s Kappa, an interrater agreement score, was 0.19. These additional tests confirm that the test retest reliability for these measures is weaker than expected.

We do not believe that measures of test-retest reliability for the stability of social norms measures exists in the literature, and so our study contributes the first benchmark. Given that social norms often influence choice without the need for explicit articulation, and that inquiries into normative and empirical expectations are novel for many respondents, it is not surprising that test-retest reliability for explicit social norms measures are not perfect. Although other benchmarks do not exist, we believe that these social norms measures performed reasonably well in the test-retest exercise. The average difference of 2-3 points on a 10-point scale, in measuring social norms, suggests that a person’s social expectations can move from, say, a view in which only 20% of people in Jordan agree with statement X, to one in which about half agree with X, but not to a view in which nearly all Jordanians agree with statement X.

> 4 The Vader lexicon is utilized to calculate these scores, w hich range from -1 to 1, and indicate the following: >0 is positive sentiment, 0 is neutral and <0 is a negative sentiment.

17