DISCRIMINATION AGAINST SEXUAL MINORITIES IN EDUCATION AND HOUSING: EVIDENCE FROM TWO FIELD EXPERIMENTS IN SERBIA

# **Rater reliability**

The agreement of the evaluations of the outcomes among the three raters was found to be quite satisfactory. The inter‐rater reliability expressed by the intraclass correlation coefficient (ICC)<sup>3</sup> was as high as 0.91.<sup>4</sup> Since the inter‐rater reliability was high, the average ratings of the three raters were used as the final measures of the outcomes.

i Lee Badgett, Laura Durso, and Alyssa Schneebaum, “New Patterns of Poverty in the Lesbian, Gay, and Bisexual ‐ ‐ ‐ ‐ Community,” Williams Institute, 2013, https://williamsinstitute.law.ucla.edu/wp <u>content/uploads/LGB Poverty Update Jun‐2013.pdf.</u> Dominik Koehler, “LGBTI People Are (Likely) Over Represented in the Bottom 40%,” World Bank Group SOGI Task Force, 2015, http://blogs.worldbank.org/governance/lgbti‐people‐are‐likely‐over‐represented‐bottom‐40. ii Fundamental Rights Agency, “EU LGBT Survey Technical Report,” European Union Agency for Fundamental Rights, Vienna, 2012, <u>http://fra.europa.eu/sites/default/files/eu‐lgbt‐survey‐technical‐report_en.pdf.</u> iii Philip Oreopoulos, ‘‘Why Do Skilled Immigrants Struggle in the Labor Market? A Field Experiment With Thirteen Thousand Resumes,” _American Economic Journal: Economic Policy_ 3, no. 4 (2011): 148– 171, <u>https://www.aeaweb.org/articles?id=10.1257/pol.3.4.148.</u> iv Katrin Auspurg, Thomas Hinz, and Laura Schmid, “Contexts and Conditions of Ethnic Discrimination: Evidence From a Field Experiment in a German Housing Market,” _Journal of Housing Economics_ 35 (2017): 26‐36, <u>https://www.sciencedirect.com/science/article/pii/S1051137717300037.</u> v Devah Pager and Hana Shepherd, “The Sociology of Discrimination: Racial Discrimination in Employment, Housing, Credit, and Consumer Markets,” _Annual Review of Sociology_ 34 (2008): 181‐209, <u>http://www.annualreviews.org/doi/abs/10.1146/annurev.soc.33.040406.131740.</u> vi András Tilcsik, “Pride and Prejudice: Employment Discrimination against Openly Gay Men in the United States,” _American Journal of Sociology_ 117, no. 2 (2011): 586‐626, https://www.ncbi.nlm.nih.gov/pubmed/22268247. vii United Nations Development Programme (UNDP), “Being LGBTI in Eastern Europe: Serbia Country Report,” UNDP, 2017, <u>http://www.eurasia.undp.org/content/dam/rbec/docs/undp‐rbec‐blee‐SERBIA%20.pdf.</u> viii ‐ European Commission, “Serbia 2016 Report,” Brussels, 2016, 3, https://ec.europa.eu/neighbourhood <u>enlargement/sites/near/files/pdf/key_documents/2016/20161109_report_serbia.pdf.</u> ix UNDP, “Being LGBTI in Eastern Europe.” x Cisgender refers to people whose identity and gender corresponds with their birth sex, i.e., who are not transgender. xi Martijn Hesselink and Ton van der Wiele, “Mystery Shopping: In‐Depth Measurement of Customer Satisfaction,” 2011, <u>http://www.mvmgroup.rs/wp‐content/uploads/2011/04/Importance‐of‐mistery‐shopping.pdf.</u> xii For example, in the SILC survey, (World Bank forthcoming), only 17 percent of LGBTI respondents stated that they were open about their sexual orientation at work in the last five years.

xiii For example, qualitative survey focus groups in Belgrade, Nis, and Novi Sad showed that in the acronym LGBT, most of the participants were familiar only with the meaning of the letters L (lesbian) and G (gay). The result was confirmed in the quantitative survey, showing that majority of the people in Serbia relate LGBT population to gay, lesbian, and bisexual people, while the awareness of transgender and intersex people is quite low. National Democratic Institute (NDI), “LGBTI Public Opinion Poll: Western Balkans,” 2015, https://www.ndi.org/LGBTI_Balkans_poll.

xiv UNESCO, “From Insult to Inclusion: Asia‐Pacific Report on Bullying, Violence and Discrimination on the Basis of Sexual Orientation and Gender Identity,” Bangkok, 2015, http://unesdoc.unesco.org/images/0023/002354/235414e.pdf. xv Mahidol University, Plan International Thailand, and UNESCO Bangkok Office, “Bullying Targeting Secondary School Students Who Are or Are Perceived to Be Transgender or Same‐Sex Attracted: Types, Prevalence, Impact, Motivation and Preventive Measures in 5 Provinces of Thailand,” Bangkok, 2014, <u>http://unesdoc.unesco.org/images/0022/002275/227518e.pdf.</u> 3 Model used for assessing the rater reliability: Shrout, P.E and Fleiss, J.L. Intraclass Correlations: Uses in assessing rater reliability, _Psychological Bulletin_ , 1979, Vol. 86, No. 2, 420‐428.

4 ANOVA results based on which the ICC was calculated

|**Sources of variance**|**df**|**MS**|
|---|---|---|
|Between raters|2|0.35|
|Between<br>landlords|319|1.70|
|(observations)|||
|Residual+error|638|0.04|

24