metadata: authors: 0: author: Himelein,Kristen
count: Timor-Leste
docna: 0: docna: Interviewer effects in subjective survey
            questions: evidence from Timor-Leste
docty: Policy Research Working Paper
subtopic: Settlement of Investment Disputes,Agricultural Economics,State Owned Enterprise Reform,Public Sector Administrative & Civil
            Service Reform,Educational Sciences,Gender and Development,Democratic Government,Economics and Finance of Public
            Institution Development,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Science and Technology Development,Social Protections and Labor,Communities and Human Settlements,Education,Gender
abstracts: cdata!: There is an inherent tension between
            traditional norms and survey protocols for quantitative data
            collected in the developing world. Unexpected interactions
            between the interviewer and respondent can lead to
            interviewer effects in the data, particularly in the case of
            subjective or sensitive questions. This paper makes use of a
            unique data set available from Timor-Leste containing
            subjective and objective questions to study these effects.
            In addition to their age and gender, data were collected
            from the interviewers regarding their opinions on the
            subjective questions prior to fieldwork. Fixed effects and
            mixed effects logit models are used to examine the main
            effects and interactions between interviewer and respondent
            characteristics. More objective measures serve as a pseudo
            control group. The paper finds interviewer effects in the
            both subjective and objective data, but the magnitude is
            considerably stronger for subjective questions. The paper
            also finds that female respondents are more susceptible to
            influence based on the interviewer's beliefs. Despite
            methodological shortcomings, the study highlights the need
            to consider more fully the impact of traditional cultural
            norms when conducting quantitative surveys in the developing
            world on topics that are outside the standard objective questions.
geo_reg_key: 517198,517191
repnme: repnme: Interviewer effects in subjective survey
            questions: evidence from Timor-Leste
display_title: Interviewer effects in subjective
            survey questions: evidence from Timor-Leste
listing_relative_url: /research/2015/03/24112749/interviewer-effects-subjective-survey-questions-evidence-timor-leste
topicv3: State Owned Enterprises m1326323 721,Cadastre and Land Registration m1326617 890,Education m1327048 470,Dispute Resolution m1327498 843,Gender m1327621 615
publishtoextweb_dt: 2015-03-09T00:00:00Z
historic_topic: Science and Technology Development,Social Protections and Labor,Communities and Human Settlements,Education,Gender
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 26
model_extractions: WPS7208 Policy Research Working Paper 7208 Interviewer Effects in Subjective Survey Questions Evidence from Timor-Leste _Kristen Himelein_ Poverty Global Practice Group March 2015
Policy Research Working Paper 7208

# **Abstract**

There is an inherent tension between traditional norms and survey protocols for quantitative data collected in the developing world. Unexpected interactions between the interviewer and respondent can lead to interviewer effects in the data, particularly in the case of subjective or sensitive questions. This paper makes use of a unique data set available from Timor-Leste containing subjective and objective questions to study these effects. In addition to their age and gender, data were collected from the interviewers regarding their opinions on the subjective questions prior to fieldwork. Fixed effects and mixed effects logit models are used to examine the main effects and interactions between interviewer and respondent characteristics. More objective measures serve as a pseudo control group. The paper finds interviewer effects in the both subjective and objective data, but the magnitude is considerably stronger for subjective questions. The paper also finds that female respondents are more susceptible to influence based on the interviewer’s beliefs. Despite methodological shortcomings, the study highlights the need to consider more fully the impact of traditional cultural norms when conducting quantitative surveys in the developing world on topics that are outside the standard objective questions.

This paper is a product of the Poverty Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at khimelein@ worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Interviewer Effects in Subjective Survey Questions: Evidence from Timor-Leste

# Kristen Himelein<sup>1</sup>

## _World Bank_

_1818 H Street NW, Washington DC, 20433, USA_

**Key words:** survey methodology, interviewer effects, subjective data, justice surveys, Timor-Leste

> 1 Kristen Himelein is a Senior Economist / Statistician in the World Bank’s Poverty Global Practice, and can be reached at khimelein@worldbank.org.  The author would like to thank Kinnon Scott, Steve Heeringa, Brady West, Melissa Plegue, the participants in the 2013 European Survey Research Association conference in Ljubljana, and two anonymous referees for very useful comments.  This analysis was supported by Research Support Budget program in the Development Economics Research Group of the World Bank.  All views are those of the author and do not reflect the views of the World Bank or its member countries.
# **1. Introduction**

Collecting data through quantitative interviews can be an awkward social interaction. An inherent tension is created between the societal norms of conversation and the survey protocols needed to standardize information across a range of interviewers and respondents (Kahn & Cannell, 1957, Suchman & Jordon, 1980). Individual interviewers may have differing levels of success in navigating this interaction, creating interviewer effects in the data. The further the interview is from normal conversation, such as discussing sensitive topics with a stranger, the more likely it is to find non-trivial impacts from the interviewer (Schnell & Kreuter, 2005).

The tension between culture and protocol is often amplified in developing country settings. Traditional societies often have strict cultural codes of behavior. Interviewing is also generally conducted face-to-face, and therefore more reminiscent of a social setting than a telephone or e- mail survey. In Timor-Leste, traditional society is particularly hierarchical, with a history of exclusion of women, minorities, and the youth. In addition data collection for this study took place in 2008, following the end of decades of conflict. While political violence and other organized aggression was rare at this time, simmering ethnic and regional tensions remained. The questions in this study focus on the respondent’s perceptions of corruption, community dispute resolution, women’s rights, and land claims, all potentially contentious topics.

Two main methodological questions are addressed in this paper. First, are interviewer effects present in the survey data, and, if so, is the magnitude greater for the sensitive subjective questions compared to more objective questions? Secondly, what, if any conclusions can be drawn regarding the nature of these interviewer effects? The main hypothesis is that if interviewer effects were present, respondents would likely show a form of social desirability bias (Crowne and Marlowe, 1960), deferring to the traditional norms of hospitality to give the answer they perceive the interviewer “guest” would like to hear. The respondent’s expectations are based on the interviewer’s observable traits and possible additional cues given by the interviewer based on their opinions. The expectation is that this would lead to pro-state responses to young or female interviewers and more conservative responses to older male interviewers.

The analysis makes use of a unique data set available from Timor-Leste which specifically examines the characteristics of the respondent, the characteristics of the interviewer, and the interviewer’s personal opinions on the survey questions. It also briefly addresses the question of salience with a more detailed analysis of gender and the women’s rights question. The analysis uses fixed effects logit and multi-level mixed-effects logistic regression modeling to test for the interviewer effects. Once the presence of these effects has been identified, I attempt to estimate the amount of total variation they are capable of explaining, and to estimate the explanatory power of the three available interviewer characteristics, age, gender, and the interviewer’s opinion, as well as the associated interaction terms.

The next section briefly reviews existing literature on interviewer effects as it relates to the research questions. Section 3 discusses the data, results are presented in section 4, and section 5 offers a brief conclusion and possible areas of future research.

2
# **2. Literature Review**

The question of the interviewers’ effects on the respondent’s answers has been addressed extensively in the social science literature, and the findings vary widely depending on the variable under discussion and the context of the survey. A common mechanism by which interviewers can introduce bias is through an action on the part of the interviewer that deviates from survey protocols, either intentional or unintentional, verbal or non-verbal. Though training is designed to prevent this behavior, interviewers often deviate (Billiet & Loosevelt, 1988; Fowler & Mangione, 1990). In particular, Cannell, Miller & Oksenberg (1981) note a tendency to elaborate and paraphrase difficult questions. The questions in the Timor-Leste study would be particularly vulnerable to this as they use an unfamiliar forced-choice format and address slightly abstract topics. These questions were also both subjective and sensitive, which the literature has found to be more vulnerable to interviewer effects, though there are some exceptions. See Cotter, Cohen & Coulter (1982), Davis (1997), Davis, Couper, Janz, Cadwell & Resnicow (2010), Groves (1989), Samples et al (2014), Schaeffer (1980), and Schnell & Kreuter (2005) for further discussion.

In addition to explicit actions, it is also possible to introduce bias even while following protocols if the respondent alters their answer based on the physical characteristics of the interviewer, such as age, race, and gender. While findings again vary with context, most literature find significant impacts, particularly with race and gender. See Collins & Butler (1983), Davis (1997), Ellison, McFarland & Krause (2011), Finkel, Geterbock & Borg (1991), Freeman & Butler (1976), Galla, Frisone, Jeffrey & Gaer (1981), Hatchett & Schuman (1975), Krysan & Cooper (2003), Rhodes (1994), Singer, Frankel & Glassman (1983) for examples. In addition, these effects may be amplified for sensitive issues, such as gender and sexual practices (Tu & Liao, 2007; Johnson & Moore, 1993; Lueptow, Moser & Pendleton, 1990; Johnson & DeLamater, 1976; Galla, et al, 1981). The results from the literature are more mixed with regard to interaction effects. For example, while Huddy et al (1997) found an average increase of five percentage points on a dichotomous question for female-to-female interviews, and similar findings are recorded by Lueptow et al (1990) and Landis, Sullivan & Sheley (1973), mixed results are found by Ballou (1990), Johnson & DeLamater (1976) and Catania et al (1996). Related to age, Wallace, Kohout & Colsher (1992) found that older respondents were more comfortable with older interviewers.

The literature is more limited on the impact of the interviewer’s beliefs on the respondent’s answers. In one of the earliest examples of interviewer effects in the literature, Rice (1929) found that an interviewer’s personal politics, in this case socialism versus prohibition, led to wide differences the main cause of destitution cited by respondents. More recently Hermann (1983, discussed in Schnell & Kreuter, 2005) found “interviewer effects were small if the item was unimportant to the interviewer and that they were large if the item was unimportant to the respondent.” There are fewer examples of studies conducted outside of the United States. Phung, Hardeweg, Praneetvatakul & Waibel (2013) found significantly higher errors based on interviewer 3
gender, ethnicity, age, and education in a study of Thailand and Vietnam, and concluded based on interaction effects that matching on respondent and interviewer characteristics improved data quality. Liu & Stainbeck (2013) found inconsistent gender of interviewer effects in a survey on marriage conducted in China. Blaydes & Gillum (2013) examined the impact of a veiled enumerator on questions of religiosity in Egypt, and found Muslim women indicated they were more religious to a veiled enumerator while Christians indicated less. These effects were the largest for younger, poorest and less educated women. Benstead (2010) conducted a similar survey in Morocco that included both male and female respondents and interviewers, finding that respondents gave more socially progressive responses to female interviewers, though less so for veiled female interviewers. She also finds that women respondents were more likely to give socially progressive responses to female interviewers. Flores-Macias & Lawson (2008) find that respondents in Mexico City gave more socially progressive answers when questioned by a woman, but found no differences in the remaining districts of the country. Meng-Li & Yu (2008) looked at the effect education differentials on biotechnology surveys in Taiwan, China, and found no significant first order effects on age, education or interviewer experience, but did find significance on certain second order interactions related to education.

There are other aspects of the interviewer effects literature that are not relevant to this discussion. For example much of the recent literature has focused on the impact of interviewercorrelated non-response on bias and the precision of estimates (see West & Olson, 2010 for a recent example). In the case of the Timor-Leste survey, agreement to participate was secured jointly by the team supervisor and village officials at the cluster level, and therefore it would not be possible to separate the supervisor and geography effects for unit non-response. Unit nonresponse is also less than five percent. For item non-response, all respondents answered all five questions considered here, though some indicated “both” or “neither” of the two choices offered. (See Tables 2 and 3 in the appendix for question-level response rates.) These options were included to decrease the probability that the interviewer would rephrase the question in order to elicit an answer from a potentially confused respondent, and to not overly pressure a respondent to answer a potentially sensitive item. This analysis is not considered in this paper but may be for future work. Other areas of the literature which are not applicable here include learning, as the survey is a cross section (see Bollinger & David, 2005; Duan et al, 2007; and Olson & Peytchev, 2007), and incentive payments as interviewers are paid a flat salary for participation (Kennickell, 2004; Rosen et al, 2010).

# **3. Data**

In 2007 the Timor-Leste Survey of Living Standards (TLSLS2) collected information on household demographics, consumption based welfare measures, and a range of other socioeconomic characteristics. The following year, the World Bank conducted an extension survey (TLSLS-2X) to collect additional information on agriculture, shocks and vulnerability, 4
financial services, and access to justice. The extension survey, which is the main data set in this study, revisited a representative subsample of 1,427 households from the original TLSLS2, though was not a panel survey as different questions were asked during the revisit.

Fieldwork was conducted between April and October 2008 by seven mobile field teams of three interviewers and one supervisor. Interviewer teams were not constructed based on specific criteria, though efforts were made for each team to contain at least one woman. Teams were then assigned to constructed geographic areas of the country called zones. (Zones, generally though not always, corresponded to groups of administrative districts.) Within teams, interviewers were randomly assigned to households.<sup>2</sup> Supervision missions were taken to monitor the quality of fieldwork. A subsequent review of data quality led to the exclusion of data from one field team, and the re-sampling and re-interview of this zone by the remaining six teams. Additionally, during the course of the survey, there were replacements made for three interviewers – for which interviewer-level information was not available. These observations are therefore excluded, leaving a total of 19 interviewers available for analysis.

The original TLSLS2 sample was a two-stage probability sample, with selection of PSUs proportional to size in the first stage and simple random sampling of households in the second stage. The extension sample was selected as a nationally representative simple random sample of PSUs. In selected PSUs, 12 of the 15 original households were randomly selected to be revisited, with the final three households being held as replacements. Overall, the survey had a seven percent replacement rate. The respondent for the justice module was randomly selected to be either the head of household or spouse of the household head to ensure an equal gender distribution of men and women in the sample. Sampling weights were calculated as the inverse probability of overall selection, with a post-estimation correction to the stratum population totals.

The access to justice module was specifically designed to capture indicators that are traditionally difficult to measure quantitatively. The questions used a similar structure to subjective questions found in the Afrobarometer and Latinobarometer surveys, which offer respondents two neutral statements and ask which is closer to the respondent’s opinion. The full text of the questions is available in the appendix but generally the first question relates to corruption (Corruption), the second to whether the main responsibility for law and order lies with the police or the community (Law & Order), the third to a woman’s role in traditional society

> 2 This was in accordance with the field directions to even distribute workload.  Since households in Timor-Leste can be quite distant and sometimes only accessible by foot, randomly assigning the interviewers within a PSU is necessary for an even workload.  The assumption was also tested, using t-tests, and taking into account the complex sample design when calculating the standard errors.  There are no statistically significant differences in the average age, gender, or years of education of the household head, or the average household consumption, between male and female interviewers, or between interviewers older or younger than the median interviewer age of 33.

5
(Women’s Rights), the fourth to the resolution of non-violent disputes (Dispute Resolution), and finally to land titling and conflict in communities (Land Titling).<sup>3</sup> Fieldworkers for the project were experienced enumerators, most of whom had participated in previous data collection efforts on behalf of the statistics bureau. They underwent a rigorous weeklong training, including numerous field sessions. After the training but prior to the start of fieldwork, interviewers and supervisors were asked to complete the perceptions of the law section based on their own beliefs. Interviewers were not told they were participating in a methodological study; rather that it was an evaluation exercise as part of the training.

In general, the characteristics of the interviewers differed from those in the general population. Interviewers were younger and more likely to be male than respondents, and more highly educated as interviewers were required to have a minimum of completed secondary education, while only 12 percent of respondents met this criterion. (See Table 1 for further details.) The interviewer’s responses to the subjective questions were generally consistent with those given by respondents with similar characteristics. Limiting the respondent sample to urban residents with secondary education, and controlling for age and gender, there are statistically significant differences between interviewers and respondents only for question 4, Dispute Resolution. This is likely related to the fact that all of the interviewers live in the capital, Dili, were residents were in general less likely to indicate a preference for community over formal dispute resolution systems, though the limited sample size precludes a more formal test. Table 2 shows the comparison between the interviewer’s responses and those of the overall population.

As a pseudo comparison group for the subjective questions, I also test four more objective dichotomous variables also collected in the extension survey: whether or not the respondent attended a community meeting in the last 12 months, if the household had experienced problems with Siam Weed (an invasive species common to rural Timor-Leste), if the household’s farm had stone terraces (a wide spread agricultural practice), and whether the household had experienced a drought in the last two years. The full text of these questions is listed in the appendix. Table 3 shows the distribution and non-response rates for the objective variables.

# **4. Results**

_4.1 Fixed Effects Model_

> 3 To simplify the presentation of results, the five questions are given names linked to the general issue to which they relate.  These names are meant as a shorthand description only.  The actual phrasing of the question was designed to be neutral, and the full text is available in the appendix.

6
The first step in the analysis was to identify if interviewer effects were present in the data, and, if so, to what degree. Table 4 shows an excerpt of the results for a fixed effects logit model for the five subjective measures described above, as well as four more objective measures. The dependent variable for the subjective measures is whether the respondent said A, or that the first statement was closed to their beliefs. For the objective questions, the dependent variable is a ‘yes’ response. In addition to the interviewer fixed effects, the full model includes the gender, age, education level of the respondent, urban/rural residence, the log per capita expenditure of the household, and the district of residence. The interviewers are identified by their gender and age. Both sets of models take into account the complex design features, stratification and clustering of the survey.

The results indicate strong jointly significant for interviewer effects in all nine models. Though the interviewer effects are significant in both the subjective and objective models, they explain a larger percentage of the total variation in the subjective models. Table 5 shows the Nagelkerke adjusted R<sup>2</sup> values for identical logit models with and without the interviewer fixed effects.<sup>4</sup> The percentage of the variation explained by the interviewer fixed effects ranges between 41.0 percent and 46.1 percent for the five subjective questions, and between 11.8 percent and 25.7 percent in the four objective models. These percentages, however, likely overstate the percentage of the variation attributable solely to interviewer effects. Because of the field team structure and grouped geographic assignment of the teams, it is not possible to fully separate the interviewer and geographic variation, even with the inclusion of district fixed effects in the model. These impacts could be substantial as post-conflict Timor-Leste was characterized by high levels of fractionalization and regional differences. This issue though would also occur for the objective measures, and therefore the comparisons are likely robust. These findings support many previous studies in the literature on interviewer effects in sensitive and subjective questions.

# _4.2 Mixed Effects Model_

Having established the presence of interviewer effects, a mixed effect logit is used to allow for the inclusion of interviewer-level covariates in addition to interviewer random effects. For this analysis, it is not possible to include the weights nor adjust the standard errors for clustering or stratification when implemented using Stata software for statistical analysis. While the design is approximately self-weighting within the stratum, the exclusion of the clustering would underestimate standard errors.

The specification for the model is, for each of the five questions:

> 4 Note that these models do not use the complex design features of the data because the ratios of likelihoods with complex sample design are not statistically valid.

7
The dependent variable Yj is a binary variable for a given question response (‘A’ in this case) for respondent _j_ , α is a constant term, _X_ is a vector of relevant household and individual characteristics for respondent _j_ ,⁡W is a set of characteristics for interviewer _i, I_ is the interviewer’s personal response to an individual question, μj is the random effect associated with the intercept for interviewer _i_ with a distribution of 2 where σi represents the variance of the random interviewer effects. The residual error for respondent _j_ with interviewer _i_ for each question is represented by εij with a distribution of the error term of The variables included in vector _X_ are the respondent’s gender, age, and level of education, and the household’s log per capita household expenditure, urban/rural residence status, and district of residence. The interviewer level characteristics include age and gender only, as the minimum level of education does not vary across interviewers. Results of the mixed effects model are presented in Table 6.<sup>5</sup> While the individual respondent-level predictors for the question response are not the main focus of this analysis, but it is important to note, as demonstrated in Table 5, some are significant and collectively they explain the majority of the variation for a given question. For the Corruption question, urban residents are less likely to sanction ‘additional payments’ to public officials, while women and wealthier respondents are more likely to condone such payments. There are no statistically significant effects on the respondent’s age or education. From the coefficients on the education variables there seemed to be a trend in the variables of higher educated people be more likely find additional payments permissible, but even tested jointly the results are not significant. In the Law & Order question, wealthier respondents are more likely to indicate that the main responsibility for maintaining law and order within the community lies with the police, as opposed to traditional leaders and the community itself. Results from the Women’s Rights question indicate, unsurprisingly, that women are more likely to believe that a woman should be allowed to speak for herself in the traditional _adat_ dispute resolution proceedings. There are no statistically significant effects with regard to age, education, urban/rural residence status, or household

> 5 Due to the nature of the data, multicollinearity may also be an issue.  Multicollinearity results when there is a correlation between independent variables in regression analysis.  In the case of severe mutlicollinearity, the variance is overestimated, thereby causing the false acceptance of the null hypothesis.  In addition, there can be incorrect signs on the coefficients for explanatory variables.  To test for multicolllinearity, the models including the interviewer’s gender, age, and opinion on the subjective questions were re-run as linear regressions following which variance inflation factors (VIF) values were calculated.  The VIF were all below the threshold of concern of five, with nearly all being also below three.

8
consumption. For the Dispute Resolution question and the Land Titling question, with the exception of some district dummy variables, none of the individual characteristic explanatory variables are statistically significant. Across the five subjective models, the geographical indicator variables for Timor-Leste’s 13 districts are in many cases significant.

The results of this model also support the finding of interview effects from the fixed effects logit model. In the mixed effects model, the random intercepts are significant for all five subjective models, indicating that the individual interviewers do impact the outcome. This is more general than fixed effects since the fixed effects model calculates only the presence of interview effects for the specific set of interviewers tested, while the mixed effects model assumes that there is a universe of possible interviews, of which a random sample is selected for participation in the survey.

To examine potential outlier effects of individual interviewers, I calculate the empirical best linear unbiased predictor for each interviewer on each of the five subjective questions and test for outlier values. Of the 93 interviewer-question combinations that were estimated by the models, five are significant at the ten percent or greater threshold of significance, and two at the five percent threshold. Those which are significant are not centered on a particular interviewer or question, and are generally consistent with what would be expected to find in terms of frequency of outliers in a normal distribution. Following Wilk and Gnanadesikan (1968), graphs comparing quantiles of empirical best linear unbiased predictors against quantiles of normal distribution are shown in Figure 1 in the appendix. These graphs show the effects generally fall on or near the 45 degree line, though there are single outlier values for Corruption and Dispute Resolution.

There are limited significant results for the first order effects on age, gender and the interviewer’s beliefs, even though the exclusion of the weights and complex design features lead to the underestimation of standard errors. Respondents were more likely to tell a female respondent that the main responsibility for maintaining law and order in the community was with the police instead of the traditional _chefe de suco_ . This is consistent with respondents giving more pro-state answers to historically disenfranchised groups. If this hypothesis were true, however, one would expect positive and significant results on the Women’s Rights and Land Titling questions, and negative and significant results on Corruption and Dispute Resolution. None of the other questions show significance, and the signs on Corruption and Land Titling move in the opposite direction. With regard to age, assuming the youth to be more progressive, the expected results would be the same as above for female interviewers. All but the Women’s Rights question show the expected sign, but only Land Titling has a significant result. For older interviewers, which assumedly would be perceived as more conservative, the expectation on the signs would be in the opposite direction. Three of the five questions have the expected sign, but only one, Law & Order, has a significant result. Overall, the first order results show general support for the hypothesis, but the results are far from conclusive.

In all five questions, the respondent is more likely to agree with choice ‘A’ if the interviewer also selected ‘A’ though this difference is weakly significant in two cases (at the 10% level), and 9
not significant in the remaining three cases. The coefficients on the two weakly significant cases indicate substantial effects however. Respondents are almost twice as likely to respond ‘A’ when that was the response of the interviewer for the Corruption and Law & Order questions.

# _4.3  Comparisons to Objective Variables_

As a verification check that the results of tests performed are not anomalies common across the data set and unrelated to interviewer effects, the mixed effects models are also calculated for the four objective variables. The main results follow expectations for Timor-Leste. For example older, while rural and less educated farmers being less likely to have stone terraces on their land. Again, many of the district level effects are significant. With regard to the interviewer-level effects, the only effect significant at the 95% confidence level is that respondents are less likely to tell interviewers under 30 years of age that they have experienced drought. There is also a weakly significant relationship between older interviewers and attending community meetings. The empirical best linear unbiased predictors were also calculated for these models. There are only two outliers interviewer-question combinations are identified, one in the stone terrace question and one related to drought. The quantile graphs for these models are also shown in Figure 1 in the appendix.

With regard to the interviewer’s prior beliefs, the interviewer’s responses to the five subjective questions are included separately for each of the four objective questions. The sign and significance level of the impact of these effects are shown in Table 7. Of the twenty possible subjective-objective combinations, there are statistically significant results in two of the cases and two of the cases are not estimable as the models do not converge. As it seems unlikely that an interviewer’s prior beliefs about dispute resolution would impact the likelihood that the respondent indicates their plot has stone terraces, the significant relationships are most likely either spurious relationships or the interviewer’s beliefs are correlated with other characteristics that impact the quality of the enumeration.

# _4.4 Interaction Terms_

Traditional relationships in Timorese society between age, gender, and deference create many potentially relevant combinations of characteristics. At the extreme, theory would support significant interaction term between a younger female respondent and an older male interviewer in which she is more likely to defer to his opinion. It is not possible to include this level of interaction terms because of dangers in over-fitting the model and having insufficient degrees of freedom for the estimation. Since there were more significant results with the first order effects related to gender, the interaction analysis focuses on those relationships, and on the interviewer’s opinion, as it is less well-studied than other relationships. Table 8 shows a sample of the possible interactions and the contrasts of the marginal predictions. The lines which note an individual covariate are the joint significance tests of the main effects, and the lines in which more than one covariate are noted are the joint tests of the two way interactions. In most cases, while the significance of the main effect remains robust to the inclusion of the interactions terms, at least in cases in which the effect was initially significant, the interaction terms are not significant.

10
An exception, however, is the interaction term between a female respondent and the interviewer’s opinion. In three of the five models, Corruption, Law & Order, and Women’s Rights, the interaction is strongly significant. These results are robust to the inclusion of the gender of interviewer and associated interactions. In two of the models, Corruption and Dispute Resolution, the interaction between a female interviewer, a female respondent, and the interviewer’s opinion are also significant. The only question which shows no evidence of this is Land Titling. In no cases, however, is the interaction between a female interviewer and the interviewer’s opinions significant. The conclusion that can be drawn is that female respondents to this survey were more likely to be influenced by the opinions of the interviewer than male respondents. This is true even if the interviewer is a woman. Also, these findings are robust to the inclusion of an education variable, and therefore the results are not driven in differences in education between genders. Overall these results suggest that female respondents are more likely to be influenced by the interviewers’ opinions.

# _4.5 Salience_

Salience may impact the magnitude of the interviewer effects if respondents are less susceptible to pressure if they believe strongly about a subject, or if interviewer are more likely to exert pressure if they feel strongly. To test this, I repeat the analysis for Women’s Rights, separating the sample into male-only and female-only samples, and into male interviewer and female interviewer sample, under the hypothesis that women’s rights issues would be less salient to men than women. The separate male and female respondent subgroups have roughly equal sample sizes (n=638 for men, n=621 for women). Using the fixed effects model, I find that the interviewer effects are jointly significant for the male-only sample (p<0.0001) but only weakly significant for the female-only sample (p<0.0670). Comparing the percentage of variance explained, again using a Nagelkerke test the overall interviewer effects explain a slightly higher proportion of the variance for the female-only sample, but overall the explanatory power of this model is much higher for the male sample (R<sup>2</sup> =0.250) compared with the female sample (R<sup>2</sup> =0.162). Therefore even the limited explanatory power of the interviewer effects contributes a large portion of the explanation for the female sample. Repeating the analysis with male and female interviewer subsamples, a Nagelkerke test shows that interviewer effects explain a higher portion of the variance in the female interviewer sample, 37.8 percent compared to 30.7 percent, but again the overall variance explained was lower (see Table 5). If these results are explained as Women’s Rights being more relevant to urban education women than to the average female respondent, the results support the finding in Hermann (1983) that issues that are salient to interviewers are more likely to show interviewer effects while those important to respondent are less likely, but the results are again not conclusive.

# **5. Conclusions**

Overall, the analysis finds strong evidence that interviewer effects are present in the data from the TLSLS-2X survey, but that these effects are stronger for subjective and potentially sensitive 11
questions compared to more objective questions. The respondent’s characteristics explain the majority of the variation, even taking into account the fact that the explained variance of the interviewer effects is likely to be overstated due to confounding with geographic effects. The findings on the first order effects of the interviewer support the hypothesis that respondents are more likely to give pro-state answers to historically disenfranchised groups, but the evidence is not conclusive. Similarly the findings suggest women respondents are more susceptible to the influence of the interviewer’s opinions on subjective questions.

The above results must be interpreted with the caveat that the study was one of convenience and is vulnerable to multicollinearity and omitted variable bias. The likely correlation between the interviewer’s opinions and other covariates such as the interviewer’s age, gender, education, and ethnic group raises the potential impact of multicollinearity of the estimates and significance generated with age, gender, and interviewer opinion, as well as omitted variable bias for education and ethnic group.

The results from this study, while not definitive, may provide a basis for future work more purposely designed to study interviewer effects. A design may employ an interpenetrating design, such as the one used in Schnell & Kreuter (2005), or a cross-classified estimation model, to separate the geographic and interviewer effects. The topics covered in a methodological study could also be selected to have more straightforward expectations and to have a sufficiently large sample size to explore a wider range of interactions. Even if future work is not forthcoming, the results highlight the need to consider more fully the impact of traditional cultural norms when conducting quantitative surveys in the developing world on topics that are outside the objective survey questions.

12