university student in the second group we created two Collect Earth “projects” including 640 randomly chosen places each, and we shared them sequentially. For places in the first project, the assessors were asked to decide without any guidance whether they were urban or rural. For the second project, on the other hand, they had to follow the structured protocol.

The procedure was similar for anonymous MTurk workers in the third and fourth groups of assessors. We created two MTurk “tasks” including links to the Google images of all 1,277 places with overlaid administrative boundaries. The first task requested impromptu assessments while for the second one the assessors had to follow the structured protocol. Because MTurk workers are anonymous, we could not ensure that the assessors participating in the first and second round were the same. But for each round, we could specify that each of the 1,277 places needed to be assessed exactly 10 times and that the same MTurk workers could not classify any place more than once.

# _Assessing consistency_

Relying on human judgment to classify places raises understandable concerns about the consistency of the outcomes depending on the characteristics of the assessors and the protocols they follow. Relying on administrative boundaries as the unit of analysis also implies that places can have very different sizes and this diversity could affect human judgment. We addressed these two concerns sequentially and found that the classification of places generated by our methodology was reassuringly stable.

First, we conducted a regression analysis linking the probability of a place being judged urban by an assessor with the characteristics of the accessor and with the protocol followed. We used a standard Logit model, with one observation per place and assessor, and with errors clustered at the place level. The two assessor characteristics considered were familiarity with India and knowledge of urban issues.

The results show that more places are considered urban by assessors familiar with India, while fewer places are judged to be urban among assessors who are knowledgeable on urban issues (table 2). It also appears that following a structured protocol leads to more places being classified as urban. However, while all coefficients are significant, the overall explanatory power of the regression is very small, with the pseudo R-square never exceeding 0.01. This means that subjective biases exist but do not have a substantial impact on the classification of places.

Another way to check whether human judgment leads to consistent results is to compare the classification of places across the seven rounds of assessments. To determine whether a place is urban or rural, we applied the majority rule to all the available assessments within each round. The few places for which we face a tie were excluded from the analysis. Overall, assessments are quite consistent. Two thirds of places have the same status in all seven rounds, an additional 13 percent are classified identically in six rounds, and yet another 10 percent in five rounds.

The classifications also appear to be highly consistent when considering pairwise comparisons between the seven rounds. The percentage of places assessed in the same way in two different rounds of classification varies between 82.3 and 96.7 (table 3). This is regardless of whether the few tied cases are included or not. The result is similar if Tetrachoric correlation coefficients are considered instead, with the estimated coefficients hovering around 0.9.

12