least in years with severe drought where marginal differences in perceptions, benchmarks, or plot characteristics matter least for shock identification.

An important caveat of the previous analysis is that it does not account for the fact that agreement may occur due to pure chance. In the LSMS-ISA data, farmers only face a binary “choice” between reporting or not reporting a shock (a loss due to a specific shock). This increases the chance for random agreement if farmers face some uncertainty when the choice is not obvious to them.<sup>56</sup> In the absence of a verifiable “true value”, there is thus a need for a _chance-corrected_ agreement measure. We take this from the literature on “interrater reliability” (or “agreement coefficients”) which has one of its most common applications in clinical studies where the suitability of a rating scale to elicit consistent answers from different raters is explored. These agreement coefficients provide a measure of homogeneity (or consensus) in ratings of one or more “subjects” given by different “raters”.<sup>57</sup> This paper adapts this framework to the context at hand: Shock occurrence (the _subject_ ) is evaluated by farmers on each plot within an EA (the _raters_ ) and classified into shock or no shock (the _rating categories_ ). Various agreement coefficients exist in the literature. They differ in their definition of “chance” and thus the correction they apply to observed rates of agreement. This paper opts for Gwet’s AC1 which features the highest flexibility regarding different numbers of raters, categories, subjects, and missing values while also being the most robust to a number of anomalies in other coefficients (Gwet, 2008, 2014). The standardized, chance-corrected agreement coefficients that are calculated for shock reports within an EA can lie in the range between -1 and 1. A value of zero means observing as much agreement in farmers’ shock reports within the same EA as would be expected by pure chance. A value of positive 1 (negative 1), conversely, would imply perfect agreement (disagreement). The interpretation of all other values in the interval between -1 and 1 is somewhat arbitrary, however, a widely cited reference for interpretation is given by the Landis and Koch (1977) benchmark scale. Following the recommendation in Gwet (2014), we do not assign the coefficients obtained deterministically to the intervals on the Landis and Koch (LK) scale but acknowledge the probabilistic nature of the point estimates. We therefore rather assign coefficients to the highest interval on the LK scale for which the cumulative probability that the value falls into the interval exceeds 95 percent. Taking this framework to the data, we analyze the homogeneity in farmers’ loss identification due to specific shocks within the same EA (Table 13, Table A 11).<sup>58</sup> We calculate agreement coefficients for three different cases of shocks.

**1.** Whether there were any maize productivity reductions on the plot

**2.** Whether there were any drought productivity reductions on the plot

**3.** Whether there were any plant productivity reductions due to too much rain on the plot

> 56 In other terms, raw agreement does not necessarily reflect _intrinsic_ agreement.

> 57 One practical example may be studying the degree of agreement between several doctors in their diagnosis of a sample of patients.

> 58 The analysis focuses on maize plots which constitute the largest share of the sample and shock reports based on damage on the crop which showed the highest agreement and greatest sensitivity to shocks thus far.

37