Furthermore, we also calculate a fourth agreement coefficient value that takes drought and rain damage as separate observations (separate “subjects”) on each plot and calculates a coefficient of chance-corrected agreement within an EA for this larger sample of observations. Throughout, we obtain a single, chance-corrected agreement coefficient based on comparing shock reports on plots within the same EA, i.e. we only compare those plots where it is reasonable to assume they face the same weather shocks. As before, we constrain our sample to those EAs where at least one plot recorded a certain shock.<sup>59</sup> _Table 13_ : Ethiopia - Chance-adjusted agreement coefficients for self-reported shock exposure in EAs where at least one plot recorded the respective shock.

|**Round**|**Shock**|**Gwet's**<br>**AC**|**SE**|**N**<br>**Raters**|**N**<br>**Subjects**|**Obs.**<br>**Agreement**|**Exp.**<br>**Agreement**|**LB**|**UB**|
|---|---|---|---|---|---|---|---|---|---|
||Any|0.359|0.045|1326|131|0.657|0.465|0.2|0.4|
|ESS18|Drought|0.389|0.080|546|56|0.685|0.485|0.2|0.4|
||Rains|0.527|0.117|327|24|0.725|0.417|0.2|0.4|
||All shocks|0.396|0.072|797|80|0.698|0.500|0.2|0.4|
||Any|0.623|0.030|2641|201|0.773|0.398|0.4|0.6|
|ESS15|Drought|0.655|0.040|1732|143|0.785|0.378|0.4|0.6|
||Rains|0.688|0.067|492|30|0.781|0.299|0.4|0.6|
||All shocks|0.604|0.041|2051|173|0.784|0.455|0.4|0.6|
||Any|0.427|0.034|2415|196|0.697|0.472|0.2|0.4|
|ESS13|Drought|0.458|0.061|877|85|0.719|0.481|0.2|0.4|
||Rains|0.512|0.093|547|35|0.698|0.381|0.2|0.4|
||All shocks|0.425|0.060|1186|120|0.712|0.500|0.2|0.4|

Note: Interrater reliability coefficients (Gwet's AC) for plots within the same enumeration area. Classification into benchmark interval is probabilistic, not deterministic; LB = lower boundary, UB = upper boundary; Landis and Koch (1977) benchmark scale for agreement coefficients: < 0 poor; 0-20 slight; 0.21 - 0.40 fair; 0.41 - 0.60 moderate; 0.61 - 0.80 substantial; 0.81- 1.0 almost perfect.

The coefficients obtained mostly fall in the range between 0.2 and 0.4 suggesting only limited agreement between farmers. Agreement tends to be lowest for whether there were any plant productivity reductions on the plot which is expected since idiosyncratic shocks may play a role here. On the other hand, rain shocks (either too much or too little rain) should be covariate within the EA which is reflected in slightly higher agreement. Coefficients are again highest in the round with the highest shock incidence, i.e. ESS15 lending further support to the hypothesis that the salience and intensity of the disaster plays a role for the consistency with which farmers record losses and attribute them to the same shock. Since we do not observe the “true value” of whether

> 59 If we do not impose this constraint and compare shock reports for the full sample, we find high rates of agreement. These, however, are driven by low shock incidences in most rounds which, naturally, lead farmers to straightforwardly rule out the occurrence of a shock, e.g. a drought. From this, we can conclude that farmers relatively consistently rule out the occurrence of a shock, however, this does not imply how trustworthy farmers’ reports are _if they claim to have incurred, for example, damage due to drought._ Arguably, the latter is more relevant for the issue of attributing losses to disasters and we therefore only compare a sample in which there seems to be at least some chance that a shock actually occurred (see also the previous discussion in this section).

38