The results for the mean poverty predictions in Table 8 show that the non-benchmarked Fay-Herriot estimates underestimate poverty, by a slight amount in Sri Lanka but by a more significant amount in Tanzania. The difference in precision is more striking. The Fay-Herriot estimators are significantly more precise than the Horvitz-Thompson direct estimates in both countries. Relative to these more conservative direct estimates, the Fay-Herriot procedure shrinks the mean square error by nearly half in both countries. However, the household level model, compared with the Horvitz-Thompson estimates, shrinks the mean squared error substantially more, by 70 percent in Sri Lanka and 85 percent in Tanzania.

_Table 8: Average Mean Squared Error (MSE) and coefficient of variation (CV) of target area estimates of headcount nonmonetary poverty, by country and method_

|Mean and Uncertainty|Sri Lank<br>|an subdi<br>|stricts<br>|Tanzania<br>|n distric<br>|ts<br>|
|---|---|---|---|---|---|---|
||Mean<br>poverty|Mean<br>MSE|Mean<br>CV|Mean<br>poverty|Mean<br>MSE|Mean<br>CV|
|Direct survey estimates|||||||
|Horvitz-Thompson approximation|4.0|15.5|65.2|20.0|73.7|43.3|
|EA-Clustered variance estimates|4.0|10.9|58.6|20.0|59.2|38.1|
|Area level Fay-Herriot model|3.7|8.3|47.9|16.8|35.8|34.2|
|Household-level EB model|3.7|4.6|32.0|18.6|11.1|17.5|

_Notes: Mean poverty refers to the average of area headcount rates prior to benchmarking. Mean MSE is the mean across areas of the mean squared error, estimated using a parametric bootstrap approach, multiplied by 10,000_ Table 9 shows the evaluation results against the area poverty rates derived from the census. Average relative bias is highest for the Fay-Herriot estimator in both countries, and lowest for the direct estimates. Of greater interest for evaluating accuracy, however, are the results for correlation and root mean squared error. In both countries, the small area estimates are far more accurate than the direct estimates, but the comparisons between the household level model and Fay-Herriot model is mixed. In Sri Lanka, the household model estimates are more highly correlated with the census than the FayHerriot estimates by a moderate amount (0.88 vs 0.84), but slightly less strongly correlated in Tanzania (0.876 vs. 0.883). The Fay-Herriot predictions have a lower root mean squared error in Sri Lanka, reflecting fewer large outliers, but the root mean squared error is virtually equal in Tanzania. Overall, with regards to accuracy, both the household level model and the Fay-Herriot model perform well.

The household model estimates are substantially more precise than the Fay-Herriot model and the direct estimates, but the tighter confidence intervals around the household model estimates may lower coverage rates. The last column of Table 8 shows the coverage rate, which for the household model in Sri Lanka is 84.3%. This is slightly higher than the Horvitz-Thompson estimates (82.2%), moderately higher than the clustered direct estimates (76.1%) but substantially lower than the Fay-Herriot estimates (91.8%). For Tanzania, the coverage rate for the household level model is 74.8%. This is the lowest of the four estimators but only modestly lower than that of the standard enumeration-area clustered estimates (76.1%). The Horvitz-Thompson and Fay-Herriot estimators achieve much higher coverage rates of 85.5% and 91.8%, respectively.

25