Furthermore, we demonstrate how our algorithmic accuracy is affected based on 1) the type of satellite data used (optical only, radar only or both) - given the considerable differences in the complexity and costs of imagery processing across the various options, and 2) whether plots under specific area thresholds are excluded from the training data - given the potential concerns around the mismatch between the relatively small scale of farming in Malawi and Ethiopia and the Sentinel-2 imagery used in our analysis.

The analysis leverages three national multi-topic household surveys that have been implemented by each country’s national statistical agency over the period of 2018-2020 with financial support from the World Bank Living Standards Measurement Study - Integrated Surveys on Agriculture (LSMS-ISA) initiative. The surveys include detailed, plot-level data on crop farming and georeferenced plot locations. Each data set offers a representative snapshot of the smallholder production system in each country for a given reference season. Linking georeferenced plot-level survey data to publicly available Sentinel-2 imagery and other ancillary geospatial data for the reference agricultural season, we conduct a rich array of sensitivity analyses to assess how crop type prediction accuracy changes when trained on different subsets of plot observations in the survey data - not only in terms of the plot observation count but also the approach to georeferencing plot location. Each data subset was designed to simulate a specific ground data collection scenario. We simulate conditions where, for example, only a certain amount or quality of data is available to train a model, and then compare the out-of-sample prediction accuracies across the scenarios. The results of 26,250 _in silico_ experiments shed light on the ground data needs that should be met for household surveys to plan a more enabling role in satellite-based crop type mapping. After identifying the best available model, we apply it to map areas cultivated with maize across Malawi and Ethiopia at 10-meter spatial resolution.

There are six headline findings that emerge from our analysis. First, a simple machine learning workflow can classify pixels with maize cultivation with up to 75 percent accuracy - though the predictive accuracy varies with the survey data collection method and the number of observations available for model training. Second, georeferencing plot locations by collecting either full plot boundaries or complete plot corner points provide the best quality of information for model training. Third, classification performance almost always peaks before or around 4,000 plots - corresponding to slightly less than 60 percent of the training data. Fourth, the seemingly-small erosion in predictive accuracy under less preferable approaches to georeferencing plot locations results in total area under maize cultivation being overestimated by 0.16 to 0.47 million hectares (8 to 24 percent) in Malawi vis-a-vis the results from the best performing model. Fifth, to avoid overfitting models, it is preferable not to exclude from the training data set any plot observations based on a minimum area threshold. Finally, in the case of maize area mapping in Malawi, optical features alone can provide sufficient signal to maximize prediction quality and the potential benefits offered by synthetic aperture radar, i.e. satellite data unaffected by cloud cover, can be offset by additional noise introduced with its use.

4