metadata: authors: 0: author: Azzari,George
1: author: Jain,Shruti
2: author: Jeffries,Graham
3: author: Kilic,Talip
4: author: Murray,Siobhan
count: Africa
docna: 0: docna: Understanding the Requirements for Surveys
            to Support Satellite-Based Crop Type Mapping : Evidence from
            Sub-Saharan Africa
docty: Policy Research Working Paper
subtopic: Food Security,Labor & Employment Law,Climate Change and Agriculture,Crops and Crop Management Systems,Natural Disasters,Trade Facilitation
teratopic: Law and Development,Agriculture,Environment,International Economics and Trade
abstracts: cdata!: With the surge in publicly available
            high-resolution satellite imagery, satellite-based
            monitoring of smallholder agricultural outcomes is gaining
            momentum. This paper provides recommendations on how
            large-scale household surveys should be conducted to
            generate the data needed to train models for satellite-based
            crop type mapping in smallholder farming systems. The
            analysis focuses on maize cultivation in Malawi and
            Ethiopia, and leverages rich, georeferenced plot-level data
            from national household surveys that were conducted in
            2018–20 and that are integrated with Sentinel-2 satellite
            imagery and complementary geospatial data. To identify the
            approach to survey data collection that yields optimal data
            for training remote sensing models, 26,250 in silico
            experiments are simulated within a machine learning
            framework. The best model is then applied to map seasonal
            maize cultivation from 2016 to 2019 at 10-meter resolution
            in both countries. The analysis reveals that smallholder
            plots with maize cultivation can be identified with up to 75
            percent accuracy. However, the predictive accuracy varies
            with the approach to georeferencing plot locations and the
            number of observations in the training data. Collecting full
            plot boundaries or complete plot corner points provides the
            best quality of information for model training.
            Classification performance peaks with slightly less than 60
            percent of the training data. Seemingly small erosion in
            accuracy under less preferable approaches to georeferencing
            plots results in total area under maize cultivation being
            overestimated by 0.16 to 0.47 million hectares (8 to 24
            percent) in Malawi.
repnme: repnme: Understanding the Requirements for Surveys
            to Support Satellite-Based Crop Type Mapping : Evidence from
            Sub-Saharan Africa
display_title: Understanding the Requirements for
            Surveys to Support Satellite-Based Crop Type Mapping :
            Evidence from Sub-Saharan Africa
listing_relative_url: /research/2021/04/32967235/understanding-requirements-surveys-support-satellite-based-crop-type-mapping-evidence-sub-saharan-africa
topicv3: Agriculture and Food Security m1326205 435,Crop Production m1326627 174,Erosion m1503662 3202,Household Surveys m1504071 2761,Performance Metrics m1504250 2600
publishtoextweb_dt: 2021-04-01T00:00:00Z
dois: 10.1596/1813-9450-9609
historic_topic: Law and Development,Agriculture,Environment,International Economics and Trade
origu: Development Data Group (DECDG)
no_of_pages: 43
author_keys: 0: author_key: 886702
1: author_key: 892365
model_extractions: Policy Research Working Paper 9609 50x2030 Data-Smart Agriculture Series Understanding the Requirements for Surveys to Support Satellite-Based Crop Type Mapping Evidence from Sub-Saharan Africa _George Azzari Shruti Jain Graham Jeffries Talip Kilic Siobhan Murray_ Development Economics Development Data Group April 2021
Policy Research Working Paper 9609

# **Abstract**

With the surge in publicly available high-resolution satellite imagery, satellite-based monitoring of smallholder agricultural outcomes is gaining momentum. This paper provides recommendations on how large-scale household surveys should be conducted to generate the data needed to train models for satellite-based crop type mapping in smallholder farming systems. The analysis focuses on maize cultivation in Malawi and Ethiopia, and leverages rich, georeferenced plot-level data from national household surveys that were conducted in 2018–20 and that are integrated with Sentinel-2 satellite imagery and complementary geospatial data. To identify the approach to survey data collection that yields optimal data for training remote sensing models, 26,250 in silico experiments are simulated within a machine learning framework. The best model is then applied to map seasonal maize cultivation from 2016 to 2019 at 10-meter resolution in both countries. The analysis reveals that smallholder plots with maize cultivation can be identified with up to 75 percent accuracy. However, the predictive accuracy varies with the approach to georeferencing plot locations and the number of observations in the training data. Collecting full plot boundaries or complete plot corner points provides the best quality of information for model training. Classification performance peaks with slightly less than 60 percent of the training data. Seemingly small erosion in accuracy under less preferable approaches to georeferencing plots results in total area under maize cultivation being overestimated by 0.16 to 0.47 million hectares (8 to 24 percent) in Malawi.

This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at tkilic@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Understanding the Requirements for Surveys to Support Satellite-Based Crop Type Mapping: Evidence from Sub-Saharan Africa**

George Azzari1, Shruti Jain2, Graham Jeffries3, Talip Kilic4 and Siobhan Murray5<sup>1</sup> Originally published in the <u>Policy Research Working Paper Series on</u> _April 2021._ This version is updated on _June 2021._ To obtain the originally published version, please email <u>prwp@worldbank.org.</u> JEL Codes: C52, C55, C83, Q12.

Keywords: Agriculture, Maize, Crop Type Mapping, Sentinel-2, Earth Observation, Remote Sensing, Household Surveys, Data Integration, Malawi, Ethiopia.

> 1 The authors are listed alphabetically. 1Chief Technology Officer, Atlas AI P.B.C., george@atlasai.us. 2 Geospatial Data Scientist, Atlas AI P.B.C, <u>shruti@atlasai.us.</u> 3Senior Technical Portfolio Manager, Atlas AI P.B.C, <u>graham@atlasai.us.</u> 4Corresponding author. Senior Economist, Development Data Group, World Bank, <u>tkilic@worldbank.org.</u> 5 Technical Specialist, Development Data Group, World Bank, smurray@worldbank.org. The authors would like to thank Calogero Carletto, Keith Garrett, and the members of the Earth Observation Task Force of the United Nations Global Working Group for Big Data for their comments on the earlier version of the paper. This is a publication of 50x2030 Initiative to Close the Agricultural Data Gap, a multi-partner program that seeks to bridge the global agricultural data gap by transforming data systems in 50 countries in Africa, Asia, the Middle East and Latin America by 2030. For more information on the Initiative, please visit https://www.50x2030.org/.
# **1. Introduction**

<mark>Agriculture is an integral part of livelihoods in Sub-Saharan Africa, where it can contribute up to 69 percent of household income in rural areas (Davis et al. 2017). As such, improving the productivity of smallholder farmers has been a long-standing goal in many African countries that aim to eliminate poverty and food insecurity.</mark> <mark>To monitor progress towards national and international development goals related to agricultural productivity, countries need accurate, crop-specific measures of area under cultivation, production and yields – not only at the national level but with sufficient within-country disaggregation that can guide targeting and evaluation of policies and programs promoting agricultural and rural development and resilience against disasters and extreme weather events.</mark> <mark>With the commencement of the European Space Agency’s Sentinel-2 mission in 2015 and the subsequent surge in the public availability of high-resolution satellite imagery, research has shown the feasibility of satellite-based monitoring of agricultural outcomes in smallholder farming systems (Becker-Reshef et al., 2020; Burke and Lobell, 2017; Jin et al., 2017, 2019; Lambert et al., 2018; Lobell et al., 2019, 2020). Latest advances in satellite imagery and remote sensing techniques have the potential to provide timely insights into conditions on the ground and can fill gaps in agricultural monitoring and statistics (Nakalembe, 2020).</mark> Satellite-based approaches to mapping agricultural outcomes, such as crop-specific estimates of cultivated areas and yields, require data for training and validating the underlying remote sensing models. The quality and spatial resolution of satellite-based estimates is directly impacted by the data used for model training and validation (Lobell et al., 2019, 2020). Recent earth observation research that has focused on low-income countries has relied largely on two sources of training and validation data: (i) manually-labeled optical imagery (DeFourny et al., 2019; Xiong et al., 2017; Wei et al., 2020),<sup>2</sup> and (ii) ground data collection, including as part of household and farm surveys (Hegarty-Craver et al., 2020; Jin et al., 2017, 2019; Kerner et al., 2020; Lambert et al., 2018; Richard et al., 2017).<sup>3</sup> Our paper is related to earth observation applications that rely on georeferenced survey data to meet model training and validation needs.

> 2 In order to map cropland across the African continent, researchers have manually labeled land cover types for sample points using high-resolution optical imagery as a reference. For example, Xiong et al (2017) created a training data set covering Africa by labeling homogenous crop/non-crop areas with a minimum area of 90 x 90 meters (0.81 ha). Several satellite-based cropland data sets have been created for the African continent (using manually-labeled satellite imagery - see Wei et al.’s (2020) comparison of the available data sets), and at sub-regional scales (DeFourny et al. 2019).

> 3 In order to train and validate remote sensing models for mapping cropland and areas cultivated with maize and nonmaize crops, including cassava and beans, at sub-national areas in Tanzania and Kenya, researchers either have used existing survey data that were collected by non-governmental organizations and government agencies to (Jin et al., 2017, 2019; Kerner et al., 2020) or have conducted new survey data collection to meet model training needs (Richard et al., 2017). Similarly, Lambert et al. (2018) rely on sub-national in situ data collected in Mali for satellite-based crop

2
On surveys, research has revealed the need to use improved methods for household and farm survey data collection for enhancing our understanding of the agricultural sector, particularly in low-income countries that stand to benefit the most from the resulting data (Abay et al. 2019; Carletto et al., 2015, 2017; Desiree and Jolliffe, 2018; Kilic et al., 2020). Gourlay et al. (2019) demonstrated that the inverse scale-productivity relationship in agriculture (i.e. the hypothesis that smallholders are more productive than their larger counterparts) may be a statistical artifact, driven by systematic measurement errors in farmer-reported survey data on crop production. The authors showcased these errors by comparing objectively-measured crop cutting and farmer-reported maize yields on the same sample of plots in Eastern Uganda.<sup>4</sup> Follow-up research has demonstrated that survey methods for measuring crop yields directly affect the utility of surveys for earth observation applications, and have provided unambiguous support for the use of objective survey methods to generate the required training and validation data for remote sensing models that integrate survey and satellite data to derive high-resolution estimates of crop yields (Lobell et al. 2019, 2020 in Uganda and Mali, respectively).

Despite the expanding knowledge base regarding the use of earth observation techniques in lowincome countries that are primarily characterized by smallholder farming, research studies have largely remained sub-national in scope and have exhibited heterogeneity in terms of the ground data used to fulfill comparable analytical objectives pursued in different settings. Lack of methodological research to identify the required volume of and approach to ground data collection for training and validating remote sensing models is arguably one of the hurdles against the scaleup of satellite-based estimation of agricultural outcomes across countries and expansive geographies. Identifying ground data requirements for key earth observation applications in lowincome countries, including high-resolution crop type mapping and crop yield estimation, would be important not only for assessing the utility of existing georeferenced household survey data for earth observation research but also informing the design of future large-scale household and farm surveys that can provide the required training and validation data for downstream earth observation efforts.<sup>5</sup> Against this background, this paper addresses several operational and inter-related research questions in the context of high-resolution maize area mapping in Malawi and Ethiopia: 1) what is the minimum volume of household survey data that is required to reach an acceptable level accuracy of a crop classification algorithm? and 2) how does the approach to georeferencing plot locations in household surveys impact the accuracy of the same crop classification algorithm?

> type mapping and yield estimation at the village-level in Konigue commune. Relatedly, Hegarty-Craver et al. (2020) mapped four crop types (maize, beans, bananas, cassava) in Senegal using training data derived from high-resolution UAV imagery which they collected in the field.

> 4 These findings are corroborated by Abay et al. (2019) and Desiere and Jolliffe (2018), in Ethiopia and Uganda, respectively.

> 5 Future large-scale surveys that can be a source of training and validation data for earth observation efforts include surveys that are supported by the 50x2030 Initiative.

3
Furthermore, we demonstrate how our algorithmic accuracy is affected based on 1) the type of satellite data used (optical only, radar only or both) - given the considerable differences in the complexity and costs of imagery processing across the various options, and 2) whether plots under specific area thresholds are excluded from the training data - given the potential concerns around the mismatch between the relatively small scale of farming in Malawi and Ethiopia and the Sentinel-2 imagery used in our analysis.

The analysis leverages three national multi-topic household surveys that have been implemented by each country’s national statistical agency over the period of 2018-2020 with financial support from the World Bank Living Standards Measurement Study - Integrated Surveys on Agriculture (LSMS-ISA) initiative. The surveys include detailed, plot-level data on crop farming and georeferenced plot locations. Each data set offers a representative snapshot of the smallholder production system in each country for a given reference season. Linking georeferenced plot-level survey data to publicly available Sentinel-2 imagery and other ancillary geospatial data for the reference agricultural season, we conduct a rich array of sensitivity analyses to assess how crop type prediction accuracy changes when trained on different subsets of plot observations in the survey data - not only in terms of the plot observation count but also the approach to georeferencing plot location. Each data subset was designed to simulate a specific ground data collection scenario. We simulate conditions where, for example, only a certain amount or quality of data is available to train a model, and then compare the out-of-sample prediction accuracies across the scenarios. The results of 26,250 _in silico_ experiments shed light on the ground data needs that should be met for household surveys to plan a more enabling role in satellite-based crop type mapping. After identifying the best available model, we apply it to map areas cultivated with maize across Malawi and Ethiopia at 10-meter spatial resolution.

There are six headline findings that emerge from our analysis. First, a simple machine learning workflow can classify pixels with maize cultivation with up to 75 percent accuracy - though the predictive accuracy varies with the survey data collection method and the number of observations available for model training. Second, georeferencing plot locations by collecting either full plot boundaries or complete plot corner points provide the best quality of information for model training. Third, classification performance almost always peaks before or around 4,000 plots - corresponding to slightly less than 60 percent of the training data. Fourth, the seemingly-small erosion in predictive accuracy under less preferable approaches to georeferencing plot locations results in total area under maize cultivation being overestimated by 0.16 to 0.47 million hectares (8 to 24 percent) in Malawi vis-a-vis the results from the best performing model. Fifth, to avoid overfitting models, it is preferable not to exclude from the training data set any plot observations based on a minimum area threshold. Finally, in the case of maize area mapping in Malawi, optical features alone can provide sufficient signal to maximize prediction quality and the potential benefits offered by synthetic aperture radar, i.e. satellite data unaffected by cloud cover, can be offset by additional noise introduced with its use.

4
The paper is organized as follows. Section 2 describes the survey and earth observation data. Section 3 presents the empirical methodology. Section 4 discusses the results and section 5 concludes.

# **2. Data**

# **2.1. Survey data**

We use nationally representative, multi-topic household survey data collected in Malawi and Ethiopia by the respective national statistical offices over the period of 2018-2020 with support from the World Bank Living Standards Measurement Study-Integrated Surveys on Agriculture (LSMS-ISA) initiative. The key variables that drive each survey’s sampling design is household consumption expenditures and poverty. However, the surveys do provide large samples of agricultural households and extensive data on their agricultural activities. Maize is the primary crop grown in Malawi, while in Ethiopia, small grains are more prevalent, but maize still plays an important role as a staple crop. More details regarding the survey data are provided below.

# **2.1.1. Malawi**

The survey data in Malawi stem from the Integrated Household Panel Survey (IHPS) 2019 and the Fifth Integrated Household Survey (IHS5) 2019/20. The surveys were implemented concurrently by the Malawi National Statistical Office.

IHPS 2019 is the fourth follow-up to a national sample of households and individuals that had been interviewed for the first time in 2010, and later in 2013 and 2016. At baseline, the IHPS was designed to be representative at the national-level and separately for rural and urban domains.<sup>6</sup> Starting in 2013, the IHPS attempted to track all household members that were interviewed in the last survey round and that were projected to be at least 12 years of age and were known to be residing in mainland Malawi during the follow-up survey round.<sup>7</sup> Once a split-off individual was located, the new household that he/she may have joined vis-a-vis the prior survey round was brought into the IHPS sample. Based on these protocols, the dynamically expanding IHPS sample included 3,181 households in 2019, which can be traced back to 1,491 original households that had been interviewed in 2010. The IHPS 2019 fieldwork was conducted from April to December 2019, and the households that were determined to have owned and/or cultivated land during the 2018/19 rainy season were attempted to be visited twice, once in the post-planting period and once

> 6 For more information on the IHPS rounds, please consult the publicly-available basic information document for each survey - IHPS 2010: https://bit.ly/ihps2010; IHPS 2013: https://bit.ly/ihps2013; IHPS 2016: <u>https://bit.ly/ihps2016; and IHPS 2019: http://bit.ly/ihps2019.</u>

> 7 The individuals who were residing in prisons, police compounds or army barracks were excluded from tracking.

5
in the post-harvest period, following the same set of fieldwork protocols that had been used in the prior IHPS rounds.

The IHS5 2019/20 is the second source of survey data in Malawi. Unlike the IHPS 2019, the IHS5 is a cross-sectional survey that is designed to be representative at the national-, urban/rural-, regional- and district-levels. The IHS5 sample includes a total of 11,434 households, distributed across 717 EAs throughout Malawi.<sup>8</sup> The fieldwork was implemented from April 2019 to April 2020, and each sampled household was visited once. The households that were determined to have owned and/or cultivated any land reported information on the last completed rainy season, which could have been 2017/18 or 2018/19 depending on the interview date.

The IHPS 2019 and the IHS5 2019/20 used identical, extensive agricultural questionnaires that elicited information at the parcel-, parcel-plot-, and parcel-plot-crop-level, depending on the topic. Of particular importance to our research is that the surveys identified each crop cultivated on each plot, and in the process, determined whether a given plot was monocropped or intercropped. Further, each cultivated plot that was determined to be within 2 hours of travel (irrespective of the mode of transport) was attempted to be visited with the farmer. The plot area was captured with a Garmin eTrex 30 handheld global positioning system (GPS) unit, and the plot location was georeferenced in two ways: (i) the enumerator captured the GPS coordinates for the corner point at which the plot area measurement commenced and manually inputted the GPS coordinate into the computer-assisted personal interviewing (CAPI) application (i.e. the corner point method), and (ii) the enumerator also captured the perimeter of the plot during the plot area measurement exercise and stored the resulting geospatial polygon on the GPS unit following a naming convention that facilitates the linking of the polygon to the plot record in the household survey data (i.e. the full boundary method).

We refined the initial data set to isolate the best quality data for the analysis. Plot records were retained only if they possessed both a corner point and a full plot boundary and had a crop type record for the reference rainy agricultural season. Furthermore, if the location information (either corner point, or plot boundary, or both) was duplicated across two or more plots, we dropped all duplicated records, except in cases where one, and only one, of the duplicated records had a high degree of confidence assigned to their location data quality - in these cases, the record with the high degree of confidence was kept and the remaining records were dropped. Lastly, only records with a high degree of confidence in the location data quality (both for the corner point and the plot boundary), as indicated by a metric provided by the GPS unit, were retained. We treated plots that were cultivated with any maize as “maize plots”, and otherwise labeled them as “non-maize.” Tables 1 and 2 show the IHPS 2019 and the IHS5 2019/20 rainy season plot observations, broken down by georeferenced information availability and by maize cultivation status, respectively. The final analysis sample includes 2,792 IHPS 2019 plots and 5,794 IHS5 plots associated with the

> 8 See the IHS5 2019/20 basic information document for more information: http://bit.ly/ihs201920.

6
2018/19 rainy season, and 3,265 IHS5 plots associated with the 2017/18 season. The total number of agricultural households that are associated with these observations is 1,470 in the IHPS and 5,432 in the IHS5.

Table 1: IHPS 2019 and IHS5 2019/20 rainy season plots by <u>georeferenced information availability</u>

|**Plot category**|**IHPS**|**2019**|**IHS5 2**|**019/20**|
|---|---|---|---|---|
||Obs|%|Obs|%|
|Plots with no geolocation information|334|6.2|1,105|6.4|
|Plots with a corner point, but no polygon boundary|1,365|25.4|4,871|28.4|
|Plots with a corner point and a polygon boundary, but dropped from analysis|874|16.3|2,139|12.5|
|Plots with a corner point and a polygon boundary, used for analysis|2,792|52.0|9,059|52.7|
|**Total # of Plots**|**5,365**|**100.0**|**17,174**|**100.0**|
|**Total # of Associated Households**|**2,3**|**35**|**8,7**|**70**|

Table 2: IHPS 2019 and IHS5 2019/20 rainy season plots by maize cultivation status, conditional on being used for analysis

||**IHPS**|**2019**||**IHS5**|**2019/20**||
|---|---|---|---|---|---|---|
|**Season**|2018|/19|201|7/18|201|8/19|
|**Crop type**|Obs|%|Obs|%|Obs|%|
|Maize|2,033|72.8|2,330|71.4|4,222|72.9|
|Non-maize|759|27.2|935|28.6|1,572|27.1|
|**Total # of Plots**|**2,792**|**100.0**|**3,265**|**100.0**|**5,794**|**100.0**|
|**Total # of Associated Households**|**1,4**|**70**|**1,9**|**26**|**3,5**|**06**|

To begin investigating how the approach to georeferencing plot locations would affect the accuracy of remote sensing models that combine survey and satellite data for high-resolution crop type mapping, we used the full plot boundaries to first derive several additional sets of coordinates that could have been generated with alternative plot geolocation methods and that include:

1. The coordinates of one plot corner that is recorded by the enumerator, i.e. “corner point.”

2. The coordinates of the plot centroid that is derived from the full boundary, i.e. “centroid.” 3. The coordinates of 4 to 8 plot corner points that are derived from the boundary, based on the complexity of the plot shape (geometric simplification) and that are in turn used to:

- a. Derive the geospatial predictors for each pixel corresponding to a given corner point and use these pixels and the associated predictors as the training data, i.e. “boundary points.”

7
- b. Randomly select 20% of the pixels within the convex hull that is formed by the corner points; derive the geospatial predictors of interest for each sampled pixel; and use these pixels and the associated predictors as the training data, i.e. “convex hull.”

- c. Derive the geospatial predictors for all pixels within the convex hull and aggregate the information to the plot-level by taking the average, for each predictor, across all pixels, i.e. “hull mean.”

4. The full plot boundary that is in turn used to:

- a. Randomly select 20% of the pixels from a 10m grid within the plot; derive the geospatial predictors of interest for each sampled pixel; and use these pixels and the associated predictors as the training data, i.e. “plot points.”

- b. Derive the geospatial predictors for all pixels from a 10m grid within the plot and aggregate the information to the plot-level by taking the average, for each predictor, across all pixels, i.e. “plot mean.”

This listing of alternative approaches to georeferencing plot locations on the ground is also indicative of the increasing operational complexity as we move from 1 to 4. Finally, Figure 1 provides a visual overview of these methods and how they impact the computation of geospatial predictors that are specific to each plot.

8
<!-- Start of picture text --> Plot boundary Corner point<br>6 approaches to sample Pixel value at one<br>full boundaries corner point (used as<br>Centroid Plot mean Plot points Boundary points<br>Pixel value at Mean of all pixels in Values at a random 20% Values at 4-8 corner<br>boundary centroid boundary ofpixels in boundary points<br>Hull mean Convex hull<br>Mean of all pixels in Values at a random 20%<br>simplified boundary of pixels in simplified<br>Map data ©2020 Google constructedcorner points from boundaryfrom corner constructed points<br><!-- End of picture text -->
starting the plot area measurement (which was then conducted with a Garmin eTrex 30 handheld GPS unit).

For our analysis, the plot records were retained only if they possessed corner point information and had a crop type record for the 2018 _meher_ season. Furthermore, if the location information was duplicated across two or more plots, we dropped all duplicated records, except in cases where one, and only one, of the duplicated records had a high degree of confidence assigned to their location data quality - in these cases, the one record with the high degree of confidence was kept and the remaining records were dropped. Lastly, a small share of plots was dropped due to low confidence in their location data quality, as captured by the Android tablet. We treated plots that were cultivated with any maize plantings as maize plots, and otherwise labeled them as “nonmaize.” Tables 3 and 4 show the breakdown of ESS 2018/19 _meher_ season plots, by georeferenced information availability and by maize cultivation status, respectively.<sup>10</sup> The final analysis sample includes 11,905 ESS 2018/19 plots, originating from 2,090 households. Since the ESS 2018/19 did not capture full plot boundaries, our analysis focuses primarily on Malawi, with the findings from Ethiopia playing a supporting role.

Table 3: ESS 2018/19 _meher_ season plots by <u>georeferenced information availability</u>

|**Plot category**|**ESS 2**<br>|**018/19**<br>|
|---|---|---|
||Obs|%|
|Plots with no geolocation information|1,168|8.7|
|Plots with a corner point, but dropped from analysis|299|2.2|
|Plots with a corner point, used for analysis|11,905|89.0|
|**Total # of Plots**|**13,372**|**100.0**|
|**Total # of Associated Households**|**2,1**|**99**|

Table 4: ESS 2018/19 _meher_ season plots by maize cultivation status, conditional on being used for analysis

|**Crop type**|**ESS 20**<br>|**18/19**<br>|
|---|---|---|
||Obs|%|
|Maize|1,867|15.7|
|Non-maize|10,038|84.3|
|**Total # of Plots**|**11,905**|**100.0**|
|**Total # of Associated Households**|**2,0**|**90**|

10 The share of plots with no geolocation information is significantly lower in the ESS 2018/19 data mainly due to the reliance on resident enumerators, as opposed to mobile survey teams in the context of the IHPS 2019 and the IHS5 2019/20.

10
# **2.2. Earth observation data sets**

Mapping crop area among smallholder plots across a large geographic scale requires satellite remote sensing data sources with high spatial resolution and temporal cadence. Building on precedents set in the research literature, we designed an array of satellite-derived metrics that can be used by a statistical model to distinguish between crop cover types. We used two types of satellite imagery in our maize area mapping experiments - optical and synthetic aperture radar (SAR). Each data source captures different crop properties useful for crop type mapping. For example, optical imagery records information that can be used to characterize a crop’s phenology, while SAR imagery captures properties of the canopy structure that may signify differences between crops (Robertson et al. 2020). We processed and extracted both optical and SAR data to the survey plot locations for maize area mapping.

# **2.2.1 Synthetic aperture radar imagery**

Sentinel-1 (S1) satellites carry a Synthetic Aperture Radar (SAR) sensor that operates in a part of the microwave region of the electromagnetic spectrum which is insensitive to water vapor and thus unaffected by clouds or haze. Sentinel-1 Interferometric Wide swath mode (IW) provides images with dual polarization (VV and VH) centered on a single frequency. Google Earth Engine provides S1 images at 10m resolution which are corrected for noise. To use this imagery, we applied Local Incidence Angle (LIA) correction, and computed RATIO and DIFF bands (Table 5).

# **2.2.2 Optical imagery**

Sentinel-2 (S2) satellites provide multispectral imagery for 13 spectral bands with a 10 m resolution for red, green, blue, and near infrared bands used to compute common vegetation indices. We used S2 Level-2A imagery hosted in Google Earth Engine (Gorelick et al. 2017) in our analysis. Image pre-processed included masking out pixels containing clouds, cloud shadows, haze, and snow using quality masks provided with the imagery product (see Appendix Table A1 for summary statistics on imagery counts). Once preprocessed, five vegetation indices (VIs) were calculated for all available S2 images (Table 5).

11
Table 5: Satellites, bands and indices used in the analysis.

|Band /<br>Index|Name|Central wavelength / Index formula|Satellite|
|---|---|---|---|
|VV|Vertically polarized backscatter|5.5465763 cm|Sentinel-1|
|VH|Horizontally polarized backscatter|5.5465763 cm|Sentinel-1|
|RATIO|Ratio|VV / VH|Sentinel-1|
|DIFF|Difference|VV – VH|Sentinel-1|
|RDED4|Red Edge 4|865 nm|Sentinel-2|
|GCVI|Green Chlorophyll Vegetation Index|(NIR – GREEN)/1|Sentinel-2|
|NBR1|Normalized Burn Ratio 1|(NIR – SWIR1) / (NIR + SWIR1)|Sentinel-2|
|NDTI|Normalized Difference Temperature Index|(SWIR1 – SWIR2) / (SWIR1 + SWIR2)|Sentinel-2|
|NDVI|Normalized Difference Vegetation Index|(NIR – RED) / (NIR + RED)|Sentinel-2|
|SNDVI|Smoothed Normalized Difference<br>Vegetation Index|(NIR – RED) / (NIR + RED + 0.16)|Sentinel-2|

# **2.2.3 Harmonic regressions for characterizing crop phenology**

We used the multi-temporal collection of bands and indices from S1 and S2 to capture changes in crop phenology over time. To identify temporal patterns that characterize crop phenology, a harmonic regression model was fit at a pixel level to the time series of each unique band and index (Deines et al. 2020, Jin et al. 2019). See Equations 1 and 2 for Malawi and Ethiopia, the latter of which includes an additional pair of harmonic terms. The algorithm produces features that capture the seasonality of different crop types and that include harmonic coefficients, seasonal mean, and goodness of fit measures. These features are useful to map crop types because a maize pixel undergoes seasonal changes in greenness that differ from those of other crops (See Figure 2).

Equation 1 G _0 1 2_ c _2 1 3_ ( _2 1 4_ c _2 2 5_ ( _2 2_ (where _1_ = _1_ and _2_ = _2_ ) tt GGII = ββ + ββ tt+ ββ cc( ππωω tt) + ββ ccs ππωω tt) + ββ cc( ππωω tt) + ββ ccs ππωω tt) + εε ωω ωω Equation 2 G _0 1 2_ c _2 1 3_ ( _2 1 4_ c _2 2 5_ ( _2 2 6_ c _2 3 7_ ( _2 3_ tt (where GGII _1_ == ββ _1_ , + ββ _2_ = _2_ tt+ ββ, and _3_ cc(= _3_ ππωω) tt) + ββ ccs ππωω tt) + ββ cc( ππωω tt) + ββ ccs ππωω tt) +ββ cc( ππωω tt) + ββ ccs ππωω tt) + εε ωω ωω ωω 12
<!-- Start of picture text --> 5 S2 GCVI - Maize S2 GCVI - Groundnut S2 GCVI - Soyabean @ Observed<br>e — Fitted<br>4 e ° .,e c<br>. ‘<br>e fe eve<br>-3 e<br>5 . ‘ e<br>1) 2 e $ ° e<br>1 es 7 e\ e e c<br>e L % e<br>0<br>Dec Feb Apr Jun Aug Dec Feb Apr Jun Aug Dec Feb Apr Jun Aug<br>2018 2019 2019 2019 2019 2018 2019 2019 2019 2019 2018 2019 2019 2019 2019<br>Date<br><!-- End of picture text -->
Table 6: Additional EO data used in the maize classification pipeline

|**Feature**|**Explanation**|**Data Source**|**Included in**|
|---|---|---|---|
|Elevation|Obtained using GEE’s<br>inbuilt_terrain_algorithm<br>|Shuttle Radar Topography<br>Mission (30-meter|Malawi, Ethiopia|
|Slope|that uses an elevation<br>raster to generate slope<br>and aspect bands|resolution)|Malawi, Ethiopia|
|Aspect (direction of slope)|||Malawi, Ethiopia|
|Average temperature|Mean daily temperature<br>during growing season|aWhere daily observed<br>weather API (0.1-degree|Malawi|
|GDD|Growing degree days*<br>accumulated during<br>growing season|resolution)|Malawi|
|Total precipitation|Total precipitation during<br>growing season||Malawi|

**Notes:** * A growing degree day is one in which the mean temperature is greater than a base value that must be exceeded for crop growth to occur. For maize, this base value is 10 °C.

# **3. Methodology**

We developed a methodological framework that is presented in Figure 3 and that is designed to quantify the ability of a machine learning model to identify pixels as maize or non-maize under scenarios with limited training data quantity, various data collection methods, and type of satellitederived variables used. The overarching approach was to

1. define a common modeling pipeline that trains and evaluates a maize classification model for a given data set,

2. feed the modeling pipeline with each data set in a sequence designed to emulate hypothetical scenarios of field data collection (varying the number of observations, the plot geolocation method, and the minimum plot size), 3. vary the type of satellite data used by the modeling pipeline (optical only, radar only, both optical and radar), and 4. compare evaluation metrics across different scenarios. Figure 3 depicts the overall structure of the study.

14
<!-- Start of picture text --> '<br>'1 Clean and aggregate i'<br>Process EO data to ground data from da ened ground<br>'\ obtain‘ features of surveysvey’ and => for differentgeneral geolocatiets \;<br>' interest: (S1, S2, 8eolocations. Perform ormethodsdilicrent representinggeolocation ''<br>|| topography, weather) necessary QA checks. different field data !<br>' collection scenarios \<br>1<br>'' Generate composites of ''<br>' labels (for each !<br>' geolocation method) '<br>' and the features }<br>! '<br>'aeCeSobtainedeee fromSEES EO data 22555555555 o aaa aneo'<br>' i! 1<br>1 For each geolocation ; For each training data |<br>\ Split all labeled data method, produce H i subset, fit a Random \<br>H into train / val / test (70 several subsets of 1 Forest pipeline (Initial \<br>1 / 15/15), making sure training data — to ‘| | model hyperparameters \<br>\ that val and test sets =» emulate scenarios of => identified using cross }<br>\ remain the same across varying amounts of ' | validation on the entire !<br>H all geolocation methods data collected with 1 training data). Use the '<br>1 varying thresholds on 1 trained model to make<br>minimum plot area predictions on val set. '<br>' i! 1<br>\ ' '<br>\;<br>' 1! 1<br>1<br>Assess the effect of, After model tuning, 1 Assess how val set<br>! satellite imagery, retrain RF pipeline on | performance changes }<br>! geolocationsample size, method, and r eachdata. subset Obtain of modeltraining hi withset increasingsize. Tune model training | |'<br>' minimum plot size on performance on test tt hyperparameters !<br>' test set performance. data. ‘ry furtherover/underfitting. to minimize ''<br>1 1<br><!-- End of picture text -->
<!-- Start of picture text --> » -<br>Vl<br><!-- End of picture text --> <!-- Start of picture text --> »».|<br>eT<br><!-- End of picture text -->

<!-- Start of picture text --> »<br>er<br><!-- End of picture text -->
JT FP) )(TN + FN)
<!-- Start of picture text --> Distribution of plot sizes<br>0.0 0.5 ist Area (ha) 2.0 2.5<br><!-- End of picture text --> <!-- Start of picture text --> el<br>Vv;<br><!-- End of picture text --> <!-- Start of picture text --> &<br>8<br><!-- End of picture text --> <!-- Start of picture text --> &L<br>eT<br><!-- End of picture text --> <!-- Start of picture text --> ae<br>&ét<br>Oe]<br><!-- End of picture text -->
modeling pipeline in steps of 2 percentage points. Subsampling of training data was also done in a stratified manner (by district and class label). Each subsample was passed through the maize classification pipeline and evaluation results were recorded. In total, we tested 26,250 scenarios, comprising:

1. 7 geolocation methods - boundary points, centroid, convex hull, corner, hull mean, plot points, and plot mean

2. 50 data subsets - 2% to 100% subsets of training data, at an increment of 2% points 3. 5 area thresholds - 0, 0.05, 0.1, 0.15, and 0.2 ha 4. 3 feature types - optical only, radar only, both optical and radar 5. 5 replications to capture variability due to random sampling To compare performance across countries, we applied a similar workflow to the Ethiopia survey data set. However, due to the limitations of that data set, we only tested the corner point geolocation method, with no area threshold, and with optical data only.

# **4. Results and discussion**

We conducted sensitivity analyses to understand how survey data properties, especially their number and geolocation method, affect the performance of maize classification predictions. We constructed a maize classification model for each unique combination of the data collection scenarios and compared performance across these scenarios. We would like to note here, that we calculated performance on plot centroids in the test subset regardless of the data collection scenario being evaluated, to ensure that our performance metrics were comparable across geolocation methods. In this section, we present the drivers of maize classification performance by first focusing on the effects of survey sample size and geolocation method, then plot area thresholds, and finally the source of satellite data used.

# **4.1. Effect of the approach to georeferencing plot location**

Maize classification accuracy scores across geolocation methods typically fell within 2.5 percentage points of each other, with gaps increasing with the number of plots used for model training (Figure 7a). The variation in maize classification performance across geolocation methods was more pronounced in the MCC curves (Figure 7c), with the “hull mean” and “plot mean” methods producing the most significant performance advantage.

The difference in the accuracy and MCC training curves arises because MCC takes into account all the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), thus providing a more balanced measure of performance. This becomes especially important when we consider geolocation strategies where a number of training points are collected for a single plot, such as “boundary points”, “convex hull”, and “plot points” - they further increase 19
the imbalance between maize and non-maize observations in the training data set. For this reason, these three geolocation methods show worse MCC performance.

In nearly all cases, the centroid method outperformed corner point method. If only a single GPS point is collected by data collectors, that location should be near the center of the plot. The performance of corner point method was similarly poor in both Malawi and Ethiopia, demonstrated by MCC plots for both countries (Figures 7c and 7d). The plot mean and hull mean methods outperformed all the other methods. Hence, if plot boundaries or multiple corner points are available, the results show that aggregating pixels in the plot (e.g. by taking the mean) is preferable to sampling many pixels from the plot.

Increasing the number of training plots led to an increase in MCC in most cases (Figure 8). When only a small fraction of training observations was available (less than 1,000 plots), geolocation methods where a number of training points can be constructed (such as “boundary points”, “convex hull”, and “plot points”) performed better than others. However, with larger sample sizes (greater than 2,000 plots), “plot mean” and “hull mean” outperformed other methods. The “centroid” geolocation method performed similarly to “plot mean” and “hull mean,” except when nearly all of the plots were used for training (around 7,000 plots) in which case “plot mean” and “hull mean” methods outperform the “centroid” method. Lastly, corner points from about 7,000 plots gave roughly the same performance as using the “plot mean” or “hull mean” geolocation strategy on approximately 3,000 plots.

20
<!-- Start of picture text --> Training curves Training curves with smoothened trials<br>0.76 pees ’ — fas j =" |‘ — Geolocation— boundary Strategypoints<br>ia Ses < cam = — centroid7 apo<br>> me a IS eg AP 4 == Seu — convex hull<br>Fs}tjL 0.74 JZG —. | < - j || ~~ + oe<a”- a—< = — |, | —— cornerhull mean<br><U 972 7Z Kalh — plot points.<br>Lva — plot mean<br>E<br>& 0.70<br>~is)<br>wa<br>© 0.68<br>0.66<br>0 1000 2000 3000 4000 5000 6000 7000 0 1000 2000 3000 4000 5000 6000 7000<br>Number of plots Number of plots<br><!-- End of picture text -->

<!-- Start of picture text --> Training curves Training curves with smoothened trials<br>0.76<br>o tk i<br>8 0.74<br>=<br>co)<br>Z A :<br>ov 0.72<br>a<br>5<br>0.70 1 7 7<br>rrfo}<br>odSs }<br>© 0.68 ~~<br>0.66 T<br>|<br>0 1000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000<br>Number of plots Number of plots<br><!-- End of picture text -->
<!-- Start of picture text --> as Training curves Training curves with smoothened trials<br>0.30 || bie | 2 i A | | | | : _ | ——GeolocationboundaryStrategypoints<br>— || 4 yy Weyey | leerIe. | | | —_——— SS | ——— convexcentroidhull<br>oe | i A ; yr — corner<br>= , ae Var hn aG y ee —__e Ae<br>wv 0.20 ia ; Sa ee ae ry G SS —— — plot points<br>Q f fp i, A 23h — ——————l<br>€$ 0.15 | di): it yo \ S ta YYY f + —— plot mean<br>65 0.10 | - IAWINHP Yi)f; A ! |_|<br>I | }/ ff<br>0.05 -4-¥ 7 ‘es ee ee ee ee a c/see ee eeee<br>0.00 +o : iw, “| —<br>0 1000 2000 3000 4000 5000 6000 7000 0 1000 2000 3000 4000 5000 6000 7000<br>Number of plots Number of plots<br><!-- End of picture text -->

<!-- Start of picture text --> _— Training curves Training curves with smoothened trials<br>3 0.20 pan ey : —<br>a > e PNK ruavwey ZA —— . 5<br>A | . Ah y ¥. | | | =~ ae = |<br>c)44i||/fj |<br>|<br>01000 2000 3000 4000 5000 6000 7000 8000 0 1000 2000 3000 4000 5000 6000 7000 8000<br>Number of plots Number of plots<br><!-- End of picture text -->
<!-- Start of picture text --> ’ Number of plots<br>0.30 a 727<br>+ Gi 2834<br>0.25 oS lam 4945<br>oO + = co 7 [=a 7030<br>Q zl x ’ ;<br>a<br>‘yE 0.20 ’ eee¢ = ‘ r) +<br>5& 0.15 —+ =+<br>5<br>©0.10 + + Ss<br>0.05 7 $<br>0.00 &<br>boundary points centroid convex hull corner hull mean plot points plot mean<br>Geolocation strategy<br><!-- End of picture text -->
<!-- Start of picture text --> boundary points centroid convex hull<br>a]<br>Ko<br>Qa<br>@ 1:5 :<br>r=)<br>a7 Oo) cs ° @<br>O= } sttees feet,°<br>= 0.5 (Nose ste ptegs-25, Ros<br>aes,cSvoD 0.0 8°26 0agtaneeel24, w.20°05,428 0°,,20080be sgc0o.atg089038, eeee et ~% 4egtelags " eee > r) OBSti62?*seaseneee 3 iFgod?” Sea: E aiitiee<br>£ °8age Pogg8 4008" jo. age? ee!<br>[s)w295 ®o,0<br>x<br>corner hull mean plot points<br>ro)Pd} e 4} , °o, g<br>a, e008<br>o 15 . Wace:<br>S ° e it) 20° 98Py<br>© Sess, ae °<br>a 1.0 Ls Od tl e,0<br>oO be 0 bigs"<br>1S] oo? *t.0 8. a ee i<br>= 0.5 joesg# Son we vo) ati weqor? beocoet ese To<br>‘oO jee " Sag on “sesey “Byece® “onstetcs of oo. 4 my) > cr<br>ro£ 0.0 oe ° 000882=rse8egeg:Maes gonngdeetecd° eee $0°0% ogeearsovnnmaeand n90°%0co (8afo IT0ggg00 eo!o58esangthge ot rae88e°wk 885 88880had co HEHo000<br>[s) ° Pcocon” PcocoK<br>a -0.5<br>2°o<br>2000 4000 6000 2000 4000 6000<br>plot mean Number of plots Number of plots<br>2<br>2<br>Qa<br>oO 1.5 Kerr<br>(a ge<br>ca<br>_—<br>B 1.0 foods<br>oO<br>2=°<br>=0.5 2 8Bte3 jssee eee<br>o 8g, -<br>5 0.0 rec |<br>es1} 56<br>~<a<br>o -0.5 oy<br>x<br>2000 4000 6000<br>Number of plots<br><!-- End of picture text -->

<!-- Start of picture text --> 2<br>2<br>o 15<br>43 Peeee°<br>al<br>@ 1.0 3<br>1}<br>= bee<br>c 0.5 § oF<br>v Pees : ' oe<br>rad leece 288 ‘ 228 ~ aseoosCOFTer Tt Hiseeen 222s)oone<br>5 °<br>PyQa<br>x -0.5<br>2000 4000 6000 8000<br>Number of plots<br><!-- End of picture text -->
<!-- Start of picture text --> Peak out of sample MCC 7000 Number of plots to reach peak MCC<br>0.30<br>6000<br>0.28<br>4<br>[e)<br>OS 0.26 gn 5000<br>va<br>aE 0.24 we2<br>a 8 4000<br>Pha 2<br>8 g<br>3000<br>0.20<br>0.18 2000<br>ow? _ss® yw’ ot «oTM gr? «oTM ge? a? yw’ ot «oe gr? oo<br>otyo er geSS ow” i ~~) ox? 3 orey) gr ge0 on? ies ~~) go? o*<br>Geolocation strategy Geolocation strategy<br><!-- End of picture text -->
<!-- Start of picture text --> Peak out of sample MCC 7000 Number of plots to reach peak MCC<br>0.30<br>0.28 6000<br>ea<br>7 2<br>rn 0.26 5° 5000<br>Qa -_<br>E 3<br>& 4000<br>B 0.24 5<br>© 0.22 E<br>S 2<br>6 3000<br>0.18 2000<br>0.20 i<br>corner corner<br>Geolocation strategy Geolocation strategy<br><!-- End of picture text --> <!-- Start of picture text --> Max out of sample MCC Number of plots to reach 90% of max MCC<br>0.34 6000<br>0.32<br>5000<br>0.30<br>y , ,<br>‘y 0.28 B 4000<br>ES 0.26 c& ;g<br>2 , E 3000<br>6 0.24 = ry<br>+<br>0.220.20 FF 2000<br>1000<br>0.18<br>ak? ey a Os oe ak? my ak? Rey Ry es oe ak? oe<br>or? om\ cerac) ee” xo ve2 <°‘> ye2 emNK)cos’ eta <n ye] <@ow yeef<br>»? ¢ co ~~) AS oe %°oo ¢ com ) oe io<br>Geolocation strategy Geolocation strategy<br><!-- End of picture text -->
<!-- Start of picture text --> Max out of sample MCC Number of plots to reach 90% of max MCC<br>0.34 6000<br>0.32<br>oO83o 0.280.30 ra 400022{e) 5000<br>© 0.26 o<br>“a a<br>© 0.24 5€ 3000<br>S2<br>°<br>020.20 ce 200<br>O18 r 1000<br>corner corner<br>Geolocation strategy Geolocation strategy<br><!-- End of picture text -->
<!-- Start of picture text --> boundary points centroid convex hull<br>0.30 ¢<br>, } |<br>O 0.25 *<br>S)<br>= ) e | | SS 4 ; {<br>‘ov 0.20 + ; j t<br>ror a eT a a |<br>Eos ——— — ; | =<br>‘S 0.10 , — t [ j<br>i a a ee |<br>© 0.05 $ . $ i<br>0.00<br>corner hull mean plot points<br>0.30 t 3 ;<br>9 0.25 r (J ( )<br>= ® ¢ t ‘ j * {<br>a?;;;$<br>E0.15 — | l b } , : 3 —~<br>‘S“ 0.10 $ ’ ‘; ¢ ———$ . i { § 4<br>1. ? ® ° 4<br>— ——<br>0.00<br>0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20<br>plot mean Area threshold Area threshold<br>0.30 + +<br>): i<br>g °<br>= 025 ae; ’ Number of plots<br>‘v 0.20 ee ee ee es (rounded to<br>= | | nearest 1000)<br>wn4 0.15 R ; ; pe j e 1000<br>‘6 0.10 —— oe e 2000<br>x | 3 * 3000<br>© 0.05 } e 4000<br>{<br>0.00<br>0.00 0.05 0.10 0.15 0.20<br>Area threshold<br><!-- End of picture text -->
<!-- Start of picture text --> boundary points centroid convex hull<br>0.30 +<br>=Q 0.25 . $ Z|<br>Lv£ 0.20 FA + co + 4 ‘i ‘ 5 ; aa<br>$ 4<br>’ .) ’ r =<br>Z 0.15 , = ¢ =a = °° coe * ¢<br>fo}<br>0.10 $ r<br>0.05 WI<br>corner hull mean plot points<br>0.30 = om<br>[e)cy): 0.25 = = ax ae =,+<br>2 0.20 es = , Ss ;<br>oO 4 r ¢<br>65uo 0.15 ry $ uM + ns +<br>fo) =<br>0.10 ’<br>0.05<br>both opt sar both opt sar<br>soEnean Features type Features type<br>Y 0.30 i$ + O Number of plots<br>625<br>v23" a Mmmmm 35281856<br>20.20 aie a mE 7030<br>un<br>ic,:© 0.15 4 ’ ae, 4<br>°<br>0.10<br>0.05<br>both opt sar<br>Features type<br><!-- End of picture text -->
<!-- Start of picture text --> &, |<br><!-- End of picture text -->

<!-- Start of picture text -->
1.0 e<br>o<br>$09 o*<br>3 . e .<br><<br>os‘ oe ° a‘ bg<br>3 e<br>0.7<br>a °. °<br>ro ° e e<br>| 0.6 : bd<br>(@} °°<br>e<br>0 100 200 300 400 500 600 700 800<br>Number of plots<br><!-- End of picture text -->

<!-- Start of picture text --> °<br>0.6<br>Oo<br>Ss e<br>€ ° p| * °<br>2 0.4 "<br>0.2 .<br>¢ e . °<br>6= 0.0 —« ° ° ad id. BS<br>+ | e<br>ar Yor e e<br>0 100 200 300 400 500 600 700 800<br>Number of plots<br><!-- End of picture text -->
satellite feature set that performed best for each geolocation method (Table 7). Henceforth in this section, each model is referenced by the name of the geolocation method.

Table 7: Testing scenarios used for model training

|**Geolocation method**|**Area threshold**|**Satellite features**|**Out of sample MCC**|
|---|---|---|---|
|Boundary points|0|Optical only|0.21|
|Centroid|0.05|Optical and SAR|0.24|
|Convex hull|0.2|Optical only|0.21|
|Corner|0.05|Optical only|0.23|
|Hull mean|0|Optical only|0.25|
|Plot points|0|Optical only|0.24|
|Plot mean|0.05|Optical only|0.26|

Each model was then used to estimate the probability that each 10-meter pixel in Malawi was maize (a 0 to 1 continuous variable) during the 2018/19 rainy season. The pixel-level maize probabilities were converted into a binary classification using a threshold. Pixels with a maize probability above 0.6 were classified as maize, and otherwise are classified as non-maize. Absent objective data on which to empirically calibrate the classification threshold value, we selected a threshold higher than the typical value (0.5) in order to reduce the overclassification of pixels in maize resulting from the overrepresentation of maize plots in our training data set. Data set users can select a threshold value that suits their use case.

We used these maizeland maps in conjunction with a cropland mask (showing seasonal cropland coverage) trained on crowdsourced land cover labels (see Appendix) over Malawi to estimate which pixels were cropped with maize in a particular season. Specifically, we first used the cropland mask to remove all pixels in Malawi that were not cropped. We then used each of our trained maize classification models to identify cropped pixels where maize was present. The process resulted in seven different maizeland maps, one for each geolocation method. We calculated the sum of maize pixel areas in the country separately for each geolocation method (Table 8).

31
Table 8: Malawi maize area as obtained by seven different classification models

|**Classification model**|**Out of sample MCC**|**Total maize area - 2018/19**<br>**rainy season (million ha)**|
|---|---|---|
|Boundary points|0.21|2.27|
|Centroid|0.24|2.17|
|Convex hull|0.21|2.46|
|Corner|0.23|2.15|
|Hull mean|0.25|1.94|
|Plot points|0.24|2.41|
|Plot mean|0.26|1.99|
|Mean across models|0.23|2.19|

We observed in the maize area tabulations that the convex hull and plot points geolocation methods tended to over-classify pixels as maize - consistent with early observations in this paper. Hull mean and plot mean methods showed the most conservative area estimates, possibly because these models take advantage of most information from a single plot while also preventing overrepresentation of maize in the training data.

Furthermore, Table 9 below shows the relative difference between the plot mean method and each other maize classification method. We chose the “plot mean” method to represent the “best available” method and for each other model we tallied the number of pixels (and their total area) that disagreed with its maize/non-maize prediction. For example, pixels classified as one crop type by the plot mean method, but as the other crop type by centroid method, are considered to be in disagreement.

Table 9: Area misclassified as maize/non-maize under different classification models, as compared to ‘Plot mean’ which was the best performing model

|**Classification model**|**Difference in out of**<br>**sample MCC**|**Total area with**<br>**disagreement (million ha)**|
|---|---|---|
|Boundary points|-0.05|0.84|
|Centroid|-0.02|0.48|
|Convex hull|-0.05|0.69|
|Corner|-0.03|0.95|
|Hull mean|-0.01|0.22|
|Plot points|-0.02|0.55|

32
These results show that while the differences in performance metrics between different modeling scenarios are not very large, small differences can multiply over space leading to substantial differences in maize area estimation. Hence, there is value in achieving small performance gains anchored in better training data.

Finally, after evaluating maize classification performance in Malawi and Ethiopia, we generated 10-meter resolution rasters of area cultivated with maize for both countries over the period of 2016-2019. Table 10 provides an overview of these rasters. After creating the rasters of probability of maize cultivation, we generated binary maizeland masks for each country and season in two steps. We first used our country- and season-specific cropland rasters to remove all pixels that were not cultivated with any crops. Pixels with probability of (any) crop cultivation less than 40 percent were assumed to be non-cultivated. Subsequently, we used our country- and seasonspecific maizeland rasters to identify which of the cultivated pixels were cultivated with maize. In Malawi, pixels with probability of maize cultivation greater than or equal to 60 percent were assumed to be cultivated with maize. The comparable threshold was 50 percent in Ethiopia.

Table 10: Specifications of predicted maizeland rasters in Malawi and Ethiopia

|**Country**|**Maize classification**<br>**model specifications**|**Seasons trained on**|**Seasons predicted on**|
|---|---|---|---|
|Malawi|Plot mean geolocation<br>method,<br>0.05 ha area threshold,<br>Optical features only|2017/18 rainy season,<br>2018/19 rainy season|2015/16 rainy season<br>2016/17 rainy season<br>2017/18 rainy season<br>2018/19 rainy season|
|Ethiopia|Corner point geolocation<br>method,<br>No area threshold,<br>Optical features only|2018 meher season|2016 meher season<br>2017 meher season<br>2018 meher season<br>2019 meher season|

# **5. Conclusions**

Satellite data sources have tremendous potential for amplifying the insights available from household and farm surveys. The research presented here advances our understanding of how to collect optimal plot-level survey data that can train and validate remote sensing models for highresolution crop type mapping. Specifically, we quantify the interactive effects of (i) plot size, (ii) approach to georeferencing plot locations, and (iii) size of the training data set on the performance of a machine learning-based maize classification model.

The results show that collecting a complete plot boundary is preferable to competing approaches to georeferencing plot locations in large-scale household surveys and that seemingly-small erosion in maize classification accuracy under less preferable approaches to georeferencing plot locations consistently results in total area under maize cultivation to be overestimated - in the range of 0.16 33
to 0.47 million hectares (8 to 24 percent) in Malawi vis-a-vis the results from the best performing model (i.e. plot mean). The analysis reveals that collecting GPS coordinates of the complete set of plot corners, as a second-best strategy, can approximate full plot boundaries and can in turn train models with comparable performance.

Furthermore, when only a few observation plots (fewer than 1,000 plots) can be visited, full plot boundaries or multiple corner points provide significant gains vis-a-vis plot corner points or plot centroid. With mid-sized samples (3,000 to 4,000 plots), plot centroids can produce similar performance to full plot boundaries. With large sample sizes (around 7,000 plots), plot centroids fall behind full plot boundaries. If only a single GPS point is to be gathered by data collectors, that location should be near the center of the plot rather than at the plot corner. However, georeferencing plot centroids should be understood as a third-best strategy for remote sensing model training purposes. The findings suggest that classification performance almost always peaks before or at around 4,000 plots under the preferred geolocation strategies - corresponding to roughly less than 60 percent of the training data. As such, it is better to collect high-quality plot boundaries from 4,000 plots as opposed to corner points from 7,000 plots.

Moreover, we demonstrate that no plot observations should be excluded from model training based on a minimum plot area threshold - another important note for future surveys. Finally, the experiments to quantify the effect of satellite data sources on crop type classification performance suggest that optical features alone can provide sufficient signal to maximize prediction quality. We observed only small differences between models built only with optical features and those using optical and SAR features. In the case of maize area mapping in Malawi, the potential benefits offered by SAR - providing signals unaffected by cloud cover - were offset by additional noise introduced with SAR imagery.

Many outstanding questions remain for future research. Crop classification accuracies of 0.9 or greater are not unusual in the literature, though small plot sizes in the African context may limit realistically attainable accuracy. Nevertheless, attempting to improve the accuracy of maize classification and gauging the sensitivity of our recommendations for alternative crops and countries should be among the foci of future research. Improvements may be gained by removing pixels with few cloud-free observations during the growing season. Experimenting with alternative machine learning approaches and an expanded set of geospatial covariates may increase performance as well. Further work is also needed to distinguish between intercropped and monocropped (“pure stand”) maize plots in order to a) improve classification performance, b) support the creation of intercrop maps and area estimates, and c) lead to continued refinements of downstream research related also to satellite-based crop yield estimation. Related to the latter, future research should similarly identify the minimum-required volume of and approach to survey data collection that would yield optimal data for training and validating remote sensing models for high-resolution crop yield estimation. Moreover, documenting the accuracy of out-of-season 34
predictions (e.g. using data collected in 2018 to train a model to predict 2019 outcomes) and the extent of decay in model accuracy over time would reveal the required temporal frequency of ground data collection and the relative importance of capturing season-specific conditions. Finally, research on object-based classification and automated detection of plot boundaries using computer vision techniques may additionally help in reducing the data collection requirements for crop area and yield estimation.

35