Learning Crop Management by Reinforcement: gym-DSSAT
Abstract
We introduce gym-DSSAT , a gym environment for crop management tasks, that is easy to use for training Rein- forcement Learning (RL) agents.
gym-DSSAT is based on
DSSAT , a state-of-the-art mechanistic crop growth simulator.
We modify DSSAT so that an external software agent can interact with it to control the actions performed in a crop field during a growing season.
The RL environment provides predefined decision problems without having to manipulate the complex crop simulator.
We report encouraging preliminary results on a use case of nitrogen fertilization for maize.
This work opens up opportunities to explore new sustainable crop management strategies with RL, and provides RL researchers with an original set of challenging tasks to investigate.
Introduction
During a growing season, a farmer performs a sequence of operations in her field in order to reach certain production objectives (Sebillotte 1974, 1978).
She makes these decisions under uncertainty, like unknown weather changes.
Reinforcement Learning (RL) addresses such problems where an agent learns to control the evolution of an unknown and uncertain dynamical system, in order to perform a given task (Sutton and Barto 2018).
In RL, addressing a complex real-world problem usually starts with the use of a highfidelity simulator which mimics real learning conditions.
We present gym-DSSAT , an RL environment based on a celebrated high-fidelity crop model, the Decision Support
System for Agrotechnology Transfer ( DSSAT , Hoogenboom et al. 2019) crop model.
Learning sustainable crop management practices is not a trivial task.
For example, nitrogen fertilization requires minimal rainfall and temperature following the application for the fertilized nitrogen to become available to plants.
Future meteorological conditions are not known with certainty at the time of fertilization decisions.
For an efficient nitrogen fertilizer management, available nitrogen in soil must match plant uptake, both in time and quantity (Meisinger and Delgado 2002).
RL is an appealing approach to help decision-makers to learn more sustainable crop management practices (Binas, Luginbuehl, and Bengio 2019; Gautron et al. 2022a)
Contributions.
We introduce gym-DSSAT , a crop management simulator to be used to train RL agents based on the DSSAT crop model system.
gym-DSSAT features three predefined problems.
We provide preliminary experimental results indicating that RL is an interesting way to discover original and efficient crop management strategies.
As another contribution, Gautron et al.
(2022b) details the original methods that we designed to turn DSSAT –a large mechanistic model written in Fortran–, into a Python gym environment (not discussed in this article).
More information is available on gym-DSSAT GitLab 1 , including installation instructions for various operating systems, and tutorials.
Related works.
The first case of an RL agent interacting with a crop simulator in order to learn crop management is found in Garcia (1999).
The author used a modification of the D´ecibl´e crop model (Chatelin et al. 2005).
The RL agent learned wheat sowing and nitrogen fertilization under pollution constraints.
During simulations, weather series were stochastically generated.
The modified version of
D´ecibl´e is not available anymore.
In Garcia (1999), the
RL agent did not outperform the crop management policy of an expert.
Recently, several works directly used crop models or surrogate models as RL environments (e.g. Sun et al. ; Wang, He, and Luo 2020; Chen et al. 2021).
However, none of these works has provided an open source and standardized crop management RL environment.
Overweg, Berghuijs, and Athanasiadis (2021) proposed CropGym , a gym interface to train an agent to perform wheat nitrogen fertilization.
The environment uses the Python Crop Simulation Environment (PCSE) LINTUL3 (Shibu et al. 2010) wheat crop model.
Fertilization is treated as a weekly choice of a discrete amount of fertilizer to apply.
In CropGym , simulations use a limited set of historical weather records, which may favor overfitting due to limited randomness, compared with the use of a stochastic weather generator, especially for data intensive algorithms used in deep RL.
Formalizing decision-making problems in RL
In most cases, RL uses a Markov Decision Process (MDP, Puterman 1994) formulation of the environment.
An MDP defines a class of controllable dynamical system.
An agent learns to control the system to optimize a certain objective function J .
At each discrete time step t ∈ { 1 , 2 , · · · , N } , N ≤∞ , the system is in some state s t ∈ S in which one action a t from a set of actions A is performed by the agent.
Then, the system transits into its next state s t [-+
-]+ according to a transition function p ( s, a, s ′ ) , which specifies the probability of the system to transit to state s ′ after action a was performed in state s .
After an action a t has been performed, a return r t ∈ R is provided to the agent according to the return function r ( s, a, s ′ ) .
The goal of an RL agent is to learn an optimal policy π ⋆ ( s ) that specifies which action should be performed in each state, in order to optimize J .
For example, when N < ∞ , the objective function can be defined as the sum of returns: J = (cid:80) N t =1 r t .
In RL, neither p nor r is known.
The agent learns an optimal policy by interacting with its environment, i.e. the dynamical system to control.
The agent tries actions to learn their consequences and, progressively, focuses on the best actions to perform to maximize J .
Current state-of-the-art RL algorithms are known to be actor-critics, such as PPO, A2C and
SAC (Kiran et al. 2021).
gym environments.
OpenAI gym (Brockman et al. 2016) is an open source toolkit initially developed by the Open AI company.
It provides light RL environments with a standardized Application Programming Interface (API).
gym API became a reference in the RL community to create standardized RL environments in order to compare performances of RL algorithms.
The user interacts with the environment through standardized methods.
The agent interacts with the environment by calling the step() method with argument a t specifying the action to take, in order to receive s t +1 and r t .
The objective function J is defined by the user.
Crop management problems in gym-DSSAT
DSSAT is meant to simulate one crop field during one growing season.
The growth of one cultivar is simulated per unit of surface.
Hence, the whole area of the field is meaningless, and the properties of the soil and the weather conditions are supposed to be identical in the whole field.
Initial conditions may be set to simulate crop rotation along seasons.
By default, gym-DSSAT simulates a maize experiment which has been carried out in 1982 in the experimental farm of the University of Florida, Gainesville, USA (Bennett et al. 1989).
An episode lasts a simulated growing season.
A simulation starts prior to planting and ends at crop harvest, which is automatically defined as the crop maturity date.
Crop maturity depends on crop growth, which depends itself on crop management and weather events, and the time to reach it is stochastic.
During a growing season (160 days on average), an RL agent daily decides on the crop management action(s) to perform: fertilize and/or irrigate.
[Table] [Table]
[Figure] [Table]
By default, for each episode, the weather is generated by the WGEN stochastic weather simulator (Richardson 1985;
Soltani and Hoogenboom 2003).The duration between the starting date of the simulation and the planting date, which lasts about one month, induces stochastic soil conditions at the time of planting (e.g. soil nitrate, and soil water content), as a result of the stochastic weather events.
The number of measurable attributes in a field is extremely large (e.g. Husson et al. 2021).
Based on agronomic knowledge, we selected a subset of DSSAT state variables with the constraint that these variables are measurable or can be estimated in real conditions.
These observation variables are mixed, and take either continuous or discrete values.
In DSSAT , the WGEN stochastic weather simulator is implemented as a firstorder Markov chain, but all other processes are deterministic.
Therefore, gym-DSSAT decision problems are Markovian.
Because the agent only accesses a subset of all DSSAT internal variables, a gym-DSSAT problem is a Partially Observable MDP (POMDP, ˚Astr ¨ om 1965), a situation alike the one faced by farmers.
In contrast with many RL toy environments, the environment is autonomous: it evolves by itself and not only because an action has been performed by the agent.
Indeed, if on a given day a farmer does not fertilize/irrigate, the plot still evolves.
DSSAT simulates the dynamics at the plot level.
Likewise, the agent performs actions on the whole plot.
Growing conditions such as soil characteristics and other crop operations such as soil tillage and cultivar choices are fixed.
By default, gym-DSSAT provides three RL problems: 1.
A fertilization problem in which the agent can apply every day a certain quantity of nitrogen (Table 1).
Crops are rainfed, and no irrigation is applied during the growing season, except a single one before planting.
We crafted the default fertilization return function as:
[Equation]
An irrigation problem in which the agent can provide every day a certain amount of water to irrigate, as indicated in Table 1.
Independently of these irrigation actions, nitrogen fertilization occurs following the schedule provided in Table 2.
[Table]
A mixed fertilization and irrigation problem which combines both the aforementioned decision problems, i.e. the agent can fertilize and/or irrigate every day.
Custom scenario definition.
A user can easily modify the observation space in the YAML configuration file.
In the same way, the definition of the return functions can be easily modified by the user by editing a standalone Python file.
Built-in DSSAT features can be directly leveraged, such as environmental modifications with changes in atmospheric
CO 2 concentration or meteorological features, to mimic the effects of climate change.
It is also possible to customize the soil conditions of the crop field, its initial conditions, as well as the weather along the season either reproducing the precipitations from a dataset, or by using the WGEN simulator.
Experimenting with gym-DSSAT A use case: learning an efficient maize fertilization
We present an example of how to address the default fertilization task .
The source code of these experiments is available in gym-DSSAT GitLab. We used gym-DSSAT v0.0.7 .
Methods.
An episode spans one growing season, i.e. a finite number of time steps, we define the objective function as the sum of returns J .
As a common practice, we pragmatically approximate this decision problem as an MDP, even though it is a POMDP.
Table 3 provides the observa - tion space.
We consider three policies
: • The “null” policy that never fertilizes .
As there is always nitrogen in soil before cultivation (Morris et al. 2018), without mineral fertilization, the reference experiment, or control , is usually the null policy.
Agronomists measure the effect of a nitrogen fertilization policy as a gain compared to the null policy, in order to decouple the effect of nitrogen fertilization from the effect of already available nitrogen in soil (Vanlauwe et al. 2011).
• An “expert” policy published in the original maize field experiment (Bennett et al. 1989) and defined in Table 2.
This expert policy consists of three deterministic nitrogen fertilizer applications, which only depends on the number of days after planting.
• A policy learned by the Proximal Policy Optimization (PPO, Schulman et al. 2017)
RL algorithm, as implemented in Stable-Baselines3 1.4.
(Hill et al. 2018).
As our goal is to establish a simple baseline, we use the default hyper-parameter values for PPO.
We trained PPO during 10 6 episodes.
During training, the performance of PPO is evaluated on a validation environment every 10 3 episodes.
We seed the validation environment with a different seed than for the training environment.
Consequently, the validation environment generates a different sequence of weather series compared to the training environment.
The model with the best validation performance is saved as the result of the training.
In order to compare fertilization policies, we measure their performances by running them for 10 3 episodes on a test environment.
With regards to the training environment, the test environment is the same except for the seed of the pseudo-random number generator.
In the performance analysis of policies, the evolution of returns r t provides information about the learning process from an RL perspective, but returns are not directly interpretable from an agronomic perspective.
Performance analysis of crop management strategies require multiple evaluation criteria (Dor ´ e et al.
Duru et al. 2015).
To remedy this problem, we use a subset of DSSAT internal state variables as performance indicators (Table 4).
Note that these variables are not necessarily contained in the observation space of the fertilization problem (Table 3) because we use them for another purpose than algorithm training.
Each of these performance criteria is correlated with the other ones.
For instance, increasing the total fertilizer amount is likely to increase the grain yield, but it is also likely to increase the pollution induced by nitrate leaching.
The agronomic nitrogen-use efficiency (ANE, Vanlauwe et al. 2011) is a common indicator of fertilization sustainability.
For a policy π , let grnwt π be the dry matter grain yield of the policy π (kg/ha), grnwt 0 be the dry matter grain yield with no fertilization (kg/ha), and cumsumfert π be the total fertilizer quantity applied with policy π
(kg/ha), we have:
[Equation]
ANE indicates the grain yield response with respect to the null policy provided by each unit of nitrogen fertilizer.
ANE is a key metric of sustainable fertilization.
Maximizing ANE relates to the economic and environmental aspects, and leads to an efficient use of fertilizer, which limits the risks of pollution.
Performance indicators listed in Table 4 show a complex trade-off between conflicting objectives.
Results.
Figure 1 illustrates the evolution of the objective function J against the day of simulation.
PPO outperforms the two other policies.
The performance obtained by PPO learned policy is less variable than that of the expert policy.
Figure 2 provides a 2D histogram of fertilizer applications, against the day of simulation.
PPO nitrogen fertilizer applications are more frequent at the beginning of the growing season and around day of simulation 60.
This date corresponds to the beginning of the floral initiation stage.
Nevertheless, the variability of rates and application dates of PPO
Variable
Definition
Comment
[Table] [Table]
[Figure]
policy shows that it does not depend solely on the number of days after planting as the expert policy, but also depends on other factors . Table 5 provides statistics of the performance indicators mentioned in Table 4. As expected, no policy is optimal for all the performance criteria.
PPO policy exhibits a good performance trade-off between the expert and the null policies.
Grain yield and nitrogen content in grains (a nutritional criteria) are close to those of the expert policy.
On average, PPO policy consumes about 28% less nitrogen than the expert policy.
Consistently, ANE for PPO is about 29% larger than that of the expert policy.
From a practical perspective, a good fertilization policy consists of a limited number of applications of the fertilizer as the expert policy suggests.
Indeed, each application costs in terms of fertilizer and its application.
The mean number of applications of PPO ( ∼ 6 ) is larger than for the expert policy ( 3 ) but still remains manageable.
Execution time.
We performed all the experiments with gym-DSSAT on a standard 8-core laptop.
The mean running time to simulate one day in gym-DSSAT , i.e. taking a single step in the environment, is 2 .
56 ± 0 .
22 ms.
Thus, each interaction is fast and this allows a large number of interactions to be considered for training the agent. [Figure]
Reproducibility.
We successfully reproduced the results of this study on the same hardware and software layers.
This means that both results of gym-DSSAT and
Stable-Baselines3 PPO are reproducible on the same platform.
Nevertheless, as a more general reproducibility issue, we cannot guarantee the cross-platform reproducibility of the experiments that we presented.
If we consider only gym-DSSAT , we have successfully reproduced the outputs of the environment across various Linux platforms.
Concluding remarks
We presented gym-DSSAT , a gym environment to train RL agents for realistic crop management tasks .
gym-DSSAT provides the RL community with a state-of-the-art crop simulator that features original challenges.
The preliminary results, which we present here, confirm that, in simulated conditions, RL can discover interesting crop management policies.
gym-DSSAT also allows world-wide growing conditions to be mimicked, using already widely available DSSAT simulation files.
gym-DSSAT can be an important tool for addressing the ongoing challenges of sustainable crop intensification through improved crop management, including those in the Global South.
Acknowledgments
The authors acknowledge the PDI team, in particular Karol
Sierocinski, for their help.
They also thank the DSSAT development team, especially Cheryl Porter for her continuous support.
We acknowledge the Consultative Group for International Agricultural Research (CGIAR) Platform for Big
Data in Agriculture and we especially thank Brian King.
Ph.
Preux, and O-A. Maillard acknowledge the support of the M´etropole Europ´eenne de Lille (MEL), ANR, Inria, Universit´e de Lille, through the AI chair Apprenf number R-PILOTE-19-004-APPRENF.
We acknowledge the support of the AIDA team at CIRAD and the outstanding working environment provided by Inria in the Scool research group.
Emilio J. Padr´on’s work was partially supported through the research projects PID2019-104184RBI00 funded by MCIN/AEI/10.13039/ and
ED431C 2021/30, ED431F 2021/11 and ED431G 2019/ funded by Xunta de Galicia.
We thank Debabrota Basu for his help in the writing of this paper.