  This paper develops a hierarchical reinforcement learning architecture for
multi-mission spaceflight campaign design under uncertainty, including vehicle
design, infrastructure deployment planning, and space transportation
scheduling. This problem involves a high-dimensional design space and is
challenging especially with uncertainty present. To tackle this challenge, the
developed framework has a hierarchical structure with reinforcement learning
(RL) and network-based mixed-integer linear programming (MILP), where the
former optimizes campaign-level decisions (e.g., design of the vehicle used
throughout the campaign, destination demand assigned to each mission in the
campaign), whereas the latter optimizes the detailed mission-level decisions
(e.g., when to launch what from where to where). The framework is applied to a
set of human lunar exploration campaign scenarios with uncertain in-situ
resource utilization (ISRU) performance as a case study. The main value of this
work is its integration of the rapidly growing RL research and the existing
MILP-based space logistics methods through a hierarchical framework to handle
the otherwise intractable complexity of space mission design under uncertainty.
We expect this unique framework to be a critical steppingstone for the emerging
research direction of artificial intelligence for space mission design.
