The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate the performance of novel algorithms and applications. Strong evaluation methods should explore the diverse scenarios of interaction between humans and robots . We propose quality diversity (QD) algorithms as a method for simultaneously exploring both environments and human actions to discover diverse failure scenarios . We focus on the shared autonomy domain, where the robot attempts to infer the goal of a human operator . We evaluate our approach by automatically generating scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that QD outperforms Monte-Carlo simulation and optimization based methods in effectively searching the scenario space, highlighting its promise for automatic evaluation of algorithms in shared autonomy.
The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring the diverse scenarios of interaction between humans and robots in simulation can improve understanding of the system and avoid potentially costly failures in real-world settings. We formulate this as a quality diversity (QD) problem, where the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions . We focus on the shared autonomy domain, where the robot attempts to infer the goal of a human operator , and adopt the QD algorithm MAP-Elites to generate scenarios for two published algorithms in this domain: shared autonomy via hindsight optimization and linear policy blending. Some of the generated scenarios confirm previous theoretical findings, while others are surprising and bring about a new understanding of state-of-the-art implementations. Our experiments show that MAP-Elites outperforms Monte-Carlo simulation and optimization based methods in effectively searching the scenario space, highlighting its promise for automatic evaluation of algorithms in shared autonomy.
D
the performance of
R
Strong evaluation methods should explore
Exploring
R
. We propose
in simulation can improve understanding of the system and avoid potentially costly failures in real-world settings. We formulate this as a
R
algorithms as a method for
problem, where the goal is to discover diverse failure scenarios by
D
to discover diverse failure scenarios
R
. We evaluate our approach by automatically generating
, and adopt the QD algorithm MAP-Elites to generate
R
QD
MAP-Elites