  In this paper, we explore using runtime verification to design safe
cyber-physical systems (CPS). We build upon the Simplex Architecture, where
control authority may switch from an unverified and potentially unsafe advanced
controller to a backup baseline controller in order to maintain system safety.
New to our approach, we remove the requirement that the baseline controller is
statically verified. This is important as there are many types of powerful
control techniques -- model-predictive control, rapidly-exploring random trees
and neural network controllers -- that often work well in practice, but are
difficult to statically prove correct, and therefore could not be used before
as baseline controllers. We prove that, through more extensive runtime checks,
such an approach can still guarantee safety. We call this approach the
Black-Box Simplex Architecture, as both high-level controllers are treated as
black boxes. We present case studies where model-predictive control provides
safety for multi-robot coordination, and neural networks provably prevent
collisions for groups of F-16 aircraft, despite occasionally outputting unsafe
actions.
