  We distinguish two general modes of testing for Deep Neural Networks (DNNs):
Offline testing where DNNs are tested as individual units based on test
datasets obtained independently from the DNNs under test, and online testing
where DNNs are embedded into a specific application environment and tested in a
closed-loop mode in interaction with the application environment. Typically,
DNNs are subjected to both types of testing during their development life cycle
where offline testing is applied immediately after DNN training and online
testing follows after offline testing and once a DNN is deployed within a
specific application environment. In this paper, we study the relationship
between offline and online testing. Our goal is to determine how offline
testing and online testing differ or complement one another and if we can use
offline testing results to run fewer tests during online testing to reduce the
testing cost. Though these questions are generally relevant to all autonomous
systems, we study them in the context of automated driving systems where, as
study subjects, we use DNNs automating end-to-end controls of steering
functions of self-driving vehicles. Our results show that offline testing is
more optimistic than online testing as many safety violations identified by
online testing could not be identified by offline testing, while large
prediction errors generated by offline testing always led to severe safety
violations detectable by online testing. Further, we cannot use offline testing
results to run fewer tests during online testing in practice since we are not
able to identify specific situations where offline testing could be as accurate
as online testing in identifying safety requirement violations.
