  Artificial intelligence (AI) systems have become increasingly common and the
trend will continue. Examples of AI systems include autonomous vehicles (AV),
computer vision, natural language processing, and AI medical experts. To allow
for safe and effective deployment of AI systems, the reliability of such
systems needs to be assessed. Traditionally, reliability assessment is based on
reliability test data and the subsequent statistical modeling and analysis. The
availability of reliability data for AI systems, however, is limited because
such data are typically sensitive and proprietary. The California Department of
Motor Vehicles (DMV) oversees and regulates an AV testing program, in which
many AV manufacturers are conducting AV road tests. Manufacturers participating
in the program are required to report recurrent disengagement events to
California DMV. This information is being made available to the public. In this
paper, we use recurrent disengagement events as a representation of the
reliability of the AI system in AV, and propose a statistical framework for
modeling and analyzing the recurrent events data from AV driving tests. We use
traditional parametric models in software reliability and propose a new
nonparametric model based on monotonic splines to describe the event process.
We develop inference procedures for selecting the best models, quantifying
uncertainty, and testing heterogeneity in the event process. We then analyze
the recurrent events data from four AV manufacturers, and make inferences on
the reliability of the AI systems in AV. We also describe how the proposed
analysis can be applied to assess the reliability of other AI systems.
