  Automated Machine Learning (AutoML) is a rapidly growing set of technologies
that automate the model development pipeline by searching model space and
generating candidate models. A critical, final step of AutoML is human
selection of a final model from dozens of candidates. In current AutoML
systems, selection is supported only by performance metrics. Prior work has
shown that in practice, people evaluate ML models based on additional criteria,
such as the way a model makes predictions. Comparison may happen at multiple
levels, from types of errors, to feature importance, to how the model makes
predictions of specific instances. We developed \tool{} to support interactive
model comparison for AutoML by integrating multiple Explainable AI (XAI) and
visualization techniques. We conducted a user study in which we both evaluated
the system and used it as a technology probe to understand how users perform
model comparison in an AutoML system. We discuss design implications for
utilizing XAI techniques for model comparison and supporting the unique needs
of data scientists in comparing AutoML models.
