  Major depressive disorder is a debilitating disease affecting 264 million
people worldwide. While many antidepressant medications are available, few
clinical guidelines support choosing among them. Decision support tools (DSTs)
embodying machine learning models may help improve the treatment selection
process, but often fail in clinical practice due to poor system integration.
  We use an iterative, co-design process to investigate clinicians' perceptions
of using DSTs in antidepressant treatment decisions. We identify ways in which
DSTs need to engage with the healthcare sociotechnical system, including
clinical processes, patient preferences, resource constraints, and domain
knowledge. Our results suggest that clinical DSTs should be designed as
multi-user systems that support patient-provider collaboration and offer
on-demand explanations that address discrepancies between predictions and
current standards of care. Through this work, we demonstrate how current trends
in explainable AI may be inappropriate for clinical environments and consider
paths towards designing these tools for real-world medical systems.
