  Explainable Artificial Intelligence (XAI) is an emerging research topic of
machine learning aimed at unboxing how AI systems' black-box choices are made.
This research field inspects the measures and models involved in
decision-making and seeks solutions to explain them explicitly. Many of the
machine learning algorithms can not manifest how and why a decision has been
cast. This is particularly true of the most popular deep neural network
approaches currently in use. Consequently, our confidence in AI systems can be
hindered by the lack of explainability in these black-box models. The XAI
becomes more and more crucial for deep learning powered applications,
especially for medical and healthcare studies, although in general these deep
neural networks can return an arresting dividend in performance. The
insufficient explainability and transparency in most existing AI systems can be
one of the major reasons that successful implementation and integration of AI
tools into routine clinical practice are uncommon. In this study, we first
surveyed the current progress of XAI and in particular its advances in
healthcare applications. We then introduced our solutions for XAI leveraging
multi-modal and multi-centre data fusion, and subsequently validated in two
showcases following real clinical scenarios. Comprehensive quantitative and
qualitative analyses can prove the efficacy of our proposed XAI solutions, from
which we can envisage successful applications in a broader range of clinical
questions.
