  In hybrid human-AI systems, users need to decide whether or not to trust an
algorithmic prediction while the true error in the prediction is unknown. To
accommodate such settings, we introduce RETRO-VIZ, a method for (i) estimating
and (ii) explaining trustworthiness of regression predictions. It consists of
RETRO, a quantitative estimate of the trustworthiness of a prediction, and VIZ,
a visual explanation that helps users identify the reasons for the (lack of)
trustworthiness of a prediction. We find that RETRO-scores negatively correlate
with prediction error across 117 experimental settings, indicating that RETRO
provides a useful measure to distinguish trustworthy predictions from
untrustworthy ones. In a user study with 41 participants, we find that
VIZ-explanations help users identify whether a prediction is trustworthy or
not: on average, 95.1% of participants correctly select the more trustworthy
prediction, given a pair of predictions. In addition, an average of 75.6% of
participants can accurately describe why a prediction seems to be (not)
trustworthy. Finally, we find that the vast majority of users subjectively
experience RETRO-VIZ as a useful tool to assess the trustworthiness of
algorithmic predictions.
