  While the need for interpretable machine learning has been established, many
common approaches are slow, lack fidelity, or hard to evaluate. Amortized
explanation methods reduce the cost of providing interpretations by learning a
global selector model that returns feature importances for a single instance of
data. The selector model is trained to optimize the fidelity of the
interpretations, as evaluated by a predictor model for the target. Popular
methods learn the selector and predictor model in concert, which we show allows
predictions to be encoded within interpretations. We introduce EVAL-X as a
method to quantitatively evaluate interpretations and REAL-X as an amortized
explanation method, which learn a predictor model that approximates the true
data generating distribution given any subset of the input. We show EVAL-X can
detect when predictions are encoded in interpretations and show the advantages
of REAL-X through quantitative and radiologist evaluation.
