  There has been increasing interest in building deep hierarchy-aware
classifiers that aim to quantify and reduce the severity of mistakes, and not
just reduce the number of errors. The idea is to exploit the label hierarchy
(e.g., the WordNet ontology) and consider graph distances as a proxy for
mistake severity. Surprisingly, on examining mistake-severity distributions of
the top-1 prediction, we find that current state-of-the-art hierarchy-aware
deep classifiers do not always show practical improvement over the standard
cross-entropy baseline in making better mistakes. The reason for the reduction
in average mistake-severity can be attributed to the increase in low-severity
mistakes, which may also explain the noticeable drop in their accuracy. To this
end, we use the classical Conditional Risk Minimization (CRM) framework for
hierarchy-aware classification. Given a cost matrix and a reliable estimate of
likelihoods (obtained from a trained network), CRM simply amends mistakes at
inference time; it needs no extra hyperparameters and requires adding just a
few lines of code to the standard cross-entropy baseline. It significantly
outperforms the state-of-the-art and consistently obtains large reductions in
the average hierarchical distance of top-$k$ predictions across datasets, with
very little loss in accuracy. CRM, because of its simplicity, can be used with
any off-the-shelf trained model that provides reliable likelihood estimates.
