  Performance prediction, the task of estimating a system's performance without
performing experiments, allows us to reduce the experimental burden caused by
the combinatorial explosion of different datasets, languages, tasks, and
models. In this paper, we make two contributions to improving performance
prediction for NLP tasks. First, we examine performance predictors not only for
holistic measures of accuracy like F1 or BLEU but also fine-grained performance
measures such as accuracy over individual classes of examples. Second, we
propose methods to understand the reliability of a performance prediction model
from two angles: confidence intervals and calibration. We perform an analysis
of four types of NLP tasks, and both demonstrate the feasibility of
fine-grained performance prediction and the necessity to perform reliability
analysis for performance prediction methods in the future. We make our code
publicly available: \url{https://github.com/neulab/Reliable-NLPPP}
