Cook's distance
In statistics, Cook's distance or Cook's "D" is a commonly used estimate of the influence of a data point when performing a least-squares regression analysis. In a practical ordinary least squares analysis, Cook's distance can be used in several ways: to indicate influential data points that are particularly worth checking for validity; or to indicate regions of the design space where it would be good to be able to obtain more data points. It is named after the American statistician R. Dennis Cook, who introduced the concept in 1977. Definition. Data points with  large residuals (outliers) and/or high leverage may distort the outcome and accuracy of a regression. Cook's distance measures the effect of deleting a given observation. Points with a large Cook's distance are considered to merit closer examination in the analysis. For the algebraic expression, first define where formula_2 is the error term, formula_3 is the coefficient matrix, formula_4 is the number of covariates or predictors for each observation, and formula_5 is the design matrix including a constant. The least squares estimator then is formula_6, and consequently the fitted (predicted) values for the mean of formula_7 are where formula_9  is the projection matrix (or hat matrix). The formula_10-th diagonal element of formula_11, given by formula_12, is known as the leverage of the formula_10-th observation. Similarly, the formula_10-th element of the residual vector formula_15 is denoted by formula_16. Cook's distance formula_17 of observation formula_18 is defined as the sum of all the changes in the regression model when observation formula_10 is removed from it where formula_21 is the fitted response value obtained when excluding formula_10, and formula_23 is the mean squared error of the regression model. Equivalently, it can be expressed using the leverage (formula_24): Detecting  highly influential observations. There are different opinions regarding what cut-off values to use for spotting highly influential points. Since Cook's distance is in the metric of an "F" distribution with formula_4 and formula_27 (as defined for the design matrix formula_5 above) degrees of freedom, the median point (i.e., formula_29) can be used as a cut-off. Since this value is close to 1 for large formula_30, a simple operational guideline of formula_31 has been suggested. Note that the Cook's distance measure does not always correctly identify influential observations. Relationship to other influence measures (and interpretation). formula_17 can  be expressed using the leverage (formula_33) and the square of the "internally" Studentized residual (formula_34), as follows: The benefit in the last formulation is that it clearly shows the relationship between formula_36 and formula_24 to formula_17 (while p and n are the same for all observations). If formula_36 is large then it (for non-extreme values of formula_24) will increase formula_17. If formula_24 is close to 0 than formula_17 will be small, while if formula_24 is close to 1 then formula_17 will become very large (as long as formula_46, i.e.: that the observation formula_10 is not exactly on the regression  line that was fitted without observation formula_10). formula_17 is related to DFFITS through the following relationship (note that formula_50 is the "externally" studentized residual, and formula_51 are defined here): formula_17 can be interpreted as the distance one's estimates move within the confidence ellipsoid that represents a region of plausible values for the parameters. This is shown by an alternative but equivalent representation of Cook's distance in terms of changes to the estimates of the regression parameters between the cases, where the particular observation is either included or excluded from the regression analysis. An alternative to formula_17 has been  proposed. Instead of considering the influence a single observation has on the overall model, the statistics formula_55 serves as a measure of how sensitive the prediction of the formula_10-th observation is to the deletion of each observation in the original data set. It can be formulated as a weighted linear combination of the formula_57's of all data points. Again, the projection matrix is involved in the calculation to obtain the required weights: In this context, formula_59 (formula_60) resembles the correlation between the predictions formula_61 and formula_62. In contrast to formula_17, the distribution of formula_55 is asymptotically normal for  large sample sizes and models with many predictors. In absence of outliers the expected value of formula_55 is approximately formula_66. An influential observation can be identified if with formula_68 as the median and formula_69 as the median absolute deviation of all formula_70-values within the original data set, i.e., a robust measure of location and a robust measure of scale for the distribution of formula_55. The factor 4.5 covers approx. 3 standard deviations of formula_70 around its centre. When compared to Cook's distance, formula_70 was found to perform well for high- and intermediate-leverage outliers, even in presence of  masking effects for which formula_17 failed. Software implementations. Many programs and statistics packages, such as R, Python, etc., include implementations of Cook's distance. Extensions. High-dimensional Influence Measure (HIM) is an alternative to Cook's distance for when formula_75 (i.e., when there are more predictors than observations). While the Cook's distance quantifies the individual observation's influence on the least squares regression coefficient estimate, the HIM measures the influence of an observation on the marginal correlations.