Kernel smoother
A kernel smoother is a statistical technique to estimate a real valued function formula_1 as the weighted average of neighboring observed data. The weight is defined by the kernel, such that closer points are given higher weights. The estimated function is smooth, and the level of smoothness is set by a single parameter. Definitions. Let formula_2 be a kernel defined by where: Popular kernels used for smoothing include parabolic (Epanechnikov), Tricube, and Gaussian kernels. Let formula_7 be a continuous function of "X". For each formula_8, the Nadaraya-Watson kernel-weighted average (smooth "Y"("X") estimation) is defined by where : In the following sections, we describe some particular cases of kernel smoothers. Gaussian kernel smoother. The Gaussian kernel is one of the most widely used kernels, and is expressed with the equation below. Here, b is the length scale for the input space. Nearest neighbor smoother. The idea of the nearest neighbor smoother is the following. For each point "X", take m nearest neighbors and estimate the value of "Y"("X") by averaging the values of these neighbors. Formally, formula_11, where formula_12 is the "m"th closest to "X" neighbor, and Example: In this example, "X" is one-dimensional. For each X , the formula_14 is an average value of 16 closest to "X" points (denoted by red). The result is not smooth enough. Kernel average smoother. The idea of the kernel average smoother is the following. For each data point "X", choose a constant distance size "λ" (kernel radius, or window width for "p" = 1 dimension), and compute a weighted average for all data points that are closer than formula_15 to "X" (the closer to "X" points get higher weights). Formally, formula_16 and "D"("t") is one of the popular kernels. Example: For each "X" the window width is constant, and the  weight of each point in the window is schematically denoted by the yellow figure in the graph. It can be seen that the estimation is smooth, but the boundary points are biased. The reason for that is the non-equal number of points (from the right and from the left to the "X") in the window, when the "X" is close enough to the boundary. Local linear regression. In the two previous sections we assumed that the underlying Y(X) function is locally constant, therefore we were able to use the weighted average for the estimation. The idea of local linear  regression is to fit locally a straight line (or a hyperplane for higher dimensions), and not the constant (horizontal line). After fitting the line, the estimation formula_17 is provided by the value of this line at "X" point. By repeating this procedure for each "X", one can get the estimation function formula_18. Like in previous section, the window width is constant formula_19 Formally, the local linear regression is computed by solving a weighted least square problem. For one dimension ("p" = 1): formula_20 The closed form solution is given by: where: Example: The resulting function is smooth, and the problem  with the biased boundary points is reduced. Local linear regression can be applied to any-dimensional space, though the question of what is a local neighborhood becomes more complicated. It is common to use k nearest training points to a test point to fit the local linear regression. This can lead to high variance of the fitted function. To bound the variance, the set of training points should contain the test point in their convex hull (see Gupta et al. reference). Local polynomial regression. Instead of fitting locally linear functions, one can fit polynomial functions. For p=1, one should minimize : formula_25 with formula_26 In general case (p>1), one should minimize: formula_27