Efficient super-resolution via image warping

This paper introduces a new algorithm for enhancing image resolution from an image sequence. The approach we propose herein uses the integrating resampler for warping. The method is a direct computation, which is fundamentally different from the iterative back-projection approaches proposed in previous work. This paper shows that image-warping techniques may have a strong impact on the quality of image resolution enhancement. By coupling the degradation model of the imaging system directly into the integrating resampler, we can better approximate the warping characteristics of real sensors, which also significantly improve the quality of super-resolution images. Examples of super-resolutions are given for gray-scale images. Evaluations are made visuallyby comparing the resulting images and those using bi-linear resampling and back-projection and quantitativelyusing OCR as a fundamental measure. The paper shows that even when the images are qualitatively similar, quantitative differences appear in machine processing. q 2000 Elsevier Science B.V. All rights reserved. Keywords: Super-resolution; Imaging-consistent restoration/reconstruction; Integrating resampling; Integrating resampler; Quantitative evaluatio n; OCR; Bilinear resampling; Image reconstruction; Image restoration; Image warping 1. Background and introduction The basic premise of super-resolution is to combine images (and in some work scene/data models) to produce an image that has higher “resolution” than the original image. It is not sufficient to just resample the image—size is not the same as resolution. Increasing resolution could be viewed as either increasing the signal-to-noise ratio while keeping the size fixed and/or approximating the image at a larger size with reasonable approximations for frequencies higher than those representable at the original size. In either case, it requires additional information and assumptions. There are, of course, some fundamental limits on what can be gained; lens and photo-sites cause blur, and the discrete sampling causes frequencies above the Nyquist rate to alias. While in general the aliasing is theoretically impossible to separate, the problem occurs mostly along sharp “intensity edges”, so multiple images with slightly different imaging positions can be used to reduce its impact. There are two different groups of research that use the term super-resolution—one is based on multiple images, which will be considered within this paper; the other uses “scene/object” models. Early research on image-based super-resolution dates back to the work by Huang and Tsai [8]. They solved the problem in the frequency domain, but they disregarded sensor blurring. Furthermore, they assumed only inter-frame translations. Gross [7] solved the problem by first merging the low resolution images by interpolation and then obtaining the high-resolution image by deblurring the merged image. Again, only inter-frame translations were considered. Peleg and Keren [11,14] solved the problem by first estimating an initial guess of the high-resolution image, then simulating the imaging process and minimizing the difference between the observed and simulated low-resolution images. Irani and Peleg [9,10] used a back-projection method similar to that used in tomography to minimize the same difference. Bascle et al. [1] used the same back-projection method to minimize the difference between the predicted and actual images except that a simple motion blur model was included. All previous work ignores the impact of image warping techniques. In this paper, we show that warping techniques may have a strong impact on the quality of image resolution enhancement. Image warping has been around almost as long as image processing, allowing users to reshape the image geometry. Image warping requires the underlying image to be “resampled” at non-integer locations; hence, it requires image reconstruction. When the goal of warping is to produce output for human viewing, only moderately accurate image intensities are needed. In these cases, techniques Image and Vision Computing 18 (2000) 761–771 0262-8856/00/$ see front matter q 2000 Elsevier Science B.V. All rights reserved. PII: S0262-8856(99)00044-X www.elsevier.com/locate/imavis * Corresponding author. Tel.: 11-610-7584061. E-mail address:tboult@eecs.lehigh.edu (T.E. Boult). using bi-linear interpolation have been found sufficient. However, as a step for applications such as super-resolution, the precision of the warped intensity values is often important. For these problems, bi-linear image reconstruction may not be sufficient. This paper shows how the integrating resampler can be used to enhance image resolution. This builds upon the ideas of imaging-consistent restoration/reconstruction algorithms proposed in Ref. [2] and the integrating resampler proposed in Ref. [5]. A final contribution of this paper is a quantitative comparison and analysis of our techniques. In this paper, we use a character recognition rate based on a commercial OCR program. We chose this because it is easy to make it quantitative, and we consider this to be a good measure for other applications such as license-plate reading and other pattern recognition problems. In what follows in this paper, the image formation process and the relationship between restoration, reconstruction, and super-resolution is briefly reviewed in Section 2. The integrating resampler—an efficient method for warping using imaging-consistent reconstruction/restoration algorithms—is given in Section 3. This algorithm is well suited for today’s pipelined micro-processors. In addition, the integrating resampler can allow for modifications of the intensity values to better approximate the warping characteristics of real sensors. The super-resolution algorithm where we compare the integrating resampler with bi-linear resampling is presented in Section 4. Quantitative measurement of super-resolution imaging using OCR is shown in Section 5. 2. Image formation and image restoration In this section, we briefly review the image formation process as well as the sensor model proposed in Ref. [2]. Image formation is generally described as a sequence of filtering operations. See Fig. 1. Let f x; y be the intensity distribution of a scene at the front aperture of a lens. That distribution is acted upon by h1x; y; the blurring component of the lens, yielding f1x; y: A geometric distortion functionh2x; y is then applied to produce imagef2u; v: At this point, f2 strikes the image sensor where it undergoes additional blurring by a point spread functionh3u; v to generate image f3u; v: This blurring reflects the limitation of the sensor to accurately resolve each point without the influence of neighboring points. We choose to use a simple model wherein this blurring takes place within one pixel because for CCD and CID cameras, the physical boundaries between photo-sites generally allow only insignificant charge transfer between pixels. Imagef3 undergoes spatial sampling as it hits the discrete photo-sites in a CCD or CID camera. The combination of h3 with sampling is known as areas sampling. It reflects the finite size of the sampling area. If h3 is taken to be an impulse, then we have point sampling. This ideal, often assumed for theoretical considerations, is generally not true in practice. In either case, intensities in the sampled image Is are now defined only for integer values of u and v. The digital imageI u; v is obtained via an analog-todigital converter that quantizes the samples of Is. This completes the image formation process, leaving I u; v as the starting point for subsequent processing, including image reconstruction and restoration. Reconstruction and restoration start with I (and models for one or morehis), and seek to solve for one or more of the fjs. Recoveringf is known as image restoration and is of considerable interest in image processing. Note that recovering a functional form off2 is an intra-pixel restoration. Reconstruction limits itself to the problem of deriving the continuous functionf3. Restoration attempts to estimate the original input functionf2 (or f1). The goal of super-resolution is to recover a discrete approximation, with resolution higher than Is, to either f1 (plain super-resolution) orf (super-resolution with deblurring). Because super-resolution increases the signal-to-noise ratio, it significantly aids in the ill-condition problem of deblurring. 3. Imaging-consistency and the integrating resampler An algorithm is called imaging-consistent if it is the exact solution for some input function, which, according to the imaging model, would have generated the measured input. For image reconstruction, we achieve this by first restoring the image to yieldf2, then blurring it again by the pixel’s PSF. Details can be found in Ref. [2]. Only one-dimensional image models are presented herein; higher dimensions are treated separately. M.-C. Chiang, T.E. Boult / Image and Vision Computing 18 (2000) 761–771 762 Fig. 1. The image formation process and the relationship between restoration, reconstruction, and super-resolution. The first imaging-consistent method to consider is based on a piecewise quadratic model for the image. If we assume a Rect PSF filter (1 inside and 0 outside the pixel), the imaging consistent algorithm is easy to derive. To ensure that the function is continuous and that the method is local, we define the value of the reconstruction at the pixel boundarieski andki11 to be equal toEi andEi11: Any method of approximation could be used to compute Ei, though our examples will only include linear interpolation and cubic convolution. See [3, Section 2.3] for more details. Given the valuesEi at the pixel edges, an additional constraint, that the integral across the pixel must equal Vi, results in exactly three constraints: qi21=2  Ei ; qi1=2  Ei11; Z1=2 2 1=2 qix dx Vi : 1 From Eq. (1), one can det