Patent Document

CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    Not Applicable 
       STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0002]    Not Applicable 
       INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC 
       [0003]    Not Applicable 
       BACKGROUND OF THE INVENTION 
       [0004]    1. Field of the Invention 
         [0005]    This invention generally relates to image color tone correction and more particularly to image color tone correction of video image streams corrected to reference still images of similar scenic content. 
         [0006]    2. Description of Related Art 
         [0007]    Still cameras typically have optimized processing parameters that result in beautiful, deeply saturated, colorful images. 
         [0008]    Video recorders, due to the need to very quickly acquire and process video frames, must make compromises with modified processing parameters. These compromises frequently result in desaturated video frames, and colors that appear “off” from both a still camera, and direct viewing of a scene. 
         [0009]    When still cameras are used for recording video, similar processing parameter tradeoffs are required, with the same reduction in picture quality of the acquired video frames. 
       BRIEF SUMMARY OF THE INVENTION 
       [0010]    An aspect of the invention is a method of image color tone correction, which may comprise: (a) providing a reference image and one or more source images, wherein all images are comprised of pixels of a scene; (b) correcting a designated portion of one of the source images based on the reference image to create a color tone corrected image; and (c) outputting the color tone corrected image to a computer readable medium. The designated portion may be any subset of one of the source images, and may be a simple cropping, or other form of selection. Additionally, the designated portion may be used for determining a set of coefficients for overall color image correction of the complement of the source images. 
         [0011]    The color correction may occur in any color space. However, as the YUV is frequently used in video processing, it will be used here as an example, without limitation. Alternative color spaces with robust gamuts may be used as well. In the following descriptions, YUV is merely indicative of one color space that may be used, and not indicative of a limitation. 
         [0012]    The correcting step may comprise: (a) forming an intermediate image from the designated portion of one of the source images using a global correction in a YUV color space; and (b) refining the intermediate image into the color tone corrected image. Alternatively, steps (a) and (b) can be combined into one step. If separate into two steps, the mapping function of step (a) can be realized in advance as a look up table. Typically, the designated portion of one of the source images corresponds to the same scenic content as that found in the reference image. 
         [0013]    The method of forming the intermediate image step may comprise: (a) mapping a source pixel Y value, Y source , from the designated portion of one of the source images, to an intermediate Y value, Y int , in the intermediate image; (b) mapping a source U pixel value, U source , from the source image to be corrected, to an intermediate pixel U value, U int , in the intermediate image; and (c) mapping a source pixel V value, V source , from the source image to be corrected, to an intermediate pixel V value, V int , in the intermediate image. These mappings may be performed independently, or in parallel for either all or just those pixels located in the designated portion of one of the source images. Further, the mappings may be performed sequentially in any order, for example, with each of the pixel values processed with the source pixel Y value, then all of the U pixel values, then all of the V pixel values. 
         [0014]    The mapping of the source image pixel YUV value to the intermediate image pixel YUV value may comprise: providing independent Y, U, and V look up tables (LUTs) that allow mapping of a source pixel YUV value, (Y source , U source , V source ), from the designated portion of one of the source images, to a corresponding intermediate pixel YUV value, (Y int , U int , V int ), in the intermediate image; wherein the independent Y, U, and V look up tables (LUTs) are comprised of nominal mapping values. 
         [0015]    The independent Y, U, and V look up tables (LUTs) may be determined from hardware characterization experiments, where, for specific lighting conditions and specific processing parameters, there is an optimal Y, U, and V look up table. 
         [0016]    The independent Y, U, and V look up tables (LUTs) may be generated off-line, without involvement or usage of any data from the reference image or the one or more source images. 
         [0017]    Each of the independent Y, U, and V look up tables (LUTs) represents a mapping between: a first set of processing parameters for the reference image, and a second set of processing parameters for the one or more source images. The term “processing parameters” is used here to capture the concept of sensor speed (ISO number), gamma, color correction, lighting, or other parameter that would affect the acquisition of the reference image or the one or more source images. 
         [0018]    The mapping of the source pixel YUV value to the intermediate YUV value may comprise: (a) calculating independent Y, U, and V look up tables (LUTs) that map the source pixel YUV value, (Y source , U source , V source ), from the source image to be corrected, to a corresponding intermediate pixel YUV value, (Y int , U int , V int ), in the intermediate image; (b) wherein the independent Y look up table (LUT) may comprise: a curve fitted Cumulative Distribution Function (CDF) based on histograms of the source image and the reference image; (c) wherein the independent U, and V look up tables (LUTs) are comprised of mean-value mapping values. In this embodiment, the independent Y, U, and V look up tables (LUTs) may be comprised of mean-value mapping values when the nominal map is not available and/or CDF matching is too computationally expensive. 
         [0019]    The curve fitted Cumulative Distribution Function (CDF) may be curve fit with a cubic polynomial for a low mid, and high range of data. Ideally, these three polynomials substantially overlap with the same or very close functional outputs for the same data input. The three cubic polynomials may also form cubic splines, thereby guaranteeing overlap and continuity of slope. The curve fit procedure may exclude noisy outlier data at the extremes of the data. 
         [0020]    Similarly, other curve fit methods may be used, as well known in the numerical analysis community. 
         [0021]    The mapping of the source pixel YUV value to the intermediate pixel YUV value may comprise: calculating independent Y, U, and V look up tables (LUTs) that map the source pixel YUV value, (Y source , U source , V source ), from the source image to be corrected, to a corresponding intermediate pixel U value, (Y int , U int , V int ), in the intermediate image; wherein the independent Y, U, and V look up tables (LUTs) are comprised of mean-value mapping value based on the designated portion of one of the source images, and a corresponding designated portion of the reference image. 
         [0022]    It is to be understood that the intermediate image may never be visualized, and may only exist as pixel values (Y int , U int , V int ) within a computer readable medium, most likely a computer or video processor. 
         [0023]    The refining of the intermediate image into the color tone corrected image step may comprise: (a) matching a color tone of the intermediate image to a color tone of the reference image; (b) wherein the color tone of the intermediate image is comprised of the Y, U, and V values of the pixels of the intermediate image, (Y int , U int , V int ); and (c) wherein the color tone of the reference image is comprised of the Y, U, and V values of the pixels of the reference image, (Y ref , U ref , V ref ). This process uses co-located pixels, that is, the intermediate image is an image of the scene, and the reference image is also an image of the scene, such that a co-located pixel in both represents the same pixel of information in the scene, despite being in both the intermediate and reference images. This remains true, even if a subset of a single source image is to be processed. 
         [0024]    The method of matching the color tone of the intermediate image to the color tone of the reference image step may comprise: (a) regressing the color tone of the intermediate image with the color tone of the reference image to form color tone correction parameters a i,j , where i ,j ε [Y, U, V]; (b) wherein the regressing step minimizes a least squares error; and c) applying the color tone correction parameters a i,j  to the intermediate image to form the color tone corrected image. 
         [0025]    While least squares error is among the lowest computational burden method of regression, other error bounds and solution methods may also be used. However, the least squares error solution routine typically is included with readily available computer software mathematics solution packages, thereby minimizing development time on this particular matrix regression solution. 
         [0026]    The subset of the reference image color tone and the subset of the intermediate image may comprise an interlaced subset. 
         [0027]    The regressing step may comprise: a) using a subset of the reference image color tone and a corresponding subset of the intermediate image to form color tone correction parameters a i,j , where i ,j ε [Y, U, V]; b) whereby the computational loading for calculating the color tone correction parameters a i,j  is substantially reduced. The method of image color tone correction may further comprise: applying the color tone correction parameters a i,j , where i ,j ε [Y, U, V], to the intermediate images of the remainder of the source images from the same scene to form the color tone corrected images. This applying step is a simple matrix multiplication of the intermediate image times the color tone correction parameters a i,j . 
         [0028]    Another aspect of the invention is a computer readable medium comprising a programming executable capable of performing on a computer the methods discussed above. 
         [0029]    Further, an image color tone correction apparatus may be constructed that comprises the methods described above. The resultant image color tone correction apparatus may be hand held. Or, rather, data acquisition of the reference and source images may be hand held, and then post processed on more capable hardware that may not be hand held. 
         [0030]    A still further aspect of the invention is a method of image color tone correction, which may comprise: (a) acquiring a still reference image of a scene, wherein the still reference image is comprised of pixels; (b) acquiring one or more source images of the scene, wherein the source images are comprised of pixels; (c) correcting a designated portion of one of the source images based on a corresponding portion of the reference image to create a color tone corrected image; and (d) outputting the color tone corrected image to a computer readable medium. This method allows for the correction of a portion of one or more of the source images to be selectively applied, much like a selected portion of a graphic image in photographic editing software. 
         [0031]    A computer readable medium comprising a programming executable capable of performing on a computer the selective method of image color tone correction discussed above. 
         [0032]    In another aspect of the invention, the method of image color tone correction may comprise: (a) detecting a change in the scene; and (b) after detecting the change in the scene, acquiring a new reference image to replace the reference image. This concept is similar to video broadcasting, where, at the beginning of a group of pictures (GOP), a full image is used, except here, when the scene changes, a new reference image is acquired. 
         [0033]    Further aspects of the invention will be brought out in the following portions of the specification, wherein the detailed description is for the purpose of fully disclosing preferred embodiments of the invention without placing limitations thereon. 
     
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S) 
         [0034]    The invention will be more fully understood by reference to the following drawings which are for illustrative purposes only: 
           [0035]      FIG. 1  is a flow chart showing an automatic color tone correction method using a nominal mapping for the initial color correction. 
           [0036]      FIG. 2  is a proposed method of constructing a nominal mapping function used in the automatic color tone correction method of  FIG. 1 . 
           [0037]      FIG. 3A  is an example of a scatter chart of the relationship between a source image (video frame) and a reference image (still picture) on the Y channel. 
           [0038]      FIG. 3B  is an example of a histogram of the video source value at  100  and its associated data range. 
           [0039]      FIG. 4  is a plot of a Gretag Macbeth Color-Checker under different lighting conditions; an example of noisy outlier data in the very dark and bright area using the Y channel data. 
           [0040]      FIG. 5  is a flow chart showing an automatic color tone correction method initial color correction processed through a Cumulative Distribution Function (CDF) matching and polynomial mapping independently with pre-determined mapping functions. 
           [0041]      FIG. 6  is a graph of the Cumulative Distribution Function histogram curves for a source and a reference image. 
           [0042]      FIG. 7  is a graph of the third order (cubic) mapping polynomial that describes the relationship between the source luminance value and the mapped luminance value. 
           [0043]      FIG. 8  is a flow chart showing an automatic color tone correction method initial color correction processed through a mean-value mapping. 
           [0044]      FIG. 9A  is a flow chart of the automatic color tone correction method, with input source image(s) and a reference still image producing a tone corrected image. 
           [0045]      FIG. 9B  is a chart showing the various computer readable media to which the color tone corrected image may be output. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0046]    Referring more specifically to the drawings, for illustrative purposes the present invention is embodied in the apparatus generally shown in  FIG. 1  through  FIG. 9B . It will be appreciated that the apparatus may vary as to configuration and as to details of the parts, and that the method may vary as to the specific steps and sequence, without departing from the basic concepts as disclosed herein. 
         [0047]    Definitions 
         [0048]    “YUV” is an acronym for the color space known by Y=Luminance, U=Normalized B-Y, V=Normalized R-Y. The YUV model defines a color space in terms of one luma and two chrominance components. 
         [0049]    YUV signals are created from an original RGB (red, green, and blue) source. The weighted values of R, G, and B are added together to produce a single Y signal, representing the overall brightness, or luminance, of that spot. The U signal is then created by subtracting the Y from the blue signal of the original RGB, and then scaling; V is created by subtracting the Y from the red, and then scaling by a different factor. This can be accomplished easily with analog circuitry. 
         [0050]    The methods and apparatus of the invention described herein may be applicable on any known color space. The reason is used in this patent application is that YUV is a common color space in conventional digital image/video capturing devices. “Video image” means a collection of frames of images typically obtained with a video recorder. Such video image may also be referred to as a video image stream. 
         [0051]    “Still image” means a single image that is not normally part of a video image. Still image may also be referred to as a still picture. 
         [0052]    “Computer” means any device capable of performing the steps, methods, or producing signals as described herein, including but not limited to: a microprocessor, a microcontroller, a video processor, a digital state machine, a field programmable gate array (FGPA), a digital signal processor, a collocated integrated memory system with microprocessor and analog or digital output device, a distributed memory system with microprocessor and analog or digital output device connected by digital or analog signal protocols. 
         [0053]    “Computer readable medium” means any source of organized information that may be processed by a computer to perform the steps described herein to result in, store, perform logical operations upon, or transmit, a flow or a signal flow, including but not limited to: random access memory (RAM), read only memory (ROM), a magnetically readable storage system; optically readable storage media such as punch cards or printed matter readable by direct methods or methods of optical character recognition; other optical storage media such as a compact disc (CD), a digital versatile disc (DVD), a rewritable CD and/or DVD; electrically readable media such as programmable read only memories (PROMs), electrically erasable programmable read only memories (EEPROMs), field programmable gate arrays (FGPAs), flash random access memory (flash RAM); and information transmitted by electromagnetic or optical methods including, but not limited to, wireless transmission, copper wires, and optical fibers. 
         [0054]    1.0 Introduction 
         [0055]    One aspect of this invention provides algorithms of color tone correction to match the colors of one image to the colors of another image with a similar scene. The colors that comprise an image are frequently referred to as the tonality of the image. 
         [0056]    Another aspect of this invention is designed to match the color tone of video images with associated still picture images taken from the same, similar, or even different image sensors. The algorithms described here may also be applicable for chrominance adjustment for other applications, such as photomontages. 
         [0057]    Video images generally do not look as vivid and colorful as still picture images. Even though a scene is captured by the same sensor (e.g., CMOS) for both still picture and video image streams an image obtained from video will not have the same color tone as the still picture image due to process variations between the initial captured raw data to the final output image data. These process variations may include: different gamma values, dynamic range adjustments, image enhancements, sensor sensitivity (ISO sensitivity numbers), exposure time, illumination, or other parameters. 
         [0058]    The invention here systematically determines a set of mapping functions between video images and corresponding still pictures. Once the mapping functions are determined, they are used to adjust the color tonal values of the input video images so that they match the colors (saturation and luminosity) of a reference still picture that has a similar scene. This method thereby imbues video images with the tonal qualities and saturation of still images, thereby greatly enhancing the perceived video image quality. 
         [0059]    The method disclosed here accurately transforms the color tone of one image (generally from a video image stream) to be the same as another one (generally from a still image). Although this invention describes the algorithm only from the standpoint of improving video images by using reference still images, one may readily extend the algorithm to other related applications. 
         [0060]    To aid in the description of the algorithms use here, still pictures will be referred to as references, an associated video image stream for correction as source, and the corrected video image stream as the mapped (or corrected) image stream. 
         [0061]    2.0 Method 
         [0062]    Refer now to  FIG. 1 . To achieve accurate color conformance between two images, a two-step approach is described  100 . Additionally, the original YUV color space is used to avoid additional color space transformations and any limitations (such as out of gamut problems) or transcription errors that may result in the use of such color space transformations. 
         [0063]    Refer now to  FIG. 1 , which is a flow chart showing an automatic color tone correction method. Here, the three color components from source image are first processed through nominal mapping independently with pre-determined mapping functions. These intermediate data are then as inputs for the refinement. Each final output channel is determined by all three channels to ensure the final colors appearance at the refinement stage. 
         [0064]    In greater detail, the source luminance Y source    102  proceeds to a nominal mapping process  104  to generate an intermediate luminance Y int    106  value. This value is in turn processed through a refinement stage  108  with a reference input luminance Y ref    110  to produce a mapped luminance Y map    112 . The other inputs to the refinement step  108  will be deferred in the description at this time. 
         [0065]    Similarly, the source U value U source    114  is used as an input to a nominal mapping process  116  to generate an intermediate U value U int    118  value. This value is in turn processed through a refinement stage  120  with a reference input U value U ref    122  to produce a mapped U value U map    124 . As above, other inputs to the refinement step  120  will be deferred in the description for a short time. 
         [0066]    Finally, the source V value V source    126  proceeds to a nominal mapping process  128  to generate an intermediate V value V int    130  value. This value is in turn processed through a refinement stage  132  with a reference input V value V ref    134  to produce a mapped V value V map    136 . 
         [0067]    The description above merely described the processing of the Y source    102 , U source    114 , and V source    126  with no interaction with each other. Now, the interactions are added in the three refinement stages Y  108 , U  120 , and V  132 . 
         [0068]    The refinement stage  108  additionally inputs U int    118  and V int    130  in the formation of the Y map    112  output value. 
         [0069]    Similarly, the refinement stage  120  additionally inputs Y int    106  and V int    130  in the formation of the U map    124  output value. 
         [0070]    Finally, the refinement stage  132  additionally inputs Y int    106  and U int    118  in the formation of the V map    136  output value. 
         [0071]    2.1 Nominal Mapping 
         [0072]    The idea of nominal mapping as a first step is to work as a global color correction to bring the color tones of two images closer together, which is a preparatory stage for the subsequent refinement stage. The nominal mapping function is independent to the image contents but is likely dependent on each particular image capturing device. It can be pre-determined or determined on the fly. 
         [0073]    The nominal mappings of the Y source    102 , U source    114 , and V source    126  occur in blocks  104 ,  116 , and  128  in a predetermined look up table (LUT) to provide an initial global correction  138  to the source input YUV values. 
         [0074]    The refinement mappings of the Y int    106 , U int    118 , and V int    130  occur in blocks  108 ,  120 , and  132  using LSE regression to provide a local correction  140  to the source intermediate YUV values. 
         [0075]    2.1.1 Nominal Mapping Function 
         [0076]    To form a nominal mapping function, a pair of images from a still picture and an associated video frame (with the same scene) is used. 
         [0077]    Refer now to  FIG. 2 , which shows one proposed method of constructing a mapping function  200 . Y, U, and V components are each processed independently at this stage. Here, the images are calibrated with Gretag Macbeth Color-Checker for the determination of nominal mapping function because it contains wider range of chrominance data. However, images with differing content are also applicable. 
         [0078]    Specifically, a modified mean value mapping  202  is determined by using an input reference picture (a still picture)  204 , and the same or similar image from a source  206  frame of similar content from the video image stream. The modified mean value mapping  202  output is used as an input into a polynomial fitting section  208 , which in turn outputs the nominal mapping function  210 . 
         [0079]    Ideally, both the reference picture  204  and the source frame  206  contain a wide gamut of color information, such as a Gretag Macbeth Color-Checker chart. Such a chart is developed and calibrated with spectrophotometers, allowing precise and consistent color samples, and typically have a very wide color gamut. 
         [0080]    Modified Mean Value Mapping 
         [0081]    Refer now to  FIGS. 3A and 3B .  FIG. 3A  shows an example of the relationship between a source image (video frame) and a reference image (still picture) on the Y channel  300 . A single value on source image is associated with a wide range of data values on reference image and vice versa. For example, as shown in  FIG. 3A , a data value  100  on video image (the vertical line  302  within the graph) is associated with data values between  24  and  207  (with mean value at  103 ) on the corresponding still picture.  FIG. 3B  shows the histogram of this associated data range for the video source value at  100 . A single value from the associated range needs to be mapped in order to have a one-to-one nominal mapping. 
         [0082]    Here the mean value is chosen as the mapped value. This is because the mean value is easy to compute with the assumption that the noise is unbiased. Although using the value with peak distribution is another potential method that could be used, the result is very close to that by average but with a greatly increased computational load. Therefore, in this instance the mean value is used as the mapped value. 
         [0083]    With this approach, the mapping functions from different lighting conditions are almost the same among each channel. Similar results are also shown in different models, such as a widely available high definition camcorder. This implies that each camera only needs three mapping functions, with only a single nominal mapping function for each channel. 
         [0084]    Refer now to  FIG. 4 , which is an example of the plot of a Gretag Macbeth Color-Checker under different lighting conditions  400 . It should be noted that the mean-valued mapped data in very dark  402  and very bright  404  regions are more sensitive to noise and interpolation errors, as circled in  FIG. 4 . These circled data should be treated as outliers and removed before forming the mapped function. This additional step is necessary to avoid uncontrollable mapping for incorrect color adjustment. Further additional end-data mapping points, mapping 0 to 0 and 255 to 255, are added prior to the polynomial fitting process to assure a meaningful mapping function result. 
         [0085]    Polynomial Fit for the Nominal Mapping 
         [0086]    Since the mean-value mapping function is not smooth and may not cover all values, polynomial fitting in least square sense is further applied to form the final nominal mapping function. This mapping function in general can be a 3rd-order polynomial to fit the “S-shape” relationship in a generic form as following: 
         [0087]    If necessary, piecewise mapping can ensure a more accurate nominal mapping function. But no more than three segments appear necessary. 
         [0088]    As aforementioned, the nominal mapping functions are device dependent and perform as global color correction to bring the color tones of an input source image closer to the reference one. The polynomial mapping functions can be pre-determined and saved as look-up tables, or determined after acquisition of a source image and a reference image with near real-time processing. 
         [0089]    2.2 Refinement 
         [0090]    Since nominal mapping step is many-to-one mapping and each channel of YUV is processed independently, the resultant colors may not be accurate. The refinement step is therefore applied to match the color tone with all channels displayed together. Actually, this refinement step is more critical than the nominal mapping, which mainly prepares the data for better color correction in the refinement step. 
         [0091]    After performing the nominal mapping for Y, U, and V channels individually, an intermediate image can be constructed from the intermediate values of Y, U, and V. This intermediate image may be placed in a 3×(n+1) matrix with the Y, U, and V values separately used as: 
         [0000]    
       
         
           
             
               [ 
               
                 
                   
                     
                       Y 
                       0 
                     
                   
                   
                     
                       U 
                       0 
                     
                   
                   
                     
                       V 
                       0 
                     
                   
                 
                 
                   
                     
                       Y 
                       1 
                     
                   
                   
                     
                       U 
                       1 
                     
                   
                   
                     
                       V 
                       1 
                     
                   
                 
                 
                   
                     ⋮ 
                   
                   
                     ⋮ 
                   
                   
                     ⋮ 
                   
                 
                 
                   
                     
                       Y 
                       n 
                     
                   
                   
                     
                       U 
                       n 
                     
                   
                   
                     
                       V 
                       n 
                     
                   
                 
               
               ] 
             
             . 
           
         
       
     
         [0000]    For each luminance value of Y in the intermediate image, two related sets of data may be determined: 1) the data from the intermediate image that shares the same Y value; and 2) the data of corresponding positions from the reference image. The associated mapping functions can be found by solving the following equations in the least square sense for color tone correction parameters a i,j , where i ,j ε [Y, U, V]: 
         [0000]    
       
         
           
             
               
                 [ 
                 
                   
                     
                       
                         Y 
                         
                           ref 
                           , 
                           0 
                         
                       
                     
                   
                   
                     
                       
                         Y 
                         
                           ref 
                           , 
                           1 
                         
                       
                     
                   
                   
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         Y 
                         
                           ref 
                           , 
                           n 
                         
                       
                     
                   
                 
                 ] 
               
               = 
               
                 
                   
                     [ 
                     
                       
                         
                           
                             Y 
                             0 
                           
                         
                         
                           
                             U 
                             0 
                           
                         
                         
                           
                             V 
                             0 
                           
                         
                       
                       
                         
                           
                             Y 
                             1 
                           
                         
                         
                           
                             U 
                             1 
                           
                         
                         
                           
                             V 
                             1 
                           
                         
                       
                       
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             Y 
                             n 
                           
                         
                         
                           
                             U 
                             n 
                           
                         
                         
                           
                             V 
                             n 
                           
                         
                       
                     
                     ] 
                   
                   int 
                 
                  
                 
                   [ 
                   
                     
                       
                         
                           a 
                           YY 
                         
                       
                     
                     
                       
                         
                           a 
                           YU 
                         
                       
                     
                     
                       
                         
                           a 
                           YV 
                         
                       
                     
                   
                   ] 
                 
               
             
             , 
             
               
 
             
              
             
               
                 [ 
                 
                   
                     
                       
                         U 
                         
                           ref 
                           , 
                           0 
                         
                       
                     
                   
                   
                     
                       
                         U 
                         
                           ref 
                           , 
                           1 
                         
                       
                     
                   
                   
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         U 
                         
                           ref 
                           , 
                           n 
                         
                       
                     
                   
                 
                 ] 
               
               = 
               
                 
                   
                     [ 
                     
                       
                         
                           
                             Y 
                             0 
                           
                         
                         
                           
                             U 
                             0 
                           
                         
                         
                           
                             V 
                             0 
                           
                         
                       
                       
                         
                           
                             Y 
                             1 
                           
                         
                         
                           
                             U 
                             1 
                           
                         
                         
                           
                             V 
                             1 
                           
                         
                       
                       
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             Y 
                             n 
                           
                         
                         
                           
                             U 
                             n 
                           
                         
                         
                           
                             V 
                             n 
                           
                         
                       
                     
                     ] 
                   
                   int 
                 
                  
                 
                   [ 
                   
                     
                       
                         
                           a 
                           UY 
                         
                       
                     
                     
                       
                         
                           a 
                           UU 
                         
                       
                     
                     
                       
                         
                           a 
                           UV 
                         
                       
                     
                   
                   ] 
                 
               
             
             , 
             
               
                 and 
                  
                 
                   
 
                 
                 [ 
                 
                   
                     
                       
                         V 
                         
                           ref 
                           , 
                           0 
                         
                       
                     
                   
                   
                     
                       
                         V 
                         
                           ref 
                           , 
                           1 
                         
                       
                     
                   
                   
                     
                       ⋮ 
                     
                   
                   
                     
                       
                         V 
                         
                           ref 
                           , 
                           n 
                         
                       
                     
                   
                 
                 ] 
               
               = 
               
                 
                   
                     [ 
                     
                       
                         
                           
                             Y 
                             0 
                           
                         
                         
                           
                             U 
                             0 
                           
                         
                         
                           
                             V 
                             0 
                           
                         
                       
                       
                         
                           
                             Y 
                             1 
                           
                         
                         
                           
                             U 
                             1 
                           
                         
                         
                           
                             V 
                             1 
                           
                         
                       
                       
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             Y 
                             n 
                           
                         
                         
                           
                             U 
                             n 
                           
                         
                         
                           
                             V 
                             n 
                           
                         
                       
                     
                     ] 
                   
                   int 
                 
                  
                 
                   [ 
                   
                     
                       
                         
                           a 
                           VY 
                         
                       
                     
                     
                       
                         
                           a 
                           VU 
                         
                       
                     
                     
                       
                         
                           a 
                           VV 
                         
                       
                     
                   
                   ] 
                 
               
             
             , 
           
         
       
     
         [0000]    where n+1 is the number of data points in the intermediate image, and the Y ref,m , U ref,m , and V ref,m  refer to reference image pixels, numbering with the second index m ranging from 0 to n+1. The Y k , U k , and V k  for k ε [0, n+1] represents the intermediate image pixel color tone values. 
         [0092]    This system of equations may also be simultaneously solved in a combined forms as: 
         [0000]    
       
         
           
             
               
                 
                   [ 
                   
                     
                       
                         
                           Y 
                           0 
                         
                       
                       
                         
                           U 
                           0 
                         
                       
                       
                         
                           V 
                           0 
                         
                       
                     
                     
                       
                         
                           Y 
                           1 
                         
                       
                       
                         
                           U 
                           1 
                         
                       
                       
                         
                           V 
                           1 
                         
                       
                     
                     
                       
                         ⋮ 
                       
                       
                         ⋮ 
                       
                       
                         ⋮ 
                       
                     
                     
                       
                         
                           Y 
                           n 
                         
                       
                       
                         
                           U 
                           n 
                         
                       
                       
                         
                           V 
                           n 
                         
                       
                     
                   
                   ] 
                 
                 ref 
               
               = 
               
                 
                   
                     [ 
                     
                       
                         
                           
                             Y 
                             0 
                           
                         
                         
                           
                             U 
                             0 
                           
                         
                         
                           
                             V 
                             0 
                           
                         
                       
                       
                         
                           
                             Y 
                             1 
                           
                         
                         
                           
                             U 
                             1 
                           
                         
                         
                           
                             V 
                             1 
                           
                         
                       
                       
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             Y 
                             n 
                           
                         
                         
                           
                             U 
                             n 
                           
                         
                         
                           
                             V 
                             n 
                           
                         
                       
                     
                     ] 
                   
                   int 
                 
                  
                 
                   [ 
                   
                     
                       
                         
                           a 
                           YY 
                         
                       
                       
                         
                           a 
                           UY 
                         
                       
                       
                         
                           a 
                           VY 
                         
                       
                     
                     
                       
                         
                           a 
                           YU 
                         
                       
                       
                         
                           a 
                           UU 
                         
                       
                       
                         
                           a 
                           YU 
                         
                       
                     
                     
                       
                         
                           a 
                           YV 
                         
                       
                       
                         
                           a 
                           UV 
                         
                       
                       
                         
                           a 
                           VV 
                         
                       
                     
                   
                   ] 
                 
               
             
             , 
           
         
       
     
         [0000]    where int refers to the intermediate image, and ref refers to reference image pixels. The Y k , U k , and V k  for both the int and ref images are indexed on k ε [0, n+1]. 
         [0093]    If the system is underdetermined, special handling is required. For example, if n=2 or the rank of coefficient matrix is 2, the Y value may be kept unchanged and thereby form a 2×2 matrix to solve for refinement parameters U and V. If n=1 or the rank of coefficient matrix is 1, there is no need for the refinement and the nominally mapped values are the final outputs. 
         [0094]    At this point, the mapped output image may be formed by: 
         [0000]    
       
         
           
             
               
                 
                   [ 
                   
                     
                       
                         
                           Y 
                           0 
                         
                       
                       
                         
                           U 
                           0 
                         
                       
                       
                         
                           V 
                           0 
                         
                       
                     
                     
                       
                         
                           Y 
                           1 
                         
                       
                       
                         
                           U 
                           1 
                         
                       
                       
                         
                           V 
                           1 
                         
                       
                     
                     
                       
                         ⋮ 
                       
                       
                         ⋮ 
                       
                       
                         ⋮ 
                       
                     
                     
                       
                         
                           Y 
                           n 
                         
                       
                       
                         
                           U 
                           n 
                         
                       
                       
                         
                           V 
                           n 
                         
                       
                     
                   
                   ] 
                 
                 mapped 
               
               = 
               
                 
                   
                     [ 
                     
                       
                         
                           
                             Y 
                             0 
                           
                         
                         
                           
                             U 
                             0 
                           
                         
                         
                           
                             V 
                             0 
                           
                         
                       
                       
                         
                           
                             Y 
                             1 
                           
                         
                         
                           
                             U 
                             1 
                           
                         
                         
                           
                             V 
                             1 
                           
                         
                       
                       
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                         
                           ⋮ 
                         
                       
                       
                         
                           
                             Y 
                             n 
                           
                         
                         
                           
                             U 
                             n 
                           
                         
                         
                           
                             V 
                             n 
                           
                         
                       
                     
                     ] 
                   
                   int 
                 
                  
                 
                   [ 
                   
                     
                       
                         
                           a 
                           YY 
                         
                       
                       
                         
                           a 
                           UY 
                         
                       
                       
                         
                           a 
                           VY 
                         
                       
                     
                     
                       
                         
                           a 
                           YU 
                         
                       
                       
                         
                           a 
                           UU 
                         
                       
                       
                         
                           a 
                           YU 
                         
                       
                     
                     
                       
                         
                           a 
                           YV 
                         
                       
                       
                         
                           a 
                           UV 
                         
                       
                       
                         
                           a 
                           VV 
                         
                       
                     
                   
                   ] 
                 
               
             
             , 
           
         
       
     
         [0000]    using the same nomenclature as before where mapped refers to the mapped output image. 
         [0095]    In addition, there is no need to provide all the associated data in the image to calculate the refinement parameters, a ij . Only a subset of the data, which are within a range of nominal mapping value, are necessary for determining the refinement parameters. This not only reduce the calculation load but also make the system robust to the noise. 
         [0096]    Rounding Strategy 
         [0097]    Still pictures may contain pixels with maximum value (max_val), e.g., max_val=255 for 8-bit data. In order to retain the mapped maximum value, the refinement output with the value greater than (max_val−1) will be set to the max_val. Others, rounding to the nearest integer will be processed for the final refinement output. 
         [0098]    3. Alternative Methods 
         [0099]    Other variations of the first step of the proposed method may be used, particularly in the case where the nominal mapping functions are not a priori determined or polynomial fittings are not available. 
         [0100]    3.1 Alternative Method 1 
         [0101]    The Cumulative Distribution Function (CDF) fit with polynomial fitting can also achieve a result similar to the one proposed for nominal mapping of the luminance Y channel. However, it is more computational expensive. 
         [0102]    In addition, CDF matching does not work well on chrominance channels U and V because the chrominance data values are more concentrated. However, they can still be mapped by modified mean-value mapping. Although mean-value mapping is not as accurate as nominal mapping, the overall result is almost the same as the result of proposed method. This is because the refinement stage is more critical and can handle the deviation between the two results. As a result, the mean-value mapping procedure appears as an alternative method as shown in  FIG. 5 . 
         [0103]    Refer now to  FIG. 5 . Here, the three color components from source image are first processed through CDT matching and polynomial mapping independently with pre-determined mapping functions. These intermediate data are then as inputs for the refinement. Each final output channel is determined by all three channels to ensure the final colors appearance at the refinement stage. 
         [0104]    In greater detail, the source luminance Y source    502  proceeds to a nominal mapping process  504  to generate an intermediate luminance Y int    506  value. This value is in turn processed through a refinement stage  508  with a reference input luminance Y ref    510  to produce a mapped luminance Y map    512 . The other inputs to the refinement step  908  will be deferred in the description at this time. 
         [0105]    Similarly, the source U value U source    514  proceeds to a nominal mapping process  516  to generate an intermediate U value U int    518  value. This value is in turn processed through a refinement stage  520  with a reference input U value U ref    522  to produce a mapped U value U map    524 . As above, other inputs to the refinement step  520  will be deferred in the description for a short time. 
         [0106]    Finally, the source V value V source    526  proceeds to a nominal mapping process  928  to generate an intermediate V value V int    530  value. This value is in turn processed through a refinement stage  532  with a reference input V value V ref    534  to produce a mapped V value V map    536 . 
         [0107]    The description above merely described the processing of the Y source    502 , U source    514 , and V source    526  with no interaction with each other. Now, the interactions are added in the three refinement stages Y  508 , U  520 , and V  532 . 
         [0108]    The refinement stage  508  additionally inputs U int    518  and V int    530  in the formation of the Y map    512  output value. 
         [0109]    Similarly, the refinement stage  520  additionally inputs Y int    506  and V int    530  in the formation of the U map    524  output value. 
         [0110]    Finally, the refinement stage  532  additionally inputs Y int    506  and U int    518  in the formation of the V map    536  output value. 
         [0111]    Cumulative Distribution Function (CDF) Matching 
         [0112]    Standard CDF matching and histogram matching techniques both require iterative operations. Here normal CDF matching is modified by applying polynomial fittings to avoid the requirement of iterative operations and to ensure a continuous mapping. 
         [0113]    The algorithm comprises: 1) adjusting the size of the source image equal to the same size as the reference image; 2) calculating the histograms of the source and the reference images; 3) determining the cumulative histograms for both images; 4) determining a mapping function between these two cumulative histograms; and 5) applying curve fitting to finalize the mapping function. 
         [0114]    Refer now to  FIG. 6 .  FIG. 6  illustrates this mapping process  600 . CDF source    602  and CDF ref    604  represent the cumulative histograms of the source and the reference images, respectively. For a particular color Y source    606  in CDF source    602 , the cumulative count is T source    608 . Locate T ref    610  from CDF ref    604 , which has the closest value to T source    608  (this closest value is at an exaggerated distance on the graph  600  for clarity). The corresponding Y value, Y ref    612 , will then be the mapped value of Y source    606 . This process is repeated for all values of CDF ref    604  to determine a correspondence set between the Y source    606  and Y ref    612 . 
         [0115]    The next step comprises applying curve fitting to finalize the mapping function between the two correspondence sets Y source    606  and Y ref    612 , while making the mapping continuous. 
         [0116]    Refer now to  FIG. 7 , which shows a graphical example  700  of the mapping polynomial. Third order polynomials are used for piecewise fitting for an input source luminance value  702 . The resulting mapping function for a particular segment has the general form of Y int =p 1 Y source   3 +p 2 Y source   2 +p 3 Y source +p 4 , where p 1 , p 2 , p 3  and p 4  are cubic polynomial coefficients for that particular segment. The output of the mapping function is the mapped luminance value  704 . 
         [0117]    Each of the three cubic segments cover the ranges (source luminance values) of [0-50], [35-225], and [220-255], respectively in bands  706 ,  708 , and  710  respectively (the mapped luminance values). The ranges overlap to ensure a smooth fit at the cubic transitions. 
         [0118]    Instead of overlapping the ranges to ensure a smooth fit, cubic splines may be use to ensure a continuous smooth fit for the data. 
         [0119]    The result of the cubic segment curve mapping is shown by an overlapping plot of the original data point  712  and the corresponding fitted value  714 . In this  FIG. 7 , it appears that the results appear virtually identical. 
         [0120]    Modified Mean-Value Mapping 
         [0121]    The modified mean-value mapping is essentially the same method as previously proposed. However, since there is no cubic polynomial curve fit here, linear interpolations from neighboring data points are used to determine missing mapping points. 
         [0122]    3.2 Alternative Method 2 
         [0123]    When the nominal mapping functions are not available and the CDF matching with polynomial fitting on Y channel is not suitable, the modified mean-value mapping can be applied on all channels, as shown in  FIG. 8 . 
         [0124]    Refer now to  FIG. 8 , the three color components from source image are first processed through mean-value mapping independently without pre-determined mapping functions. These intermediate data are then as inputs for the refinement. Each final output channel is determined by all three channels to ensure the final colors appearance at the refinement stage. 
         [0125]    In greater detail, the source luminance Y source    802  proceeds to a nominal mapping process  804  to generate an intermediate luminance Y int    806  value. This value is in turn processed through a refinement stage  808  with a reference input luminance Y ref    810  to produce a mapped luminance Y map    812 . The other inputs to the refinement step  808  will be deferred in the description at this time. 
         [0126]    Similarly, the source U value U source    814  proceeds to a nominal mapping process  816  to generate an intermediate U value U int    818  value. This value is in turn processed through a refinement stage  820  with a reference input U value U ref    822  to produce a mapped U value U map    824 . As above, other inputs to the refinement step  820  will be deferred in the description for a short time. 
         [0127]    Finally, the source V value V source    826  proceeds to a nominal mapping process  828  to generate an intermediate V value V int    830  value. This value is in turn processed through a refinement stage  832  with a reference input V value V ref    834  to produce a mapped V value V map    836 . 
         [0128]    The description above merely described the processing of the Y source    802 , U source    814 , and V source    830  with no interaction with each other. Now, the interactions are added in the three refinement stages Y  808 , U  820 , and V  832 . 
         [0129]    The refinement stage  808  additionally inputs U int    818  and V int    830  in the formation of the Y map    812  output value. 
         [0130]    Similarly, the refinement stage  820  additionally inputs Y int    806  and V int    830  in the formation of the U map    824  output value. 
         [0131]    Finally, the refinement stage  832  additionally inputs Y int    806  and U int    818  in the formation of the V map    836  output value. 
         [0132]    4.0 Implementation Overview 
         [0133]    Backing up now for the details of automatic image color tone correction, now look at overall implementations of the method. 
         [0134]    Refer now to  FIG. 9A , which is a flow chart of the overall color tome correction method  900 . Here, one or more source images  902 , as well as at least one reference image  904  (preferably a still image, that has highest color tone quality) are input into the color tone correction method of this application. An initial mapping  908  may or may not be used as an input to produce a tone corrected output  910 . 
         [0135]    Refer now to  FIG. 9B , which is a diagram of the various computer readable media to which the tone corrected image  910  may be output. It may be sent to a microwave transmitter  914  for propagation through microwaves  916  to a distant connection. The tone corrected image  910  may be stored on a memory device faster or slower than the depicted USB storage device  918 , as well as stored on a digital or analog recording medium exemplified as a high definition video recorder  920 , or a still or moving digital camera  922 . Similarly, the tone corrected image  910  may be displayed, with or without further processing, on a graphical device such as a plasma or liquid crystal display  924 . The tone corrected image  910  may be transmitted through a network  926 , which may be wireless, or may use TCP/IP or other interconnection techniques to any other device in this  FIG. 9B . Still another output of the tone corrected image  910  may be to a multiple media device  928 , which may have a VHS video recorder  930 , or a DVD recorder  932 . Yet another output of the tone corrected image  910  may be to a computer  934  that may or may not be connected to a display device  936 , to be ultimately viewed by a person  938 . The computer  934  may itself have a wide variety of computer readable media available for subsequent secondary output of the tone corrected image  910 . Finally, the tone corrected image  910  may be transmitted through a wired connection  940 , which may have one or more conductors  942  of electromagnetic information, ultimately being transmitted to one or more of the devices present in this  FIG. 9B . 
         [0136]    Conclusion 
         [0137]    Although the description above contains many details, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments of this invention. Therefore, it will be appreciated that the scope of the present invention fully encompasses other embodiments which may become obvious to those skilled in the art, and that the scope of the present invention is accordingly to be limited by nothing other than the appended claims, in which reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” All structural, chemical, and functional equivalents to the elements of the above-described preferred embodiment that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present invention, for it to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase “means for.”

Technology Category: 5