PATENT DOCUMENT

Publication Number: US-9874975-B2
Application Number: US-201615144706-A
Country: US
Kind Code: B2

Title: Reconstruction of original touch image from differential touch image

Abstract:
Reconstruction of an original touch image from a differential touch image is disclosed. Reconstruction can include aligning columns of the differential touch image relative to each other and aligning the image to a baseline DC value. The column and baseline alignment can be based on the differential image data indicative of no touch or hover, because such data can more clearly show the amount of alignment needed to reconstruct the original image. The reconstruction can be performed using the differential image data alone. The reconstruction can also be performed using the differential image data and common mode data indicative of the missing image column average.

Claims:
What is claimed is: 
     
       1. A method for reconstructing an original touch image from a differential touch image, comprising:
 capturing the differential touch image at a touch sensitive device; 
 capturing common mode data at the device; 
 combining the captured differential touch image data and the common mode data; 
 performing a column alignment to align columns of the differential touch image to each other based on the combined data; and 
 performing a baseline alignment to align the column-aligned differential touch image with a baseline value. 
 
     
     
       2. The method of  claim 1 , further comprising:
 sorting data values based on the combined data into bins of a histogram; 
 identifying a highest-count bin of the histogram; and 
 determining an offset value based on the highest-count bin. 
 
     
     
       3. The method of  claim 2 , wherein the baseline value is based on the determined offset value. 
     
     
       4. The method of  claim 2 , wherein values used to align columns of the differential touch image to each other are based on the determined offset value. 
     
     
       5. The method of  claim 2 , further comprising:
 identifying a second highest-count bin of the histogram; 
 in accordance with a determination that the first and second highest-count bins are adjacent, the determined offset value is a weighted average of the center values for the two identified bins; and 
 in accordance with a determination that the two identified bins are not adjacent, the determined offset value is the center value of the highest-count bin. 
 
     
     
       6. The method of  claim 1 , wherein combining the captured differential touch image data and the common mode data comprises scaling the differential touch image data and the common mode data based on a number of data values in the differential image indicative of a touching or hovering object. 
     
     
       7. The method of  claim 6 , wherein the scaling reduces contribution of the common mode data to the combined data as a number of data values indicating the touching or hovering object in differential touch image data increases. 
     
     
       8. The method of  claim 1 , wherein the baseline value is based on the combined data. 
     
     
       9. A touch sensitive device comprising:
 a touch panel capable of sensing a proximate object; 
 scan logic couplable to the touch panel and capable of performing a scan of the touch panel so as to capture a first touch image; and 
 a processor capable of reconstructing a second touch image from the first touch image, the reconstructing including: 
 capturing the differential touch image at a touch sensitive device; 
 capturing common mode data at the device; 
 combining the captured differential touch image data and the common mode data; 
 performing a column alignment to align columns of the differential touch image to each other based on the combined data; and 
 performing a baseline alignment to align the column-aligned differential touch image with a baseline value. 
 
     
     
       10. The touch sensitive device of  claim 9 , wherein the reconstructing further comprises:
 sorting data values based on the combined data into bins of a histogram; 
 identifying a highest-count bin of the histogram; and 
 determining an offset value based on the highest-count bin. 
 
     
     
       11. The touch sensitive device of  claim 10 , wherein the baseline value is based on the determined offset value. 
     
     
       12. The touch sensitive device of  claim 10 , wherein values used to align columns of the differential touch image to each other are based on the determined offset value. 
     
     
       13. The touch sensitive device of  claim 10 , the reconstruction further comprising:
 identifying a second highest-count bin of the histogram; 
 in accordance with a determination that the first and second highest-count bins are adjacent, the determined offset value is a weighted average of the center values for the two identified bins; and 
 in accordance with a determination that the two identified bins are not adjacent, the determined offset value is the center value of the highest-count bin. 
 
     
     
       14. The touch sensitive device of  claim 9 , wherein combining the captured differential touch image data and the common mode data comprises scaling the differential touch image data and the common mode data based on a number of data values in the differential image indicative of a touching or hovering object. 
     
     
       15. The touch sensitive device of  claim 14 , wherein the scaling reduces contribution of the common mode data to the combined data as an amount of data indicating a touching or hovering object in differential touch image data increases. 
     
     
       16. The touch sensitive device of  claim 9 , wherein the baseline value is based on the combined data. 
     
     
       17. A method for reconstructing an original touch image from a differential touch image, comprising:
 capturing the differential touch image at a touch sensitive device; 
 capturing common mode data at the device; 
 performing a column alignment to align columns of the differential touch image to each other based on the captured common mode data; and 
 performing a baseline alignment to align the column-aligned differential touch image with a baseline value. 
 
     
     
       18. The method of  claim 17 , further comprising:
 sorting data values based on the combined data into bins of a histogram; 
 identifying a highest-count bin of the histogram; and 
 determining an offset value based on the highest-count bin. 
 
     
     
       19. The method of  claim 18 , further comprising:
 identifying a second highest-count bin of the histogram; 
 in accordance with a determination that the first and second highest-count bins are adjacent, the determined offset value is a weighted average of the center values for the two identified bins; and 
 in accordance with a determination that the two identified bins are not adjacent, the determined offset value is the center value of the highest-count bin. 
 
     
     
       20. The method of  claim 17 , wherein combining the captured differential touch image data and the common mode data comprises scaling the differential touch image data and the common mode data based on a number of data values in the differential image indicative of a touching or hovering object.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 13/448,182, filed Apr. 16, 2016 and published on Oct. 17, 2013 as U.S. Publication No. 2013-0271427, the contents of which are incorporated by reference herein in their entirety for all intended purposes. 
    
    
     FIELD 
     This relates generally to a differential touch image and more specifically to reconstructing an original touch image from a differential touch image. 
     BACKGROUND 
       FIG. 1  illustrates an exemplary touch image captured by a touch sensitive device, such as a touch panel. In the example of  FIG. 1 , touch image  100  can include rows  101  and columns  102  of image data (depicted by small circles), where the image data values can indicate an area  110  of the device at which an object touched or hovered over the panel (depicted by a broken circle), and the remaining untouched area  120 . 
     Two types of touch images that can be captured include an original touch image and a differential touch image, depending on the scan configuration of the device.  FIG. 2  illustrates an exemplary column  102  from the touch image  100  of  FIG. 1  showing what the image data  225  can look like in an original touch image and what the image data  235  can look like in a differential touch image. For explanatory purposes, the image data is shown in  FIG. 2  as continuous, although it should be understood that the image data can also be discrete as illustrated in  FIG. 1 . In the original touch image, the image data indicative of an untouched area  225   a  can have a value of zero and the image data indicative of a touched area  225   b  can have values greater than zero, depending on the proximity of the touching or hovering object. The original touch image can include actual magnitudes of the image values. In the differential touch image, each column of image data can be balanced to have an average value of zero, i.e., having the DC information removed. As such, the image data indicative of an untouched area  235   a  can have a negative value and the image data indicative of a touched area  235   b  can have negative and positive values depending on the proximity of the touching or hovering object. As such, the column&#39;s negative and positive values can balance out to zero. The differential touch image can include differential (or relative) magnitudes of the image values. 
     A touch sensitive device capable of generating a differential touch image can have some advantages. For example, some device hardware used to generate an original touch image can be eliminated, thereby freeing up space to expand the device&#39;s viewing area. Also, image effects, such as thermal drift, can have little or no effect on a differential image. However, because many touch sensitive devices require the actual image value magnitudes (such as provided by an original touch image) to perform various functions, a differential touch image can have limited use. 
     SUMMARY 
     This relates to reconstructing an original touch image from a differential touch image in a touch sensitive device. A reconstruction method can include aligning columns of the differential touch image relative to each other based on image data indicative of no touch or hover (“untouched data”). Untouched data can more clearly show how much column alignment is needed compared to image data indicative of a touch or hover (“touched data”). The method can further include aligning the image to a baseline value based on the untouched data in order to restore the previously removed DC information to the differential image. The untouched data can more clearly show how much baseline alignment is needed compared to the touched data. In one example, the reconstruction can be performed using the differential image data alone. In another example, the reconstruction can be performed using the differential image data and common mode data indicative of the missing image column average. The ability to use the differential touch image to reconstruct the original touch image can advantageously provide the benefits of the differential image while performing device functions with the reconstructed original image. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an exemplary touch image according to various embodiments. 
         FIG. 2  illustrates an exemplary column of an original touch image and a differential touch image according to various embodiments. 
         FIG. 3  illustrates an exemplary method for reconstructing an original touch image from a differential touch image according to various embodiments. 
         FIG. 4  illustrates an exemplary row of a differential touch image before and after column alignment according to various embodiments. 
         FIG. 5  illustrates an exemplary row of a differential touch image before and after baseline alignment according to various embodiments. 
         FIG. 6  illustrates an exemplary method for reconstructing an original touch image from a differential touch image based on image data according to various embodiments. 
         FIGS. 7A-7B  illustrate an exemplary method for detecting baseline inversion in a differential touch image according to various embodiments. 
         FIGS. 8A-8B  illustrate an exemplary method for reconstructing an original touch image from a differential touch image based on image data and common mode data according to various embodiments. 
         FIGS. 9A-9B  illustrate another exemplary method for reconstructing an original touch image from a differential touch image based on image data and common mode data according to various embodiments. 
         FIG. 10  illustrates another exemplary method for reconstructing an original touch image from a differential touch image based on image data according to various embodiments. 
         FIG. 11  illustrates an exemplary computing system that can perform original touch image reconstruction according to various embodiments. 
         FIG. 12  illustrates an exemplary mobile telephone that can perform original touch image reconstruction according to various embodiments. 
         FIG. 13  illustrates an exemplary digital media player that can perform original touch image reconstruction according to various embodiments. 
         FIG. 14  illustrates an exemplary portable computer that can perform original touch image reconstruction according to various embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description of example embodiments, reference is made to the accompanying drawings in which it is shown by way of illustration specific embodiments that can be practiced. It is to be understood that other embodiments can be used and structural changes can be made without departing from the scope of the various embodiments. 
     This relates to reconstructing an original touch image from a differential touch image in a touch sensitive device. A reconstruction method can include aligning columns of the differential touch image relative to each other based on image data indicative of no touch or hover (“untouched data”). Untouched data can more clearly show how much column alignment is needed compared to image data indicative of a touch or hover (“touched data”). The method can further include aligning the image to a baseline value, representative of the previously removed DC information, based on the untouched data. The untouched data can more clearly show how much baseline alignment is needed compared to the touched data. In some embodiments, the reconstruction can be performed using the differential image data alone. In other embodiments, the reconstruction can be performed using the differential image data and common mode data (i.e., output from a touch device when there is no proximate object). The ability to use the differential touch image to reconstruct the original image can advantageously provide the benefits of the differential image while also performing device functions with the reconstructed original image. 
     In some embodiments, the differential touch image can be a fully differential touch image. Although reconstruction from offsets based on DC balancing is described herein, it should be understood that reconstruction based on any image offsets can be performed according to various embodiments. 
       FIG. 3  illustrates an exemplary method for reconstructing an original touch image from a differential touch image. In the example of  FIG. 3 , a differential touch image can be captured during a differential scan of a touch panel ( 310 ). In some embodiments, the differential scan can perform a differential drive operation as follows. All the rows of the touch panel or a subset of rows can be driven during each step of the scan. For the driven rows, some can be driven with stimulation signals having positive phase and others with stimulation signals having negative phase. The static (non-signal related) capacitance that is approximately uniform across the touch panel may not appear in the output, because of the combination of positive and negative phases in the drive signals. After several scan steps in which different combinations of rows are driven with various stimulation signal phases and patterns, the difference between the touch data values can be calculated, thereby forming the differential touch image. 
     In some embodiments, the differential scan can perform a differential sense operation as follows. The rows of the touch panel can be driven one at a time, for example, with a stimulation signal. The sensing amplifiers on the columns of the touch panel can be differential sensing amplifiers, where each measures the difference between two columns, between a column and the average of all the other columns, or between one subset of columns and another subset of columns. The static (non-signal related) capacitance that is approximately uniform across the touch panel may not appear in the output, because the sensing amplifiers measure only differences. After the scan, the difference between the touch image values on each row may be calculated, thereby forming the differential touch image. 
     In some embodiments, the differential scan can combine the differential drive and differential sense operations to capture the differential touch image. 
     As described previously in  FIG. 2 , each column of image data can have an average value of zero, with untouched data being negative and touched data being negative and positive depending on the object&#39;s proximity, thereby rendering the sum of the column zero. Due to differing touch levels (as well as noise and other effects), each column of image data can have different average values. These differences can be most apparent in the untouched data, where negative values for all the columns can tend to differ. To correct these differences, the columns can be aligned relative to each other using the untouched data as a guide ( 320 ). Methods for performing the column alignment will be described in detail below. 
       FIG. 4  illustrates an example of column alignment in the touch image of  FIG. 1 . In the example of  FIG. 4 , an exemplary row (labeled “A-A” in  FIG. 1 ) of untouched data is shown for a differential touch image. Prior to column alignment, the row shows the erroneous differences between the untouched data values  435  (depicted by small circles) across all the columns  102  in the image. After column alignment, the row shows the untouched data values  445  across all the columns  102  having the same negative value N. The data values for the touched data in each column (not shown) can be adjusted according to the adjustments made to their corresponding untouched data. 
     Referring again to  FIG. 3 , after the columns have been aligned, the image data still lacks the DC information. This can be most apparent in the column-aligned untouched data, where the data value including the DC information should be around zero, not negative. To restore the DC information, the image can be aligned relative to a baseline value B, representative of the DC information, using the column-aligned untouched data as a guide ( 330 ). Methods for performing the baseline alignment will be described in detail below. 
       FIG. 5  illustrates an example of baseline alignment in the column-aligned row of  FIG. 4 . In the example of  FIG. 5 , an exemplary row of column-aligned untouched data from  FIG. 4  is shown. Prior to baseline alignment, the row shows untouched data  545  (depicted by small circles) with a negative data value N. After baseline alignment, the row shows the image data  555  having a data value corresponding to the baseline value. In most embodiments, the baseline value is zero. The data values for the touched data in each column (not shown) can be adjusted according to the adjustments made to their corresponding untouched data, resulting in all positive values for the touched data. 
     Reconstructing an original touch image from a differential touch image can be formulated in mathematical terms. Mathematically, aligning the columns of a differential touch image with each other can be formulated as selecting a set of relative column offsets which minimize the row edge energy, where each edge has a weight vector associated with it. The weights can be chosen so as to weight the untouched data heavily. This is because the required column alignments can be more apparent in the untouched data, as described previously. Mathematically, aligning the image to a baseline value can be formulated as finding an overall offset of the image which minimizes the total image energy using the edge weights.  FIGS. 6 through 10  illustrate exemplary methods for performing the column and baseline alignments according to these mathematical formulations as will be described in detail below. 
       FIG. 6  illustrates an exemplary method for reconstructing an original touch image from a differential touch image using the differential image data. In the example of  FIG. 6 , a differential touch image can be captured during a differential scan of a touch panel ( 610 ). Consider a 3×3 differential touch image C as follows. 
                     C   =           C   11           C   12           C   13               C   21           C   22           C   23               C   31           C   32           C   33             ,           (   1   )               
where C ij =image data value, i=row, and j=column. For example, for 3×3 image C, C 23 =the image data value for row 2 and column 3.
 
     Row edge energy can be calculated from the image C and formed in a matrix E as follows ( 615 ). 
                     E   =           E   11           E   12           E   13               E   21           E   22           E   23               E   31           E   32           E   33             ,           (   2   )               
where E ij =row edge energy value defined as (C ij −C i,j+1 ). The first and last columns of the image C can be mathematically connected to treat the image as a cylinder for calculation purposes. For example, for 3×3 matrix E,
 
 E   11 =( C   11   −C   12 )  (3)
 
 E   12 =( C   12   −C   13 )  (4)
 
 E   13 =( C   13   −C   11 ).  (5)
 
     Weights associated with each edge can be calculated and formed in a matrix Was follows ( 620 ). 
                     W   =           W   11           W   12           W   13               W   21           W   22           W   23               W   31           W   32           W   33             ,           (   6   )               
where W ij =edge weight for E ij . For example, W 23 =edge weight for E 23 .
 
     The success of the column and baseline alignment can depend to some extent on choosing the appropriate edge weights. Ideally, the weights can be chosen based on the probability of the image data making up that edge being untouched data. A variety of weight calculations can be selected from. In one embodiment,
 
 W   ij =[100−( C   ij −MIN j )],  (7)
 
where MIN j =minimum data value of image column j.
 
     In another embodiment,
 
 W   ij =[100−abs( C   ij −MED j )],  (8)
 
where MED j =median of the negative data values of image column j. Here, only the negative data values are considered because the median of the negative values can be a more robust estimator of the current baseline value of the differential touch image. The estimated baseline value of the differential image can then be indicative of the baseline alignment to be done to reconstruct the original image.
 
     In still another embodiment,
 
 W   ij =max[( W   max −abs( C   ij −MED j )), W   min ],  (9)
 
where W min , W max =minimum and maximum edge weights, respectively. These weights can be set according to design or operational parameters of the device. For example, in some embodiments, W min =⅕, W max =75.
 
     In other embodiments, mode can be used instead of median,
 
 W   ij [100−abs( C   ij −MOD j )]  (10)
 
 W   ij =max[( W   max −abs( C   ij −MOD j )), W   min ],  (11)
 
where MOD j =modal of the negative data value of image column j.
 
     In other embodiments, a mean or a weighted mean of the negative data values of each image column j can be used rather than minimum, median, or mode to calculate weights, similar to Equations (7)-(11). Any other suitable image data parameters can be used according to various embodiments. 
     In other embodiments, any of the above weighting schemes can be used for an initial weighting. This initial weighting can then be modified to account for the presence of other non-touch/non-baseline effects, such as the negative pixel effect. One such weighting can including calculating the affected image data N ij  as follows. N ij =1, if there exists at least one image data value in image row i and image column j that is greater than zero. N ij =0, otherwise. Accordingly,
 
 W   ij,a   =W   ij   −W   ij   ·N   ij   ·k,   (12)
 
where k=an attenuation factor. In some embodiments, k=0.5. Here, weights W ij  for negative pixel influenced image data can be attenuated to produce attenuated weights W ij,a .
 
     In the above embodiments, the edge weights can be the minimum of two weights. In other embodiments, the maximum of the two weights and their arithmetic and geometric mean can be considered. 
     Also, in the above embodiments, the edge weights can be determined for the general case. In other embodiments, the edge weights can be determined based on variance in the untouched data of each column. To do this, the mode of each column can be calculated and the number of occurrences of the mode determined. If the mode is sufficiently strong, i.e., the number of occurrences high, a smaller maximum edge weight and the modal formulation (Equations (10), (11)) can be used. As the mode weakens, i.e., the number of occurrences decreases, the variance can increase. If the mode is sufficiently weak, i.e., the number of occurrences is low enough, a larger edge weight can be used with the median formulation (Equations (8), (9)). This can allow the weight calculation to better reflect the available data. 
     Relative column offsets R, i.e., the amount of column alignment to be done for each column relative to an adjacent column, can be calculated as follows ( 625 ). First, the following mean square error for R j  can be minimized,
 
 E   ij   W   ij ( E   ij   +R   j ) 2 ,  (13)
 
where R j =relative offset for image column j.
 
     The relationship between the relative column offsets R and absolute column offsets A, i.e., the absolute amount of column alignment to be done for each column, as illustrated in Equation (17), can be formulated as follows.
 
 R   j   =A   j   −A   j-1   (14)
 
where A j =absolute offset for image column j. The first and last columns of the differential image C can be mathematically connected to treat the image as a cylinder for calculation purposes. For example, for 3×3 image C, R 1 =A 1 −A 3 .
 
     The partial derivatives of R j  can be set to zero,
 
∂ R   j =2Σ i   W   ij ( E   ij   +R   j )=0.  (15)
 
     R j  can be found, 
     
       
         
           
             
               
                 
                   
                     R 
                     j 
                   
                   = 
                   
                     
                       
                         
                           ∑ 
                           i 
                         
                         ⁢ 
                         
                           
                             W 
                             
                               i 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               j 
                             
                           
                           ⁢ 
                           
                             E 
                             
                               i 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               j 
                             
                           
                         
                       
                       
                         
                           ∑ 
                           i 
                         
                         ⁢ 
                         
                           W 
                           
                             i 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             j 
                           
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
           
         
       
     
     Absolute column offsets A can be calculated based on the relative offsets R as follows ( 630 ). 
     
       
         
           
             
               
                 
                   
                     
                       
                         M 
                         
                           i 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           j 
                         
                       
                       ⁢ 
                       
                         A 
                         j 
                       
                     
                     = 
                     
                       R 
                       j 
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   17 
                   ) 
                 
               
             
             
               
                 
                   M 
                   = 
                   
                     
                       
                         
                           1 
                         
                         
                           
                             - 
                             1 
                           
                         
                         
                           0 
                         
                       
                       
                         
                           0 
                         
                         
                           1 
                         
                         
                           
                             - 
                             1 
                           
                         
                       
                       
                         
                           
                             - 
                             1 
                           
                         
                         
                           0 
                         
                         
                           1 
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   18 
                   ) 
                 
               
             
           
         
       
     
     Equation (18) can have multiple solutions, such that there is no true inverse of matrix M. Accordingly, a pseudo-inverse matrix P can be generated and applied as follows to get the absolute column offsets A j  for each column.
 
 A   j   =P   ij   R   j ,  (19)
 
where P ij =pseudo-inverse value of corresponding M ij  value.
 
     Next, a global offset A g  can be calculated for the baseline alignment ( 635 ). The global offset A g  can be calculated based on image data at the borders of the image, because the border data can be assumed to be untouched data, as follows. 
                     A   g     =           ∑   border     ⁢       W     i   ⁢           ⁢   j       ⁡     (       C     i   ⁢           ⁢   j       +     A     i   ⁢           ⁢   j         )             ∑   border     ⁢     W     i   ⁢           ⁢   j           .             (   20   )               
The global offset A g  can also be formulated as a sum over the entire touch image, not just the border data.
 
     The absolute column offsets A j  can be adjusted based on the global offset as follows ( 640 ).
 
 A   j,f   =A   j   +A   g ,  (21)
 
where A j,f =adjusted absolute offset for image column j.
 
     The adjusted absolute offsets A j,f  can be applied to the differential touch image C as follows ( 645 ).
 
 I   ij   =C   ij   +A   j,f ,  (22)
 
where I ij =reconstructed original touch image data from differential touch image data C ij .
 
     Applying the offsets A j,f  can align the columns of the image C relative to each other and the image to the baseline, thereby reconstructing an original touch image I from the differential touch image C. 
     Although the example refers to 3×3 matrices, it is to be understood that the method applies to any suitable matrix sizes according to various embodiments. 
     An inverted baseline can occur when a touching or hovering object is present at the time that a baseline value for the touch panel is established. This can be problematic in a differential touch image because the inherent DC balancing of the image data can make the inverted case look like an intended actual touch or hover and because the current baseline value is estimated based on the median of the negative touch values (as described previously). For example, a differential touch image with a single inverted touch or hover can appear as two touching or hovering objects in the reconstructed original touch image. 
       FIGS. 7A-7B  illustrate an exemplary method for detecting these inversions in the method of  FIG. 6 . In the example of  FIGS. 7A-7B , common mode data can be captured at the touch panel ( 705 ). Common mode data can refer to data output from a touch panel when the rows in the panel are driven simultaneously, for example. In other words, common mode data can be the average values for the image columns. Note that the common mode data need not be accurate, i.e., need not have low noise and/or variance. Common mode data can be captured during a separate scan period than the differential touch image scan period. In some embodiments, multiple scans can be performed to capture common mode data for each column and the captured data averaged to provide an average common mode value for each column. 
     The method of  FIGS. 7A-7B  can then proceed in a similar manner as the method of  FIG. 6  (blocks  610 - 630 ) to capture a differential touch image C ( 710 ), calculate row edge energy E ( 715 ), calculate edge weights W ( 720 ), calculate relative column offsets R ( 725 ), and calculate absolute column offsets A ( 730 ). 
     After calculating the offsets A ( 730 ), common mode parameters can be calculated based on the captured common mode data as follows ( 735 ). The common mode mean CMM is 
     
       
         
           
             
               
                 
                   
                     
                       C 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       M 
                     
                     ⁢ 
                     
                         
                     
                     = 
                     
                       
                         
                           ∑ 
                           j 
                         
                         ⁢ 
                         
                           C 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             M 
                             j 
                           
                         
                       
                       
                         n 
                         cols 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   23 
                   ) 
                 
               
             
           
         
       
     
     where CM j =common mode data, and n cols =number of columns j in the image. The mean-subtracted common mode MCM is
 
MCM j =CM j −CMM.  (23)
 
     The mean Ā of the absolute column offsets A can be calculated as follows ( 740 ). The offsets A can effectively be an estimate of the common mode data. 
     
       
         
           
             
               
                 
                   
                     A 
                     _ 
                   
                   = 
                   
                     
                       
                         
                           ∑ 
                           j 
                         
                         ⁢ 
                         
                           A 
                           j 
                         
                       
                       
                         n 
                         cols 
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   25 
                   ) 
                 
               
             
           
         
       
     
     A first inversion detection test can be performed, in which a comparison can be made between the offset mean and the common mode mean ( 745 ). If the common mode mean CMM is substantially smaller than the offset mean Ā, a large-scale inversion is likely present. A panel scan can be performed to capture another baseline ( 785 ) and the reconstruction can be stopped for this image. If a large-scale inversion is not detected, a second inversion detection test can be performed to look for a smaller scale inversion as described below. 
     First, the mean-subtracted column offsets for each column Â j  can be calculated as follows ( 750 ).
 
 Â   j   =A   j   −Ā.   (26)
 
     The second inversion detection test can compare the mean-subtracted column offsets Â j  to the mean-subtracted common mode values MCM j  for each column ( 755 ). The column with the largest absolute value between the two can be selected ( 760 ). For that column, if the signed difference is substantially negative, an inversion is likely present. A panel scan can be performed to capture another baseline ( 785 ) and the reconstruction can be stopped for this image. If a smaller scale inversion is not detected, the image data can be deemed inversion-free. 
     The method can then perform in a similar manner as  FIG. 6  (blocks  635 - 645 ) for the inversion-free data to calculate a global offset A g  ( 770 ), adjust the absolute column offsets A with the global offset A g  ( 775 ), and apply the adjusted offset A j,f  to the differential touch image C to reconstruct an original touch image I ( 780 ). 
       FIGS. 8A-8B  illustrate an exemplary method for reconstructing an original touch image from a differential touch image using differential image data and common mode data. The method of  FIGS. 8A-8B  is similar to the method of  FIG. 6  with the addition of the use of common mode data to estimate the baseline value of an image column (rather than using a median of the negative touch values), thereby bypassing the inversion detection method of  FIGS. 7A-7B . 
     In the example of  FIGS. 8A-8B , common mode data CM can be captured at a touch panel in a similar manner as described in  FIGS. 7A-7B  ( 802 ). A differential touch image C can be captured at the touch panel in a similar manner as described in  FIG. 6  ( 804 ). The common mode data CM can be added to the captured image touch data C to create new image C′ as follows ( 806 ).
 
 C′   ij   =C   ij +CM j .  (27)
 
     The common mode data can include substantial noise in some instances, which can lead to significant variations between successive captured images. To reduce the noise effects, a histogram approach can be used to estimate a common mode mean offset CM o  as follows, under the assumption that most of the image data is untouched data, e.g., data indicative of no touching or hovering object. First, histogram bins over the range of data values in image C′ can be generated for a desired bin size ( 808 ). For example, in some embodiments, the bin size can be 16. For each data value in image C′ that falls into a given bin, that bin&#39;s count can be incremented by 2. 
     Half of the bin size can be added to the image C′ to produce image C (2)  ( 810 ). For example, in some embodiments, for a bin size of 16, a value of 8 can be added to each data value C′ ij  in the image C′. For each new data value (increased by half the bin size) in image C (2)  that falls into a given bin of the histogram, that bin&#39;s count can be incremented by 1 ( 812 ). Half the bin size can then be subtracted from the image C′ to produce image C (3)  ( 814 ). For example, in some embodiments, for a bin size of 16, a value of 8 can be subtracted from each data value C′ ij  in the image C′. For each new data value (decreased by half the bin size) in image C (3)  that falls into a given bin of the histogram, that bin&#39;s count can be incremented by 1 ( 816 ). 
     The histogram bins with the highest and second highest counts can be identified ( 818 ). If the identified bins are adjacent ( 820 ), the common mode mean offset CM o  can be the weighted average of the center values for the two bins ( 822 ). If not, the common mode mean offset CM o  can be the center value of the identified highest-count bin. ( 824 ). The offsets CM o  can be subtracted from the common mode values CM j  as follows ( 826 ).
 
BASE j =CM j −CM o .  (28)
 
These resulting values BASEj can be estimates of the baseline values for the image columns.
 
     Next, the method of  FIGS. 8A-8B  can perform in a similar manner as the method of  FIG. 6  (blocks  615 - 645 ). Row edge energy can be calculated ( 828 ). Edge weights W can be calculated using BASE j  as follows.
 
 W   ij =[100−abs( C   ij −BASE j )].  (29)
 
Equation (29) is similar to Equation (8) with BASE j  replacing MED j . Relative column offsets R can be calculated ( 832 ). Absolute column offsets A can be calculated ( 834 ). A global offset A g  can be calculated ( 836 ). The absolute column offsets A can be adjusted with the global offset A g  ( 838 ). The adjusted offsets A j,f  can be applied to the differential touch image C to reconstruct an original touch image I ( 840 ).
 
       FIGS. 9A-9B  illustrate another exemplary method for reconstructing an original touch image from a differential touch image using differential image data and common mode data. The method of  FIGS. 9A-9B  is similar to the method of  FIGS. 8A-8B  with the differences noted below. In the example of  FIGS. 9A-9B  (similar to  FIGS. 8A-8B  in blocks  802 - 826 ), common mode data can be captured ( 902 ), a differential touch image C captured ( 904 ), the common mode data CM added to the captured image C ( 906 ), and the histogram approach applied ( 908 - 926 ). 
     Common mode errors, such as noise and other errors, can directly couple into the reconstructed original touch image. To minimize these effects, common mode data can be limited to being used to directly reconstruct the original touch image only when there is no other source of information about the magnitudes of the differential image columns. The sum of the edge weights along a given edge, e.g., W 11 , W 21 , W 31 , etc., formed by image columns 1 and 2, can be indicative of the amount of magnitude information available for performing column alignment. This knowledge can be integrated into the method as follows. 
     After using the histogram approach to calculate BASE j , estimates of the baseline values for the image columns ( 926 ), BASE j  can be added to the differential touch image C to form image C′ as follows ( 928 ).
 
 C′   ij   =C   ij +BASE j .  (30)
 
     The row edge energy can be calculated for image C′ ( 930 ) in a similar manner as the method of  FIGS. 8A-8B  (block  828 ). Edge weights W can be calculated as follows.
 
 W   ij =max[( W   max −abs( C   ij )), W   min ].  (31)
 
     If common mode data is trustworthy, a penalty term can be applied for deviating from direct reconstruction, i.e., adding the common mode data to the differential image data to directly to reconstruct the original touch image, making the mean square error for R j ,
 
Σ ij   W   ij ( E   ij   +R   j ) 2   +γR   j   2 ,  (32)
 
where γ=a penalty factor. In general, γ can be inversely proportional to common mode noise. For example, in some embodiment, γ=1.0. If a differential image column includes mostly touched data, i.e., data that indicates a touching or hovering object, the method can weigh more heavily toward direct reconstruction because of the limited amount of untouched data, i.e., data that does not indicate a touching or hovering object, in that column to perform the column and baseline alignment.
 
     To minimize the mean square error, the partial derivatives of R j  can be set to zero,
 
∂ R   j =2Σ i   W   ij ( E   ij   +R   j )+2γ R   j =0.  (33)
 
     R j  can be found, 
     
       
         
           
             
               
                 
                   
                     R 
                     j 
                   
                   = 
                   
                     
                       
                         
                           ∑ 
                           i 
                         
                         ⁢ 
                         
                           
                             W 
                             
                               i 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               j 
                             
                           
                           ⁢ 
                           
                             E 
                             
                               i 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               j 
                             
                           
                         
                       
                       
                         
                           
                             ∑ 
                             i 
                           
                           ⁢ 
                           
                             W 
                             
                               i 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               j 
                             
                           
                         
                         + 
                         y 
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   34 
                   ) 
                 
               
             
           
         
       
     
     Next, the method of  FIGS. 9A-9B  can perform in a similar manner as the method of  FIGS. 8A-8B  (blocks  834 - 840 ) to calculate absolute column offsets A ( 936 ), calculate a global offset A g  ( 938 ), adjust the absolute column offsets A with the global offset A g  ( 940 ), and apply the adjusted offsets A j,f  to the differential touch image C to reconstruct an original touch image I ( 942 ). 
       FIG. 10  illustrates another exemplary method for reconstructing an original touch image from a differential touch image using the differential image data. In the example of  FIG. 10 , a differential touch image C can be captured ( 1005 ). In some embodiments, a minimum data value MIN j  for each image column j can be calculated ( 1010 ). The minimum data value can be added to the differential image data, thereby reconstructing the original touch image I from the differential touch image C as follows ( 1015 ).
 
 I   ij   =C   ij −MIN j .  (35)
 
     In some embodiments, a median MED j  of the negative data values of image column j can be calculated rather than MIN j  ( 1010 ). The median can be added to the differential image data, thereby reconstructing the original touch image I from the differential touch image C as follows ( 1015 ).
 
 I   ij   =C   ij −MED j .  (36)
 
     In some embodiments, a mode MOD j  of the negative data values of image column j can be calculated rather than MIN j  or MED j  ( 1010 ). The mode can be added to the differential image data, thereby reconstructing the original touch image I from the differential touch image C as follows ( 1015 ).
 
 I   ij   =C   ij −MOD j .  (37)
 
     In some embodiments, a mean or a weighted mean of the negative data values of each image column j can be calculated rather than MIN j , MED j , or MOD j  and then added to the differential image data to reconstruct the original touch image, similar to Equations (35)-(37). Any other suitable image data parameters can be used according to various embodiments. 
     It is to be understood that reconstruction methods are not limited to those of  FIGS. 6 through 10 , but can include other or additional actions capable of reconstructing an original touch image from a differential touch image according to various embodiments. It is further to be understood that the methods are not limited to differential touch images, but can be applied to any appropriate images in need of reconstruction. Although reconstruction from offsets due to DC balancing is described, it is to be understood that the methods can be used to correct for any arbitrary set of column offsets. 
       FIG. 11  illustrates an exemplary computing system  1100  that can reconstruct an original touch image from a differential touch image according to various embodiments. In the example of  FIG. 11 , computing system  1100  can include touch controller  1106 . The touch controller  1106  can be a single application specific integrated circuit (ASIC) that can include one or more processor subsystems  1102 , which can include one or more main processors, such as ARM968 processors or other processors with similar functionality and capabilities. However, in other embodiments, the processor functionality can be implemented instead by dedicated logic, such as a state machine. The processor subsystems  1102  can also include peripherals (not shown) such as random access memory (RAM) or other types of memory or storage, watchdog timers and the like. The touch controller  1106  can also include receive section  1107  for receiving signals, such as touch signals  1103  of one or more sense channels (not shown), other signals from other sensors such as sensor  1111 , etc. The touch controller  1106  can also include demodulation section  1109  such as a multistage vector demodulation engine, panel scan logic  1110 , and transmit section  1114  for transmitting stimulation signals  1116  to touch sensor panel  1124  to drive the panel. The panel scan logic  1110  can access RAM  1112 , autonomously read data from the sense channels, and provide control for the sense channels. In addition, the panel scan logic  1110  can control the transmit section  1114  to generate the stimulation signals  1116  at various frequencies and phases that can be selectively applied to rows of the touch sensor panel  1124 . 
     The touch controller  1106  can also include charge pump  1115 , which can be used to generate the supply voltage for the transmit section  1114 . The stimulation signals  1116  can have amplitudes higher than the maximum voltage by cascading two charge store devices, e.g., capacitors, together to form the charge pump  1115 . Therefore, the stimulus voltage can be higher (e.g., 6V) than the voltage level a single capacitor can handle (e.g., 3.6 V). Although  FIG. 11  shows the charge pump  1115  separate from the transmit section  1114 , the charge pump can be part of the transmit section. 
     Touch sensor panel  1124  can include a capacitive sensing medium having drive lines and multiple strips of sense lines according to various embodiments. The drive and sense line strips can be formed from a transparent conductive medium such as Indium Tin Oxide (ITO) or Antimony Tin Oxide (ATO), although other transparent and non-transparent materials such as copper can also be used. The drive lines and sense line strips can be formed on a single side of a substantially transparent substrate separated by a substantially transparent dielectric material, on opposite sides of the substrate, on two separate substrates separated by the dielectric material, etc. 
     Computing system  1100  can also include host processor  1128  for receiving outputs from the processor subsystems  1102  and performing actions based on the outputs that can include, but are not limited to, moving an object such as a cursor or pointer, scrolling or panning, adjusting control settings, opening a file or document, viewing a menu, making a selection, executing instructions, operating a peripheral device coupled to the host device, answering a telephone call, placing a telephone call, terminating a telephone call, changing the volume or audio settings, storing information related to telephone communications such as addresses, frequently dialed numbers, received calls, missed calls, logging onto a computer or a computer network, permitting authorized individuals access to restricted areas of the computer or computer network, loading a user profile associated with a user&#39;s preferred arrangement of the computer desktop, permitting access to web content, launching a particular program, encrypting or decoding a message, and/or the like. The host processor  1128  can also perform additional functions that may not be related to panel processing, and can be coupled to program storage  1132  and display device  1130  such as an LCD display for providing a UI to a user of the device. In some embodiments, the host processor  1128  can be a separate component from the touch controller  1106 , as shown. In other embodiments, the host processor  1128  can be included as part of the touch controller  1106 . In still other embodiments, the functions of the host processor  1128  can be performed by the processor subsystem  1102  and/or distributed among other components of the touch controller  1106 . The display device  1130  together with the touch sensor panel  1124 , when located partially or entirely under the touch sensor panel or when integrated with the touch sensor panel, can form a touch sensitive device such as a touch screen. 
     Note that one or more of the functions described above can be performed, for example, by firmware stored in memory (e.g., one of the peripherals) and executed by the processor subsystem  1102 , or stored in the program storage  1132  and executed by the host processor  1128 . The firmware can also be stored and/or transported within any non-transitory computer readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “non-transitory computer readable storage medium” can be any non-transitory medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device. The non-transitory computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disc such a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as compact flash cards, secured digital cards, USB memory devices, memory sticks, and the like. 
     The firmware can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “transport medium” can be any non-transitory medium that can communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The transport medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic or infrared wired or wireless propagation medium. 
     It is to be understood that the touch panel is not limited to touch, as described in  FIG. 11 , but can be a proximity panel or any other panel according to various embodiments. In addition, the touch sensor panel described herein can be either a single-touch or a multi-touch sensor panel. 
     It is further to be understood that the computing system is not limited to the components and configuration of  FIG. 11 , but can include other and/or additional components in various configurations capable of compensating for a negative pixel effect according to various embodiments. 
       FIG. 12  illustrates an exemplary mobile telephone  1200  that can include touch sensor panel  1224 , display  1236 , and other computing system blocks, capable of reconstructing an original touch image from a differential touch image according to various embodiments. 
       FIG. 13  illustrates an exemplary digital media player  1300  that can include touch sensor panel  1324 , display  1336 , and other computing system blocks, capable of reconstructing an original touch image from a differential touch image according to various embodiments. 
       FIG. 14  illustrates an exemplary personal computer  1400  that can include touch sensor panel (trackpad)  1424 , display  1436 , and other computing system blocks, capable of reconstructing an original touch image from a differential touch image according to various embodiments. 
     The mobile telephone, media player, and personal computer of  FIGS. 12 through 14  can realize improved accuracy with capability to reconstruct an original touch image from a differential touch image according to various embodiments. 
     Although embodiments have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the various embodiments as defined by the appended claims.

Metadata:
Filing Date: 20160502
Publication Date: 20180123
Grant Date: 20180123
Priority Date: 20120416
Inventors: BENBASAT ARI Y.
VIETA WILLIAM MATTHEW
WESTERMAN WAYNE CARL
HOTELLING STEVEN P.
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/0418", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0418", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0418", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 48191034