PATENT DOCUMENT

Publication Number: US-11475865-B2
Application Number: US-202117149415-A
Country: US
Kind Code: B2

Title: Backlight reconstruction and compensation

Abstract:
A processor or other circuitry may obtain emissive element strength information for an array of emissive elements of an electronic display. The processor or other circuitry may reconstruct backlight information at multiple locations within the electronic display. The processor or other circuitry also compensates display of image data based at least in part on the reconstructed backlight information.

Claims:
What is claimed is: 
     
       1. A method, comprising:
 at a processor, obtaining emissive element strength information for an array of emissive elements of an electronic display, wherein the emissive element strength information comprises singular value decomposition sets for a plurality of locations within the electronic display, and the singular value decomposition sets each comprise decomposed horizontal and vertical weights; 
 reconstructing, using the processor, backlight information at the plurality of locations within the electronic display; and 
 compensating display of image data based at least in part on the reconstructed backlight information. 
 
     
     
       2. The method of  claim 1 , wherein the emissive element strength information comprises a strength function of luminance of a respective emissive element of the array of emissive elements relative to driving levels. 
     
     
       3. The method of  claim 1 , wherein the array of emissive elements comprises a two-dimensional array of emissive elements. 
     
     
       4. The method of  claim 3 , wherein the plurality of locations are dispersed between locations of the emissive elements of the two-dimensional array of the emissive elements. 
     
     
       5. The method of  claim 1 , wherein compensating the electronic display of the image data comprises compensating the image data for different strengths of respective emissive elements of the array of emissive elements effecting emissivity at each location of the plurality of locations. 
     
     
       6. The method of  claim 5 , wherein compensating the image data comprises determining a backlight level for a plurality of pixels of the electronic display. 
     
     
       7. The method of  claim 6 , wherein compensating the image data comprises compensating image data at the plurality of pixels. 
     
     
       8. The method of  claim 6 , wherein determining the backlight level for the plurality of pixels comprises determining the backlight level at each of the plurality of pixels. 
     
     
       9. The method of  claim 8 , wherein determining the backlight level at each of the plurality of pixels comprises interpolating a respective pixel location backlight level from two or more of the plurality of locations. 
     
     
       10. The method of  claim 1 , wherein the emissive element strength information comprises chromaticity information for the array of emissive elements. 
     
     
       11. The method of  claim 10 , wherein compensating the image data comprises compensating for color drift due to a changing backlight level of the array of emissive elements. 
     
     
       12. The method of  claim 1 , wherein reconstructing the backlight information comprises selectively normalizing the backlight reconstruction information to a profile of gain values mapped for the plurality of locations by multiplying weighted luminance values from the backlight reconstruction information by respective gain values of the profile of gain values to normalize to the reconstructed backlight information to the profile of gain values when the profile is enabled. 
     
     
       13. A system comprising:
 statistics circuitry configured to generate statistics relating to display of image data on an electronic display, wherein the statistics comprise strength information for a plurality of emissive elements configured to backlight the electronic display, wherein the strength information comprises singular value decomposition sets for a plurality of locations within the electronic display, and the singular value decomposition sets each comprise decomposed horizontal and vertical weights; 
 backlight reconstruction and compensation system configured to receive the image data and the strength information, wherein the backlight reconstruction and compensation system comprises:
 backlight reconstruction circuitry configured to receive the strength information and reconstruct luminance levels of the backlight at the plurality of locations in the electronic display; and 
 backlight compensation circuitry configured to:
 receive the reconstructed luminance levels from the backlight reconstruction circuitry and the image data; and 
 adjust the image data to compensate for backlight variance at the plurality of locations based at least in part on the reconstructed luminance levels. 
 
 
 
     
     
       14. The system of  claim 13  comprising the electronic display. 
     
     
       15. The system of  claim 13 , wherein the plurality of emissive elements comprises a two-dimensional array of emissive elements. 
     
     
       16. The system of  claim 15 , where the plurality of locations comprises a plurality of grid points with the grid in a plane of the two-dimensional array of emissive elements. 
     
     
       17. The system of  claim 16 , wherein the reconstructed luminance levels comprises an amount of luminance at each grid point from one or more respective emissive elements of the plurality of emissive elements. 
     
     
       18. The system of  claim 16 , wherein adjusting the image data comprises determining a backlight luminance level for a pixel by interpolating two or more grid points of the plurality of grid points. 
     
     
       19. The system of  claim 13 , wherein the strength information comprises color drift information for the plurality of emissive elements, and adjusting the image data comprises compensating for the color drift information. 
     
     
       20. A method, comprising:
 at a processor, obtaining emissive element strength information for an array of emissive elements of an electronic display, wherein the emissive element strength information comprises singular value decomposition sets for a plurality of locations within the electronic display, and the singular value decomposition sets each comprise decomposed horizontal and vertical weights; 
 reconstructing, using the processor, backlight luminance information at the plurality of locations within the electronic display; 
 reconstructing, using the processor, backlight chromaticity information at the plurality of locations within the electronic display; 
 interpolating backlight luminance information for a pixel from two or more locations of the plurality of locations; 
 interpolating backlight chromaticity information for the pixel from the two or more locations; and 
 compensating display of image data based at least in part on the interpolated backlight luminance information and the interpolated backlight chromaticity.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to U.S. Provisional Patent Application No. 63/072,091, entitled “Backlight Reconstruction and Compensation,” filed Aug. 28, 2020, which this application incorporates in its entirety for all purposes. 
    
    
     BACKGROUND 
     The present disclosure relates generally to reconstructing a brightness and/or a color of a backlight at one or more pixels based on a strength (e.g., point spread function (PSF)) of backlight emissive elements (e.g., light emitting diode (LEDs)). 
     This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present techniques, which are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art. 
     Electronic displays may use one or more emissive elements (e.g., LEDs) to provide backlighting to display images on the electronic display. In embodiments where more than a single backlight emissive element is used, the response of the one or more emissive elements may have different strengths of emissivity. In other words, sending a signal to uniformly backlight at least a portion of the display may appear differently due to different strengths of emissivity of different backlight emissive elements of the display. These different strengths of the emissivity of the different emissive elements may be attributable to manufacturing process differences, different emissive element batches, differences in the different lines of transmission between a power supply and the respective emissive elements, and/or other differences in driving circuitry, the emissive elements, and/or the connections therebetween that may cause the different emissive elements to display different brightness levels. These differing brightness levels may cause artifacts to be visible on the display during operation of the display. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which: 
         FIG. 1  is a block diagram of an electronic device with a display having emissive elements, where the electronic device includes backlight reconstruction and compensation (BRC) unit to reconstruct and compensate differences in strengths of emissive elements, in accordance with an embodiment of the present disclosure; 
         FIG. 2  is one example of the electronic device of  FIG. 1 , in accordance with an embodiment of the present disclosure; 
         FIG. 3  is another example of the electronic device of  FIG. 1 , in accordance with an embodiment of the present disclosure; 
         FIG. 4  is another example of the electronic device of  FIG. 1 , in accordance with an embodiment of the present disclosure; 
         FIG. 5  is another example of the electronic device of  FIG. 1 , in accordance with an embodiment of the present disclosure; 
         FIG. 6  is a flow diagram of a process for driving a display using backlight reconstruction, in accordance with an embodiment of the present disclosure; 
         FIG. 7  is a block diagram of pixel contrast control (PCC) circuitry including the BRC unit of  FIG. 1 , in accordance with an embodiment of the present disclosure; 
         FIG. 8  is a graph of overlapping and non-overlapping portions of a display that may be used by the PCC circuitry of  FIG. 7 , in accordance with an embodiment of the present disclosure; 
         FIG. 9  is a graph of a backlight array with emissive elements and grid locations interspersed between the emissive elements and used to reconstruct the backlight, in accordance with an embodiment; and 
         FIG. 10  is a block diagram of the BRC unit of  FIG. 1 , in accordance with an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     One or more specific embodiments of the present disclosure will be described below. These described embodiments are only examples of the presently disclosed techniques. Additionally, in an effort to provide a concise description of these embodiments, all features of an actual implementation may not be described in the specification. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers&#39; specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be complex and time consuming, but may nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure. 
     When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” and “the” are intended to mean that there are one or more of the elements. The terms “including” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to “one embodiment,” “an embodiment,” “embodiments,” and “some embodiments” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. 
     An electronic display may utilize multiple emissive elements (e.g., LEDs) in an array (e.g., a two-dimensional array) to provide backlighting to the display in localized backlighting zones. Due to properties of the various emissive elements and/or other local backlighting differences between different backlighting zones, the backlight emissive elements may have differing strengths (e.g., point spread functions, referred to herein as PSFs) that may produce display artifacts. A point spread function may be used to model how light spreads and/or is distributed in space from some or from all backlight emissive elements. In some embodiments, the PSF for each backlight emissive element may be uniquely determined/modeled for a specific emissive element. As discussed in detail below, to address such issues, backlight reconstruction may be employed to determine the brightness and/or color at each pixel value based on the PSFs of the emissive elements and estimated brightness levels. Using the backlight reconstruction, the pixel values may be modified to account for the brightness and/or color of the backlight at each pixel position. 
     As will be described in more detail below, an electronic device  10  that uses such backlight reconstruction and compensation, such as the electronic device  10  shown in  FIG. 1 , may be any suitable electronic device, such as a computer, a mobile phone, a portable media device, a wearable device, a tablet, a television, a virtual-reality headset, and the like. Thus, it should be noted that  FIG. 1  is merely an example of a particular implementation and is intended to illustrate the types of components that may be present in the electronic device  10 . 
     In the depicted embodiment, the electronic device  10  includes the electronic display  12 , one or more input devices  14 , one or more input/output (I/O) ports  16 , a processor core complex  18  having one or more processor(s) or processor cores, local memory  20 , a main memory storage device  22 , a network interface  24 , a power source  25 , and a backlight reconstruction and compensation (BRC) unit  26 . The various components described in  FIG. 1  may include hardware elements (e.g., circuitry), software elements (e.g., a tangible, non-transitory computer-readable medium storing instructions), or a combination of both hardware and software elements. For example, the BRC unit  26  may be implemented as dedicated circuitry and/or instructions stored in the main memory storage device  22  that are executed using the processor core complex  18 . Moreover, while the BRC unit  26  is referred to here as a “unit,” this is meant to describe one example form that backlight reconstruction and compensation may take in an electronic device. Indeed, it may be unitary or modular in some cases, but may represent separate, non-unitary components implemented by separate components of the electronic device  10  in other cases. To provide one non-limiting example, backlight reconstruction may be independent of compensation (e.g., backlight reconstruction may be performed using software running on the processor core complex  18  while compensation may be performed by image processing circuitry in display pipeline). It should also be noted that the various depicted components may be combined into fewer components or separated into additional components. For example, the local memory  20  and the main memory storage device  22  may be included in a single component. 
     The processor core complex  18  may execute instruction stored in local memory  20  and/or the main memory storage device  22  to perform operations, such as generating and/or transmitting image data. As such, the processor core complex  18  may include one or more processors, such as one or more microprocessors, one or more application specific processors (ASICs), one or more field programmable logic arrays (FPGAs), one or more graphics processing units (GPUs), or the like. Furthermore, as previously noted, the processor core complex  18  may include one or more separate processing logical cores that each process data according to executable instructions. 
     The local memory  20  and/or the main memory storage device  22  may store the executable instructions as well as the data to be processed by the cores of the processor core complex  18 . Thus, the local memory  20  and/or the main memory storage device  22  may include one or more tangible, non-transitory, computer-readable media. For example, the local memory  20  and/or the main memory storage device  22  may include random access memory (RAM), read only memory (ROM), rewritable non-volatile memory such as flash memory, hard drives, optical discs, and the like. 
     The network interface  24  may facilitate communicating data with other electronic devices via network connections. For example, the network interface  24  (e.g., a radio frequency system) may enable the electronic device  10  to communicatively couple to a personal area network (PAN), such as a Bluetooth network, a local area network (LAN), such as an 802.11x Wi-Fi network, and/or a wide area network (WAN), such as a 4G, LTE, or 5G cellular network. The network interface  24  includes one or more antennas configured to communicate over network(s) connected to the electronic device  10 . 
     The power source  25  may include any suitable source of energy, such as a rechargeable lithium polymer (Li-poly) battery and/or an alternating current (AC) power converter. 
     The I/O ports  16  may enable the electronic device  10  to receive input data and/or output data using port connections. For example, a portable storage device may be connected to an I/O port  16  (e.g., Universal Serial Bus (USB)), thereby enabling the processor core complex  18  to communicate data with the portable storage device. The I/O ports  16  may include one or more speakers that output audio from the electronic device  10 . 
     The input devices  14  may facilitate user interaction with the electronic device  10  by receiving user inputs. For example, the input devices  14  may include one or more buttons, keyboards, mice, trackpads, and/or the like. The input devices  14  may also include one or more microphones that may be used to capture audio. 
     The input devices  14  may include touch-sensing components in the electronic display  12 . In such embodiments, the touch sensing components may receive user inputs by detecting occurrence and/or position of an object touching the surface of the electronic display  12 . 
     The electronic display  12  may include a display panel with one or more display pixels. The electronic display  12  may control light emission from the display pixels to present visual representations of information, such as a graphical user interface (GUI) of an operating system, an application interface, a still image, or video content, by display image frames based at least in part on corresponding image data. In some embodiments, the electronic display  12  may be a display using liquid crystal display (LCD), a self-emissive display, such as an organic light-emitting diode (OLED) display, or the like. 
     The BRC unit  26  may be used to reconstruct a backlight for the electronic display  12  using PSFs of emissive elements of the electronic display  12 . The backlight reconstruction is used to determine the brightness and/or color of the backlight at each pixel value based on the PSFs and estimated brightnesses. Using the determined brightnesses and/or colors, the BRC unit  26  is used to compensate for the different brightnesses and/or colors of the emissive elements backlighting specific pixel locations. For example, the BRC unit  26  may modify the image values for the respective pixel locations inverse to any color and/or brightness fluctuations of the local backlights at the pixel locations. 
     As described above, the electronic device  10  may be any suitable electronic device. To help illustrate, one example of a suitable electronic device  10 , specifically a handheld device  10 A, is shown in  FIG. 2 . In some embodiments, the handheld device  10 A may be a portable phone, a media player, a personal data organizer, a handheld game platform, and/or the like. For example, the handheld device  10 A may be a smart phone, such as any IPHONE® model available from Apple Inc. 
     The handheld device  10 A includes an enclosure  28  (e.g., housing). The enclosure  28  may protect interior components from physical damage and/or shield them from electromagnetic interference. In the depicted embodiment, the electronic display  12  is displaying a graphical user interface (GUI)  30  having an array of icons  32 . By way of example, when an icon  32  is selected either by an input device  14  or a touch-sensing component of the electronic display  12 , a corresponding application may launch. 
     The input devices  14  may extend through the enclosure  28 . As previously described, the input devices  14  may enable a user to interact with the handheld device  10 A. For example, the input devices  14  may enable the user to record audio, to activate or deactivate the handheld device  10 A, to navigate a user interface to a home screen, to navigate a user interface to a user-configurable application screen, to activate a voice-recognition feature, to provide volume control, and/or to toggle between vibrate and ring modes. The I/O ports  16  may also extend through the enclosure  28 . In some embodiments, the I/O ports  16  may include an audio jack to connect to external devices. As previously noted, the I/O ports  16  may include one or more speakers that output sounds from the handheld device  10 A. 
     Another example of a suitable electronic device  10  is a tablet device  10 B shown in  FIG. 3 . For illustrative purposes, the tablet device  10 B may be any IPAD® model available from Apple Inc. A further example of a suitable electronic device  10 , specifically a computer  10 C, is shown in  FIG. 4 . For illustrative purposes, the computer  10 C may be any MACBOOK® or IMAC® model available from Apple Inc. Another example of a suitable electronic device  10 , specifically a wearable device  10 D, is shown in  FIG. 5 . For illustrative purposes, the wearable device  10 D may be any APPLE WATCH® model available from Apple Inc. As depicted, the tablet device  10 B, the computer  10 C, and the wearable device  10 D each also includes an electronic display  12 , input devices  14 , and an enclosure  28 . 
       FIG. 6  is a flow diagram of a process  100  that may be utilized by the BRC unit  26 . Specifically, the BRC unit  26  may obtain emissive element strengths for an array of emissive elements of the electronic display  12  (block  102 ). The strengths may pertain to an overall brightness of the individual emissive elements and/or may refer to brightnesses at different wavelengths (e.g., different colors) of the emissive elements. The strengths of the pixels may be indicated using a point spread function (PSF) that provides different brightnesses and/or colors for different pixel values for one or more emissive elements of the display. Using the strengths, the BRC unit  26  reconstructs the backlight for the electronic display  12  (block  104 ). For instance, the BRC unit  26  may determine a brightness and/or color for one or more pixels of the electronic display  12 . For instance, the BRC unit  26  may determine what the backlight looks like at a point (e.g., a pixel) of the electronic display  12 . The reconstruction may include defining two or more overlapped zones and/or non-overlapped zones of pixels to determine the brightnesses and/or color. The overlapped zones may be defined as extensions of the non-overlapped zones. Using the determined brightness and/or color, the BRC unit  26  compensates for the backlight variance based at least in part on the strengths (block  106 ). For instance, the image data values (e.g., in a linear or gamma domain) of respective pixels may be compensated. In addition to or alternative to modifying image data values, the BRC unit  26  may cause the backlight driving to be compensated to increase uniformity. 
       FIG. 7  is a block diagram of pixel contrast control (PCC) circuitry I/O that includes the BRC unit  26 . The BRC unit  26  receives emissive element strengths  112  and image data  113 . As illustrated, the BRC unit  26  includes a backlight reconstruction component  114  and a backlight compensation component  116 . The BRC unit  26  also receives brightness estimations  118  from brightness estimation circuitry  120 . Brightness estimation is used to estimate the brightness of individual addressable backlight zones based on pixel values of the content to enhance contrast while preserving detail and reducing (e.g., minimizing) halo and flicker and to generate compensated image data  122  that compensates for backlight brightnesses and/or colors. Statistics circuitry  124  generates statistics including local statistics based on overlapped zones of the electronic display  12 , local statistics based on non-overlapped zones of the electronic display  12 , and/or global statistics. An emissive element processor  126  uses the statistics to compute brightnesses for the individually addressable backlight zones based on the pixel values of the content. The local statistics may be particularly useful in displays with local dimming while global statistics may be applicable to displays with global backlight and to displays with local dimming. The statistics calculated in the statistics circuitry  124  may include brightness maximums, brightness minimums, brightness averages, en-gamma/de-gamma information, uniformity statistics, and/or other information. 
       FIG. 8  is a graph of portions  130  and  131  of the electronic display  12 . In the portions  130  and  131 , non-overlapped zones  132  (individually referred to as non-overlapped zones  132 A,  132 B,  132 C,  132 D,  132 E,  132 F,  132 G, and  132 H). The portions  130  and  131  also includes include overlapped zones  134  (individually referred to as  134 A and  134 B). At edges of an active area of the electronic display  12 , the overlapped zones  134  start at an edge of a respective non-overlapping zone  132  and extends beyond the borders of the non-overlapping zone  132 . As illustrated, the overlapped zone  134 A includes a significant portion (e.g., all) of the non-overlapped zone  132 A and a vertical overlap  136  that extends into portions of the non-overlapped zones  132 B and  132 D. Similarly, the overlapped zone  134 A includes a horizontal overlap  138  that extends into portions of the non-overlapped zones  132 C and  132 D. 
     Away from the edge of the active area, the overlapped zones  134  may extend around a single non-overlapped zone  132  in multiple directions. For example, the overlapped zone  134 B includes a significant portion of the non-overlapped zone  132 F and a first vertical overlap  140  that extends above the non-overlapped zone  132 F into non-overlapping zone  132 E and  132 G. The overlapped zone  134 B also includes a second vertical overlap  142  extending below the non-overlapped zone  132 F. The overlapped zone  134 B also includes a first horizontal overlap  144  and a second horizontal overlap  146  that extends into non-overlapped zones  132 G and  132 H. 
     Returning to  FIG. 7 , the emissive element processor  126  may be included in the processor core complex  18 , may be performed by the processor core complex  18 , and/or may include a dedicated coprocessor that supplements processing of the processor core complex  18 . The brightness estimations  118  are computed from the gathered statistics from the statistics circuitry  124  for emissive elements in a two-dimensional array of the emissive elements. 
     The emissive element processor  126  also utilizes a two-dimensional convolution filter  148 . The two-dimensional convolution filter  148  applies any suitable filter that may provide filtering in two dimensions. In one example, the two-dimensional convolution filter  148  includes a two-dimensional FIR filter on elements of data sets sent over from the emissive element processor  126 . 
     The emissive element processor  126  may also utilize a two-dimensional bilateral filter  150 . The two-dimensional bilateral filter  150  applies a bilateral filter to values of a number (e.g., 7) of emissive elements and takes a weighted average of the number of emissive element values. The weighting in the two-dimensional bilateral filter  150  may be based on distance of the emissive elements from a reference point and/or intensity of the values of the respective emissive elements. In some embodiments, the weighting average may be based on long division. However, since the range of expected values is limited, an approximation of the results may be made from one or more data sets. If the initial approximation is sufficiently precise, the bilateral filtration process proceeds. If additional precision is to be used, a number (e.g., 1) of Newton-Raphson update steps may be used to converge from the initial approximation to the desired precision. 
     The emissive element processor  126  may also utilize a temporal filter  152  that is used to temporally filter data from the emissive element processor  126 . For instance, when the temporal filter  152  is activated, it may function as an infinite impulse response (IIR) filter. The temporal filter  152  may be configured in a global filtering mode that causes the temporal filter to function as a classic IIR filter with asymmetric gains to allow for different transition speeds for dark-to-bright transitions and bright-to-dark transitions. When configured in a local filtering mode, for each emissive element, a local parameter is computed based on previous local parameters and emissive element differences. 
     A copy engine  154  may be used to write the brightness estimations  118  to the backlight reconstruction component  114 . The copy engine  154  copies the elements of the input data set to multiple output locations with optional processing for each output. For instance, the optional processing may include enabling/disabling scaling using a scale factor, a minimum limit for a brightness threshold, scaling based on system level brightness settings, and/or other processing of the brightness estimations  118  from the emissive element processor  126 . 
     A power function  156  may utilize hardware and/or software to adjust the brightness estimations based on power/power settings for the electronic device  10 . A division function  158  may utilize hardware and/or software to perform division. For example, the division function  158  may include a hardware accelerator that utilizes a polynomial approximation of the division where the polynomial used to approximate the division is based on the input range of the value being divided. When an additional precision is to be used for the long division, the polynomial approximation may converge to the point of precision using a Newton-Raphson update step. 
     Backlight reconstruction may utilize a backlight grid. The backlight grid includes a grid of the emissive elements and specifies a number of intermediate points in between the emissive elements. For example,  FIG. 9  illustrates an example grid  160  that represents at least a portion of backlighting for the electronic display  12 . As illustrated, the grid  160  includes twelve emissive elements  162  in three rows. As illustrated, grid points  164  are dispersed between the emissive elements  162 . The distribution, location, and/or number of the grid points  164  may be set using corresponding input parameters. For instance, an offset and/or spacing parameter may be used to set how far to offset a grid point  164  from an edge of the active area of the electronic display  12 , from another grid point  164 , and/or from an emissive element  162 . Furthermore, a number of rows or columns of grid points  164  may be set using respective number parameters. 
       FIG. 10  illustrates a block diagram of an embodiment of the BRC unit  26 . As illustrated, the BRC receives emissive element strengths  112 . The emissive element strengths  112  may be received in singular value decomposition (SVD) sets  190 . Accordingly, in such embodiments, the reconstruction of the backlight may be performed by applying the strengths for one or more (e.g., each) emissive element  162  of the backlight of the electronic display  12 . The SVD sets  190  may be fetched from the local memory  20  using a direct memory access (DMA) channel. In some embodiments, the SVD sets  190  may be stored in the local memory  20  in a raster-scan order of the associated emissive elements  162  associated the emissive element strengths  112 . The number of SVD sets  190  may be controlled using a parameter set for the BRC unit  26  using an SVD number parameter. 
     The reconstruction of the backlight at each grid point  164  is achieved by applying the strengths for each emissive element  162  to the brightness value for the emissive element  162  using the brightness estimation discussed above. In some embodiments, only a portion of the emissive elements  162  are used to apply the strengths for backlight reconstruction. For each emissive element  162  used in the backlight reconstruction, the emissive element strengths  112  of the emissive element  162  is included in the SVD sets  190  (e.g., up to a number of sets selectable using a set parameter). In each SVD set  190  a grid point coordinate  192  is used to determine how much effect the respective emissive element has on the backlight at the grid point coordinate  192 . For instance, a horizontal weight  194  and a vertical weight  196  may be applied to the emissive element strengths  112  using one or more multipliers  198  to apply the horizontal weight  194  and the vertical weight  196 . Weighted strengths  204  from the SVD sets  190  are summed together in one or more adders  206  to form weight sum  208 . 
     In some embodiments, the emissive element strengths  112  may indicate a non-uniformity in color. For example, the emissive element strengths  112  may be related to color shifts in the International Commission on Illumination (CIE)  1931  XYZ color space. Based on the non-uniformity in color, chrominance (e.g., (X, Z)) compensation may be activated in the backlight reconstruction. Chrominance compensation data may be stored in the form of ratios Z/Y  210  and X/Y  212 . The weighted sum  208  is multiplied by the brightness estimations  118  in multipliers  214 ,  216 , and  218 . In the multiplier  214 , the weighted sum is multiplied by the ratio Z/Y  219  in addition to the brightness estimations  118 , and in the multiplier  216 , the weighted sum  208  is multiplied by the ratio X/Y  212  in addition to the brightness estimations  118 . Summing circuitries  220 ,  222 , and  224  may be used to sum the scaled weighted sums  208  for the respective paths in the backlight reconstruction component  114 . The outputs of the summing circuitries  220 ,  222 , and  224  are each submitted to a XYZ-to-RGB converter  226  that is used to reconstruct the backlight into RGB when backlight color compensation is enabled. For instance, a 3×3 transform may be used to convert the XYZ values computed at each grid point to linear RGB values. When color compensation is not enabled, in some embodiments, luminance may be solely compensated using the Y channel (through the summing circuitry  222 ). 
     Furthermore, when backlight color compensation is enabled, a global target color (e.g., an XY color) or a local target color (e.g., an XY color) may be calculated in a target-to-RGB converter  228 . This conversion to target color is based at least in part on the luminance in the Y channel using the Z/Y ratio  210  and the X/Y ratio  212  and with Z equaling 1−X−Y. 
     When color compensation is enabled, the RGB values of the target color (global or local) and the reconstructed values are transmitted to an RGB gain calculator  230  that calculates gains for in RGB values. The RGB gains may be calculated using component-wise division followed by global scaling of the ratios. The component-wise division may be estimated using one of a number (e.g., 16) of polynomials. If additional precision is to be used, the RGB gain calculator  230  may apply one or more update steps using the Newton-Raphson method. Accordingly, the reconstructed backlight at each of the grid points  164  may be converted to RGB gain values using an interpolation engine  234  and pixel coordinates  232 . 
     As may be appreciated, the grid points  164  may be at a lower resolution than pixels of the electronic display  12  to reduce processing/storage costs for determining and/or storing information for each individual pixel. Accordingly, to accommodate compensation at the pixels with a different resolution than the emissive elements  162 , the RGB gain values for each grid point  164  may be used to interpolate for pixels between the grid points  164  based on a location of the respective pixels in relation to respective grid points  164 . For example, the interpolation may include bilinear interpolation for both vertical and horizontal directions from respective closest grid points  164 . In some embodiments, the grid points  164  may have a same resolution as the pixels of the electronic display  12  where backlight information may be determined and/or stored for each individual pixel. 
     In some embodiments, the backlight reconstruction is to be normalized to an all-on profile  236 . The all-on profile  236  represents all emissive elements  162  being set to a same brightness. The all-on profile  236  may be conceptualized as a map of gains. This all-on profile  236  or map of gains is static and defined with the resolution of the grid points  164 . The all-on profile  236  is fetched and stored prior to a first frame being displayed following a power up of the electronic display  12 . This all-on profile  236  is combined with the weighted luminance in the Y channel using a multiplier  238 . The result of the multiplier is then interpolated in an interpolation engine  240  similar to how the output of the RGB gain calculator  230  is interpolated to the pixel resolution. 
     The interpolated values from the interpolation engines  234  and  240  are transmitted to the backlight compensation component  116  that includes a pixel modifier  242 . The pixel modifier  242  modifies the image data  113  to generate the compensated image data  122 . In some embodiments, the compensated image data  122  may undergo additional manipulation. For example, the compensated image data  122  may be used to cause a liquid crystal (LC) to open more fully when a backlight is lower than an expected value. Additionally or alternatively, the backlight level of one or more locations may be lowered to reduce power when one or more grid locations indicate that the blacklight level is above a target value. 
     Components/units discussed herein may include software implemented in the processor, LED processor, other processors/coprocessors using instructions stored in the storage device(s)  22  and/or the memory  20 . Additionally or alternatively, various components and/or units of the components/units discussed herein may be implemented with application-specific hardware circuitry, such as an application-specific integrated circuit (ASIC). 
     The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure. 
     The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function] . . . ” or “step for [perform]ing [a function] . . . ”, it is intended that such elements are to be interpreted under 35 U.S.C. 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. 112(f).

Metadata:
Filing Date: 20210114
Publication Date: 20221018
Grant Date: 20221018
Priority Date: 20200828
Inventors: CHAPPALLI, MAHESH B.
Assignee: APPLE INC
CPC Classifications: [{"code": "G09G3/3426", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G3/2003", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G2310/0237", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0242", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0233", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2360/16", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0233", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G5/10", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G2340/06", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0646", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G3/2003", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G2320/0233", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G5/10", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G2310/0237", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G3/3426", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G2360/16", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/06", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0646", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0242", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2320/0233", "inventive": false, "first": false, "tree": "[]"}]
Family ID: 77802241