PATENT DOCUMENT

Publication Number: US-7652678-B2
Application Number: US-95755704-A
Country: US
Kind Code: B2

Title: Partial display updates in a windowing system using a programmable graphics processing unit

Abstract:
Techniques to generate partial display updates in a buffered window system in which arbitrary visual effects are permitted to any one or more windows (e.g., application-specific window buffers) are described. Once a display output region is identified for updating, the buffered window system is interrogated to determine which regions within each window, if any, may effect the identified output region. Such determination considers the consequences any filters associated with a window impose on the region needed to make the output update.

Claims:
1. A method to generate a partial display update in a windowing system having a plurality of display layers presented on a display device communicatively coupled to one or more general purpose central processing units, comprising:
 identifying, by one of the one or more general purpose central processing units, an output region associated with a top-most display layer, the output region having an associated output size and location; 
 identifying, by one of the one or more general purpose central processing units, a buffer having a size and location corresponding to the output size and location; 
 identifying the top-most display layer as a current display layer; 
 determining if a filter is associated with the current display layer and, if there is, 
 determining an input region for the filter, said input region having an associated size and location, and 
 adjusting the buffer size and location to correspond to the union of the input region&#39;s size and location and the buffer&#39;s size and location; 
 setting the display layer immediately lower than the current display layer to the current display layer; 
 repeating the act of determining for each relevant display layer in the windowing system; 
 establishing an output buffer having a size and location to accommodate the size and location of the buffer; 
 compositing that portion of each display layer that overlaps the output buffer&#39;s location into the established output buffer; and 
 displaying the partial display update from the established output buffer on the display device. 
 
     
     
       2. The method of  claim 1 , wherein the act of identifying comprises obtaining output region information from a windowing subsystem. 
     
     
       3. The method of  claim 1 , wherein the act of establishing comprises instantiating an output buffer. 
     
     
       4. The method of  claim 1 , wherein the act of compositing comprises compositing each display layer that overlaps the output buffer&#39;s location beginning with a bottom-most display layer and proceeding in a linear fashion to the top-most display layer. 
     
     
       5. The method of  claim 1 , wherein the act of compositing uses one or more graphics processing units. 
     
     
       6. The method of  claim 5 , wherein the acts of identifying an output region, identifying a buffer, identifying the top-most display layer, determining if a filter is associated with the current display layer, setting the display layer immediately lower than the current display layer to the current display layer and establishing an output buffer use one or more general purpose central processing units. 
     
     
       7. The method of  claim 1 , further comprising transferring that portion of the output buffer corresponding to the output region&#39;s location to a frame buffer. 
     
     
       8. The method of  claim 1 , wherein the relevant display layers in the windowing system comprise those layers associated with a specified display unit. 
     
     
       9. A program storage device having computer-executable instructions stored therein for performing the method recited in any one of  claims 1  through  8 . 
     
     
       10. A method to generate a partial display update on a display device communicatively coupled to one or more general purpose central processing units, comprising:
 identifying, by one of the one or more general purpose central processing units, an output region associated with a top-most display layer, the output region having an associated output size and location; 
 determining an input region for each of one or more filters, each of said one or more filters associated with a display layer and having an associated input size and location; 
 establishing a buffer having a size and location to accommodate the union of the output region&#39;s location and each of the one or more input regions&#39;locations; 
 compositing that portion of each display layer that overlaps the buffer&#39;s location into the established buffer; and 
 displaying the partial display update from the established buffer on the display device. 
 
     
     
       11. The method of  claim 10 , wherein the act of identifying comprises obtaining output region information from a windowing subsystem. 
     
     
       12. The method of  claim 10 , wherein the top-most display layer comprises an associated filter. 
     
     
       13. The method of  claim 10 , wherein the act of compositing comprises compositing each display layer that overlaps the buffer&#39;s location beginning with a bottom-most display layer and proceeding in a linear fashion to the top-most display layer. 
     
     
       14. The method of  claim 10 , wherein the act of compositing uses one or more graphics processing units. 
     
     
       15. The method of  claim 14 , wherein the acts of identifying, determining and establishing uses one or more general purpose central processing units. 
     
     
       16. The method of  claim 10 , further comprising transferring that portion of the buffer corresponding to the output region&#39;s location to a frame buffer. 
     
     
       17. A program storage device having computer-executable instructions stored therein for performing the method recited in any one of  claims 10  through  16 . 
     
     
       18. A computer system, comprising:
 a central processing unit; 
 memory, operatively coupled to the central processing unit, said memory adapted to provide a plurality of application-specific window buffers, at least one assembly buffer and a frame buffer; 
 graphics processing unit operatively coupled to the frame buffer; 
 a display port operatively coupled to the frame buffer and adapted to couple to a display device; and 
 instructions stored in the memory for causing the central processing unit to
 identify an output region associated with a top-most application-specific window buffer, the output region having an associated output size and location, 
 determine an input region for each of one or more filters, each of said one or more filters associated with an application-specific window buffer and having an associated input size and location, 
 establish the assembly buffer to have a size and location corresponding to the union of the output region&#39;s location and the one or more input regions&#39;locations, and 
 use the graphics processing unit to composite that portion of each application-specific window buffer that overlaps the assembly buffer&#39;s established location into the assembly buffer taking into account any filter associated with the application-specific window buffer. 
 
 
     
     
       19. The system of  claim 18 , wherein the instructions further comprise instructions to transfer that portion of the assembly buffer corresponding to the output region&#39;s location to the frame buffer. 
     
     
       20. The system of  claim 18 , wherein the instructions to composite comprise instructions to composite each application-specific window buffer proceeds in a linear fashion from a bottom-most application-specific window buffer to a top-most application-specific window buffer. 
     
     
       21. The system of  claim 18 , wherein the instructions to identify, determine and establish use the general purpose central processing unit. 
     
     
       22. The system of  claim 18 , further comprising one or more additional central processing units operatively coupled to the memory.

Description:
BACKGROUND 
     This application claims priority to U.S. patent application Ser. No. 10/877,358, entitled “Display-Wide Visual Effects for a Windowing System using a Programmable Graphics Processing Unit,” filed 25 Jun. 2004 and which is hereby incorporated by reference. 
     The invention relates generally to computer display technology and, more particularly, to the application of visual effects using a programmable graphics processing unit. The subject matter of the invention is generally related to the following jointly owned and co-pending patent applications: “System for Reducing the Number of Programs Necessary to Render an Image,” by John Harper, Ser. No. 10/826,773; “System for Optimizing Graphics Operations” by John Harper, Ralph Brunner, Peter Graffagnino, and Mark Zimmer, Ser. No. 10/825,694; “System for Emulating Graphics Operations,” by John Harper, Ser. No. 10/826,744; and “High-Level Program Interface for Graphics Operations,” by John Harper, Ralph Brunner, Peter Graffagnino, and Mark Zimmer, Ser. No. 10/826,762, each incorporated herein by reference in its entirety. 
     Referring to  FIG. 1 , in prior art buffered window computer system  100 , each application (e.g., applications  105  and  110 ) has associated with it one or more window buffers or backing stores (e.g., buffers  115  and  120 —only one for each application is shown for convenience). Backing store&#39;s represent each application&#39;s visual display. Applications produce a visual effect (e.g., blurring or distortion) through manipulation of their associated backing store. At the operating system (“OS”) level, compositor  125  combines each application&#39;s backing store (in a manner that maintains their visual order) into a single “image” stored in assembly buffer  130 . Data stored in assembly buffer  130  is transferred to frame buffer  135  which is then used to drive display unit  140 . As indicated in  FIG. 1 , compositor  125  (an OS-level application) is implemented via instructions executed by computer system central processing unit (“CPU”)  145 . 
     Because of the limited power of CPU  145 , it has not been possible to provide more than rudimentary visual effects (e.g., translucency) at the system or display level. That is, while each application may effect substantially any desired visual effect or filter to their individual window buffer or backing store, it has not been possible to provide OS designers the ability to generate arbitrary visual effects at the screen or display level (e.g., by manipulation of assembly buffer  130  and/or frame buffer  135 ) without consuming virtually all of the system CPU&#39;s capability—which can lead to other problems such as poor user response and the like. 
     Thus, it would be beneficial to provide a mechanism by which a user (typically an OS-level programmer or designer) can systematically introduce arbitrary visual effects to windows as they are composited or to the final composited image prior to its display. 
     SUMMARY 
     Methods, devices and systems in accordance with the invention provide a means for performing partial display updates in a windowing system that permits layer-specific filtering. One method in accordance with the invention includes: identifying an output region associated with a top-most display layer (e.g., an application-specific window buffer), wherein the output region has an associated output size and location; determining an input region for each of one or more filters, wherein each of the one or more filters is associated with a display layer and has an associated input size and location (substantially any known visual effect filter may be accommodated); establishing a buffer (e.g., an assembly buffer) having a size and location that corresponds to the union of the output region&#39;s location and each of the one or more input regions&#39; locations; and compositing that portion of each display layer that overlaps the buffer&#39;s location into the established buffer. In one embodiment, that portion of the buffer corresponding to the identified output region is transferred to a frame buffer where it is used to update a user&#39;s display device. In another embodiment, the acts of identifying, determining and establishing are performed by one or more general purpose central processing units while the act of compositing is performed by one or more special purpose graphical processing units in a linear fashion (beginning with the bottom-most display layer and proceeding to the top-most display layer). 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a prior art buffered window computer system. 
         FIG. 2  shows a buffered window computer system in accordance with one embodiment of the invention. 
         FIGS. 3A and 3B  show a below-effect in accordance with one embodiment of the invention. 
         FIGS. 4A and 4B  show an on-effect in accordance with one embodiment of the invention. 
         FIGS. 5A and 5B  show an on-effect in accordance with another embodiment of the invention. 
         FIGS. 6A and 6B  show an above-effect in accordance with one embodiment of the invention. 
         FIGS. 7A and 7B  show a full-screen effect in accordance with one embodiment of the invention. 
         FIG. 8  shows, in block diagram form, a display whose visual presentation has been modified in accordance with the invention. 
         FIG. 9  shows, in flowchart form, an event processing technique in accordance with one embodiment of the invention. 
         FIG. 10  shows a system in which a partial display update in accordance with the prior art is performed. 
         FIG. 11  shows, in flowchart format, a partial display update technique in accordance with one embodiment of the invention. 
         FIG. 12  shows an illustrative system in accordance with the invention in which a partial display update is performed. 
     
    
    
     DETAILED DESCRIPTION 
     Methods and devices to generate partial display updates in a buffered window system in which arbitrary visual effects are permitted to any one or more windows are described. Once a display output region is identified for updating, the buffered window system is interrogated to determine which regions within each window, if any, may effect the identified output region. Such determination considers the consequences any filters associated with a window impose on the region needed to make the output update. The following embodiments of the invention, described in terms of the Mac OS X window server and compositing application, are illustrative only and are not to be considered limiting in any respect. (The Mac OS X operating system is developed, distributed and supported by Apple Computer, Inc. of Cupertino, Calif.) 
     Referring to  FIG. 2 , buffered window computer system  200  in accordance with one embodiment of the invention includes a plurality of applications (e.g., applications  205  and  210 ), each of which is associated with one or more backing stores, only one of which is shown for clarity and convenience (e.g., buffers  215  and  220 ). Compositor  225  (one component in an OS-level “window server” application) uses fragment programs executing on programmable graphics processing unit (“GPU”)  230  to combine, or composite, each application&#39;s backing store into a single “image” stored in assembly buffer  235  in conjunction with, possibly, temporary buffer  240 . Data stored in assembly buffer  235  is transferred to frame buffer  245  which is then used to drive display unit  250 . In accordance with one embodiment, compositer  225 /GPU  230  may also manipulate a data stream as it is transferred into frame buffer  245  to produce a desired visual effect on display  250 . 
     As used herein, a “fragment program” is a collection of program statements designed to execute on a programmable GPU. Typically, fragment programs specify how to compute a single output pixel—many such fragments being run in parallel on the GPU to generate the final output image. Because many pixels are processed in parallel, GPUs can provide dramatically improved image processing capability (e.g., speed) over methods that rely only on a computer system&#39;s CPU (which is also responsible for performing other system and application duties). 
     Techniques in accordance with the invention provide four (4) types of visual effects at the system or display level. In the first, hereinafter referred to as “before-effects,” visual effects are applied to a buffered window system&#39;s assembly buffer prior to compositing a target window. In the second, hereinafter referred to as “on-effects,” visual effects are applied to a target window as it is being composited into the system&#39;s assembly buffer or a filter is used that operates on two inputs at once to generate a final image—one input being the target window, the other being the contents of the assembly buffer. In the third, hereinafter referred to as “above-effects,” visual effects are applied to a system&#39;s assembly buffer after compositing a target window. And in the fourth, hereinafter referred to as “full-screen effects,” visual effects are applied to the system&#39;s assembly buffer as it is transmitted to the system&#39;s frame-buffer for display. 
     Referring to  FIGS. 3A and 3B , below-effect  300  in accordance with one embodiment of the invention is illustrated. In below-effect  300 , the windows beneath (i.e., windows already composited and stored in assembly buffer  235 ) a target window (e.g., contained in backing store  220 ) are filtered before the target window (e.g., contained in backing store  220 ) is composited. As shown, the contents of assembly buffer  235  are first transferred to temporary buffer  240  by GPU  230  (block  305  in  FIG. 3A  and ← in  FIG. 3B ). GPU  230  then filters the contents of temporary buffer  240  into assembly buffer  235  to apply the desired visual effect (block  310  in  FIG. 3A  and ⇑ in  FIG. 3B ). Finally, the target window is composited into (i.e., on top of the contents of) assembly buffer  235  by GPU  230  (block  315  and → in  FIG. 3B ). It will be noted that because the target window is composited after the visual effect is applied, below-effect  300  does not alter or impact the target window. Visual effects appropriate for a below-effect in accordance with the invention include, but are not limited to, drop shadow, blur and glass distortion effects. It will be known by those of ordinary skill that a filter need not be applied to the entire contents of the assembly buffer or target window. That is, only a portion of the assembly buffer and/or target window need be filtered. In such cases, it is known to use the bounding rectangle or the alpha channel of the target window to determine the region that is to be filtered. 
     Referring to  FIGS. 4A and 4B , on-effect  400  in accordance with one embodiment of the invention is illustrated. In on-effect  400 , a target window (e.g., contained in backing store  220 ) is filtered as it is being composited into a system&#39;s assembly buffer. As shown, the contents of window buffer  220  are filtered by GPU  230  (block  405  in  FIG. 4A  and ← in  FIG. 4B ) and then composited into assembly buffer  235  by GPU  230  (block  410  in  FIG. 4A  and ⇑ in  FIG. 4B ). Referring to  FIGS. 5A and 5B , on-effect  500  in accordance with another embodiment of the invention is illustrated. In on-effect  500 , a target window (e.g., contained in backing store  220 ) and assembly buffer  235  (block  505  in  FIG. 5A  and ← in  FIG. 5B ) are filtered into temporary buffer  240  (block  510  in  FIG. 5A  and ⇑ in  FIG. 5B ). The resulting image is transferred back into assembly buffer  235  (block  515  in  FIG. 5A  and → in  FIG. 5B ). Visual effects appropriate for an on-effect in accordance with the invention include, but are not limited to, window distortions and color correction effects such as grey-scale and sepia tone effects. 
     Referring to  FIGS. 6A and 6B , above-effect  600  in accordance with one embodiment of the invention is illustrated. In above-effect  600 , the target window (e.g., contained in backing store  220 ) is composited into the system&#39;s assembly buffer prior to the visual effect being applied. Accordingly, unlike below-effect  300 , the target window may be affected by the visual effect. As shown, the target window is first composited into assembly buffer  235  by GPU  230  (block  605  in  FIG. 6A  and ← in  FIG. 6B ), after which the result is transferred to temporary buffer  240  by GPU  230  (block  610  in  FIG. 6A  and ⇑ in  FIG. 6B ). Finally, GPU  230  filters the contents of temporary buffer  240  into assembly buffer  235  to apply the desired visual effect (block  615  in  FIG. 6A  and → in  FIG. 6B ). Visual effects appropriate for an on-effect in accordance with the invention include, but are not limited to, glow effects. 
     Referring to  FIGS. 7A and 7B , full-screen effect  700  in accordance with one embodiment of the invention is illustrated. In full-screen effect  700 , the assembly buffer is filtered as it is transferred to the system&#39;s frame buffer. As shown, the contents of assembly buffer  235  are filtered by GPU  230  (block  705  in  FIG. 7A  and ← in  FIG. 7B ) as the contents of assembly buffer  235  are transferred to frame buffer  245  (block  710  in  FIG. 7A  and ⇑ in  FIG. 7B ). Because, in accordance with the invention, programmable GPU  230  is used to apply the visual effect, virtually any visual effect may be used. Thus, while prior art systems are incapable of implementing sophisticated effects such as distortion, tile, gradient and blur effects, these are possible using the inventive technique. In particular, high-benefit visual effects for a full-screen effect in accordance with the invention include, but are not limited to, color correction and brightness effects. For example, it is known that liquid crystal displays (“LCDs”) have a non-uniform brightness characteristic across their surface. A full-screen effect in accordance with the invention could be used to remove this visual defect to provide a uniform brightness across the display&#39;s entire surface. 
     It will be recognized that, as a practical matter, full-screen visual effects must conform to the system&#39;s frame buffer scan rate. That is, suitable visual effects in accordance with  700  include those effects in which GPU  230  generates filter output at a rate faster than (or at least as fast as) data is removed from frame buffer  245 . If GPU output is generated slower than data is withdrawn from frame buffer  245 , potential display problems can arise. Accordingly, full-screen effects are generally limited to those effects that can be applied at a rate faster than the frame buffer&#39;s output scan rate. 
     Event routing in a system employing visual effects in accordance with the invention must be modified to account for post-application effects. Referring to  FIG. 8 , for example, application  210  may write into window buffer  220  such that window  800  includes button  805  at a particular location. After being modified in accordance with one or more of effects  300 ,  400 ,  600  and  700 , display  250  may appear with button  805  modified to display as  810 . Accordingly, if a user (the person viewing display  250 ) clicks on button  810 , the system (i.e., the operating system) must be able to map the location of the mouse click into a location known by application  210  as corresponding to button  805  so that the application knows what action to take. 
     It will be recognized by those of ordinary skill in the art that filters (i.e., fragment programs implementing a desired visual effect) operate by calculating a destination pixel location (i.e., x d , y d ) based on one or more source pixels. Accordingly, the filters used to generate the effects may also be used to determine the source location (coordinates). Referring to  FIG. 9 , event routing  900  in accordance with one embodiment of the invention begins when an event is detected (block  905 ). As used herein, an event may be described in terms of a “click” coordinate, e.g., (x click , y click ). Initially, a check is made to determine if the clicked location comports with a filtered region of the display. If the clicked location (x click , y click ) has not been subject to an effect (the “No” prong of block  910 ), the coordinate is simply passed to the appropriate application (block  925 ). If the clicked location (x click , y click ) has been altered in accordance with the invention (the “Yes” prong of block  910 ), the last applied filter is used to determine a first tentative source coordinate (block  915 ). If the clicked location has not been subject to additional effects in accordance with the invention (the “Yes” prong of block  920 ), the first tentative calculated source coordinate is passed to the appropriate application (block  925 ). If the clicked location has been subject to additional effects in accordance with the invention (the “No” prong of block  920 ), the next most recently applied filter is used to calculate a second tentative source coordinate. Processing loop  915 - 920  is repeated for each filter applied to clicked location (x click , y click ). 
     In addition to generating full-screen displays utilizing below, on and above filtering techniques as described herein, it is possible to generate partial screen updates. For example, if only a portion of a display has changed only that portion need be reconstituted in the display&#39;s frame buffer. 
     Referring to  FIG. 10 , consider the case where user&#39;s view  1000  is the result of five (5) layers: background layer L 0   1005 , layer L 1   1010 , layer L 2   1015 , layer L 3   1020  and top-most layer L 4   1025 . In the prior art, when region  1030  was identified by the windowing subsystem as needed to be updated (e.g., because a new character or small graphic is to be shown to the user), an assembly buffer was created having a size large enough to hold the data associated with region  1030 . Once created, each layer overlapping region  1030  (e.g., regions  1035 ,  1040  and  1045 ) was composited into the assembly buffer—beginning at background layer L 0   1005  (region  1045 ) up to top-most layer L 4   1025  (region  1030 ). The resulting assembly buffer&#39;s contents were then transferred into the display&#39;s frame buffer at a location corresponding to region  1030 . 
     When layer-specific filters are used in accordance with the invention, the prior art approach of  FIG. 10  does not work. For example, a specified top-layer region comprising (a×b) pixels may, because of that layer&#39;s associated filter, require more (e.g., due to a blurring type filter) or fewer (e.g., due to a magnification type filter) pixels from the layer below it. Thus, the region identified in the top-most layer by the windowing subsystem as needing to be updated may not correspond to the required assembly buffer size. Accordingly, the effect each layer&#39;s filter has on the ability to compute the ultimate output region must be considered to determine what size of assembly buffer to create. Once created, each layer overlapping the identified assembly buffer&#39;s extent (size and location) may be composited into the assembly buffer as described above with respect to  FIG. 10  with the addition of applying that layer&#39;s filter—e.g., a below, on or above filter as previously described. 
     Referring to  FIG. 11 , assembly buffer extent (size and location) determination technique  1100  in accordance with one embodiment of the invention includes receiving identification of a region in the user&#39;s display that needs to be updated (block  1105 ). One of ordinary skill in the art will recognize that this information may be provided by conventional windowing subsystems. The identified region establishes the initial assembly buffer&#39;s (“AB”) extent (block  1110 ). Starting at the top-most layer (that is, the windowing layer closest to the viewer, block  1115 ) a check is made to determine if the layer has an associated filter (block  1120 ). Illustrative output display filters include below, on and above filters as described herein. If the layer has an associated filter (the “Yes” prong of block  1120 ), the filter&#39;s region of interest (“ROI”) is used to determine the size of the filter&#39;s input region required to generate a specified output region (block  1125 ). As described in the filters identified in paragraph [ 0002 ], a filter&#39;s ROI is the input region needed to generate a specified output region. For example, if the output region identified in accordance with block  1110  comprises a region (a×b) pixels, and the filter&#39;s ROI identifies a region (x×y) pixels, then the identified (x×y) pixel region is required at the filter&#39;s input to generate the (x×y) pixel output region. The extent of the AB is then updated to be equal to the combination (via the set union operation) of the current AB extent and that of the region identified in accordance with block  1125  (block  1130 ). If there are additional layers to interrogate (the “Yes” prong of block  1135 ), the next layer is identified (block  1140 ) and processing continues at block  1120 . If no additional layers remain to be interrogated (the “No” prong of block  1135 ), the size of AB needed to generate the output region identified in block  1105  is known (block  1145 ). With this information, an AB of the appropriate size may be instantiated and each layer overlapping the identified AB region composited into it in a linear fashion—beginning at the bottom-most or background layer and moving upward toward the top-most layer (block  1150 ). Once compositing is complete, that portion of the AB&#39;s contents corresponding to the originally identified output region (in accordance with the acts of block  1105 ) may be transferred to the appropriate location within the display&#39;s frame buffer (“FB”) (block  1155 ). For completeness, it should be noted that if an identified layer does not have an associated filter (the “No” prong of block  1120 ) processing continues at block  1135 . In one embodiment, acts in accordance with blocks  1110 - 1145  may be performed by one or more cooperatively coupled general purpose CPUs, while acts in accordance with blocks  1150  and  1155  may be performed by one or more cooperatively coupled GPUs. 
     To illustrate how process  1100  may be applied, consider  FIG. 12  in which user&#39;s view  1200  is the result of compositing five (5) display layers: background layer L 0   1205 , layer L 1   1210 , layer L 2   1215 , layer L 3   1220  and top-most layer L 4   1225 . In this example, assume region  1230  has been identified as needing to be update on display  1200  and that (i) layer L 4   1225  has a filter whose ROI extent is shown as  1235 , (ii) layer L 3   1220  has a filter whose ROI extent is shown as  1245 , (iii) layer L 2   1225  has a filter whose ROI extent is shown as  1255 , and (iv) layer L 1   1210  has a filter whose ROI extent is shown as  1265 . 
     In accordance with process  1100 , region  1230  is used to establish an initial AB size. (As would be known to those of ordinary skill in the art, the initial location of region  1230  is also recorded.) Next, region  1240  in layer L 3   1220  needed by layer L 4   1225 &#39;s filter is determined. As shown, the filter associated with layer L 4   1225  uses region  1240  from layer L 3   1220  to compute or calculate its display (L 4  Filter ROI  1235 ). It will be recognized that only that portion of layer L 3   1220  that actually exists within region  1240  is used by layer L 4   1225 &#39;s filter. Because the extent of region  1240  is greater than that of initial region  1230 , the AB extent is adjusted to include region  1240 . A similar process is used to identify region  1250  in layer L 2   1215 . As shown in  FIG. 12 , the filter associated with layer L 3   1220  does not perturb the extent/size of the needed assembly buffer. This may be because the filter is the NULL filter (i.e., no applied filter) or because the filter does not require more, or fewer, pixels from layer L 2   1215  (e.g., a color correction filter). 
     The process described above, and outlined in blocks  1120 - 1130 , is repeated again for layer L 2   1215  to identify region  1260  in layer L 1   1210 . Note that region  1260  is smaller than region  1250  and so the size (extent) of the AB is not modified. Finally, region  1270  is determined based on layer L 1 &#39;s filter ROI  1265 . If region  1270  covers some portion of background layer L 0   1205  not yet “within” the determined AB, the extent of the AB is adjusted to do so. Thus, final AB size and location (extent)  1275  represents the union of the regions identified for each layer L 0   1205  through L 4   1225 . With region  1275  known, an AB of the appropriate size may be instantiated and each layer that overlaps region  1275  is composited into it—starting at background layer L 0   1205  and finishing with top-most layer L 4   1225  (i.e., in a linear fashion). That portion of the AB corresponding to region  1230  may then be transferred into display  1200 &#39;s frame buffer (at a location corresponding to region  1230 ) for display. 
     As noted above, visual effects and display updates in accordance with the invention may incorporate substantially any known visual effects. These include color effects, distortion effects, stylized effects, composition effects, half-tone effects, transition effects, tile effects, gradient effects, sharpen effects and blur effects. 
     Various changes in the components as well as in the details of the illustrated operational methods are possible without departing from the scope of the following claims. For instance, in the illustrative system of  FIG. 2  there may be additional assembly buffers, temporary buffers, frame buffers and/or GPUs. Similarly, in the illustrative system of  FIG. 12 , there may be more or fewer display layers (windows). Further, not all layers need have an associated filter. Further, regions identified in accordance with block  1125  need not overlap. That is, regions identified in accordance with the process of  FIG. 11  may be disjoint or discontinuous. In such a case, the union of disjoint regions is simply the individual regions. One of ordinary skill in the art will further recognize that recordation of regions may be done in any suitable manner. For example, regions may be recorded as a list of rectangles or a list of (closed) paths. In addition, acts in accordance with  FIGS. 3A ,  4 A,  6 A,  7 A and  9  may be performed by two or more cooperatively coupled GPUs and may, further, receive input from one or more system processing units (e.g., CPUs). It will further be understood that fragment programs may be organized into one or more modules and, as such, may be tangibly embodied as program code stored in any suitable storage device. Storage devices suitable for use in this manner include, but are not limited to: magnetic disks (fixed, floppy, and removable) and tape; optical media such as CD-ROMs and digital video disks (“DVDs”); and semiconductor memory devices such as Electrically Programmable Read-Only Memory (“EPROM”), Electrically Erasable Programmable Read-Only Memory (“EEPROM”), Programmable Gate Arrays and flash devices. 
     The preceding description was presented to enable any person skilled in the art to make and use the invention as claimed and is provided in the context of the particular examples discussed above, variations of which will be readily apparent to those skilled in the art. Accordingly, the claims appended hereto are not intended to be limited by the disclosed embodiments, but are to be accorded their widest scope consistent with the principles and features disclosed herein.

Metadata:
Filing Date: 20041001
Publication Date: 20100126
Grant Date: 20100126
Priority Date: 20040625
Inventors: BRUNNER RALPH
HARPER JOHN
Assignee: APPLE INC
CPC Classifications: [{"code": "G09G5/14", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G5/393", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G5/14", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G5/393", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 34971412