PATENT DOCUMENT

Publication Number: US-8432396-B2
Application Number: US-76070307-A
Country: US
Kind Code: B2

Title: Reflections in a multidimensional user interface environment

Abstract:
A graphical user interface has a back surface disposed from a viewing surface to define a depth. A visualization object receptacle is disposed between the back surface and a viewing surface and contains a visualization object. A reflection surface is defined such that a reflection of the visualization object is displayed on the reflection surface.

Claims:
What is claimed is: 
     
       1. A method, comprising:
 providing for display a display object in a three-dimensional user interface; 
 providing for display an object receptacle and a visualization object located on the object receptacle, the object receptacle being displayed in front of the display object such that the visualization object obscures at least a portion of the display object; 
 producing a reflection of the obscured portion of the display object on a reflection surface in the three-dimensional user interface; and 
 providing the reflection for display on a display device such that the portion of the display object obscured by the visualization object is visible in the reflection. 
 
     
     
       2. The method of  claim 1 , wherein the three-dimensional user interface comprises a back surface and a side surface, the method comprising:
 providing the object receptacle and the reflection surface for display on the side surface, and 
 providing the display object for display on the back surface. 
 
     
     
       3. The method of  claim 2 , comprising:
 providing for display a shadow of the visualization object along a viewing angle of the three-dimensional user interface towards the back surface. 
 
     
     
       4. The method of  claim 3 , comprising:
 obscuring at least a portion of the reflection by the visualization object. 
 
     
     
       5. The method of  claim 1 , wherein;
 the display object includes an application interface, and 
 the visualization object includes an interactive icon. 
 
     
     
       6. The method of  claim 1 , comprising:
 providing a reflection of the visualization object for display on the reflection surface; 
 determining changes to the visualization object; 
 and transforming the changes to update the reflection surface. 
 
     
     
       7. The method of  claim 1 , comprising:
 scaling the reflection to indicate a curvature of the reflection surface. 
 
     
     
       8. The method of  claim 1 , comprising:
 applying an affine transformation to determine the reflection of the obscured portion of the display object. 
 
     
     
       9. The method of  claim 1 , comprising
 determining transparency information associated with surfaces between a back surface and a viewing surface; and 
 producing a shadow that is cast from visual content in a surface closer to the viewing surface onto a surface further from the viewing surface. 
 
     
     
       10. The method of  claim 1 , comprising:
 scaling the reflection of the obscured portion of the display object relative to the display object. 
 
     
     
       11. The method of  claim 1 , wherein the reflection surface is located beneath the object receptacle. 
     
     
       12. A system, comprising:
 one or more computers configured to perform operations comprising:
 providing for display a display object in a three-dimensional user interface; 
 providing for display an object receptacle and a visualization object located on the object receptacle, the object receptacle being displayed in front of the display object such that the visualization object obscures at least a portion of the display object; 
 producing a reflection of the obscured portion of the display object on a reflection surface in the three-dimensional user interface; and 
 providing the reflection for display on a display device such that the portion of the display object obscured by the visualization object is visible in the reflection. 
 
 
     
     
       13. The system of  claim 12 , wherein the three-dimensional user interface comprises a back surface and a side surface, the operations comprising:
 providing the object receptacle and the reflection surface for display on the side surface, and 
 providing the display object for display on the back surface. 
 
     
     
       14. The system of  claim 13 , wherein:
 the display object includes an application interface, and 
 the visualization object is an interactive three-dimensional icon. 
 
     
     
       15. The system of  claim 14 , wherein:
 the reflection is partially obscured by the visualization object. 
 
     
     
       16. The system of  claim 12 , wherein:
 the reflection of the obscured portion of the display object is scaled relative to the display object. 
 
     
     
       17. The system of  claim 12 , wherein:
 a shadow is cast from the visualization object toward a back surface. 
 
     
     
       18. The system of  claim 12 , wherein:
 an amount transparency is applied to the object receptacle such that the reflection is partially visible through the object receptacle. 
 
     
     
       19. The system of  claim 12 , wherein:
 the object receptacle comprises a dock, and the visualization object comprises an icon. 
 
     
     
       20. A non-transitory computer readable medium storing instructions that are executable by a processing device, and upon such execution cause the processing device to perform operations comprising:
 providing for display a display object in a three-dimensional user interface; 
 providing for display an object receptacle and a visualization object located on the object receptacle, the object receptacle being displayed in front of the display object such that the visualization object obscures at least a portion of the display object; 
 producing a reflection of the obscured portion of the display object on a reflection surface in the three-dimensional user interface; and 
 providing the reflection for display on a display device such that the portion of the display object obscured by the visualization object is visible in the reflection. 
 
     
     
       21. The computer readable medium of  claim 20 , the operations comprising applying a transform to the obscured portion of the display object to determine the reflection of the obscured portion of the display object. 
     
     
       22. The computer readable medium of  claim 21 , wherein the display object comprises an application interface, the application interface including a window containing visual content, the visual content being obscured by the visualization object. 
     
     
       23. The computer readable medium of  claim 22 , the operations comprising:
 applying a transform to the visual content; and 
 providing for display a reflection of the visual content on the reflection surface. 
 
     
     
       24. The computer readable medium of  claim 20 , the operations comprising scaling a size of the reflection of the obscured portion of the display object relative to the obscured portion of the display object. 
     
     
       25. The computer readable medium of  claim 20 , the operations comprising creating a shadow that is cast behind the visualization object toward a back surface.

Description:
BACKGROUND 
     A graphical user interface allows a large number of graphical objects or items to be displayed on a display screen at the same time. Leading personal computer operating systems, such as the Apple Mac OS®, provide user interfaces in which a number of visualizations of system objects can be displayed according to the needs of the user. Example system objects include system functions, alerts, windows, peripherals, files, and applications. Taskbars, menus, virtual buttons, a mouse, a keyboard, and other user interface elements provide mechanisms for accessing and/or activating the system objects corresponding to the displayed representations. 
     The graphical objects and access to the corresponding system objects and related functions, however, should be presented in a manner that facilitates an intuitive user experience. The use of metaphors that represent concrete, familiar ideas can facilitate such an intuitive user experience. For example, the metaphor of file folders can be used for storing documents; the metaphor of a file cabinet can be used for storing information on a hard disk; and the metaphor of the desktop can be used for an operating system interface. 
     As the capabilities of processing devices progress, however, so do the demands on the graphical user interface to convey information to the users in an intuitive manner. 
     SUMMARY 
     Disclosed herein is a multidimensional desktop graphical user interface. In one implementation, a method includes axially disposing a back surface from a viewing surface in a user interface. A reflection surface can be defined between the viewing surface and the back surface. A visualization receptacle containing a visualization object can be disposed between the reflection surface and the viewing surface. Visualization data associated with the visualization object can be transformed to produce a reflection of the visualization object on the reflection surface, which can be displayed on the viewing surface as emanating from the reflection surface. 
     In another implementation, a graphical user interface includes a viewing surface and a back surface axially disposed from the viewing surface. A reflection surface is disposed between the viewing surface and the back surface, and a visualization receptacle is disposed between the reflection surface and the viewing surface. The visualization receptacle contains a visualization object and a reflection of the visualization object can be displayed on the reflection surface. 
     In another implementation, a computer readable medium stores instructions that can be executable by a processing device. Upon such execution, the instructions can cause the processing device to display a back surface axially disposed from the viewing surface, define a reflection surface disposed between the viewing surface and the back surface, display a visualization receptacle having visualization object, and display a reflection of the visualization object on the reflection surface. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of an example system that can be utilized to implement the systems and methods described herein. 
         FIG. 2  is a block diagram of an example user interface architecture. 
         FIG. 3  is an image of an example visualization receptacle. 
         FIG. 4  is a block diagram of an example system layer structure that can be utilized to implement the systems and methods described herein. 
         FIG. 5A  is a block diagram of an example multidimensional desktop environment. 
         FIG. 5B  is a block diagram of an exemplary physical arrangement which the reflection content surface mimics. 
         FIG. 6A  is a block diagram of another exemplary multidimensional desktop environment. 
         FIG. 6B  is a block diagram of a reflection surface. 
         FIG. 7  is a flow diagram of an example process for generating a multidimensional desktop environment having reflected visual content. 
         FIG. 8  is a flow diagram of another example process for generating a multidimensional desktop environment having reflected visual content. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a block diagram of an example system  100  that can be utilized to implement the systems and methods described herein. The system  100  can, for example, be implemented in a computer device, such as any one of the personal computer devices available from Apple Inc., or other electronic devices. Other example implementations can also include video processing devices, multimedia processing devices, portable computing devices, portable communication devices, etc. 
     The example system  100  includes a processing device  102 , a memory  104 , a data store  106 , a graphics device  108 , input devices  110 , output devices  112 , and a network interface  114 . A bus system  116 , such as a data bus and a motherboard, can be used to establish and control data communication between the components  102 ,  104 ,  106 ,  108 ,  110 ,  112  and  114 . Other example system architectures, however, can also be used. 
     The processing device  102  can, for example, include one or more microprocessors. The memory  104  can, for example, include a random access memory storage device, such as a dynamic random access memory, or other types of computer-readable medium memory devices. The data store  106  can, for example, include one or more hard drives, a flash memory, and/or a read only memory, or other types of computer-readable medium memory devices. 
     The graphics device  108  can, for example, include a video card, a graphics accelerator card, or a display adapter, and is configured to generate and output images to a display device. In one implementation, the graphics device  108  can be realized in a dedicated hardware card connected to the bus system  116 . In another implementation, the graphics device  108  can be realized in a graphics controller integrated into a chipset of the bus system  116 . Other implementations can also be used. 
     Example input devices  110  can include a keyboard, a mouse, a stylus, a video camera, a multi-touch surface, etc., and example output devices  112  can include a display device, an audio device, etc. 
     The network interface  114  can, for example, include a wired or wireless network device operable to communicate data to and from a network  118 . The network  118  can include one or more local area networks (LANs) or a wide area network (WAN), such as the Internet. 
     In an implementation, the system  100  includes instructions defining an operating system stored in the memory  104  and/or the data store  106 . Example operating systems can include the MAC OS® X series operating system, the WINDOWS® based operating system, or other operating systems. Upon execution of the operating system instructions, access to various system objects is enabled. Example system objects include data files, applications, functions, windows, etc. To facilitate an intuitive user experience, the system  100  includes a graphical user interface that provides the user access to the various system objects and conveys information about the system  100  to the user in an intuitive manner. 
       FIG. 2  is a block diagram of an example user interface architecture  200 . The user interface architecture  200  includes a user interface (UT) engine  202  that provides the user access to the various system objects  204  and conveys information about the system  100  to the user. 
     Upon execution, the UI engine  202  can cause the graphics device  108  to generate a graphical user interface on an output device  112 , such as a display device. In one implementation, the graphical user interface can include a multidimensional desktop  210  and a multidimensional application environment  212 . In an implementation, the multidimensional desktop  210  and the multidimensional application environment  212  include x-, y- and z-axis aspects, e.g., a height, width and depth aspect. The x-, y- and z-axis aspects may define a three-dimensional environment, e.g., a “3D” or “2.5D” environment that includes a z-axis, e.g., depth, aspect. 
     In an implementation, the multidimensional desktop  210  can include visualization objects  220 , a visualization object receptacle  222 , and stack items  224 . In some implementations, the visualization objects  220 , the visualization object receptacle  222  and the stack items  224  can be presented in a pseudo-three dimensional (i.e., “2.5D”) or a three-dimensional environment as graphical objects having a depth aspect. 
     A visualization object  220  can, for example, be a visual representation of a system object. In some implementations, the visualization objects  220  are icons. Other visualization objects can also be used, e.g., alter notification windows, menu command bars, windows, or other visual representations of system objects. 
     In an implementation, the multidimensional application environment  212  can include an application environment distributed along a depth aspect. For example, a content frame, e.g., an application interface, can be presented on a first surface, and control elements, e.g., toolbar commands, can be presented on a second surface. 
     In an implementation, the UI engine  202  generates the visualization of the multidimensional desktop  210  and multidimensional application environment  212  by compositing a plurality of layers (e.g., the first surface and the second surface) into a composite layer. The UI engine  202  can include utilize the memory  104  to store the composite layer. A graphical pipeline can be coupled to the UI engine  202  and may be adapted to transport the composite graphical layer to an output to a display device. 
       FIG. 3  is an image of an example visualization object receptacle  300 . In one implementation, the visualization object receptacle  300  can include x-, y- and z-axis aspects, e.g., a height, width and depth. In another implementation, the visualization object receptacle  300  can have only a y- and z-axis aspect, e.g., a width and depth. An example implementation of a visualization object receptacle  300  is the “Dock” user interface in the MAC OS® X Leopard operating system. Other implementations can also be used. 
     In some implementations, or more visualization objects, e.g., icons  304 ,  306 ,  308  and  310  can be disposed within the visualization object receptacle  300 , e.g., a visualization receptacle  300 . In one implementation, a lighting and shading effect is applied to emphasize the depth aspect of the visualization receptacle  300 , as illustrated by the corresponding shadows  305 ,  307 ,  309  and  311  and reflections  312 ,  314 ,  316  and  318  beneath each of the icons  304 ,  306 ,  308  and  310 . 
     In some implementations, the visualization receptacle  300  can include front surface  319  to generate a height aspect. In some implementations, a notch  320  can be included in the visualization receptacle. The notch  320  can, for example, be utilized to arrange icons related to particular programs or functions, e.g., files and folders can be dispose on a first side of the notch  320  and applications can be disposed on a second side of the notch  320 ; or a user may define arrangements according to the notch  320 , etc. 
     In some implementations, the visualization receptacle  300  can include status indicators, e.g.,  330  and  332 , disposed on the front surface  319 . The status indicators  330  and  332  can, for example, appear as illuminations to indicate a status of a system object or function associated with a corresponding icon. In some implementations, the status indicators can be color coded based on an identified status. For example, the status indicator  330  may be illuminate in a yellow color to indicate that the folder  304  is receiving a file download, and the status indicator  332  may be illuminate in a green color to indicate that a program associated with the icon  308  is running. 
     In some implementations, the visualization receptacle  300  may only define a depth aspect, e.g., the visualization receptacle  300  may not include a front surface  319 . Additional features of visualization receptacles and icons disposed therein are described in more detail below. 
       FIG. 4  is block diagram of example system layers  400  that can be utilized to implement the systems and methods described herein. Other system layer implementations, however, can also be used. 
     In an implementation, a user interface engine, such as the UI engine  202 , or another UI engine capable of generating a three-dimensional user interface environment, operates at an application level  402  and implements graphical functions and features available through an application program interface (API) layer  404 . Example graphical functions and features include graphical processing, supported by a graphics API, image processing, support by an imaging API, and video processing, supported by a video API. 
     The API layer  404 , in turn, interfaces with a graphics library layer  406 . The graphics library layer  404  can, for example, be implemented as a software interface to graphics hardware, such as an implementation of the OpenGL specification. A driver/hardware layer  408  includes drivers and associated graphics hardware, such as a graphics card and associated drivers. 
       FIG. 5A  is a block diagram of an example multidimensional desktop environment  500 . In an implementation, the multidimensional desktop environment  500  can include a back surface  502 , application interfaces  504  and  506 , a reflection surface  508 , and a visualization object receptacle  510 . Layers of graphical content from the surfaces can be composited and displayed on a display surface  520 . 
     In an implementation, a reflection content surface  512  can be defined between the back surface  502  and the display surface  520 . Visual content (e.g., wallpaper, visualization objects, an application user interface) displayed on the back surface  502  and/or the application interfaces  504 ,  506  passes through the reflection content surface  512 . In an implementation, the visual content that passes through the reflection content surface  512  can be processed by a transform matrix  514  to generate reflected visual content that can be displayed on the reflection surface  508  as viewed at a viewing angle  522 . In an implementation, the transform matrix  514  can use an affine transform, which consists of a linear transformation followed by a translation to preserve co-linearity between points and ratios of distances between any two points in the reflection surface  508  and the reflection content surface  512 . 
     A visualization object receptacle  510  can include one or more visualization objects  516  and  518 . In an implementation, the visualization objects  516  and  518  can be transformed and displayed on the reflection surface  508  as reflected visualization objects (see,  FIGS. 6A and 6B ). The visualization objects  516  and  518  can be, for example, icons or other visualizations of system objects. 
     As the visual content or placement of the back surface  502  and/or application interfaces  504  and  506  changes, a compositor  524  can be executed to refresh the contents of the display surface  520 . In an implementation, the compositor can be implemented in the UI engine  202 . When changes occur to the visual content passing through the reflection content surface  512 , the compositor  524  can be executed to collect the changes and the transform  514  can be applied to update the reflected visual content associated with the reflection surface  508 . The updated content associated with the reflection content surface  512  can be passed to a compositor  526  to update the reflection displayed behind the visualization objects  516  and  518  as visible at the viewing angle  522 . 
     In an implementation, the displayed content on the display surface  520  can be a composite of all the visual content logically beneath that the display surface  520 . The visual content can be blended together under control of transparency information associated with each point of visual content beneath the display surface  520 . As such, visual effects can be provided, such as translucent windows and drop shadows from one window cast upon another window in the multidimensional desktop environment  500 . 
       FIG. 5B  illustrates a block diagram of an exemplary physical arrangement which the reflection surface  508  mimics. In an implementation, the reflection surface  508  can appear to a viewer observing along viewing angle  522  as a mirror at right angle to the visualization objects  516  and  518 , and containing a reflection of the visual content of the back surface  502  and the application interfaces  504  and  506 . The reflection surface  508  is not limited to being placed at the bottom of a multidimensional desktop environment  500  (i.e., to reflect upward), and can be placed anywhere in the multidimensional desktop environment  500  to reflect content in any orientation. In an implementation, scaling of the reflected content can be applied so as to simulate a curved reflecting surface. 
       FIG. 6A  is a block diagram of an exemplary multidimensional desktop environment. The multidimensional desktop environment is defined by a back surface  650  and includes a reflection surface  674  logically disposed behind a visualization object receptacle  652 . The visualization object receptacle  652  can include visualization objects  662 ,  664 ,  666 ,  668 ,  670  and  672  (e.g., icons) that are disposed on top of the visualization object receptacle  652 . In some implementations, a status indicator can illuminate to indicate a status. For example, the stack item  668  may correspond to recent downloads, e.g., system updates, documents, etc., and the illumination may be lit to indicate that a download is currently in progress. The status indicator can, for example, illuminate according to a color code to indicate different status states. 
     In an implementation, reflections of the visualization objects  662 ,  664 ,  666 ,  668 ,  670  and  672  can be displayed on the reflection surface  674 . In some implementations, reflections of other system objects, e.g., the folder icon  656  and the application interface  658  can be generated and displayed on the reflection surface  674 . In some implementations, the reflections of the visualization objects  662 ,  664 ,  666 ,  668 ,  670  and  672 , the folder icon  656  or the application interface  658  can be scaled when displayed on the reflection surface  674   
     In some implementations, the transparency of the visualization objects  662 ,  664 ,  666 ,  668 ,  670  and  672 , the folder icon  656  or the application interface  658  can vary. In an implementation, shadowing  676  and  678  can be provided behind visualization objects to produce an effect as if the visualization objects were casting a shadow along the viewing angle toward the back surface  650 . 
       FIG. 6B  is a block diagram of the reflection surface  674 . In an implementation, the reflection surface  674  can include parts of the application interface  658  (or other object in the multidimensional desktop environment) that would be partially obscured by the visualization objects on the visualization object receptacle  652 . For example, visual content  680  within the application interface  658  would be rendered on the reflection surface  674 , however the visual content  680  will obscured by visualization objects  662  and  664  when displayed to the user as shown in  FIG. 6A . 
       FIG. 7  is a flow diagram of an example process  700  for generating multidimensional desktop environment having reflected visual content. The process  700  can, for example, be implemented in a processing device, such as the system  100  of  FIG. 1 , implementing user interface software and/or hardware, such as the example implementations described with respect to  FIGS. 2 ,  3  and  4 . 
     Stage  702  provides user interface surfaces. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can provide the back surface  502 , the reflection surface  508 , the reflection content surface  512  and the display surface  520  as shown in  FIG. 5A . Optionally, one or more application interfaces  504  or  506  can be provided in response to a user instantiation of an application instance as shown in  FIG. 5A . 
     Stage  704  displays a visualization receptacle. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can display the visualization receptacle  510  having a visualization object  516  or  518  as shown in  FIG. 5A  or  5 B. 
     Stage  706  transforms visualization data. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can transform visualization data associated with one of the visualization objects  516  or  518  to create a reflected visualization object as shown in  FIG. 6A . 
     Stage  708  displays the reflected visual content. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can display the reflected visualization objects as shown in  FIG. 6B . 
       FIG. 8  is a flow diagram of another example process  800  for generating multidimensional desktop environment having a simulated reflection. The process  800  can, for example, be implemented in a processing device, such as the system  100  of  FIG. 1 , implementing user interface software and/or hardware, such as the example implementations described with respect to  FIGS. 2 ,  3  and  4 . 
     Stage  802  provides user interface surfaces. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can provide the back surface  502 , the reflection surface  508 , the reflection content surface  512  and the display surface  520  as shown in  FIG. 5A . Optionally, one or more application interfaces  504  or  506  can be provided in response to a user instantiation of an application instance as shown in  FIG. 5A . 
     Stage  804  determines visual content changes. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can determine if visual content within the back surface  502  and/or application interface  504  or  506  has changed. 
     Stage  806  refreshes the visual content. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can run the compositor  524  to update the display surface  520  to reflect the visual content changes in the back surface  502  or application interface  504  or  506 . 
     Stage  810  transforms the changes. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , transforms the changes to the visual content within the reflection content surface  512  to create reflective content to be applied to the reflection surface  508 . 
     Stage  812  displays the transformed content. For example, the system  100 , implementing any one of the UI engines described in  FIGS. 2 ,  3  and  4 , can run a compositor to update the reflection surface  508  to create a reflection as shown in  FIGS. 5A ,  6 A and  6 B. 
     The apparatus, methods, flow diagrams, and structure block diagrams described in this patent document may be implemented in computer processing systems including program code comprising program instructions that are executable by the computer processing system. Other implementations may also be used. Additionally, the flow diagrams and structure block diagrams described in this patent document, which describe particular methods and/or corresponding acts in support of steps and corresponding functions in support of disclosed structural means, may also be utilized to implement corresponding software structures and algorithms, and equivalents thereof. 
     This written description sets forth the best mode of the invention and provides examples to describe the invention and to enable a person of ordinary skill in the art to make and use the invention. This written description does not limit the invention to the precise terms set forth. Thus, while the invention has been described in detail with reference to the examples set forth above, those of ordinary skill in the art may effect alterations, modifications and variations to the examples without departing from the scope of the invention.

Metadata:
Filing Date: 20070608
Publication Date: 20130430
Grant Date: 20130430
Priority Date: 20070608
Inventors: PAQUETTE MICHAEL JAMES
ORDING BAS
LOUCH JOHN O.
CHAUDHRI IMRAN A.
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/0483", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0483", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/048", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/048", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 40097046