PATENT DOCUMENT

Publication Number: US-9052743-B2
Application Number: US-201213447443-A
Country: US
Kind Code: B2

Title: Motion tracking user interface

Abstract:
A method to transition focus of a display corresponding to an object&#39;s motion tracked by a video camera or like device is disclosed. In one implementation, the display shows one or more windows or user interfaces on the display. The object&#39;s motion can be used to select one of the windows or user interfaces on the display and manipulate content presented in the window or user interface. In another implementations, the object&#39;s motion can manipulate a three-dimensional graphical icon in a three-dimensional display environment, for example, by rotating it. In another implementation, the method further tracks motion of a second object and shifts focus of the display corresponding to the motion of the second object. In another implementation, a second display may be added to mirror the focus transition corresponding to the object&#39;s motion.

Claims:
What is claimed is: 
     
       1. A method comprising:
 registering a focus on a display to a first portion of a graphical object on the display according to a current position of an object within a space outside the display; 
 receiving a selection of a target zone of an image of the object to be tracked; 
 after the registering and the receiving, tracking visually the target zone&#39;s motion from the current position to a second position within the space outside the display; and 
 while visually tracking the target zone&#39;s motion within the space outside the display, transitioning the registered focus on the display presented to a user from a first visual presentation comprising the focus registered on the first portion of the graphical object to a second visual presentation that is different than the first visual presentation, the transitioning corresponding to the target zone&#39;s tracked motion, wherein the transitioning comprises at least one of the following:
 rotating the graphical object as a three-dimensional object in a three-dimensional space represented on the display for registering the focus to a second portion of the graphical object, the rotating corresponding to at least a portion of the target zone&#39;s tracked motion, the at least a portion of the target zone&#39;s tracked motion comprises rotation of the target zone about an axis within the space outside the display; and 
 moving the registered focus from the first portion of the graphical object to a first portion of another graphical object on the display, the moving corresponding to at least a portion of the target zone&#39;s tracked motion, the at least a portion of the target zone&#39;s tracked motion comprises movement of the target zone along an axis within the space outside the display, the first visual presentation comprising the first portion of the graphical object and the first portion of the other graphical object on the display. 
 
 
     
     
       2. The method of  claim 1 , further comprising:
 using a camera to visually track the target zone&#39;s motion. 
 
     
     
       3. The method of  claim 1 , wherein:
 the transitioning comprises moving the registered focus from the first portion of the graphical object to a first portion of another graphical object on the display; 
 the moving corresponding to at least a portion of the target zone&#39;s tracked motion; 
 the at least a portion of the target zone&#39;s tracked motion comprises movement of the target zone along an axis within the space outside the display; and 
 the first visual presentation comprising the first portion of the graphical object and the first portion of the other graphical object on the display. 
 
     
     
       4. The method of  claim 3 , wherein the graphical object and the other graphical object comprise one of windows and user interfaces that present content that is operative to be manipulated by the target zone&#39;s motion. 
     
     
       5. The method of  claim 3 , wherein:
 the transitioning further comprises rotating the other graphical object as a three-dimensional object in a three-dimensional space represented on the display for registering the focus to a second portion of the other graphical object, the rotating corresponding to at least another portion of the target zone&#39;s tracked motion, the at least another portion of the target zone&#39;s tracked motion comprises rotation of the target zone about an axis within the space outside the display. 
 
     
     
       6. The method of  claim 3 , wherein at least one of the graphical object and the other graphical object is one of a browser and an editor. 
     
     
       7. The method of  claim 3 , the transitioning further comprising:
 scrolling registered focus one of up, down, left and right within the other graphical object, the scrolling corresponding to at least another portion of the target zone&#39;s motion. 
 
     
     
       8. The method of  claim 1 , the transitioning comprising:
 rotating the graphical object as a three-dimensional object in a three-dimensional space represented on the display for registering the focus to a second portion of the graphical object; 
 the rotating corresponding to at least a portion of the target zone&#39;s tracked motion; and 
 the at least a portion of the target zone&#39;s tracked motion comprises rotation of the target zone about an axis within the space outside the display. 
 
     
     
       9. The method of  claim 8 , wherein:
 the transitioning further comprises varying the size of the graphical object on the display; 
 the varying corresponding to at least another portion of the target zone&#39;s tracked motion; and 
 the other portion of the target zone&#39;s tracked motion comprises movement of the target zone one of towards the display within the space outside the display and away from the display within the space outside the display. 
 
     
     
       10. The method of  claim 1 , further comprising:
 tracking visually a second object&#39;s motion from a third position to a fourth position; and 
 transitioning focus of the display from the second visual presentation to a third visual presentation corresponding to the second object&#39;s motion. 
 
     
     
       11. The method of  claim 1 , further comprising:
 displaying the focus transition on a second display. 
 
     
     
       12. A method comprising:
 displaying a graphical icon on a display in a first visual presentation of the graphical icon when an object is in a first location and a first visual state within a space outside the display; 
 tracking visually the object&#39;s motion from the first location and the first visual state to a second location and a second visual state within the space outside the display; and 
 transitioning the graphical icon displayed on the display from the first visual presentation to a second visual presentation in real-time corresponding to the object&#39;s tracked motion, wherein the transitioning comprises rotating the graphical icon in a three-dimensional space represented in the display from the first visual presentation to the second visual presentation when the object&#39;s motion comprises rotation of the object about an axis within the space outside the display from the first visual state to the second visual state. 
 
     
     
       13. The method of  claim 12 , wherein the transitioning the graphical icon further comprises moving the graphical icon across the display from the first visual presentation to the second visual presentation when the object&#39;s motion comprises movement of the object along another axis within the space outside the display from the first location to the second location. 
     
     
       14. The method of  claim 12 , further comprising:
 tracking visually a second object&#39;s motion from a third visual state to a fourth visual state; and 
 transitioning the graphical icon on the display from the second visual presentation to a third visual presentation corresponding to the second object&#39;s motion. 
 
     
     
       15. The method of  claim 12 , further comprising:
 using a camera to visually track the object&#39;s transition. 
 
     
     
       16. A system comprising:
 a processor; and 
 a computer-readable medium coupled to the processor and operable for storing instructions, which, when executed by the processor, causes the processor to perform operations comprising:
 registering a focus on a display to a first portion of a graphical object on the display according to a current position of an object within a space outside the display; 
 receiving a selection of a target zone of an image of the object to be tracked; 
 after the registering and the receiving, tracking visually the target zone&#39;s motion from the current position to a second position within the space outside the display; and 
 while visually tracking the target zone&#39;s motion within the space outside the display, transitioning the registered focus on the display presented to a user from a first visual presentation comprising the focus registered on the first portion of the graphical object to a second visual presentation that is different than the first visual presentation, the transitioning corresponding to the target zone&#39;s tracked motion, wherein the transitioning comprises at least one of the following:
 rotating the graphical object as a three-dimensional object in a three-dimensional space represented on the display for registering the focus to a second portion of the graphical object, the rotating corresponding to at least a portion of the target zone&#39;s tracked motion, the at least a portion of the target zone&#39;s tracked motion comprises rotation of the target zone about an axis within the space outside the display; and 
 moving the registered focus from the first portion of the graphical object to a first portion of another graphical object on the display, the moving corresponding to at least a portion of the target zone&#39;s tracked motion, the at least a portion of the target zone&#39;s tracked motion comprises movement of the target zone along an axis within the space outside the display, the first visual presentation comprising the first portion of the graphical object and the first portion of the other graphical object on the display. 
 
 
 
     
     
       17. A method comprising:
 displaying a graphical icon on a display in a first visual presentation of the graphical icon when an object is in a first location and a first visual state within a space outside the display; 
 tracking visually the object&#39;s motion from the first location and the first visual state to a second location and a second visual state within the space outside the display; and 
 transitioning the graphical icon displayed on the display from the first visual presentation to a second visual presentation in real-time corresponding to the object&#39;s tracked motion, wherein the transitioning comprises rotating the graphical icon in a three-dimensional space represented in the display from the first visual presentation to the second visual presentation when the object&#39;s motion comprises rotation of the object about an axis within the space outside the display from the first visual state to the second visual state. 
 
     
     
       18. The method of  claim 17 , wherein the transitioning the graphical icon further comprises moving the graphical icon across the display from the first visual presentation to the second visual presentation when the object&#39;s motion comprises movement of the object along another axis within the space outside the display from the first location to the second location.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation (and claims the benefit of priority under 35 USC 120) of U.S. application Ser. No. 11/956,275, filed Dec. 13, 2007, now allowed, and titled “Motion Tracking User Interface,” which is incorporated by reference. 
    
    
     TECHNICAL FIELD 
     The subject matter of this patent application generally relates to user interfaces. 
     BACKGROUND 
     Input devices for computer systems commonly include a mouse, a keyboard, or other input devices (e.g., a pen-based device, a track ball, etc). Each of these input devices requires a user&#39;s spare hand to operate. In some cases, a user may need both hands free to complete a task (e.g., typing on a keyboard) or the user may suffer from a disability that precludes the use of one hand. When the user desires to interact with the computer system, the user must interrupt the task to manipulate an input device. 
     SUMMARY 
     A method to transition focus of a display corresponding to an object&#39;s motion tracked by a video camera or like device is disclosed. In one implementation, the display shows one or more windows or user interfaces on the display. The object&#39;s motion can be used to select one of the windows or user interfaces on the display and manipulate content presented in the window or user interface. In another implementations, the object&#39;s motion can manipulate a three-dimensional graphical icon in a three-dimensional display environment, for example, by rotating it. In another implementation, the method further tracks a second object&#39;s motion in the display and shifts focus of the display corresponding to the motion of the second object. In another implementation, a second display may be added to mirror the focus transition corresponding to the object&#39;s motion. 
     Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. A user can free the use of both hands while navigating a user interface or interacting with an application on a computer system. 
    
    
     
       DESCRIPTION OF DRAWINGS 
         FIG. 1  is a screenshot showing a visual tracking of an object, namely, a user&#39;s head by a camera or a like device. 
         FIG. 2  is a screenshot showing visual tracking of an object, a pen, which is manipulated by a user. 
         FIG. 3  is a screenshot showing a visual tracking of a target zone of a user&#39;s head moving from one position to another position. 
         FIG. 4  is a screenshot showing a pointer indicating focus transitioning of a display among multiple windows concurrently shown on the display. 
         FIG. 5  is a screenshot showing a three-dimensional graphical icon, namely, a teapot, which is being manipulated by a user using his head motion. 
         FIG. 6  is a screenshot showing the icon being moved and manipulated at the same time by a user using his head motion. 
         FIG. 7  is a screenshot showing optical flow of a user motion turning his head from left to right. 
         FIG. 8  is a flow diagram of an example process for transitioning focus of a display corresponding to an object&#39;s motion. 
         FIG. 9  is a flow diagram of an example process for transitioning a graphical icon&#39;s visual presentation corresponding to an object&#39;s motion. 
         FIG. 10  is a flow diagram of an example process for tracking two objects&#39; motions and transitioning focus of display corresponding to the two objects&#39; motions. 
         FIG. 11  is a flow diagram of an example process for tracking an object&#39;s motion and transitioning focus of a second display corresponding to the object&#39;s motion. 
         FIG. 12  is a block diagram of an example system architecture that can be utilized to implement the systems and methods described in this application. 
         FIG. 13A-13C  are screenshots showing a three-dimensional graphical icon (e.g., a teapot) being displayed from different angles depending on a user&#39;s perspectives. 
         FIG. 14A-14B  illustrates changing of views from a user&#39;s perspective in a three-dimensional space. 
         FIG. 15  is a flow diagram of an example process for transitioning a graphical icon&#39;s visual presentation concurrently with its visual location corresponding to an object&#39;s motion. 
     
    
    
     DETAILED DESCRIPTION 
     Selection of a Target-Zone 
       FIG. 1  is a screenshot showing a visual tracking of an object, namely, a head  102  of a user  108  by a video camera or a like device. In the example shown, a user  108  is in front of the video camera or the like device (e.g., Apple Inc.&#39;s iSight® camera). An image  104  of the user&#39;s head  102  is captured by the video camera or the like device and displayed on the top right corner of a display  100  (e.g., a computer monitor). The image  104  conveys a message to the user  108  about which region of his head  102  is to be tracked by the video camera. 
     In some implementations, the user  108  can select a target zone to be traced by clicking on a zone of the image  104 . In the example shown, the user  108  has selected the target zone  106  to be his head  102 . The video camera puts the target zone  106  in focus and tracks the motion of the target zone  106  relative to the video camera. 
     In some implementations, the object to be tracked is not necessarily a human head. Any tangible objects can be tracked, such as a pen, a box, a hand, etc. 
       FIG. 2  is a screenshot showing the visual tracking of a pen  202  by the video camera. A user  204  is holding the pen  202  in front of a video camera. An image  206  of the pen  202  captured by the video camera is displayed on the top right corner of the screen  200 , for example. Similar to  FIG. 1 , the image  206  conveys a message to the user  204  about which region of the pen  202  is to be tracked by the video camera. As can be seen in the image  206 , the region targeted is the top of the pen  202 . The video camera will accordingly track the top of the pen  202 . 
     In some implementations, selecting a target zone to be tracked by physical clicking is not necessary. For example, optical flow information can be used to automatically detect a moving object without additional effort by a user to select a target zone to be tracked. 
     Optical flow is a known technology often used in the motion picture industry. For each pixel in a frame, a motion vector is calculated which describes a sub pixel x and y movement. A measure of how good a motion vector is, or how different a next frame is, can also be computed. For example, an absolute difference can be computed between a pixel in a first frame shifted by its motion vector and a corresponding pixel in a second frame. There can also be a confidence level or a penalty computed for describing how different the pixel is from neighboring vectors. Optical flow can be performed in floating point and often can be stored in OpenEXR file formats. 
     Target Tracking 
       FIG. 3  is a screenshot showing tracking of a target zone  302  of a head  304  of a user moving from one position to another position. In  FIG. 3 , the target zone  302  of the user is moving his head  304  from a left position to a right position. At the same time, the video camera is tracking and, optionally, recording the motion of the target zone  302 , such that the tracking information can be used for any of a number of purposes (e.g., transitioning focus of a display  300  corresponding to the motion of the target zone  302 ). 
     Focus Transition of a Display 
       FIG. 4  is a screenshot showing using a pointer  402  to indicate focus transitioning of a display  400  among multiple windows  404   a - 404   e . In the example shown, windows  404   a ,  404   b ,  404   c ,  404   d , and  404   e  are concurrently shown on the display  400 , and each window is displaying content (e.g., a digital image). 
     The pointer  402  is used to indicate the current focus of the display (e.g., the window currently active). More specifically, the pointer  402  is used to indicate the transition of focus of the display  400  in correspondence with the motion of the human head or other object. For example, when the human head moves from left to right, the pointer  402  shown on the display  400  originally pausing at the window  404   a , shifts from the window  404   a  to  404   d  in response to the motion, to indicate that the focus of the display (e.g., the currently active window) has been transitioned from the window  404   a  to  404   d . Similarly, if the human head or other object move from right to left, the pointer  402  shifts from  404   d  to  404   c . This way the human head being tracked is capable of arbitrarily changing the focus of the display  400  from a first window to a second window, as well as changing the position of the pointer on the display  400 . 
       FIG. 4  illustrates an example of transitioning focus among five windows  404   a ,  404   b ,  404   c ,  404   d , and  404   e , which might or might not overlap with each other on the display  400 . The method disclosed in this application can detect the movement of the human head or other objects, and in turn place the pointer on the window intended by the user to be focused on. For example, starting from a point in the window  404   d , if the human head moves from upper left to lower right, the pointer will move correspondingly from upper left to lower right and then bring the window  404   e  to focus. When the window  404   e  is in focus, the window  404   e  will be automatically raised to the top of all the windows such that the user can clearly see the current focus of the display. 
     In some implementations, after the user selects one of the multiple windows shown on the display, the user can start to hover the pointer within the selected window by slowly moving his head over relatively short distances. For example, say the pointer  402  is now placed at the window  404   a  and the user would like to hover the pointer within the window  404   a . He could just slowly move his head over short distances, and the pointer will hover within the window  404   a  corresponding to his head motion. Other patterns of motion can be used to signal a hover. For example the motion can be described in terms of a path. The path can define a pattern, which can be mapped to an appropriate action. 
     In some implementations, there might be only one window shown on the display at one time. Similar to controlling the pointer to hover within one of the multiple windows, the user can move his head to shift focus within the single window. 
     In some implementations, the one or more windows (or applications user interfaces) shown on a display include an editor or a browser. A user can scroll a digital file presented by the editor or the browser in different directions such as up, down, left, and right simply by moving his head or the other object. For example, say there is a five-page file presented by a word processor on a display, and only the first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page. Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance. 
     In some implementations, the one or more windows (or application user interfaces) shown on the display present digital images or include three-dimensional effects (e.g., three-dimensional icons). In some implementations, the pointer indicating focus of the display can be a cursor. In certain computer operating systems, it is advantageous to use multiple types of cursors. 
     Manipulation of a Graphical Icon 
       FIG. 5  is a screenshot showing a three-dimensional graphical icon  502  (e.g., a teapot), being manipulated by a user  506  moving his head  508 . The pointer  504  indicates the current focus of the display  500 . The icon  502  as well as the pointer  504  are controlled by the head motion of the user  506 . As described earlier, when the user  506  moves his head  508  in a direction, the pointer  504  will move on the display in response to the head motion. Likewise, the graphical icon  502  will move along with the pointer  504  in response to the head motion. 
     In some implementations, the user  506  can manipulate the icon  502  linearly (e.g., follow a straight motion path on the display  500 ) and/or non-linearly (e.g., follow a curved motion path on the display  500 ). For example, the user  506  can turn his head  508  to the right to make the teapot  502  turn to the right, or turn his head to the left to make the teapot  502  turn to the left. Thus, in a three-dimensional display environment, the head or object motion being tracked by the video camera or like device, can cause a graphical object to move in six degrees of freedom, including up, down, left, right, backward, forward, rotations, etc. 
     In some implementations, the graphical icon  502  can serve as an avatar representing a user. A user could control eye positions of an avatar through head motions. For example, when the user looks right, his head motion will trigger the avatar to look right. When the user looks left, his head motion will trigger the avatar to look left. In other implementations, the avatar could simulate movements that the user makes, other than rotating or linear moving. 
     The avatar representation can be broadly applied in computer technology, whether in the form of a three-dimensional model used in computer games, a two-dimensional icon (picture) used on Internet forums or chatting websites. 
     Manipulation of the Graphical Icon and Focus Transition of the Display 
       FIG. 6  is a screenshot showing a graphical icon  602  (e.g., a teapot), being moved and manipulated at the same time by the user  604  moving his head  606 . In the example shown, two windows  608   a  and  608   b  are concurrently shown on a display  600 . The icon  602  indicates the current focus of the display  600  and follows the user&#39;s head motion. The arrow  610  in the window  608   b  shows the moving direction of the teapot  602 . 
     As shown in  FIG. 6 , the user  604  is moving the teapot  602  by moving his head in a linear path from right to left. At the same time, the user  604  is arbitrarily manipulating the teapot, like rotating it counterclockwise. This way the user  604  can switch the current focus of the display  600  between the two windows  608   a  and  608   b , as well as manipulate the teapot  602  at the same time. In some implementations, there are more than two windows concurrently shown on the display  600 . 
     Extraction of Optical Flow Information 
       FIG. 7  is a screenshot showing optical flow of the user  702  when turning his head  704  from left to right. Optical flow as used in this application is a visual representation approximating a motion of an object and would likely not be visible to the user in an actual application. As can be seen in  FIG. 7 , lines  706  (e.g., motion vectors) on the left side of the display  700  represent optical flow from left to right simulating the motion of the user  702  turning his head from left to right. In contrast, dots  708  on the right side of the display  700  appear relatively still because there is no object moving in the corresponding area (e.g., the motion vector has zero components). 
     In some implementations, it can be advantageous to extract optical flow information of an object motion from a background. The optical flow provides information on direction, intensity, and number of moving parts which can be captured in real-time by the video camera. Subsequently, the optical flow information can be used for transitioning the current focus of a display corresponding to the object motion. 
     Example Process of Display for Visual Motion-Tracking 
       FIG. 8  is a flow diagram of an example of transitioning a current focus of a display corresponding to an object&#39;s motion tracked by a video camera or like device. Some implementations of this process have already been described in reference to  FIGS. 1-7 . 
     The process includes visually tracking an object&#39;s motion  800  from a first position to a second position. In some implementations, visually tracking an object&#39;s motion includes using a video camera or a like device. The object to be tracked can be any tangible articles, such as a human head, a pen, etc. 
     Responsive to tracking the object&#39;s motion, the display transitions its current focus from a first visual presentation (e.g., a first window or user interface) to a second visual presentation (e.g., a second window or user interface) corresponding to the object&#39;s motion  810 . The transitioning of focus on the display can be indicated by a pointer or a graphical icon shown on the display. 
     In some implementations, there are one or more windows shown on the display. The pointer or the graphical icon can hover to signal transitioning of focus within a window or among multiple windows. If the pointer or the graphical icon is within one window, it can also manipulate a digital file presented by the window, by scrolling the file up, down, left and right, for example. 
     Example Process of Display for Visual Transition Tracking 
       FIG. 9  is a flow diagram of an example process for indicating focus of a display and transitioning a graphical icon&#39;s visual presentation corresponding to an object&#39;s motion. Some implementations of this process are described in reference to  FIGS. 5-7 . 
     The process includes indicating a current focus of a display by a graphical icon  900 . The focus indication can be facilitated by a pointer in the center of the graphical icon, or simply by the position of the graphical icon. 
     The process further includes visually tracking an object&#39;s motion from a first visual state to a second visual state  910 . For instance, the camera tracks a non-linear motion of a user&#39;s head, such as rotating about an axis clockwise. 
     Responsive to tracking the object&#39;s motion from a first visual state to a second visual state, the process transitions the graphical icon on the display from a first visual presentation to a second visual presentation corresponding to the object&#39;s motion  920 . Following the previous example illustrated by the preceding paragraph, the graphical icon shown on the display will rotate clockwise, corresponding to the clockwise rotating of the user&#39;s head. 
     In some implementations, the graphical icon indicates a changing of the focus of a display and rotates itself at the same time while the object being tracked moves both linearly and/or non-linearly. In some implementations, the graphical icon&#39;s motion is not constrained to rotating. 
     Example Process of Display for Multiple Objects 
       FIG. 10  is a flow diagram of an example process for tracking two objects&#39; motions and transitioning focus of display corresponding to the two objects&#39; motions. Some steps of this process have been described in reference to  FIG. 8 . 
     The process includes visually tracking an object&#39;s motion from a first position to a second position  1000 . After tracking the object&#39;s motion, the process transitions focus of a display from a first visual presentation to a second presentation  1010 . The process repeats itself by visually tracking a second object&#39;s motion from a third position to a fourth position  1020 . Then the process transitions focus of the display from a third visual presentation to a fourth presentation  1030 . 
     The process can be further explained by an example where two or more people are sharing and observing a single desktop. Each of them is able to transition the focus of the desktop by simply moving his or her head. 
     In some implementations, the two objects being visually tracked can take turns to move as well as manipulate a graphical icon shown on the display. In an example where the graphical icon is a volleyball, two people can take turns to move and spin the volleyball by simply moving their heads. 
     Example Process of Multiple Displays 
       FIG. 11  is a flow diagram of an example process for tracking an object motion and transitioning focus of display corresponding to the object motion on a second display. 
     The process includes visually tracking an object&#39;s motion from a first position to a second position  1100 . Responsive to tracking the object&#39;s motion, the process transitions focus of a display from a first visual presentation to a second visual presentation corresponding to the object&#39;s motion  1110 . The process also transitions focus of a second display from a third visual presentation to a fourth presentation corresponding to the object&#39;s motion  1120 . 
     In some implementations, the multiple displays are connected by networks. Two or more users can, on their respective displays, observe and control focus of a single desktop, or that of a presentation of a shared digital file. By way of illustration, when two or more users chat online and discuss a file presented on their respective displays, each of them can transition focus of the presentation of the file by moving his or her head. As another example, when two or more users play games online, each of them can manipulate a graphical icon (e.g., a volleyball or a basketball), simply by moving his or her head. 
     System Architecture 
       FIG. 12  is a block diagram of an example system architecture  1200  that can be utilized to implement the systems and methods described in reference to  FIGS. 1-11 . The system  1200  can, for example, be implemented in a computer device, such as any one of the personal computer devices available from Apple Inc. (Cupertino, Calif., USA), or other personal computer devices. Other example implementations can also include video processing devices, multimedia processing devices, portable computing devices, etc. 
     The example system  1200  includes a processing device  1202 , a first data store  1204 , a second data store  1206 , a graphics device  1208 , input devices  1210 , output devices  1212 , and a network interface  1214 . A bus system  1216 , such as a data bus and a motherboard, can be used to establish and control data communication between the components  1202 ,  1204 ,  1206 ,  1208 ,  1210 ,  1212  and  1214 . Other example system architectures, however, can also be used. 
     The processing device  1202  can, for example, include one or more microprocessors having one or more processing cores. The first data store  1204  can, for example, include a random access memory storage device, such as a dynamic random access memory, or other types of computer-readable medium memory devices. The second data store  1206  can, for example, include one or more hard drives, a flash memory, and/or a read only memory, or other types of computer-readable medium memory devices. 
     The graphics device  1208  can, for example, include a video card, a graphics accelerator card, a graphics processing unit (GPU) or a display adapter, and is configured to generate and output images to a display device. In one implementation, the graphics device  1208  can be realized in a dedicated hardware card connected to the bus system  1216 . In another implementation, the graphics device  1208  can be realized in a graphics controller integrated into a chipset of the bus system  1216 . 
     Example input devices  1210  can include a video camera, a keyboard, a mouse, a stylus, etc., and example output devices  1212  can include a display device, an audio device, etc. 
     The network interface  1214  can, for example, include a wired or wireless network device operable to communicate data to and from a network  1218 . The network  1218  can include one or more local area networks (LANs) or a wide area network (WAN), such as the Internet. 
     In an implementation, the system  1200  includes instructions defining an operating system stored in the first data store  1204  and/or the second data store  1206 . Example operating systems can include the MAC OS® X series operating system, the WINDOWS® based operating system, or other operating systems. Upon execution of the operating system instructions, access to various system objects is enabled. Example system objects include data files, applications, functions, windows, etc. To facilitate an intuitive user experience, the system  1200  includes a graphical user interface that provides the user access to the various system objects and conveys information about the system  1200  to the user in an intuitive manner. 
     This written description sets forth the best mode of the invention and provides examples to describe the invention and to enable a person of ordinary skill in the art to make and use the invention. This written description does not limit the invention to the precise terms set forth. Thus, while the invention has been described in detail with reference to the examples set forth above, those of ordinary skill in the art may effect alterations, modifications and variations to the examples without departing from the scope of the invention. 
     View Changing of a Three-Dimensional Graphical Icon 
       FIG. 13A-13C  are screenshots showing a three-dimensional graphical icon  1302   a - 1302   c  (e.g., a teapot) being displayed from different angles depending on a user&#39;s perspectives. In the examples shown, a position of the user&#39;s head or other object, is represented by an indicator  1304   a - 1304   c  on a display  1300   a - 1300   c . Note  FIG. 13A-13C  merely serve as examples to illustrate how a three-dimensional graphical icon can be viewed on a display from various angles in response to the motion of the user&#39;s head or other object. In some implementations, the indicator  1304   a - 1304   c  is not actually displayed. 
     In  FIG. 13A , the position of the user&#39;s head, for example, is represented by the indicator  1304   a . In the example shown, the user is looking at the icon  1302   a  from a left of the icon  1302   a , after moving the user&#39;s head to the left, relative to the position of the icon  1302   a  shown on the display  1300   a . In response to the user&#39;s head motion, the icon  1302   a  is presented on the display  1300   a  in a way as if the icon  1302   a  is actually viewed by the user from a left angle. 
     In  FIG. 13B , the position of the user&#39;s head is represented by the indicator  1304   b . In the example shown, the user is looking at the icon  1302   b  from a center front of the icon  1302   b  shown on the display  1300   b , after moving the user&#39;s head to the center front, relative to the position of the icon  1302   b . In response to the user&#39;s head motion, the icon  1302   b  is presented on the display  1300   b  in a way as if the icon  1302   b  is actually viewed by the user from a center front angle. 
     In  FIG. 13C , the position of the user&#39;s head is represented by the indicator  1304   c . In the example shown, the user is looking at the icon  1302   c  from a top right of the icon  1302   c , after moving the user&#39;s head to the top right, relative to the position of the icon  1302   c  shown on the display  1300   c . In response to the user&#39;s head motion, the icon  1302   c  is presented on the display  1300   c  in a way as if the icon  1302   c  is actually viewed by the user from a top right angle. 
     In some implementations, the three-dimensional graphical icon  1302   a - 1302   c  is presented to appear closer or farer to the user depending on a distance from the user to the icon  1302   a - 1302   c  shown on the display  1300   a - 1300   c . In various implementations, the appearance of the icon  1302   a - 1302   c  shown on the display  1300   a - 1300   c  is adjusted according to both the distance and the angle of the user. 
     The technology of displaying the three-dimensional graphical icon  1302   a - 1302   c  from different angles and/or in various distances, corresponding to the motion of the user&#39;s head or other object, can be broadly employed in computer graphics, including two-dimensional graphics, three-dimensional graphics, and image processing. By way of illustration, in the field of three-dimensional modeling (a process of developing a representation of a three-dimensional object via specialized software), the user can easily see the object being modeled on the display from different angles and/or in various distances by moving the user&#39;s head or other object. 
     Changing of Views in a Three-Dimensional Universe 
       FIG. 14A-14B  illustrates changing of views from a user&#39;s perspective in a three-dimensional space  1402 . In  FIG. 14A , a box-shaped space shown on a display  1400   a  represents a three-dimensional space  1402 . A display  1400   a - 1400   b  correspondingly presents different aspects of the three-dimensional space  1402  in response to user&#39;s head motion. 
     For purposes of illustration, an axis  1410   a - 1410   b  in  FIG. 14A-14B  represents a motion path of the user&#39;s head. For example, when the user&#39;s head turns right by 60° from a initial point 0°, an index  1412   a - 1412   b  correspondingly points at 60° on the right end of the axis  1410   a - 1410   b . In various implementations, the axis  1410   a - 1410   b  and/or the index  1412   a - 1412   b  is not actually displayed. 
     In  FIG. 14A , the user initially looks to the front facing the display  1400   a , as the index  1412   a  indicates. Correspondingly, a side A  1406  of the space  1402  is presented on the display to the user, to simulate what the user sees as if the user is actually situated in the space  1402 . 
     Following the example illustrated in  FIG. 14A , the user in  FIG. 14B  turn his head to the right by 30°, as the index  1412   b  indicates on the axis  1410   b . Imagine the user is situated in the three-dimensional space  1402 . After the user turns his head to the right by 30°, the user is supposed to face a side B  1408   b  of the space  1402 . To simulate the user&#39;s view as a result of the head motion, the display  1400   b  correspondingly presents the side B  1408   b  of the space  1402  to the user. 
     In some implementations, the aspects of the three-dimensional space  1402  look closer or farther depending on the user&#39;s distance to the aspects shown on the display  1400   a - 1400   b . In various implementations, the appearance of the aspects of the space  1402  is adjusted according to both the angle and the distance of the user. 
     The technology of showing different aspects of the three-dimensional space  1402 , in response to the motion of the user&#39;s head or other object, can be broadly employed in video games implemented on various platforms. By way of illustration, in a three-dimensional computer game that correspondingly presents aspects of a three-dimensional universe reflecting the user&#39;s viewpoints, the user can conveniently change views by motion of the user&#39;s head or other object. 
     Example Process of Display for Tracking of Visual Change in States and Locations 
       FIG. 15  is a flow diagram of an example process for transitioning a graphical icon&#39;s visual presentation concurrently with its visual location corresponding to an object&#39;s motion. Some implementations of the process are described in reference to  FIG. 7-9 . 
     The process includes visually determining a current state and a current location of an object  1500 . In some implementations, visually determining the object&#39;s current state and location involves using a video camera or a like device. 
     After determining the object&#39;s current state and location, the process visually tracks motion of the object from a first location (e.g., the object on the left of the display) and a first visual state (e.g., the object facing the display) to a second location (e.g., the object on the right of the display) and a second visual state (e.g., the object turning to the right by 30°)  1510 . Responsive to tracking the object&#39;s motion, the display transitions a graphical icon on a display from a first visual position (e.g., a first window or user interface) and a first visual presentation (e.g., the graphical icon facing front) to a second visual position (e.g., a second window or user interface) and a second visual presentation (e.g., the graphical icon turning to the right by 30°) corresponding to the object&#39;s motion  1520 .

Metadata:
Filing Date: 20120416
Publication Date: 20150609
Grant Date: 20150609
Priority Date: 20071213
Inventors: QUENNESSON KEVIN
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/04817", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10016", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G5/14", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/005", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04815", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/30201", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T19/00", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/30201", "inventive": false, "first": false, "tree": "[]"}, {"code": "H04N13/366", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/10016", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2219/028", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/269", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N13/366", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/012", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/012", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G5/14", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T19/00", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2219/028", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T7/269", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T19/00", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/2066", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T2207/30201", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2207/10016", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06T2219/028", "inventive": false, "first": false, "tree": "[]"}, {"code": "H04N13/0468", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/012", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G5/14", "inventive": false, "first": false, "tree": "[]"}]
Family ID: 40752535