PATENT DOCUMENT

Publication Number: US-11169611-B2
Application Number: US-201313849517-A
Country: US
Kind Code: B2

Title: Enhanced virtual touchpad

Abstract:
A method, including receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface and segmenting the physical surface into one or more physical regions. A functionality is assigned to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, and a sequence of three-dimensional (3D) maps is received, the sequence of 3D maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions. The 3D maps are analyzed to detect a gesture performed by the user, and based on the gesture, an input is simulated for the tactile input device corresponding to the one of the physical regions.

Claims:
The invention claimed is: 
     
       1. An apparatus, comprising:
 a sensing device configured to receive a two dimensional (2D) image containing at least a physical surface, and to receive a sequence of three dimensional (3D) maps containing at least a hand of a user, the hand positioned on the physical surface; 
 a display; and 
 a computer coupled to the sensing device and the display, and configured to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, such that one of the physical regions on the physical surface has the functionality of a touchpad or touchscreen, to analyze the 3D maps to detect a gesture performed by the user with the hand positioned on the one of the physical regions that has the functionality of the touchpad or touchscreen, and to simulate, based on the gesture, an input from the touchpad or touchscreen. 
 
     
     
       2. The apparatus according to  claim 1 , and comprising a projector coupled to the computer and configured to project an image on the physical surface in response to the gesture. 
     
     
       3. The apparatus according to  claim 1 , wherein the computer is configured to determine, based the sequence of 3D maps, a pressure applied by one or more fingers of the hand against the physical surface, and to incorporate the pressure into the simulated input. 
     
     
       4. The apparatus according to  claim 1 , wherein the computer is configured to identify, based on the 2D image, a color of an object held by the hand and in contact with the physical surface, and to incorporate the color into the simulated input. 
     
     
       5. The apparatus according to  claim 1 , wherein the computer is configured to determine, based on the sequence of 3D maps, a position and a velocity of one or more fingers of the hand, and to incorporate the position and the velocity into the simulated input. 
     
     
       6. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a two-dimensional image (2D) containing at least a physical surface, to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, such that one of the physical regions on the physical surface has the functionality of a touchpad or touchscreen, to analyze the 3D maps to detect a gesture performed by the user with the hand positioned on the one of the physical regions that has the functionality of the touchpad or touchscreen, and to simulate, based on the gesture, an input from the touchpad or touchscreen. 
     
     
       7. A method, comprising:
 Receiving, in a computer, a sequence of three-dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface; 
 analyzing the 3D maps to detect a gesture performed by the user; 
 projecting, onto the physical surface, a respective contour image encompassing each of the one or more physical objects; 
 projecting, onto the physical surface, an animation in response to the gesture; and 
 incorporating the respective contour image encompassing the one or more physical objects into the animation. 
 
     
     
       8. An apparatus, comprising:
 a sensing device configured to receive a sequence of three dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user, the hand positioned in proximity to the physical surface; 
 a projector; and 
 a computer coupled to the sensing device and the projector, and configured to analyze the 3D maps to detect a gesture performed by the user, to present on the physical surface, using the projector, a respective contour image encompassing each of the one or more physical objects and an animation, in response to the gesture, incorporating the respective contour image encompassing the one or more physical objects. 
 
     
     
       9. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a sequence of three-dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface, to analyze the 3D maps to detect a gesture performed by the user, to project, onto the physical surface, a respective contour image encompassing each of the one or more physical objects and an animation in response to the gesture, and to incorporate the respective contour image encompassing the one or more physical objects into the animation. 
     
     
       10. A method, comprising:
 receiving, by a computer, a two-dimensional image (2D′ containing at least a physical surface; 
 segmenting the physical surface into one or more physical regions; 
 assigning a functionality to each of the one or more physical regions, such that one of the physical regions on the physical surface has the functionality of a touchpad or touchscreen; 
 receiving a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, with the hand positioned on the one of the physical regions that has the functionality of the touchpad or touchscreen; 
 analyzing the 3D maps to detect a gesture performed by the user on the one of the physical regions that has the functionality of the touchpad or touchscreen; and 
 
       simulating, based on the gesture, an input from the touchpad or touchscreen wherein simulating the input comprises drawing a line in response to a first gesture performed by the hand on the physical surface, and erasing the line in response to a second gesture performed by the hand on the physical surface. 
     
     
       11. A method, comprising:
 receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface; 
 segmenting the physical surface into one or more physical regions; 
 assigning a functionality to each of the one or more physical regions, such that one of the physical regions on the physical surface has the functionality of a touchpad or touchscreen; 
 receiving a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, with the hand positioned on the one of the physical regions that has the functionality of the touchpad or touchscreen; 
 
       analyzing the 3D maps to detect a gesture performed by the user on the one of the physical regions that has the functionality of the touchpad or touchscreen; and 
       simulating, based on the gesture, an input from the touchpad or touchscreen; and
 comprising determining, based the sequence of 3D maps, a pressure applied by one or more fingers of the hand against the physical surface, and incorporating the pressure into the simulated input wherein simulating the input comprises drawing a line in response to movement of the one or more fingers, while controlling a thickness of the drawn line responsively to the pressure. 
 
     
     
       12. A method, comprising:
 receiving, by a computer, a two-dimensional color image (2D containing at least a physical surface; 
 segmenting the physical surface into one or more physical regions; 
 assigning a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device; 
 receiving a sequence of three-dimensional (3D) maps containing at least an object held by a hand of a user of the computer the object positioned on one of the physical regions; 
 recognizing, in the 2D color image, a color of the object held by the hand; 
 analyzing the 3D maps to detect a gesture performed using the object; and simulating, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions while incorporating the recognized color of the object into the simulated input wherein simulating the input comprises projecting an image that incorporates the recognized color onto the physical surface.

Description:
RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Patent Application 61/615,403, filed Mar. 26, 2012, and U.S. Provisional Patent Application 61/663,638, filed Jun. 25, 2012, which are incorporated herein by reference. This application is related to another U.S. patent application, filed on even date, entitled, “Gaze-Enhanced Virtual Touchscreen.” 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to user interfaces for computerized systems, and specifically to user interfaces that are based on three-dimensional sensing. 
     BACKGROUND 
     Many different types of user interface devices and methods are currently available. Common tactile interface devices include a computer keyboard, a mouse and a joystick. Touch screens detect the presence and location of a touch by a finger or other object within the display area. Infrared remote controls are widely used, and “wearable” hardware devices have been developed, as well, for purposes of remote control. 
     Computer interfaces based on three-dimensional (3D) sensing of parts of a user&#39;s body have also been proposed. For example, PCT International Publication WO 03/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A 3D sensor, typically positioned in a room in proximity to the user, provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on the shape of the body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device. 
     Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered. 
     As another example, U.S. Pat. No. 7,348,963, whose disclosure is incorporated herein by reference, describes an interactive video display system, in which a display screen displays a visual image, and a camera captures 3D information regarding an object in an interactive area located in front of the display screen. A computer system directs the display screen to change the visual image in response to changes in the object. 
     Three-dimensional human interface systems may identify not only the user&#39;s hands, but also other parts of the body, including the head, torso and limbs. For example, U.S. Patent Application Publication 2010/0034457, whose disclosure is incorporated herein by reference, describes a method for modeling humanoid forms from depth maps. The depth map is segmented so as to find a contour of the body. The contour is processed in order to identify a torso and one or more limbs of the subject. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map. 
     Some user interface systems track the direction of the user&#39;s gaze. For example, U.S. Pat. No. 7,762,665, whose disclosure is incorporated herein by reference, describes a method of modulating operation of a device, comprising: providing an attentive user interface for obtaining information about an attentive state of a user; and modulating operation of a device on the basis of the obtained information, wherein the operation that is modulated is initiated by the device. Preferably, the information about the user&#39;s attentive state is eye contact of the user with the device that is sensed by the attentive user interface. 
     SUMMARY OF THE INVENTION 
     There is provided, in accordance with an embodiment of the present invention a method, including receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface, segmenting the physical surface into one or more physical regions, assigning a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, receiving a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions, analyzing the 3D maps to detect a gesture performed by the user, and simulating, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device configured to receive a two dimensional (2D) image containing at least a physical surface, and to receive a sequence of three dimensional (3D) maps containing at least a hand of a user, the hand positioned on the physical surface, a display, and a computer coupled to the sensing device and the display, and configured to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to analyze the 3D maps to detect a gesture performed by the user, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product, including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a two-dimensional image (2D) containing at least a physical surface, to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to receive a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, the hand positioned on one of the physical regions, to analyze the 3D maps to detect a gesture performed by the user, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
     There is additionally provided, in accordance with an embodiment of the present invention a method, including receiving a sequence of three-dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface, analyzing the 3D maps to detect a gesture performed by the user, projecting, onto the physical surface, an animation in response to the gesture, and incorporating the one or more physical objects into the animation. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device configured to receive a sequence of three dimensional (3D) maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user, the hand positioned in proximity to the physical surface, a projector, and a computer coupled to the sensing device and the projector, and configured to analyze the 3D maps to detect a gesture performed by the user, to present, using the projector, an animation onto the physical surface in response to the gesture, and to incorporate the one or more physical objects into the animation. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product, including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a sequence of three-dimensional maps containing at least a physical surface, one or more physical objects positioned on the physical surface, and a hand of a user of the computer, the hand positioned in proximity to the physical surface, to analyze the 3D maps to detect a gesture performed by the user, to project, onto the physical surface, an animation in response to the gesture, and to incorporate the one or more physical objects into the animation. 
     There is additionally provided, in accordance with an embodiment of the present invention a method, including receiving, by a computer, a two-dimensional image (2D) containing at least a physical surface, segmenting the physical surface into one or more physical regions, assigning a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, receiving a sequence of three-dimensional (3D) maps containing at least an object held by a hand of a user of the computer, the object positioned on one of the physical regions, analyzing the 3D maps to detect a gesture performed by the object, and simulating, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus, including a sensing device configured to receive a two dimensional (2D) image containing at least a physical surface, and to receive a sequence of three dimensional (3D) maps containing at least an object held by a hand of a user, the object positioned on the physical surface, a display, and a computer coupled to the sensing device and the display, and configured to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to analyze the 3D maps to detect a gesture performed by the object, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product, including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a two-dimensional image (2D) containing at least a physical surface, to segment the physical surface into one or more physical regions, to assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, to receive a sequence of three-dimensional (3D) maps containing at least an object held by a hand of a user of the computer, the object positioned on one of the physical regions, to analyze the 3D maps to detect a gesture performed by the object, and to simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosure is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
         FIG. 1  is a schematic, pictorial illustration of a computer system implementing a non-tactile three-dimensional (3D) user interface, in accordance with an embodiment of the present invention; 
         FIG. 2  is a block diagram that schematically illustrates functional components of the computer system implementing the non-tactile 3D user interface, in accordance with an embodiment of the present invention; 
         FIG. 3  is a flow diagram that schematically illustrates a method of detecting gazes and gestures, in accordance with an embodiment of the present invention; 
         FIGS. 4A-4G , referred to collectively as  FIG. 4 , are schematic pictorial illustrations of gestures that can be used to interact with the computer system, in accordance with an embodiment of the present invention; 
         FIG. 5  is a schematic pictorial illustration of a pictures library application executing on the computer and presented on a display, in accordance with an embodiment of the present invention; 
         FIGS. 6A and 6B  are schematic pictorial illustrations of a calendar application executing on the computer and presented on the display, in accordance with an embodiment of the present invention; 
         FIGS. 7A and 7B  are schematic pictorial illustrations of a virtual keyboard presented on the display, in accordance with an embodiment of the present invention; 
         FIGS. 8A-8D , referred to collectively as  FIG. 8 , are schematic pictorial illustrations of physical regions on a physical surface, in accordance with an embodiment of the present invention; 
         FIGS. 9A-9C , referred to collectively as  FIG. 9 , are schematic pictorial illustrations showing how movement of a user&#39;s hand on or near the physical surface can provide “inertial” input to the computer; 
         FIGS. 10A-10D , referred to collectively as  FIG. 10  are schematic pictorial illustrations of the physical surface configured as an input device for a drawing application, in accordance with an embodiment of the present invention; 
         FIG. 11  is a schematic pictorial illustration showing how a “pie menu” may be incorporated into the drawing application; 
         FIGS. 12A and 12B , referred to collectively as  FIG. 12 , are schematic pictorial illustrations of the physical surface illuminated by a projector, in accordance with an embodiment of the present invention; and 
         FIGS. 13A-13D , referred to collectively as  FIG. 13 , are schematic pictorial illustrations of the computer system incorporating, into an animation projected onto the physical surface, one or more physical objects positioned on the physical surface. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Overview 
     When using physical tactile input devices such as buttons, rollers or touch screens, a user typically engages and disengages control of a user interface by touching and/or manipulating the physical device. Embodiments of the present invention describe gestures that can be performed by a user in order to engage interactive items presented on a display coupled to a computer executing a user interface that includes three-dimensional (3D) sensing. 
     As explained hereinbelow, a user can select a given one of the interactive items by gazing at the given interactive item, and manipulate the given interactive item by performing two-dimensional (2D) gestures on a tactile input device, such as a touchscreen or a touchpad. In some embodiments the computer can defines a virtual surface that emulates a touchpad or a touchscreen. The virtual surface can be implemented on a physical surface such as a book or a desktop, and the user can interact with the user interface by performing 2D gestures on the physical surface. In alternative embodiments, the virtual surface can be implemented in space in proximity to the user, and the user can interact with the computer by performing 3D gestures, as described hereinbelow. 
     In further embodiments, when configuring the physical surface as a virtual surface, the physical surface can be configured as a single input device, such as a touchpad. Alternatively, the physical surface can be divided into physical regions, and a respective functionality can be assigned to each of the physical regions. For example, a first physical region can be configured as a keyboard, a second physical region can be configured as a mouse, and a third physical region can be configured as a touchpad. 
     In additional embodiments, as described hereinbelow, a projector can be configured to project graphical images onto the physical surface, thereby enabling the physical surface to function as an interactive touchscreen on which visual elements can be drawn and manipulated in response to gestures performed by the user. 
     System Description 
       FIG. 1  is a schematic, pictorial illustration of a non-tactile 3D user interface  20  for operation by a user  22  of a computer  26 , in accordance with an embodiment of the present invention. (Although for the sake of simplicity, only a single user and user interface are shown in the figure, in practice interface  20  may interact with multiple users concurrently. Alternative embodiments of the present invention may use different user interfaces and/or support multiple user interfaces across different devices). User interface  20  in the pictured embodiment is based, by way of example, on a 3D sensing device  24 , which captures 3D scene information that includes a body, or at least parts of the body, such as a finger  30 , a hand  31 , a head  32 , or eyes  34 . Device  24  or a separate camera (not shown in the figures) may also capture color video images of the scene. The information captured by device  24  is processed by computer  26 , which drives a display screen  28  accordingly to present and manipulate on-screen interactive items  36  (also referred to herein as interactive items). Alternatively, the user interface may be used in conjunction with any type of computerized equipment, such as a laptop, a tablet computer, a television, etc. 
     While  FIG. 1  shows computer  26  in a tower configuration, other configurations of the computer are considered to be within the spirit and scope of the present invention. For example, computer  26  may be configured as a desktop computer, a portable computer (e.g., a laptop) or an all-in-one computer. 
     Computer  26  processes data generated by device  24  in order to reconstruct a 3D map of user  22 . The term “3D map” (or equivalently, “depth map”) refers to a set of 3D coordinates representing a surface of a given object, in this case the user&#39;s body. In one embodiment, device  24  projects a pattern of spots onto the object and captures an image of the projected pattern. Computer  26  then computes the 3D coordinates of points on the surface of the user&#39;s body by triangulation, based on transverse shifts of the spots in the imaged pattern. The 3D coordinates are measured, by way of example, with reference to a generally horizontal X-axis  40 , a generally vertical Y-axis  42  and a depth Z-axis  44 , based on device  24 . Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, system  20  may use other methods of 3D mapping, using single or multiple cameras or other types of sensors, as are known in the art. 
     In some embodiments, device  24  detects the location and direction of eyes  34  of user  22 , typically by processing and analyzing an image comprising light (typically infrared and/or a color produced by the red-green-blue additive color model) reflecting from one or both eyes  34 , in order to find a direction of the user&#39;s gaze. In alternative embodiments, computer  26  (either by itself or in combination with device  24 ) detects the location and direction of the eyes  34  of the user. The reflected light may originate from a light projecting source of device  24 , or any other natural (e.g., sunlight) or artificial (e.g., a lamp) source. Using techniques that are known in the art such as detecting pupil center and corneal reflections (PCCR), device  24  may process and analyze an image comprising light reflecting from an element of eye  34 , such as a pupil  38 , an iris  39  or a cornea  41 , in order to find the direction of the user&#39;s gaze. Additionally, device  24  may convey (to computer  26 ) the light reflecting from the cornea as a glint effect. 
     The location and features of the user&#39;s head (e.g., an edge of the eye, a nose or a nostril) that are extracted by computer from the 3D map may be used in finding coarse location coordinates of the user&#39;s eyes, thus simplifying the determination of precise eye position and gaze direction, and making the gaze measurement more reliable and robust. Furthermore, computer  26  can readily combine the 3D location of parts of head  32  (e.g., eye  34 ) that are provided by the 3D map with gaze angle information obtained via eye part image analysis in order to identify a given on-screen object  36  at which the user is looking at any given time. This use of 3D mapping in conjunction with gaze tracking allows user  22  to move head  32  freely while alleviating the need to actively track the head using sensors or emitters on the head, as in some eye tracking systems that are known in the art. 
     By tracking eye  34 , embodiments of the present invention may reduce the need to re-calibrate user  22  after the user moves head  32 . In some embodiments, computer  26  may use depth information for head  32 , eye  34  and pupil  38 , in order to track the head&#39;s movement, thereby enabling a reliable gaze angle to be calculated based on a single calibration of user  22 . Utilizing techniques that are known in the art such as PCCR, pupil tracking, and pupil shape, computer  26  may calculate a gaze angle of eye  34  from a fixed point of head  32 , and use the head&#39;s location information in order to re-calculate the gaze angle and enhance the accuracy of the aforementioned techniques. In addition to reduced recalibrations, further benefits of tracking the head may include reducing the number of light projecting sources and reducing the number of cameras used to track eye  34 . 
     In addition to processing data generated by device  24 , computer  26  can process signals from tactile input devices such as a keyboard  45  and a touchpad  46  that rest on a physical surface  47  (e.g., a desktop). Touchpad  46  (also referred to as a gesture pad) comprises a specialized surface that can translate the motion and position of fingers  30  to a relative position on display  28 . In some embodiments, as user  22  moves a given finger  30  along the touchpad, the computer can responsively present a cursor (not shown) at locations corresponding to the finger&#39;s motion. For example, as user  22  moves a given finger  30  from right to left along touchpad  46 , computer  26  can move a cursor from right to left on display  28 . 
     In some embodiments, display  28  may be configured as a touchscreen comprising an electronic visual display that can detect the presence and location of a touch, typically by one or more fingers  30  or a stylus (not shown) within the display area. When interacting with the touchscreen, user  22  can interact directly with interactive items  36  presented on the touchscreen, rather than indirectly via a cursor controlled by touchpad  46 . 
     In additional embodiments a projector  48  may be coupled to computer  26  and positioned above physical surface  47 . As explained hereinbelow projector  48  can be configured to project an image on physical surface  47 . 
     Computer  26  typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on non-transitory tangible computer-readable media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the computer processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer  26  is shown in  FIG. 1 , by way of example, as a separate unit from sensing device  24 , some or all of the processing functions of the computer may be performed by suitable dedicated circuitry within the housing of the sensing device or otherwise associated with the sensing device. 
     As another alternative, these processing functions may be carried out by a suitable processor that is integrated with display  28  (in a television set, for example) or with any other suitable sort of computerized device, such as a game console or a media player. The sensing functions of device  24  may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output. 
     Various techniques may be used to reconstruct the 3D map of the body of user  22 . In one embodiment, computer  26  extracts 3D connected components corresponding to the parts of the body from the depth data generated by device  24 . Techniques that may be used for this purpose are described, for example, in U.S. patent application Ser. No. 12/854,187, filed Aug. 11, 2010, whose disclosure is incorporated herein by reference. The computer analyzes these extracted components in order to reconstruct a “skeleton” of the user&#39;s body, as described in the above-mentioned U.S. Patent Application Publication 2010/0034457, or in U.S. patent application Ser. No. 12/854,188, filed Aug. 11, 2010, whose disclosure is also incorporated herein by reference. In alternative embodiments, other techniques may be used to identify certain parts of the user&#39;s body, and there is no need for the entire body to be visible to device  24  or for the skeleton to be reconstructed, in whole or even in part. 
     Using the reconstructed skeleton, computer  26  can assume a position of a body part such as a tip of finger  30 , even though the body part (e.g., the fingertip) may not be detected by the depth map due to issues such as minimal object size and reduced resolution at greater distances from device  24 . In some embodiments, computer  26  can auto-complete a body part based on an expected shape of the human part from an earlier detection of the body part, or from tracking the body part along several (previously) received depth maps. In some embodiments, computer  26  can use a 2D color image captured by an optional color video camera (not shown) to locate a body part not detected by the depth map. 
     In some embodiments, the information generated by computer as a result of this skeleton reconstruction includes the location and direction of the user&#39;s head, as well as of the arms, torso, and possibly legs, hands and other features, as well. Changes in these features from frame to frame (i.e. depth maps) or in postures of the user can provide an indication of gestures and other motions made by the user. User posture, gestures and other motions may provide a control input for user interaction with interface  20 . These body motions may be combined with other interaction modalities that are sensed by device  24 , including user eye movements, as described above, as well as voice commands and other sounds. Interface  20  thus enables user  22  to perform various remote control functions and to interact with applications, interfaces, video programs, images, games and other multimedia content appearing on display  28 . 
       FIG. 2  is a block diagram that schematically illustrates functional components of user interface  20 , in accordance with an embodiment of the present invention. Sensing device  24  comprises an illumination subassembly  50 , which projects a pattern onto the scene of interest. A depth imaging subassembly  52 , such as a suitably-configured video camera, captures images of the pattern on the scene. Typically, illumination subassembly  50  and imaging subassembly  52  operate in the infrared range, although other spectral ranges may also be used. Optionally, a color video camera (not shown) in device  24  captures 2D color images of the scene, and a microphone  54  may also capture sound. 
     A processor  56  receives the images from subassembly  52  and compares the pattern in each image to a reference pattern stored in a memory  58 . The reference pattern is typically captured in advance by projecting the pattern onto a reference plane at a known distance from device  24 . Processor  56  computes local shifts of parts of the pattern over the area of the 3D map and translates these shifts into depth coordinates. Details of this process are described, for example, in PCT International Publication WO 2010/004542, whose disclosure is incorporated herein by reference. Alternatively, as noted earlier, device  24  may be configured to generate 3D maps by other means that are known in the art, such as stereoscopic imaging, sonar-like devices (sound based/acoustic), wearable implements, lasers, or time-of-flight measurements. 
     Processor  56  typically comprises an embedded microprocessor, which is programmed in software (or firmware) to carry out the processing functions that are described hereinbelow. The software may be provided to the processor in electronic form, over a network, for example; alternatively or additionally, the software may be stored on non-transitory tangible computer-readable media, such as optical, magnetic, or electronic memory media. Processor  56  also comprises suitable input and output interfaces and may comprise dedicated and/or programmable hardware logic circuits for carrying out some or all of its functions. Details of some of these processing functions and circuits that may be used to carry them out are presented in the above-mentioned Publication WO 2010/004542. 
     In some embodiments, a gaze sensor  60  detects the gaze direction of eyes  34  of user  22  by capturing and processing two dimensional images of user  22 . In alternative embodiments, computer  26  detects the gaze direction by processing a sequence of 3D maps conveyed by device  24 . Sensor  60  may use any suitable method of eye tracking that is known in the art, such as the method described in the above-mentioned U.S. Pat. No. 7,762,665 or in U.S. Pat. No. 7,809,160, whose disclosure is incorporated herein by reference, or the alternative methods described in references cited in these patents. For example, sensor  60  may capture an image of light (typically infrared light) that is reflected from the fundus and/or the cornea of the user&#39;s eye or eyes. This light may be projected toward the eyes by illumination subassembly  50  or by another projection element (not shown) that is associated with sensor  60 . Sensor  60  may capture its image with high resolution over the entire region of interest of user interface  20  and may then locate the reflections from the eye within this region of interest. Alternatively, imaging subassembly  52  may capture the reflections from the user&#39;s eyes (ambient light, reflection from monitor) in addition to capturing the pattern images for 3D mapping. 
     As another alternative, processor  56  may drive a scan control  62  to direct the field of view of gaze sensor  60  toward the location of the user&#39;s face or eye  34 . This location may be determined by processor  60  or by computer  26  on the basis of a depth map or on the basis of the skeleton reconstructed from the 3D map, as described above, or using methods of image-based face recognition that are known in the art. Scan control  62  may comprise, for example, an electromechanical gimbal, or a scanning optical or optoelectronic element, or any other suitable type of scanner that is known in the art, such as a microelectromechanical system (MEMS) based mirror that is configured to reflect the scene to gaze sensor  60 . 
     In some embodiments, scan control  62  may also comprise an optical or electronic zoom, which adjusts the magnification of sensor  60  depending on the distance from device  24  to the user&#39;s head, as provided by the 3D map. The above techniques, implemented by scan control  62 , enable a gaze sensor  60  of only moderate resolution to capture images of the user&#39;s eyes with high precision, and thus give precise gaze direction information. 
     In alternative embodiments, computer  26  may calculate the gaze angle using an angle (i.e., relative to Z-axis  44 ) of the scan control. In additional embodiments, computer  26  may compare scenery captured by the gaze sensor  60 , and scenery identified in 3D depth maps. In further embodiments, computer may compare scenery captured by the gaze sensor  60  with scenery captured by a 2D camera having a wide field of view that includes the entire scene of interest. Additionally or alternatively, scan control  62  may comprise sensors (typically either optical or electrical) configured to verify an angle of the eye movement. 
     Processor  56  processes the images captured by gaze sensor  60  in order to extract the user&#39;s gaze angle. By combining the angular measurements made by sensor  60  with the 3D location of the user&#39;s head provided by depth imaging subassembly  52 , the processor is able to derive accurately the user&#39;s true line of sight in 3D space. The combination of 3D mapping with gaze direction sensing reduces or eliminates the need for precise calibration and comparing multiple reflection signals in order to extract the true gaze direction. The line-of-sight information extracted by processor  56  enables computer  26  to identify reliably the interactive item at which the user is looking. 
     The combination of the two modalities can allow gaze detection without using an active projecting device (i.e., illumination subassembly  50 ) since there is no need for detecting a glint point (as used, for example, in the PCCR method). Using the combination can solve the glasses reflection problem of other gaze methods that are known in the art. Using information derived from natural light reflection, the 2D image (i.e. to detect the pupil position), and the 3D depth map (i.e., to identify the head&#39;s position by detecting the head&#39;s features), computer  26  can calculate the gaze angle and identify a given interactive item  36  at which the user is looking. 
     As noted earlier, gaze sensor  60  and processor  56  may track either one or both of the user&#39;s eyes. If both eyes  34  are tracked with sufficient accuracy, the processor may be able to provide an individual gaze angle measurement for each of the eyes. When the eyes are looking at a distant object, the gaze angles of both eyes will be parallel; but for nearby objects, the gaze angles will typically converge on a point in proximity to an object of interest. This point may be used, together with depth information, in extracting 3D coordinates of the point on which the user&#39;s gaze is fixed at any given moment. 
     As mentioned above, device  24  may create 3D maps of multiple users who are in its field of view at the same time. Gaze sensor  60  may similarly find the gaze direction of each of these users, either by providing a single high-resolution image of the entire field of view, or by scanning of scan control  62  to the location of the head of each user. 
     Processor  56  outputs the 3D maps and gaze information via a communication link  64 , such as a Universal Serial Bus (USB) connection, to a suitable interface  66  of computer  26 . The computer comprises a central processing unit (CPU)  68  with a memory  70  and a user interface  72 , which drives display  28  and may include other components, as well. As noted above, device  24  may alternatively output only raw images, and the 3D map and gaze computations described above may be performed in software by CPU  68 . Middleware for extracting higher-level information from the 3D maps and gaze information may run on processor  56 , CPU  68 , or both. CPU  68  runs one or more application programs, which drive user interface  72  based on information provided by the middleware, typically via an application program interface (API). Such applications may include, for example, games, entertainment, Web surfing, and/or office applications. 
     Although processor  56  and CPU  68  are shown in  FIG. 2  as separate functional elements with a certain division of processing tasks between them, the functions of the processor and CPU may alternatively be carried out by a single processing unit, or these functions may be divided among three or more processing units. Furthermore, although device  24  is shown as containing a certain combination of components in a particular arrangement, other device configurations may be used for the purposes described herein, and are considered to be within the scope of the present invention. 
     Interaction with on-Screen Objects 
       FIG. 3  is a flow diagram that schematically illustrates a method of detecting gaze and gestures in order to select and perform an operation on a given interactive item  36 , in accordance with an embodiment of the present invention. In a presentation step  80 , computer  26  presents multiple interactive items  36  on display  28 , and in a first receive step  82 , the processor receives an input from sensing device  24  indicating a direction of a gaze performed by the user. 
     In some embodiments, receiving the input may comprise receiving, from depth imaging subassembly  52 , a 3D map containing at least head  32 , and receiving, from gaze sensor  60 , a 2D image containing at least eye  34 . Computer  26  can then analyze the received 3D depth map and the 2D image in order to identify a gaze direction of user  22 . Gaze detection is described in PCT Patent Application PCT/IB2012/050577, filed Feb. 9, 2012, whose disclosure is incorporated herein by reference. 
     As described supra, illumination subassembly  50  may project a light toward user  22 , and the received 2D image may comprise light reflected off the fundus and/or the cornea of eye(s)  34 . In some embodiments, computer  26  can extract 3D coordinates of head  32  by identifying, from the 3D map, a position of the head along X-axis  40 , Y-axis  42  and Z-axis  44 . In alternative embodiments, computer  26  extracts the 3D coordinates of head  32  by identifying, from the 2D image a first position of the head along X-axis  40  and Y-axis  42 , and identifying, from the 3D map, a second position of the head along Z-axis  44 . 
     In a selection step  84 , computer  26  identifies and selects a given interactive item  36  that the computer is presenting, on display  28 , in the gaze direction. Subsequent to selecting the given interactive item, in a second receive step  86 , computer  26  receives, from depth imaging subassembly  52 , a sequence of 3D maps containing at least hand  31 . 
     In an analysis step  88 , computer  26  analyzes the 3D maps to identify a gesture performed by user  22 . As described hereinbelow, examples of gestures include, but are not limited to a Press and Hold gesture, a Tap gesture, a Slide to Hold gesture, a Swipe gesture, a Select gesture, a Pinch gesture, a Swipe From Edge gesture, a Select gesture, a Grab gesture and a Rotate gesture. To identify the gesture, computer  26  can analyze the sequence of 3D maps to identify initial and subsequent positions of hand  31  (and/or fingers  30 ) while performing the gesture. 
     In a perform step  90 , the computer performs an operation on the selected interactive item in response to the gesture, and the method ends. Examples of operations performed in response to a given gesture when a single item is selected include, but are not limited to:
         Presenting, on display  28 , context information on the selected interactive item.   Executing an application associated with the selected interactive item.   Switching to an application associated with the selected interactive item (i.e., task switching).   Changing, on display  28 , the size of the selected interactive item.       

     In some embodiments user  22  can select the given interactive item using a gaze related pointing gesture. A gaze related pointing gesture typically comprises user  22  pointing finger  30  toward display  28  to select a given interactive item  36 . As the user points finger  30  toward display  28 , computer  26  can define a line segment between one of the user&#39;s eyes  34  (or a point between eyes  34 ) and the finger, and identify a target point where the line segment intersects the display. Computer  26  can then select a given interactive item  36  that is presented in proximity to the target point. Gaze related pointing gestures are described in PCT Patent Application PCT/IB2012/050577, filed Feb. 9, 2012, whose disclosure is incorporated herein by reference. 
     In additional embodiments, computer  26  can select the given interactive item  36  using gaze detection in response to a first input (as described supra in step  82 ), receive a second input, from touchpad  46 , indicating a (tactile) gesture performed on the touchpad, and perform an operation in response to the second input received from the touchpad. 
     In further embodiments, user  22  can perform a given gesture while finger  30  is in contact with physical surface  47  (e.g., the desktop shown in  FIG. 1 ), thereby “transforming” the physical surface into a virtual touchpad. In supplementary embodiments, as described hereinbelow, projector  48  can project an image on physical surface  47 , thereby transforming the physical surface into a virtual touchscreen. 
     As described supra, embodiments of the present invention enable computer  26  to emulate touchpads and touchscreens by presenting interactive items  36  on display  28  and identifying three-dimensional non-tactile gestures performed by user  22 . For example, computer  26  can configure the Windows 8™ operating system produced by Microsoft Corporation (Redmond, Wash.), to respond to three-dimensional gestures performed by user  22 . 
       FIGS. 4A-4G  are schematic pictorial illustrations of gestures that correspond to tactile gestures used when interacting with a computer executing the Windows 8™ operating system, in accordance with an embodiment of the present invention. In some embodiments, user  22  can perform the gestures described in  FIG. 4  as two-dimensional gestures on touchpad  46 . Additionally or alternatively, computer  26  may use inputs received from sensing device  24  to define a virtual surface (e.g., a virtual touchpad, a virtual touchscreen, a virtual keyboard or a virtual mouse) on physical surface  47 , or in space in proximity to user  22 . In operation, computer  26  can interpret three-dimensional gestures performed on the virtual surface as a corresponding two-dimensional gesture performed on touchpad  46  or touchscreen  28 . While interacting with the virtual surface hand  31  typically “hovers” over the virtual surface until user  22  performs one of the gestures described hereinbelow. 
       FIG. 4A  is a schematic pictorial illustration of hand  31  performing the Press and Hold gesture, in accordance with an embodiment of the present invention. The Press and Hold gesture is similar to the Point Touch gesture described in PCT/IB2012/050577, referenced above, and comprises user  22  gazing toward a given interactive item  36 , pushing finger  30  toward display  28  (“Press”), and holding the finger relatively steady for at least a specified time period (“Hold”). Upon identifying the gaze direction and the Press and Hold gesture, computer  26  can present context information on the selected interactive item  36 . 
     As described supra, user  22  can select a given interactive item  36  using a gaze related pointing gesture, or perform a tactile gesture on gesture pad  46 . To interact with computer  26  using a gaze related pointing gesture and the Press and Hold gesture, user  22  can push finger  30  toward a given interactive item  36  (“Press”), and hold the finger relatively steady for at least the specified time period (“Hold”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward a given interactive  36 , touch gesture pad  46  with finger  30 , and keep the finger on the gesture pad for at least the specified time period. 
       FIG. 4B  is a schematic pictorial illustration of hand  31  performing the Tap gesture, in accordance with an embodiment of the present invention. The Tap gesture is similar to the Point Select gesture described in PCT/IB2012/050577, referenced above, and comprises user  22  gazing toward a given interactive item  36 , pushing finger  30  toward display  28  (“Press”), and pulling the finger back (“Release”). Upon identifying the gaze direction and the Tap gesture, computer  26  can perform an operation associated with the given interactive item. For example, if the given interactive item comprises an application icon, the computer can execute an application associated with the application icon in response the Tap gesture. 
     To interact with computer  26  using a gaze related pointing gesture and the Tap gesture, user  22  can push finger  30  toward a given interactive item  36  (“Press”), and pull the finger back (“Release”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward a given interactive  36 , touch gesture pad  46  with finger  30 , and lift the finger off the gesture pad. 
       FIG. 4C  is a schematic pictorial illustration of hand  31  performing the Slide to Drag gesture, in accordance with an embodiment of the present invention. The Slide to Drag gesture enables user  22  to scroll interactive items  36 , thereby panning display  28 . To perform the Slide to Drag gesture, user  22  gazes toward any part of display  28 , pushes finger  30  toward the display (“Press”), moves the finger side-to-side in the direction of the requested scroll direction (“Drag”), and pulls the finger back (“Release”). Upon identifying a Slide to Drag gesture, computer  26  can “move the screen” by scrolling the interactive items on display  28  in the direction of the gesture. Therefore when gazing at display  28  and performing the Slide to Drag gesture, user  22  is in effect selecting and performing an operation on all the interactive items presented on the display. 
     In some embodiments, user  22  can control the direction of the scrolling by gazing left or right, wherein the gesture performed by finger  30  only indicates the scrolling action and not the scrolling direction. In additional embodiments, computer  26  can control the scrolling using real-world coordinates, where the computer measures the finger&#39;s motion in distance units such as centimeters and not in pixels. When using real-world coordinates, the computer can apply a constant or a variant factor to the detected movement. For example, the computer can translate one centimeter of finger motion to 10 pixels of scrolling on the display. 
     Alternatively, the computer may apply a formula with a constant or a variable factor that compensates a distance between the user and the display. For example, to compensate for the distance, computer  26  can calculate the formula P=D*F, where P=a number of pixels to scroll on display  28 , D==a distance of user  22  from display  28  (in centimeters), and F=a factor. 
     There may be instances in which computer  26  identifies that user  22  is gazing in a first direction and moving finger  30  in a second direction. For example, user  22  may be directing his gaze from left to right, but moving finger  30  from right to left. In these instances, computer  26  can stop any scrolling due to the conflicting gestures. However, if the gaze and the Slide to Drag gesture performed by the finger indicate the same direction but different scrolling speeds (e.g., the user moves his eyes quickly to the side while moving finger  30  more slowly), the computer can apply an interpolation to the indicated scrolling speeds while scrolling the interactive items. 
     To interact with computer  26  using a gaze related pointing gesture and the Slide to Drag gesture, user  22  can push finger toward display  28  (“Press”), move the finger from side to side (“Drag”), and pull the finger back (“Release”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward display  28 , touch gesture pad  46  with finger  30 , move the finger side to side, and lift the finger off the gesture pad. 
       FIG. 4D  is a schematic pictorial illustration of hand  31  performing the Swipe gesture, in accordance with an embodiment of the present invention. The Swipe gesture can be used for operations such as selecting an interactive item  36  sliding on display  28 , or switching to another application executing on the computer (similar to the Alt-Tab keyboard combination in Microsoft Windows™). To perform the Swipe gesture, user  22  gazes towards a given interactive item  36  that is sliding on display  28 , pushes finger  30  toward the display (“Push”), moves the finger at a 90° angle to the direction that the given interactive item is sliding (“Drag”), and pulls the finger back (“Release”). 
     To interact with computer  26  using a gaze related pointing gesture and the Swipe gesture, user  22  can push finger  30  toward a given interactive item  36  (“Press”), move the finger at a 90° angle to the direction that the given interactive object is sliding (“Drag”), and pull the finger back (“Release”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward a given interactive  36 , touch gesture pad  46  with finger  30 , move the finger at a 90° angle to the direction that the given interactive object is sliding (e.g., up or down if the interactive items are sliding left or right) and lift the finger off the gesture pad. 
     In an alternative embodiment, user  22  can select an interactive item sliding on display  28  by performing the Select Gesture. To perform the Select gesture, user  22  gazes toward an interactive item  36  sliding on display  28  and swipe finger  30  in a downward motion (i.e., on the virtual surface). To interact with computer  26  using a gaze related pointing gesture and the Select gesture, user  22  can push finger  30  toward a given interactive item  36  sliding on display  28 , and swipe the finger in a downward motion. 
       FIG. 5  is a schematic pictorial illustration of a pictures library application  100  executing on computer  26  and presented on display  28 , and a map application  102  executing on the computer and “sliding” horizontally across the display. User  22  user can select the sliding map application  102  by performing the Swipe or Select gestures described supra. 
       FIG. 4E  is a schematic pictorial illustration of hand  31  performing the Pinch (to zoom) gesture, in accordance with an embodiment of the present invention. The Pinch gesture is similar to the Grab gesture described in U.S. patent application Ser. No. 13/423,314 filed on Mar. 19, 2012, whose disclosure is incorporated herein by reference. To perform the Pinch gesture, user  22  gazes toward a given interactive item  36 , pushes two or more fingers  30  toward the display (“Press”), moves the fingers toward each other, e.g., pinching together an index and/or a middle finger with a thumb as shown in  FIG. 4E  (“Pinch”), and pulls the fingers back (“Release”). In response to the Pinch gesture, computer  26  can change the size (i.e., zoom) of the given interactive item presented on the display. 
     To interact with computer  26  using a gaze related pointing gesture and the Pinch gesture, user  22  can push two fingers  30  toward a given interactive item  36  (“Press”), move the fingers toward each other (“Pinch”), and pull the finger back (“Release”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward a given interactive  36 , touch gesture pad  46  with two or more fingers  30 , move the fingers towards or away from each other, and lift the finger off the gesture pad. 
     The Grab gesture has the same functionality as the Swipe gesture. To perform the Grab gesture, user  22  gazes toward a given interactive item  36 , folds one or more fingers  30  toward the palm, either pushes hand  31  toward display  28  or pulls the hand back away from the display, and performs a Release gesture. To interact with computer  26  using a gaze related pointing gesture and the Grab gesture, user  22  can perform the Grab gesture toward a given interactive item  36 , either push hand  31  toward display  28  or pull the hand back away from the display, and then perform a Release gesture. The Release gesture is described in U.S. patent application Ser. No. 13/423,314, referenced above. 
       FIG. 4F  is a schematic pictorial illustration of the Swipe From Edge gesture, in accordance with an embodiment of the present invention. In operation, the Swipe From Edge gesture enables user  22  to view hidden menus or to switch between applications executing on computer  26 . To perform the Swipe from Edge gesture, user  22  gazes toward an (outer) edge of display  28  (i.e., top bottom, left or right), pushes finger  30  toward the display, and moves the finger “into” the display (i.e., away from the edge). Alternatively, user  22  can direct a gaze toward an edge of display  28 , and perform the Swipe gesture by moving hand  31  in a horizontal swiping motion to the opposite side of the display. In embodiments described herein, “close to the edge” of the display can be set as a maximum distance from the edge of the display (e.g., 6 inches outside or from both sides of the edge). 
     To interact with computer  26  using a gaze related pointing gesture and the Swipe from Edge gesture, user  22  can push finger  30  toward an edge of display  28 , and move the finger into the display. Alternatively, user  22  can perform the Swipe gesture away from an edge of display  28 . To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward an edge of display  28 , touch gesture pad  46 , move the finger in a direction corresponding to moving into the display, and lift the finger off the gesture pad. 
     Upon identifying a Swipe From Edge gesture, computer  26  can perform an operation such as presenting a “hidden” menu on the “touched” edge. 
       FIGS. 6A and 6B  are schematic pictorial illustrations of a calendar application  110 , in accordance with an embodiment of the present invention. Initially, computer  26  presents calendar application  110 , as shown in  FIG. 6A . Upon detecting user  22  performing a Swipe From Edge gesture starting from the right edge of the calendar, computer  26  presents a hidden menu  112  (also referred to as a “Charms” menu) on the right side of the calendar (as well as time and date information presented in a black box  114  positioned in the lower-left corner of the display), as shown in  FIG. 7B . In some configurations, there may be a hidden menu  112  for each side of the screen (i.e., left, right, up, down). 
     In additional embodiments, computer  26  can present the hidden menu solely on identifying the user&#39;s gaze directed at the specific edge (the right edge in the example shown in  FIGS. 6A and 6B ), and not require any gesture to be performed by finger  30 . 
       FIG. 4F  is a schematic pictorial illustration of the Rotate gesture, in accordance with an embodiment of the present invention. The Rotate gesture enables user  22  to rotate and thereby control a given interactive item  36 . For example, the selected interactive item  36  may comprise a volume knob that user  22  can control by rotating the knob clockwise or counterclockwise. The Rotate gesture is described in U.S. patent application Ser. No. 13/423,314, referenced above. 
     To perform the Rotate gesture, user  22  gazes toward a given interactive item  36  presented on display  28 , pushes two or more fingers  30  toward the display (“Press”), rotates the fingers in a circular (i.e., clockwise/counterclockwise) motion (“Rotate”), and pulls the fingers back (“Release”). In some embodiments, computer  26  may allow the user to pinch together two or more fingers  30  from different hands  31  while performing the Rotate gesture. 
     To interact with computer  26  using a gaze related pointing gesture and the Rotate gesture, user  22  can push two or more fingers  30  toward a given interactive item  36  (“Press”), rotate the fingers (“Rotate”), and pull the finger back (“Release”). To interact with computer  26  using a gaze and gesture pad  46 , user  22  can gaze toward a given interactive  36 , touch gesture pad  46  with two or more fingers  30 , move the fingers in a circular motion on the gesture pad, and lift the finger off the gesture pad. 
     In addition to manipulating interactive items  36  via the virtual surface, user  22  may also interact with other types of items presented on display  28 , such as an on-screen virtual keyboard as described hereinbelow. 
       FIG. 7A  is a first schematic pictorial illustration of a virtual keyboard  120 , in accordance with an embodiment of the present invention. In the example shown in  FIG. 7A , user  22  interacts with a virtual keyboard  120  via a cursor  122  that computer  26  positions on display  28  in response to the motion of hand  31  and/or finger  30 . Virtual keyboard  120  is described in U.S. patent application Ser. No. 13/244,490, filed Sep. 25, 2011, whose disclosure is incorporated herein by reference. 
     In some embodiments, computer  62  may present interactive items  36  (i.e., the virtual surface) and keyboard  120  simultaneously on display  28 . Computer  26  can differentiate between gestures directed toward the virtual surface and the keyboard as follows:
         A Tap gesture directed outside keyboard  120  can be associated with the virtual surface (i.e., a virtual touchpad or a virtual touchscreen).   Any gesture by two or more connected fingers  30  directed within keyboard  120  can be interpreted as a virtual touchpad gesture.   Single finger gestures directed within keyboard  120  can be interpreted as keys being pressed on the virtual keyboard.       

     In addition to pressing single keys with a single finger, the computer can identify, using a language model, words that the user can input by swiping a single finger over the appropriate keys on the virtual keyboard. 
       FIG. 7B  is a second schematic pictorial illustration of virtual keyboard  120 , in accordance with an embodiment of the present invention. In the example shown in  FIG. 7B , user  22  first moves finger  30  to position cursor  122  at a position  124 , and moves finger  30 , along the path segments shown, so that the cursor changes direction at a position  126  (by the letter “N”), a position  128  (“O”) and a position  130  (“T”). Interpreting letters input via path segments on the virtual keyboard is described in U.S. patent application Ser. No. 13/244,490, referenced above. 
     Additional features that can be included in the virtual surface, using the depth maps and/or color images provided by device  24 , for example, include:
         Finger Detection. Computer  26  can identify which one or more fingers  30  on which hand  31  are interacting with the virtual surface. Different gestures can be defined for different fingers and/or hands.   Color Aware Touchscreen. Computer  26  can identify a color of an object held by hand  31 , and use the identified color in an application. For example, if computer  26  is executing a paint program, and user  22  picks up a colored pen (not shown), then the computer can recognize the color of the pen and use that color when presenting content “drawn” by the user on the virtual surface.   Hand-aware virtual surface. Computer  26  can determine which hand (left/right)  30  is touching the virtual surface.   User-aware virtual surface. Computer  26  can determine an identity of a given user  22  who is touching and interacting with the virtual surface.   Head orientation-aware user interface. When a gaze related pointing gesture is used to control the virtual surface, computer  26  can change the user interface as a function of head movement.   User-position aware user interface. Computer  26  can change the user interface as a function of user position, distance, and/or pose. For example, when the user moves closer to sensing device  24 , computer  26  can present interactive items  36  using a smaller size. Likewise, when the user moves further from sensing device  24 , computer  26  can present interactive items  36  using a larger size. If user  22  shifts horizontally, computer  26  can rearrange the interactive items presented on display  26  to enable better interactivity.       

     While the embodiments described herein have computer  26  processing a series of 3D maps that indicate gestures performed by a limb of user  22  (e.g., finger  30  or hand  31 ), other methods of gesture recognition are considered to be within the spirit and scope of the present invention. For example, user  22  may use input devices such as lasers that include motion sensors, such as a glove controller or a game controller such as Nintendo&#39;s Wii Remote™ (also known as a Wiimote), produced by Nintendo Co., Ltd (KYOTO-SHI, KYT 601-8501, Japan). Additionally or alternatively, computer  26  may receive and process signals indicating a gesture performed by the user from other types of sensing devices such as ultrasonic sensors and/or lasers. 
     Gaze-Based Touchscreen Enhancement 
     As described supra, embodiments of the present invention can be used to implement a virtual touchscreen on computer  26  executing user interface  20 . In some embodiments, the touchpad gestures described hereinabove (as well as the pointing gesture and gaze detection) can be implemented on the virtual touchscreen as well. In operation, the user&#39;s hand “hovers above” the virtual touchscreen until the user performs one of the gestures described herein. 
     For example, the user can perform the Swipe From Edge gesture in order to view hidden menus (also referred to as “Charms Menus”) or the Pinch gesture can be used to “grab” a given interactive item  36  presented on the virtual touchscreen. 
     Physical Surface Enhancement 
     In addition to detecting three-dimensional gestures performed by user  22  in space, computer  26  can be configured to detect user  22  performing two-dimensional gestures on physical surface  47 , thereby transforming the physical surface into a virtual tactile input device such as a virtual keyboard, a virtual mouse, a virtual touchpad or a virtual touchscreen. 
     In some embodiments, the 2D image received from sensing device  24  contains at least physical surface  47 , and the computer  26  can be configured to segment the physical surface into one or more physical regions. In operation, computer  26  can assign a functionality to each of the one or more physical regions, each of the functionalities corresponding to a tactile input device, and upon receiving a sequence of three-dimensional maps containing at least hand  31  positioned on one of the physical regions, the computer can analyze the 3D maps to detect a gesture performed by the user, and simulate, based on the gesture, an input for the tactile input device corresponding to the one of the physical regions. 
       FIGS. 8A-8D , referred to collectively as  FIG. 8 , are schematic pictorial illustrations of a physical regions  142  and  144  on physical surface  47 , in accordance with an embodiment of the present invention. In the example shown in  FIG. 8 , computer  26  configures region  142  as a virtual touchscreen, and configures region  144  as a virtual mouse. 
     In  FIG. 8A , computer  26  uses information provided by 3D sensing device  24  to detect the location of the user&#39;s hand and any fingers  30  that are touching the region  142 . Each point within region  142  (and physical surface  47 ) can be mapped to a corresponding point on display  28 . Although the example in  FIG. 8A  shows a single finger  30  in contact with region  142 , the 3D sensing device and computer  26  are can be configured to detect any number of fingers  30 , thereby enabling user  22  to perform complex, multi-finger control gestures, including scrolling, zoom, pan, and so forth. In some embodiments, computer  26  can configure region  142  as a virtual keyboard able to accept “input” from all fingers  30 . 
       FIG. 8B , shows the use of the region  144  to the right of the keyboard as a mouse region. Here the user&#39;s hand is assumed to hold a mouse (actual or non-existent). In operation, computer  26  can reposition a cursor on display  28  responsively to the movement of the user&#39;s hand in region  144 . Motions (such as tapping motions) of the user&#39;s fingers, as detected by the 3D sensing device, can be interpreted by the computer as clicks on mouse buttons, with each finger  30  assigned to correspond to a different button. 
       FIG. 8C  shows the use of the left hand to select “charms” in the area to the left of the keyboard, and  FIG. 8D  shows the use of the space above the keyboard as an interaction region, for 3D gestures that do not necessarily involve contact with physical surface  47 . 
       FIGS. 9A-9C , referred to collectively as  FIG. 9 , are schematic pictorial illustrations showing how a movement of hand  31  on or near the physical surface can provide “inertial” input to the computer, in accordance with an embodiment of the present invention. Based on input from the 3D sensing device (i.e., the sequence of 3D maps), the computer can determine both the position and the velocity of each of the user&#39;s fingers, as illustrated by lines  150  and points  152  superimposed on physical surface  47  in the left side of  FIGS. 9B and 9C . The computer can incorporate the position and the velocity information into the simulated input for controlling the direction and speed of movement of one or more interactive items  36  presented on the display. 
       FIGS. 10A-10D , referred to collectively as  FIG. 10  are schematic pictorial illustrations of physical surface  47  configured as an input device for a drawing application, in accordance with an embodiment of the present invention. IN operation, computer  26  can form the drawing in an off-screen buffer by transforming the user&#39;s interaction into drawing commands in that buffer, using a defined coordinate transformation between touch coordinates on the physical surface and pixel coordinates in the drawing buffer. 
     In the example shown in  FIG. 10 , lines  160  comprise historical positions of fingers  30  as user  22  “paints” a picture. As shown in  FIG. 10A , computer  26  can configure physical surface  47  as a multi-touch input device configured to accept input from one or more fingers  30 . 
       FIG. 10B  illustrate that the thickness of the drawn line may be controlled by how the user presses finger  30  on physical surface  47 . Computer  26  can execute an algorithm to detect, using the sequence of 3D maps, where the user touches the physical surface and to compute how many pixels of the finger are close to the physical surface (for example, how many pixels are within proximity of 1 cm from the surface). By changing the finger&#39;s proximity to the physical surface, or the angle at which it is held, the user can generate a virtual “pressure” against the physical surface, which computer  26  can incorporate into the simulated input for determining the line&#39;s thickness. In the example shown in  FIG. 10B  line  160 B is thicker than line  160 A due the increased pressure applied by finger  30  when line  160 B was drawn. 
       FIG. 10C  illustrates color awareness that may be incorporated into this drawing application. The user holds an object such as a pen  162  (or a marker with the cap closed) in hand  31 , and the 2D image received from the color video camera in device  24  can detect the color of the pen and computer  26  can incorporate this same color into the simulated input for use when presenting the currently drawn line. 
     Therefore, user  22  can pick up an object (e.g., a colored pen, as described supra), and perform a gesture while holding the object. In some embodiments, the received sequence of 3D maps contain at least the object, since hand  31  may not be within the field of view of sensing device  24 . Alternatively, hand  31  may be within the field of view of sensing device  24 , but the hand may be occluded so that the sequence of 3D maps does not include the hand. In other words, the sequence of 3D maps can indicate a gesture performed by the object held by hand  31 . All the features of the embodiments described above may likewise be implemented, mutatis mutandis, on the basis of sensing movements of a handheld object of this sort, rather than of the hand itself. 
       FIG. 10D  illustrates a possible eraser mode, whereby the user  22  rubs hand  31  over the physical surface, and this gesture causes computer  26  to erase the relevant regions of the drawing in the drawing buffer. One possibility for entering “eraser” mode is to detect that the user has placed his or her palm on the physical surface, rather than individual fingers (which would indicate “drawing” mode). Another option is to allow the user to explicitly enter “eraser” mode using a separate gesture. 
     In order to enrich the set of interactions available to user  22  in this paint application, it is also possible to add menus and other user interface elements as part of the application&#39;s usage. 
       FIG. 11  is a schematic pictorial illustration showing how a “pie menu”  170  may be incorporated into the application. The pie menu illustrated has 4 sectors  172 , each corresponding to a different option. In general, the number of sectors can be varied. User  22  can activate pie menu  170  by pressing finger  30  on physical surface  47 , and keeping the finger still for a short timeout period. This timeout enables the system to distinguish between a drawing interaction (in which case the user will very quickly start to move the finger after placing it on the physical surface), and user interface interaction (in which case the user keeps the finger still for this timeout). User  22  may select a given sector  172  from the pie menu in one of two ways: One way to perform a selection is to drag finger  30  into the desired sector  172  (during which time the sector will be highlighted in yellow), and raise the finger from the physical surface to confirm the selection. Another way is to swipe finger  30  across the desired sector  172  (from the center of the pie menu, out beyond the outer radius of the pie menu). In this latter case, the selection is performed as soon as the finger exits the pie menu&#39;s outer radius, with no need to raise the finger from the table. 
       FIGS. 12A and 12B , referred to collectively as  FIG. 12 , are schematic pictorial illustrations of physical surface  47  illuminated by projector  48 , in accordance with an embodiment of the present invention. To enrich the user&#39;s experience, projector  48  may be added to the configuration, typically above the physical surface, to physically project a drawing on the physical surface, thereby simulating a touchscreen on physical surface  47 .  FIG. 12A  shows a hand  31  drawing on the physical surface, while projector  48  projects this virtual drawing  180  onto the physical surface. This projection gives users a more immersive experience in that they do not need to look at the computer monitor to see the intermediate results of their drawing. Projector  48  can also project pie menu  170  onto physical surface  47 , as seen in  FIG. 12B . 
     In some embodiments, one or more physical objects can be positioned on physical surface  47 , and upon computer  26  receiving, from sensing device  24 , a sequence of three-dimensional maps containing at least the physical surface, the one or more physical objects, and hand  31  positioned in proximity to (or on) physical surface  47 , the computer can analyze the 3D maps to detect a gesture performed by the user, project an animation onto the physical surface in response to the gesture, and incorporate the one or more physical objects into the animation. 
     In operation, 3D maps captured from depth imaging subassembly  52  can be used to identify each physical object&#39;s location and shape, while 2D images captured from sensor  60  can contain additional appearance data for each of the physical objects. The captured 3D maps and 2D images can be used to identify each of the physical objects from a pre-trained set of physical objects. An example described in  FIG. 13  hereinbelow incorporates the physical objects into a game application where animated balls are projected onto physical surface  47  in response to user  22  swiping fingers  30  on the physical surface. In operation, the animated balls can “collide” with the physical objects by detecting the locations of the physical objects, and instantiating virtual collision objects co-located with the physical objects. 
       FIGS. 13A-13D , referred to collectively as  FIG. 13 , are schematic pictorial illustrations showing one or more physical objects  190  positioned on physical surface  47  while user  22  performs a gesture, in accordance with an embodiment of the present invention. In some embodiments, projector  48  can project, onto physical surface  47 , a respective contour image  192  encompassing each of the one or more physical objects, thereby indicating a location of each of the one or more physical objects. 
     In  FIG. 13A , user  22  is resting hand  31  on physical surface  47  and in  FIG. 13B , the user  22  starts performing a gesture by moving the hand toward a given one of physical objects  190 . In response to the user&#39;s gesture, computer  26  can project, onto hand  31  and/or surface  47 , an animation comprising multiple balls  194  with respective trailing paths  196 , the respective trailing paths indicating recent historical positions of the balls. 
     In  FIGS. 13B and 13C , upon user  22  completing the gesture, computer  26  projects the animation comprising balls  194  and their respective trailing paths  196  colliding with and reflecting off of the contour image for the given physical object, thereby incorporating the respective contour image into the animation. While the example in  FIG. 13  shows the animation projected onto physical surface  47 , presenting the animation on display  28  is considered to be within the spirit and scope of the present invention. 
     It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20130324
Publication Date: 20211109
Grant Date: 20211109
Priority Date: 20120326
Inventors: 
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/0425", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0488", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0425", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0488", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0425", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0488", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0488", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0425", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0486", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0425", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0488", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/03", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 49258337