PATENT DOCUMENT

Publication Number: US-9218063-B2
Application Number: US-201213592352-A
Country: US
Kind Code: B2

Title: Sessionless pointing user interface

Abstract:
A method, including receiving, by a computer, a sequence of three-dimensional maps containing at least a hand of a user of the computer, and identifying, in the maps, a device coupled to the computer. The maps are analyzed to detect a gesture performed by the user toward the device, and the device is actuated responsively to the gesture.

Claims:
We claim:  
     
       1. A method, comprising:
 identifying, by a computer coupled to a three-dimensional (3D) sensing device and a display, different, respective locations of multiple controllable devices other than the display; 
 receiving, by the computer from the 3D sensing device, a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer; 
 detecting in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices; 
 defining a region in space with an apex at the head of the user and a base encompassing the given device; 
 defining an interaction zone within the region; 
 defining an angle threshold; 
 defining a minimum time period; 
 analyzing the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, 
 wherein the gesture comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the device and pausing the hand for the minimum time period; and 
 actuating the given device responsively to the gesture. 
 
     
     
       2. The method according to  claim 1 , wherein the gesture is selected from a list comprising a pointing gesture, a grab gesture and a release gesture. 
     
     
       3. The method according to  claim 1 , and comprising receiving a vocal command from the user, and actuating the device in response to the gesture and the vocal command. 
     
     
       4. The method according to  claim 1 , and comprising communicating between the computer and a further device upon detecting, in the maps, a subsequent gesture directed toward the further device. 
     
     
       5. The method according to  claim 1 , wherein identifying the respective locations comprises performing, by the computer, an initialization step comprising:
 identifying, by the computer, the controllable devices that are in proximity to the 3D sensing device; and 
 directing the user to point to each of the identified controllable devices. 
 
     
     
       6. An apparatus, comprising:
 a three-dimensional sensing device; 
 a display; and 
 a computer configured to identify different, respective locations of multiple controllable devices other than the display, to receive from the three-dimensional sensing device a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer, to detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices, to define a region in space with an apex at the head of the user and a base encompassing the given device, to define an interaction zone within the region, to analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and to actuate the given device responsively to the gesture, 
 wherein the computer is configured to define an angle threshold and a minimum time period, and wherein the gesture detected by the computer comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the given device, and pausing the hand for the minimum time period. 
 
     
     
       7. The apparatus according to  claim 6 , wherein the computer is configured to select the gesture from a list comprising a pointing gesture, a grab gesture and a release gesture. 
     
     
       8. The apparatus according to  claim 6 , wherein the computer is configured to receive a vocal command from the user, and to actuate the given device in response to the gesture and the vocal command. 
     
     
       9. The apparatus according to  claim 6 , wherein the computer is configured to communicate with a further device among the multiple controllable devices upon detecting, in the maps, a subsequent gesture directed toward the further device. 
     
     
       10. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface and coupled to a display, cause the computer to:
 identify different, respective locations of multiple controllable devices other than the display, 
 receive from a three-dimensional sensing device that is coupled to the computer a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer, 
 detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices, 
 define a region in space with an apex at the head of the user and a base encompassing the given device, 
 define an interaction zone within the region, 
 analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and 
 actuate the given device responsively to the gesture, 
 wherein the instructions cause the computer to define an angle threshold and a minimum time period, and wherein the gesture detected by the computer comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the given device, and pausing the hand for the minimum time period.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Patent Application 61/526,692 filed on Aug. 24, 2011, which is incorporated herein by reference. 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to user interfaces for computerized systems, and specifically to user interfaces that are based on three-dimensional sensing. 
     BACKGROUND OF THE INVENTION 
     Many different types of user interface devices and methods are currently available. Common tactile interface devices include the computer keyboard, mouse and joystick. Touch screens detect the presence and location of a touch by a finger or other object within the display area. Infrared remote controls are widely used, and “wearable” hardware devices have been developed, as well, for purposes of remote control. 
     Computer interfaces based on three-dimensional (3D) sensing of parts of the user&#39;s body have also been proposed. For example, PCT International Publication WO 03/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A 3D sensor provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on a shape of a body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device. 
     As another example, U.S. Pat. No. 7,348,963, whose disclosure is incorporated herein by reference, describes an interactive video display system, in which a display screen displays a visual image, and a camera captures 3D information regarding an object in an interactive area located in front of the display screen. A computer system directs the display screen to change the visual image in response to changes in the object. 
     Three-dimensional human interface systems may identify not only the user&#39;s hands, but also other parts of the body, including the head, torso and limbs. For example, U.S. Patent Application Publication 2010/0034457, whose disclosure is incorporated herein by reference, describes a method for modeling humanoid forms from depth maps. The depth map is segmented so as to find a contour of the body. The contour is processed in order to identify a torso and one or more limbs of the subject. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map. 
     Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered. 
     The description above is presented as a general overview of related art in this field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application. 
     SUMMARY OF THE INVENTION 
     There is provided, in accordance with an embodiment of the present invention a method including receiving, by a computer, a sequence of three-dimensional maps containing at least a hand of a user of the computer, identifying, in the maps, a device coupled to the computer, analyzing the maps to detect a gesture performed by the user toward the device, and actuating the device responsively to the gesture. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus including a three-dimensional sensing device, and a computer configured to receive from the three-dimensional sensing device a sequence of three-dimensional maps containing at least a hand of a user of the computer, to identify, in the maps, an entity coupled to the computer, to analyze the maps to detect a gesture performed by the user toward the entity, and to actuate the entity responsively to the gesture. 
     There is additionally provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to receive a sequence of three-dimensional maps containing at least a hand of a user of the computer, to identify, in the maps, a device coupled to the computer, to analyze the maps to detect a gesture performed by the user toward the device, and to actuate the device responsively to the gesture. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosure is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
         FIG. 1  is a schematic, pictorial illustration of a computer system executing a sessionless pointing user interface (SPUI), in accordance with an embodiment of the present invention; 
         FIG. 2  is a flow diagram that schematically illustrates a method of using a gesture to actuate a device coupled to the computer, in accordance with an embodiment of the present invention; 
         FIG. 3  is a schematic pictorial illustration of a first scene comprising a user interacting with the SPUI, in accordance with an embodiment of the present invention; and 
         FIG. 4  is a schematic pictorial illustration of a second scene comprising multiple users interacting with the SPUI, in accordance with an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     When using a tactile input device such as a mouse, the user typically manipulates the physical device in a two-dimensional plane comprising a horizontal X-axis and a vertical Y-axis. However, when interacting with a non-tactile three-dimensional (3D) user interface (also referred to herein as a 3D user interface), the user may perform gestures in mid-air, and perform the gestures from different positions within a field of view of a 3D sensor coupled to the interface. 
     U.S. patent application Ser. No. 13/314,210, filed on Dec. 8, 2011, whose disclosure is incorporated herein by reference, describes focus gestures that enable a user to activate a 3D user interface and unlock gestures that enable the user to engage a locked 3D user interface. However, in instances where a user&#39;s interaction with a device controlled by a 3D user interface is very brief, requiring the user to initiate a session (with the device) by performing a focus gesture followed by an unlock gesture can be cumbersome. 
     Embodiments of the present invention provide methods and systems for a user to actuate a device (e.g., a television or a lighting fixture) via gestures described hereinbelow. In some embodiments the device is driven by a computer executing a sessionless pointing user interface (SPUI) that enables sessionless control of the device, i.e., wherein no session is initiated by the user. For example, the user can turn on a light by pointing at the light. 
     Additionally or alternatively, the computer can be configured to reduce false positives (i.e., the user may point at the light inadvertently during a conversation) by actuating the device in response to a gesture and a vocal command. For example, the computer can turn on the light in response to the user saying “Light” while pointing at the light. 
     DETAILED DESCRIPTION OF EMBODIMENTS 
       FIG. 1  is a schematic, pictorial illustration of a sessionless pointing user interface (SPUI)  20  for operation by a user  22  of a computer  26 , in accordance with an embodiment of the present invention. The SPUI is based on a 3D sensing device  24  coupled to the computer, which captures 3D scene information of a scene that includes the body (or at least a body part, such as hands  29 ) of the user. Device  24  or a separate camera (not shown in the figures) may also capture video images of the scene. The information captured by device  24  is processed by computer  26 , which drives a display  28  accordingly. 
     Computer  26 , executing SPUI  20 , processes data generated by device  24  in order to reconstruct a 3D map of user  22 . The term “3D map” refers to a set of 3D coordinates measured with reference to a fixed set of axes based on device  24 . The 3D coordinates represent the surface of a given object, in this case the user&#39;s body. In one embodiment, device  24  projects a pattern of spots onto the object and captures an image of the projected pattern. Computer  26  then computes the 3D coordinates of points on the surface of the user&#39;s body by triangulation, based on transverse shifts of the spots in the pattern. Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, interface  20  may use other methods of 3D mapping, using single or multiple cameras or other types of sensors, as are known in the art. 
     In some configurations, sensing device  24  may include a microphone (not shown) configured to convey audio signals generated in response to speech or sound from user  22  to computer  26 . Computer  26  can be configured to process the audio signals, thereby enabling the computer to respond to vocal commands from user  22 , in addition to physical gestures performed by the user. 
     Computer  26  typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on tangible media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the image processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer  26  is shown in  FIG. 1 , by way of example, as a separate unit from sensing device  24 , some or all of the processing functions of the computer may be performed by suitable dedicated circuitry within the housing of the sensing device or otherwise associated with the sensing device. 
     As another alternative, these processing functions may be carried out by a suitable processor that is integrated with display  28  (in a television set, for example) or with any other suitable sort of computerized device, such as a game console or media player. The sensing functions of device  24  may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output. 
     Various techniques may be used to reconstruct the 3D map of the body of user  22 . In one embodiment, computer  26  extracts 3D connected components corresponding to the parts of the body from the depth data generated by device  24 . Techniques that may be used for this purpose are described, for example, in U.S. patent application Ser. No. 12/854,187, filed Aug. 11, 2010, whose disclosure is incorporated herein by reference. The computer analyzes these extracted components in order to reconstruct a “skeleton” of the user&#39;s body, as described in the above-mentioned U.S. Patent Application Publication 2010/0034457, or in U.S. patent application Ser. No. 12/854,188, filed Aug. 11, 2010, whose disclosure is also incorporated herein by reference. In alternative embodiments, other techniques may be used to identify certain parts of the user&#39;s body, and there is no need for the entire body to be visible to device  24  or for the skeleton to be reconstructed, in whole or even in part. 
     Using the reconstructed skeleton, computer  26  can assume a position of a body part such as a tip of finger  30 , even though the body part (e.g., the fingertip) may not be detected by the depth map due to issues such as minimal object size and reduced resolution at greater distances from device  24 . In some embodiments, computer  26  can auto-complete a body part based on an expected shape of the human part from an earlier detection of the body part, or from tracking the body part along several (previously) received depth maps. 
     In some embodiments, the information generated by computer  26  as a result of this skeleton reconstruction includes the location and direction of the user&#39;s head, as well as of the arms, torso, and possibly legs, hands and other features, as well. Changes in these features from frame to frame (i.e. depth maps) or in postures of the user can provide an indication of gestures and other motions made by the user. User posture, gestures and other motions may provide a control input for user interaction with interface  20 . These body motions may be combined with other interaction modalities that are sensed by device  24 , including user eye movements, as described above, as well as voice commands and other sounds. Interface  20  thus enables user  22  to perform various remote control functions and to interact with applications, interfaces, video programs, images, games and other multimedia content appearing on display  28 . 
     POINTING USER INTERFACE 
       FIG. 2  is a flow diagram that schematically illustrates a method of actuating display  28 , and  FIG. 3  is a schematic pictorial illustration of a first scene  40  comprising user  22  interacting with computer  26 , in accordance with an embodiment of the present invention. In the configuration shown in  FIG. 3 , 3D sensing device  24  and computer  26  are both integrated into display  28 . 
     In a receive step  30 , computer  26  receives, from sensing device  24 , a sequence of 3D maps containing at least hand  29 , and in an identification step  32 , the computer identifies in the maps the hand and at least one device or entity coupled to, and driven by (i.e., controlled by), the computer. In the configuration shown in  FIG. 3 , the device comprises display  28  that incorporates sensing device  24 . Therefore, computer  26  can interpret a gesture toward sensing device  24  as a gesture toward display  28 . 
     In other configurations as described hereinbelow, computer  26  may control multiple devices in proximity to user  22 . Examples of devices that can be driven by computer  26  include, but are not limited to lamp fixtures, ventilation (i.e., heating/cooling) units, ceiling fans and electronic entertainment systems. If computer  26  is controlling multiple devices in proximity to user  22 , then the user can identify the location of the devices during an initialization step. During the initialization step, computer  26  can identify devices in proximity to the sensing device (e.g., via Bluetooth or another communication protocol known in the art), present interactive items on display  28  corresponding to each of the identified devices, and direct the user to first point at a given one of the interactive items and then point to the corresponding device. 
     Upon identifying hand  29  and display  28 , computer  26  can initialize sessionless pointing user interface (SPUI)  20  by defining a pyramid shaped region  42  within a field of view of 3D sensor  24 . In the example shown in  FIG. 3 , pyramid shaped region  42  is rectangular and comprises an apex (i.e., the narrow tip)  56  that meets user  22 , and a base (i.e., the wide end)  58  that encompasses display  28 . In some embodiments, computer  26  positions apex  56  at a head  44  or an eyeball  46  of user  22 . 
     In a detect step  34 , computer  26  detects, in the sequence of 3D maps, a gesture directed toward display  28 . Examples of gestures that user  22  can direct toward display  28  include, but are not limited to a grab gesture and pointing gestures. The grab gesture is described in U.S. patent application Ser. No. 13/423,314, filed on Mar. 19, 2012, whose disclosure is incorporated herein by reference, and comprises user  22  closing at least some fingers of hand  29 . 
     Pointing gestures are described in PCT International Application PCT/IB2012/050577, filed Feb. 9, 2012, whose disclosure is incorporated herein by reference, and include a point-select, a point-touch, and a point-hold gesture. For example, to perform the point-touch gesture, user  22  points hand  29  toward display  28 , stops or slows down the hand, and then pulls the hand back toward the user. 
     To reduce instances of false positives, computer  26  may be configured to define conditions for identifying a pointing gesture when analyzing the 3D maps. Examples of conditions include:
         Defining an interaction region  48  within pyramid shaped region  42 , and identifying a pointing gesture upon the 3D maps indicating user  22  positioning hand  29  within region  48  and moving the hand toward the display.   Defining an angle threshold (e.g., 90 degrees) for elbow  50 , and identifying a pointing gesture upon the 3D maps indicating user  22  extending hand  29  toward the display and extending elbow  50  at an angle  54  greater than or equal to the angle threshold.   Defining a minimum time period (e.g., 200 milliseconds), and identifying a pointing gesture upon the 3D maps indicating user  22  pausing hand  29  for the minimum time period after extending hand  29  toward the display. Requiring a minimum time period enables computer  26  to mimic the natural behavior of an individual pointing at an object.       

     In some embodiments, computer  26  may be configured to detect gestures performed by user  22  with fingers of hand  29 , such as an index finger  52 . Gestures performed by user  22  with hand  29  are referred to herein as hand gestures, and gestures performed with the fingers of the hand are referred to herein as finger gestures. Performing finger gestures can help reduce fatigue while interacting with computer  26 . For example, rather than keeping hand  29  raised to perform a hand gesture, user  22  can keep the hand resting on a lap (i.e., while sitting), and perform gestures with the fingers of hand  29 . 
     In some embodiments, computer  26  can be configured at either a higher resolution (also referred to herein as finger resolution) to detect the position (i.e., location and orientation) of individual fingers of hand  29 , or a lower resolution (also referred to herein as hand resolution) to detect the position of hand  29 . Additionally or alternatively, computer  26  may be configured to detect hand gestures and finger gestures at shorter distances between user  22  and 3D sensing device  24 , and to detect hand gestures at longer distances between the user and the 3D sensing device. When configured for finger resolution, computer can respond to finger gestures such as a finger pointing gesture or a grab gesture. 
     Returning to the flow diagram, in an actuation step  36 , computer  26  actuates the device to which the user is pointing, and the method ends. In the configuration shown in  FIG. 3 , computer  26  can actuate (i.e., turn on) display  28  in response to a pointing gesture performed by user  22 . In alternative embodiments, computer  26  can present multiple interactive items (not shown) on display  28 , identify a given interactive item to which the user is pointing, and actuate the identified interactive item. For example, computer  26  may present advertisements for multiple restaurants on display  28 , and present detailed information on a given restaurant upon detecting a gesture directed toward the advertisement of the given restaurant. 
     In some embodiments, selecting a given interactive item comprises executing a software application associated with the given interactive item. In further embodiments the given interactive item is associated with a media item (e.g., a music track or a movie), and selecting a given interactive item comprises playing a media file associated with the given interactive item. 
     In operation, computer  26  can calculate a pointing geometry for any number of users  22  interacting with the computer. In embodiments where computer  26  is configured for hand resolution, the computer typically responds to a given user  22  performing a pointing gesture within pyramid shaped region  42 . Therefore, if there is more than one user interacting with computer  26 , then the computer can define a separate pyramid shaped region  42  for each of the users. In alternative embodiments wherein computer is configured for finger resolution, the computer can be configured to respond to pointing gestures performed by user  22  both inside and outside pyramid shaped region  42 . 
       FIG. 4  is a schematic pictorial illustration of a second scene  60  (also referred to herein as room  60 ) comprising multiple users  22  interacting with computer  26 , in accordance with an embodiment of the present invention. In the description of  FIG. 4  herein, users  22  may be differentiated by appending a letter to the identifying numeral, so that the users comprise users  22 A and  22 B. As explained in detail hereinbelow, computer  26  can be configured to control display  28 , lighting fixtures  64  and  66 , and air vents  68  and  70 . 
     Using gestures described herein, a given user  22  can individually control multiple devices configured to be driven by computer  26  (i.e., in addition to display  28 ). In one example, computer  26  can turn on (or off) a given one of the lighting fixtures in response to detecting, in the 3D maps, a pointing gesture directed toward the given lighting fixture. 
     In another example, computer  26  can open and close a given one of the air vents in response to detecting, in the 3D maps, a gesture directed toward the given air vent. To open the given air vent, user  22  can perform a release gesture toward the given air vent. The release gesture, described in U.S. patent application Ser. No. 13/423,314, filed on Mar. 19, 2012, whose disclosure is incorporated herein by reference, comprises user relaxing hand  29  so as to open the hand from a closed or folded state. Similarly, to close the given air vent the user can perform a grab gesture (as described supra) toward the given air vent. 
     While the configuration of scene  60  includes a single 3D sensing device  24 , other configurations may include multiple 3D sensors, and are considered to be within the spirit and scope of the present invention. In some embodiments additional sensing devices  24  may be positioned in room  60 , thereby enabling computer  26  to monitor the entire room (i.e., no blind spots) for gestures performed by the users. In alternative embodiments, the devices (e.g., the lighting fixtures and the air vents) may contain 3D sensors that are configured to communicate with computer  26 . 
     In some embodiments, in addition to controlling multiple devices, a given user  22  can perform a pointing gesture to select one or more additional users in the user&#39;s vicinity. For example, user  22 A can perform a first pointing gesture to select an icon on display  28  to start a multiplayer game, and then perform a subsequent pointing gesture directed toward user  22 B user, thereby indicating user  22 B as an additional participant in the game. 
     In additional embodiments, computer  26  can be configured to have the actuated device communicate with a further device in response to a subsequent gesture directed toward the further device. In a first example, user  22 A wants to transfer a digital photograph  72  from a smartphone  62  to a photo album application  74  that computer  26  is presenting on display  28 . Computer  26  can update photo album  74  upon detecting, in the 3D maps, user  22 A performing an initial pointing gesture directed toward smartphone  62  and a subsequent pointing gesture directed toward display  28 . 
     In a second example, computer  26  presents a web page on display  28 , and user  22 B wants to copy the uniform resource locator (URL) of the web page to a web browser application executing on a laptop computer  74 . Computer  26  can copy the URL to the web browser application upon detecting, in the 3D maps, user  22 B performing an initial pointing gesture directed toward display  28  and a subsequent pointing gesture directed toward laptop computer  74 . 
     In a third example, user  22 A wants to adjust the temperature in room  60 . Computer  26  can present an air conditioning control menu (e.g., with icons for controlling settings such as temperature and fan speed) on display  28  in response to detecting, in the 3D maps, user  22 A performing an initial pointing gesture directed toward one of the air vents and a subsequent pointing gesture directed toward display  28 . 
     While the examples hereinabove describe using pointing gestures for multiple device interaction, other gestures are considered to be within the spirit and scope of the present invention. For example, computer  26  can copy the URL of the web page presented on display  28  to laptop computer  74  in response to detecting, in the 3D maps, user  22 B performing a grab gesture directed toward display  28  and subsequent performing a release gesture directed toward laptop computer  74 . 
     In scene  60 , light fixture  64  is positioned in proximity to air vent  68 , and light fixture  66  is positioned in proximity to air vent  70 . In operation, if a given user  22  does not accurately “aim” a gesture toward an intended device, the computer may actuate a different device responsively to the gesture. For example, if user  22 B intends to turn on light  66  but performs a pointing gesture toward air vent  70 , computer  26  may turn on air vent  70  in response to the pointing gesture. 
     In some embodiments, computer  26  be configured to actuate a given device in response to a combination of a vocal (i.e., audio) command and a gesture. For example, to turn on lighting fixture  66 , the user can say “Light” while pointing in proximity to lighting fixture  66 . Combining vocal command processing with gesture recognition can enhance the accuracy of computer  26  controlling multiple devices in proximity to each other. 
     There may be instances user  22  performs a pointing gesture without intending to actuate a device controlled by computer  26 . For example, user  22 A can point toward user  22 B during a conversation. To reduce the instances of false positives (i.e., unintentional pointing gestures), computer  26  can be configured to determine the intent of user  22  by detecting (and tracking), in the 3D maps, a gaze of the user, and actuating a given device in response to the gaze and a gesture directed toward the given device. In some embodiments tracking the gaze of the user comprises detecting an orientation of head  44  and/or eyeball  46 . 
     For example, computer  26  can be configured to respond to a pointing gesture directed towards display  28  only if the 3D maps indicate that the user&#39;s gaze is directed toward the display. Identifying a gaze direction of user  22  is described in PCT International Application PCT/IB2012/050577, filed Feb. 9, 2012, whose disclosure is incorporated herein by reference. 
     It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features, including the transformations and the manipulations, described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20120823
Publication Date: 20151222
Grant Date: 20151222
Priority Date: 20110824
Inventors: GALOR MICHA
POKRASS JONATHAN
HOFFNUNG AMIR
OR OFIR
Assignee: APPLE INC
CPC Classifications: [{"code": "G06T2207/30196", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/013", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/167", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06T7/75", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T7/75", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V40/28", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 47745504