PATENT DOCUMENT

Publication Number: US-8959013-B2
Application Number: US-201113244490-A
Country: US
Kind Code: B2

Title: Virtual keyboard for a non-tactile three dimensional user interface

Abstract:
A method, including presenting, by a computer system executing a non-tactile three dimensional user interface, a virtual keyboard on a display, the virtual keyboard including multiple virtual keys, and capturing a sequence of depth maps over time of a body part of a human subject. On the display, a cursor is presented at positions indicated by the body part in the captured sequence of depth maps, and one of the multiple virtual keys is selected in response to an interruption of a motion of the presented cursor in proximity to the one of the multiple virtual keys.

Claims:
The invention claimed is 
     
       1. A method, comprising:
 presenting, by a computer system executing a non-tactile three dimensional user interface, a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys; 
 capturing a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane; 
 presenting, on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane; and 
 selecting one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor&#39;s trajectory path segment over the virtual keys, wherein the trajectory of the user&#39;s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs. 
 
     
     
       2. The method according to  claim 1 , wherein selecting one of the multiple virtual keys comprises using a language model. 
     
     
       3. The method according to  claim 2 , wherein the language model is selected from a list consisting of a dictionary, a statistical dictionary, an n-gram model, a Markov model and a dynamic Bayesian network. 
     
     
       4. The method according to  claim 2 , wherein the language model applies rules specific to a given language. 
     
     
       5. The method according to  claim 4 , wherein the rules are selected from a list consisting of word rules, short phrase rules, parts of speech rules and grammatical rules. 
     
     
       6. The method according to  claim 2 , wherein the language model utilizes a custom dictionary based on text previously entered by a user interacting with the non-tactile three dimensional user interface. 
     
     
       7. The method according to  claim 2 , wherein the language model utilizes a custom dictionary specific to an application executing on the computer system. 
     
     
       8. The method according to  claim 2 , and comprising the language model selecting none of the virtual keys if none of the virtual keys that are in proximity to the cursor are sufficiently probable. 
     
     
       9. The method according to  claim 1 , and comprising presenting visual feedback of the selected one of the multiple virtual keys on the display. 
     
     
       10. The method according to  claim 1 , wherein each of the multiple virtual keys is selected from a list consisting of alphanumeric characters, symbol characters, punctuation characters and control commands. 
     
     
       11. The method according to  claim 1 , wherein presenting the cursor in proximity to the one of the multiple virtual keys comprises presenting the cursor within a border surrounding the one of the multiple virtual keys. 
     
     
       12. The method according to  claim 1 , and comprising conveying visual feedback upon presenting the cursor in proximity to the one of the multiple virtual keys. 
     
     
       13. The method according to  claim 1 , and comprising conveying visual feedback when a user engages and disengages from the virtual keyboard. 
     
     
       14. An apparatus, comprising:
 a display; and 
 a computer executing a non-tactile three dimensional user interface and configured to present a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys, to capture a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane, to present, on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane, and to select one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor&#39;s trajectory path segment over the virtual keys, wherein the trajectory of the user&#39;s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs. 
 
     
     
       15. The apparatus according to  claim 14 , wherein the computer is configured to select one of the multiple virtual keys by using a language model. 
     
     
       16. The apparatus according to  claim 15 , wherein the computer is configured to select the language model from a list consisting of a dictionary, a statistical dictionary, an n-gram model, a Markov model and a dynamic Bayesian network. 
     
     
       17. The apparatus according to  claim 15 , wherein the language model applies rules specific to a given language. 
     
     
       18. The apparatus according to  claim 17 , wherein the computer is configured to select the rules from a list consisting of word rules, short phrase rules, parts of speech rules and grammatical rules. 
     
     
       19. The apparatus according to  claim 15 , wherein the language model utilizes a custom dictionary based on text previously entered by a user interacting with the non-tactile three dimensional user interface. 
     
     
       20. The apparatus according to  claim 15 , wherein the language model utilizes a custom dictionary specific to an application executing on the computer system. 
     
     
       21. The apparatus according to  claim 15 , wherein the computer is configured to select none of the virtual keys if none of the virtual keys that are in proximity to the cursor are sufficiently probable. 
     
     
       22. The apparatus according to  claim 14 , wherein the computer is configured to present visual feedback of the selected one of the multiple virtual keys on the display. 
     
     
       23. The apparatus according to  claim 14 , wherein the computer is configured to select each of the multiple virtual keys from a list consisting of alphanumeric characters, symbol characters, punctuation characters and control commands. 
     
     
       24. The apparatus according to  claim 14 , wherein the computer is configured to present the cursor in proximity to the one of the multiple virtual keys by presenting the cursor within a border surrounding the one of the multiple virtual keys. 
     
     
       25. The apparatus according to  claim 14 , wherein the computer is configured to conveying visual feedback upon presenting the cursor in proximity to the one of the multiple virtual keys. 
     
     
       26. The apparatus according to  claim 14 , wherein the computer is configured to conveying visual feedback when a user engages and disengages from the virtual keyboard. 
     
     
       27. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile three dimensional user interface, cause the computer:
 to present a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys, to capture a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane, to present on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane, and to select one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor&#39;s trajectory path segment over the virtual keys, wherein the trajectory of the user&#39;s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the benefit of U.S. Provisional Patent Application 61/386,591, filed Sep. 27, 2010, which is incorporated herein by reference. 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to user interfaces for computerized systems, and specifically to user interfaces that are based on three-dimensional sensing. 
     BACKGROUND OF THE INVENTION 
     Many different types of user interface devices and methods are currently available. Common tactile interface devices include a computer keyboard, a mouse and a joystick. Touch screens detect the presence and location of a touch by a finger or other object within the display area. Infrared remote controls are widely used, and “wearable” hardware devices have been developed, as well, for purposes of remote control. 
     Computer interfaces based on three-dimensional (3D) sensing of parts of a user&#39;s body have also been proposed. For example, PCT International Publication WO 03/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A 3D sensor, typically positioned in a room in proximity to the user, provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on the shape of the body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device. 
     Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered. 
     As another example, U.S. Pat. No. 7,348,963, whose disclosure is incorporated herein by reference, describes an interactive video display system, in which a display screen displays a visual image, and a camera captures 3D information regarding an object in an interactive area located in front of the display screen. A computer system directs the display screen to change the visual image in response to changes in the object. 
     SUMMARY OF THE INVENTION 
     There is provided, in accordance with an embodiment of the present invention a method, including presenting, by a computer system executing a non-tactile three dimensional user interface, a virtual keyboard on a display, the virtual keyboard including multiple virtual keys, capturing a sequence of depth maps over time of a body part of a human subject, presenting, on the display, a cursor at positions indicated by the body part in the captured sequence of depth maps, and selecting one of the multiple virtual keys in response to an interruption of a motion of the presented cursor in proximity to the one of the multiple virtual keys. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus, including a display, and a computer executing a non-tactile three dimensional user interface and configured to present a virtual keyboard on a display, the virtual keyboard including multiple virtual keys, to capture a sequence of depth maps over time of a body part of a human subject, to present, on the display, a cursor at positions indicated by the body part in the captured sequence of depth maps, and to select one of the multiple virtual keys in response to an interruption of a motion of the presented cursor in proximity to the one of the multiple virtual keys. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile three dimensional user interface, cause the computer present a virtual keyboard on a display, the virtual keyboard including multiple virtual keys, to capture a sequence of depth maps over time of a body part of a human subject, to present, on the display, a cursor at positions indicated by the body part in the captured sequence of depth maps, and to select one of the multiple virtual keys in response to an interruption of a motion of the presented cursor in proximity to the one of the multiple virtual keys. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosure is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
         FIG. 1  is a schematic pictorial illustration of a system configured to implement a virtual keyboard for a non-tactile three dimensional user interface, in accordance with an embodiment of the present invention; 
         FIG. 2  is a flow diagram that schematically illustrates a method of interacting with the virtual keyboard, in accordance with an embodiment of the present invention; 
         FIG. 3  is a schematic pictorial illustration of the virtual keyboard, in accordance with an embodiment of the present invention; 
         FIG. 4  is a schematic pictorial illustration of character input via the virtual keyboard, in accordance with an embodiment of the present invention; and 
         FIG. 5  is a schematic pictorial illustration of character input via the virtual keyboard, in accordance with an alternative embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Overview 
     Computer keyboards typically comprise an arrangement of physical keys which act electronic switches. Despite the development of alternative input devices such as mice, touchscreens and pen devices, computer keyboards remain a commonly used versatile device for direct input into computers. 
     When using a tactile input device such as a computer keyboard, a user typically presses the physical keys in order to convey alphanumeric text and system commands (e.g., an Enter key or cursor keys) to a computer coupled to the keyboard. However, when interacting with a non-tactile 3D user interface (also referred to herein as a 3D user interface), the user may perform gestures in mid-air, and perform the gestures from different positions within a field of view of a 3D sensor coupled to the 3D user interface. 
     Embodiments of the present invention provide methods and systems for conveying input to a non-tactile 3D user interface via a virtual keyboard presented on a display. The virtual keyboard may comprise multiple virtual keys that represent alphanumeric characters (i.e., “A”-“Z” and “0”-“9”), symbol characters (e.g., “@” and “+”), punctuation characters and control commands (e.g., an Enter key, and cursor and function keys). The virtual keyboard may also comprise a box that is configured to present any text or other characters that were input by the user via the virtual keyboard. In the description and in the claims, the term “virtual keyboard” is to be understood as a graphic representation of a keyboard that does not operate tactilely, and is presented on a display. 
     The 3D user interface can be configured to track the user&#39;s hand (or any other limb), and to position a cursor on a display at positions indicated by the hand&#39;s position. In one embodiment, the user can input a given virtual key by keeping the hand relatively steady as the cursor is presented over the given virtual key for a specified time period. In an additional embodiment, the specified time period may be shortened if a language model indicates that given virtual key is predicted based on previously entered virtual keys. For example, if the user previously entered the letters “bl”, and then positions the cursor over the virtual key “i”, the 3D user interface may accept the letter “i” after the cursor is presented in proximity to the virtual key “i” for a shorter specified time period, (e.g., 0.2 seconds). The 3D user interface can accept the letter “i” after the shorter time period since the language model can identify “bli” as first characters in the words “blink”, “blind”, etc. However, if after entering the letters “bl”, the user positions the cursor over the virtual key “z”, then the 3D user interface may accept “z” after the cursor is positioned over the virtual key “z” for a longer specified time period (e.g., one second). 
     In an alternative embodiment, as the user makes a smooth change of direction of a trajectory of the hand, the 3D user interface can apply a language model to select a given virtual key that the user intended to input. For example, if the letters “bac” were previously input by the user, and the user changes the direction of the hand&#39;s trajectory as the cursor is presented in the vicinity of virtual keys “i”, “o”, “j” and “k”, the language model can select the letter “k”, thereby completing the word “back”. 
     Utilizing a language model can provide a best guess of the user&#39;s intended input that enables the user to enter characters (i.e., via the virtual keyboard) more rapidly. Additionally, the smooth change of direction is natural during fast text input, and may have ergonomic advantages. 
     System Description 
       FIG. 1  is a schematic, pictorial illustration of a non-tactile 3D user interface  20  (also referred to herein as the 3D user interface) for operation by a user  22  of a computer  26 , in accordance with an embodiment of the present invention. The non-tactile 3D user interface is based on a 3D sensing device  24  coupled to the computer, which captures 3D scene information of a scene that includes the body or at least a body part, such as a hand  30 , of the user. Device  24  or a separate camera (not shown in the figures) may also capture video images of the scene. The information captured by device  24  is processed by computer  26 , which drives a display  28  accordingly. 
     Computer  26 , executing 3D user interface  20 , processes data generated by device  24  in order to reconstruct a 3D map of user  22 . The term “3D map” refers to a set of 3D coordinates measured with reference to a generally horizontal X-axis  32 , a generally vertical Y-axis  34  and a depth Z-axis  36 , based on device  24 . The set of 3D coordinates can represent the surface of a given object, in this case the user&#39;s body. In operation, user  22  moves hand  30  in an X-Y plane  38  to interact with a virtual keyboard  40  and a cursor  42 , which are both presented on the display. 
     In one embodiment, device  24  projects a pattern of spots onto the object and captures an image of the projected pattern. Computer  26  then computes the 3D coordinates of points on the surface of the user&#39;s body by triangulation, based on transverse shifts of the spots in the pattern. Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, interface  20  may use other methods of 3D mapping, using single or multiple cameras or other types of sensors, as are known in the art. 
     Computer  26  is configured to capture, via 3D sensing device  24 , a sequence of depth maps over time. Each of the depth maps comprises a representation of a scene as a two-dimensional matrix of pixels, where each pixel corresponds to a respective location in the scene, and has a respective pixel depth value that is indicative of the distance from a certain reference location to the respective scene location. In other words, pixel values in the depth map indicate topographical information, rather than a brightness level and/or a color of any objects in the scene. For example, depth maps can be created by detecting and processing an image of an object onto which a laser speckle pattern is projected, as described in PCT International Publication WO 2007/043036 A1, whose disclosure is incorporated herein by reference. 
     In some embodiments, computer  26  can process the depth maps in order to segment and identify objects in the scene. Specifically, computer  26  can identify objects such as humanoid forms (i.e., 3D shapes whose structure resembles that of a human being) in a given depth map, and use changes in the identified objects (i.e., from scene to scene) as input for controlling computer applications. 
     For example, PCT International Publication WO 2007/132451, whose disclosure is incorporated herein by reference, describes a computer-implemented method where a given depth map is segmented in order to find a contour of a humanoid body. The contour can then be processed in order to identify a torso and one or more limbs of the body. An input can then be generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the captured depth map. 
     In some embodiments, computer  26  can process captured depth maps in order to track a position of hand  30 . By tracking the hand position, 3D user interface  20  can use hand  30  as a pointing device in order to control the computer or other devices such as a television and a set-top box. Additionally or alternatively, 3D user interface  20  may implement “digits input”, where user  22  uses hand  30  as a pointing device to select a digit presented on display  28 . Tracking hand points and digits input are described in further detail in PCT International Publication WO IB2010/051055. 
     Computer  26  typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on non-transitory tangible media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the image processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer  26  is shown in  FIG. 1 , by way of example, as a separate unit from sensing device  24 , some or all of the processing functions of the computer may be performed by suitable dedicated circuitry within the housing of the sensing device or otherwise associated with the sensing device. 
     As another alternative, these processing functions may be carried out by a suitable processor that is integrated with display  28  (in a television set, for example) or with any other suitable sort of computerized device, such as a game console or media player. The sensing functions of device  24  may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output. 
     Virtual Keyboard Interaction 
       FIG. 2  is a flow diagram that schematically illustrates a method of character input using virtual keyboard  40 , in accordance with an embodiment of the present invention, and  FIG. 3  is a schematic pictorial illustration of the virtual keyboard, in accordance with an embodiment of the present invention.  FIG. 4  is a schematic pictorial illustration of user  22  interacting with virtual keyboard  40 , in accordance with the embodiment of the present invention, and  FIG. 5  is a schematic pictorial illustration of the user interacting with the virtual keyboard, in accordance with an alternative embodiment of the present invention. 
     In a presentation step  50  in the flow diagram, 3D user interface  20  presents virtual keyboard  40  on display  28 . In the configuration shown in  FIG. 3 , virtual keyboard  40  comprises virtual keys  70 , which can be presented with a surrounding border  72 . As user  22  selects one of virtual keys  70 , which is alphanumeric, using the embodiments described herein, 3D user interface  20  can present the corresponding alphanumeric character in a text box  74 . 
     Virtual keys  70  may comprise alphanumeric characters, a backspace key, a space bar, symbols and punctuation (e.g., “@” and “?”). Additionally virtual keys  70  may include control keys (e.g., an Enter key and cursor keys) and function keys (e.g., F1, F2, F3, etc.). In some embodiments, 3D user interface  20  can toggle the virtual keys between different modes (e.g., upper and lower case characters) and character sets (e.g., English, Arabic, Chinese and Hebrew). Additionally, the design of virtual keyboard  40  may include “empty” areas  76  between each of the virtual keys, so that user  22  can easily direct cursor  42  to an empty location, thereby reducing the probability of a false positive input. 
     Returning to the flow diagram, in an initialization step  52 , computer  26  sets initial values for a standard time period and an override time period that can be used to by the 3D user interface for deciding when to accept a keystroke on virtual keyboard  40 , as described in further detail hereinbelow. Typically, the standard time period is shorter than the override time period, and are both stored as parameters in 3D user interface  20 . 
     In some embodiments, 3D user interface  20  can automatically adjust the standard and the override time periods in response to a proficiency of user  22 . In other words, 3D user interface  20  can initially set the standard and the override time periods period to first values, and then modify the standard and the override time periods according to a skill level of user  22 . In an embodiment, 3D user interface  20  may measure the user&#39;s skill level by calculating an average time interval that is required for the user to transition from a first given virtual key  70  to a second given virtual key  70  (e.g., from “a” to “t”). For example, for every five alphanumeric inputs (i.e., via the virtual keyboard) computer  26  can calculate the average time period between each of the inputs and classify the user&#39;s skill level to one of several (e.g., three) levels, where each of the levels is associated with different standard and override time period parameters. 
     Additionally or alternatively, 3D user interface  20  can adjust the specified time period using factors such as:
         A number of typing errors user  22  made using virtual keyboard  40 . The number of errors may be recorded according to the number of times a “Backspace” virtual key is selected. For example, if user  22  has a high error rate, 3D user interface  20  can increase the specified time period.   3D user interface  20  can present a given virtual key  70  that is dedicated to changing the specified time period (e.g., a “+” or a “−” key).   A unique specified time period that is associated with each user of 3D user interface  20 .   A profile of hand  30 . For example, a closed hand can be associated with a shorter specified time period, and an open hand can be associated with a longer specified time period.       

     In a first comparison step  54 , 3D user interface  20  waits for user  22  to engage virtual keyboard  40 . If 3D user interface  20  is engaged, then in a capture step  56 , computer  26  captures a sequence of depth maps of a body part such as hand  30 . 
     To engage virtual keyboard  40  (i.e., so that user  22  can input characters via the virtual keyboard), user  22  can move hand  30  so that the 3D user interface presents cursor  42  within the presented virtual keyboard. To disengage from virtual keyboard  40 , user  22  can move hand  30  randomly so that 3D user interface  20  does not present cursor  42  in the vicinity of any given virtual key  70  for more than the specified time period. Alternatively, user  22  can disengage from virtual keyboard  40  by moving hand  30  so that the 3D user interface presents cursor  42  outside virtual keyboard  40 . In some embodiments, 3D user interface  20  can convey visual feedback when user  20  engages and disengages from virtual keyboard  40 . For example, the 3D user interface can change the shading (or color) of virtual keyboard  40  when the user engages and disengages the virtual keyboard. 
     As discussed supra, user  22  can control cursor  42  by moving hand  30  (or any other limb) in X-Y plane  38 , and select a given virtual key  70  by positioning hand  30  so that cursor  42  is positioned in proximity to the given virtual key (i.e., either over the given virtual key or within the border of the given virtual key) for the specified time period. As user  22  moves hand  30  in X-Y plane  38 , 3D user interface  20 , in a presentation step  58  presents cursor  42  at positions indicated by the hand in the captured sequence of depth maps. 
     In embodiments of the present invention, computer  26  selects one of virtual keys  70  upon the captured sequence of depth maps indicating an interruption of a motion of cursor  42  (i.e., in response to an interruption of a motion of hand  30  or any other body part) in proximity to the one of the multiple virtual keys. As described in detail hereinbelow, the interruption of the motion may comprise (a) user  30  maintaining hand  30  relatively stationary for either a standard or an override time period as computer  26  presents cursor  42  in proximity to the one of the multiple virtual keys, or (b) user  30  changes direction of hand  30  in proximity to the one of the multiple virtual keys. 
     In a second comparison step  60 , if the captured sequence of depth maps indicate a specified change in direction of a trajectory of hand  30  (i.e., without the hand pausing for at least the standard time period), then in a model application step  62 , computer  26  executes a language model that attempts to select one of virtual keys  70  that is in proximity to cursor  42  as the cursor changes direction. However, if the captured sequence of depth maps does not indicate a specified change in direction of a trajectory of hand  30 , then the method continues with a third comparison step  64 . 
     In the third comparison step, if user  22  keeps hand  30  relatively steady so that computer  26  presents cursor  42  in proximity to a given virtual key  70  (i.e., within border  72 , or adjacent to the given virtual key) for the standard time period (e.g., 0.1 seconds), then the method continues with step  62 , where the language model checks if a character associated with the given virtual key comprises a character predicted by the language model. However, if user  22  moves hand  30  so that computer  26  does not present cursor  42  in proximity to a given key  70  for the standard time period (i.e., less than the standard time period), then the method continues with step  54 . 
     Typically, the language model executed in step  62  analyzes the virtual keys that are in proximity to cursor  42  as the cursor changes direction, and selects one or more virtual keys  70  that best appends to any text (i.e., a sequence of one or more virtual keys  70 ) previously selected and presented in text box  74 . Note that there may be instances when the language model does not select any virtual key  70 , if none of the virtual keys that are in proximity to cursor  42  as the cursor changes direction are sufficiently probable. 
     In some embodiments, the language model may apply rules specific to a given language (e.g., English), including but not limited to word rules, short phrase rules, parts of speech rules and grammatical rules. In additional embodiments the language model may utilize information on user  22  who is interacting with the virtual keyboard, including but not limited to a custom dictionary based on text previously entered by the user during a related input session (i.e., text input via the virtual keyboard or any other input device). 
     For example, if user  22  previously entered the words “Mozart” and “Beethoven” via virtual  40 , the language model may set a parameter that indicates that the user prefers classical music. Therefore, if the user enters the word “Bavj” via the virtual keyboard, the language model may correct “Bavj” to “Bach” (“v” is adjacent to “c” and “j” is adjacent” and “h” on the virtual keyboard), even though “Bach” was not explicitly added to the dictionary during previous input session to the music selection field. Note that “navy” is another interpretation for similar motion (with a single key shift relative to the intended “Bach”), but will be less favorable by the language model, as previous text associated with classical music was already entered by the user. 
     In further embodiments, the language model may utilize an expected semantic domain. For example, the language model may select a response using a dictionary custom tailored to a question or a field type that 3D user interface  20  presents on display  28 . In other words, the language model may utilize a custom dictionary specific to an application executing on computer  26 . For example, if 3D user interface  20  presents an input field on display  28  for a movie title or a book title, the language model can utilize a dictionary of movie and/or book titles. As an additional example, if computer  26  is executing an adventure-type game, the language model can look for specific commands (e.g., RUN, STOP, FIRE, HIDE, etc.). As a further example, if 3D user interface is presenting a personal information form to be filled out by user  22 , the language model can look for specific values for each field (e.g., “M” or “F” for the user&#39;s sex). 
     Examples of language models that can be implemented by computer  26  include a dictionary and statistical models including but not limited to a statistical dictionary, an n-gram model, a Markov model, and a dynamic Bayesian network. Language models are described in further detail in the book “Foundations of Statistical Natural Language Processing”, by Christopher D. Manning and Hinrich Schütze, MIT Press, 1999, Chapters 6, 7, 9 and 12, which is incorporated herein by reference. 
     In a fourth comparison step  66 , if computer  26  selects one or more relevant (i.e., to the language model) virtual keys  70  that are in proximity to cursor  42  (i.e., as the cursor either changes direction or is in proximity to the given virtual key for the standard time period), then the computer presents the one or more selected virtual keys in text box  74  as visual feedback in a presentation step  67 , and the method continues with step  54 . However, if computer  26  (i.e., since the language model did not select any of the virtual keys) does not select any virtual key  70  in the fourth comparison step, then the method continues with a fifth comparison step  68 . 
     In the fifth comparison step, if user  22  keeps hand  30  relatively steady so that computer  26  presents cursor  42  in proximity to the given virtual key  70  (i.e., within border  72 , or adjacent to the given virtual key) for the override time period (e.g., 0.5 seconds), then the computer selects the given virtual key in a selection step  69 , and the method continues with step  67 . However, if user  22  moves hand  30  so that computer  26  does not present cursor  42  in proximity to the given key  70  for the override time period, then the method continues with step  54 . 
     In some embodiments, 3D user interface  20  can convey visual feedback to user  22  while selecting a given virtual key  70 . For example, the 3D user interface can gradually change the shading (e.g., a gray level) of the given character presented on the given virtual key as user  22  maintains the cursor over the given virtual key. The 3D user interface can accept the given virtual key as an input when the shading reaches a certain level. Additionally or alternatively, 3D user interface  20  can increase the size of the given virtual key after the specified time period, thereby conveying an indication that the given virtual key is being “pressed”. 
     In additional embodiments, user  22  can repeat the input of a given virtual key  70  twice (e.g., “tt”) by keeping hand  30  relatively stationary so that the 3D user interface  20  maintains the cursor&#39;s position over the given virtual key twice as long as the relevant time period (i.e., either the standard or the override time periods). In a similar fashion, user  22  can repeat the input of the given virtual key three or more times. In alternative embodiments, 3D user interface  20  can limit the input of the given virtual key to a single character, regardless of how long cursor  42  is positioned over the given virtual key. To repeat the given virtual key, user  22  moves hand  30  to first position the cursor outside the border of the given virtual key, and then moves the hand a second time to position the cursor back within the border of the given virtual key. 
     In further embodiments, 3D user interface  20  can be configured to accelerate the rate of virtual keyboard  40  input by monitoring both hands  30  of user  22 . The 3D user interface can measure separate distances between each hand  30  and 3D sensing device  24 , and identify the hand closer to the 3D sensing device as active, and identify the other hand as inactive. Therefore, while “pressing” a given virtual key  70  with the active hand, the user can position the inactive hand above the next virtual key  70  that the user intends to “press”. 
     When monitoring both hands of user  22 , 3D user interface  20  may present either one or two cursors  42 . When presenting a single cursor  42 , 3D user interface  20  can toggle the cursor between the active and the inactive hand. In other words, 3D user interface  20  can first position cursor  42  in response to a position of the active hand. Once user  22  has selected a given virtual key  70  with the active hand, 3D user interface  20  can then position cursor  42  in response to a position of the inactive hand. When presenting two cursors  42 , user interface  20  may position a first cursor  42  in response to a position of the active hand, and position a second cursor  42  in response to a position of the inactive hand. 
       FIG. 4  shows cursor  42  traversing path segments  80 ,  82  and  84  in response to user  22  moving hand  30  in X-Y plane  38 , as the user enters the word “no” via virtual keyboard  40 . Initially, computer  26  positions cursor  42  over the virtual keys “q” and “w”. User  22  inputs the letter “n” by moving hand  30  in X-Y plane  38 , so that 3D user interface  20  moves cursor  42  along path segment  80  to a position over the “n” virtual key. As user  22  keeps hand  30  relatively steady over the “n” virtual key for the standard time period, the 3D user interface accepts “n” as an input, and presents “n” in text box  74 . 
     User  22  then inputs the letter “o” by moving hand  30  in X-Y plane  38 , so that 3D user interface  20  moves cursor  42  along path segment  82  to a position over the “o” virtual key. As user  22  keeps hand  30  relatively steady over the “o” virtual key for the standard time period, the 3D user interface accepts “o” as an input and presents “o” in text box  74 . Finally, user  22  presses the Enter virtual key by moving hand  30  in X-Y plane  38 , so that 3D user interface  20  moves cursor  42  along path segment  84  to a position over the Enter virtual key. As user  22  keeps hand  30  relatively steady over the Enter virtual key for the standard time period, the 3D user interface accepts the Enter key as an input. Note that the example described in  FIG. 4  uses the standard time period, since the word “no” is a word that can be predicted by a language model. 
       FIG. 5  shows cursor  42  traversing path segments  90 ,  92  and  94  in response to user  22  moving hand  30  in X-Y plane  38 , as the user enters the word “not” via virtual keyboard  40 . Initially, 3D user interface  20  positions cursor  42  over the virtual keys “q” and “w”. User  22  inputs the letter “n” by moving hand  30  in X-Y plane  38 , so that 3D user interface  20  moves cursor  42  along path segment  90  to a position over the “n” virtual key. As user  22  keeps hand  30  relatively steady over the “n” virtual key for the specified time period, the 3D user interface accepts “n” as an input, and presents “n” in text box  74 . 
     User  22  then inputs the letter “o” by moving hand  30  in X-Y plane  38 , so that 3D user interface  20  moves cursor  42  along path segment  92  to a position over the “o” virtual key. As user  22  keeps hand  30  relatively steady over the “o” virtual key for the specified time period, the 3D user interface accepts “o” as an input and presents “n” in text box  74 . 
     After entering the letters “n” and “o”, user  22  moves hand  30  in X-Y plane  38  so that 3D user interface  20  moves cursor  42  along a path  94  in response to the hand&#39;s movement. Using the captured sequence of depth maps, computer  26  calculates a point  96  along path segment  94 , which indicates a change in direction of a trajectory of the cursor, as the cursor crosses over the virtual keys “t”, “r”, “f” and “c”. Computer  26  applies a language model to resolve the ambiguity of multiple possible letters and selects the most likely virtual key  70  that user  22  intended to “press”. In the example shown in  FIG. 5 , the language model evaluates the probability of the strings “not”, “nor”, “nof” and “noc”, and identifies “not” as the most probable text string. 
     It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20110925
Publication Date: 20150217
Grant Date: 20150217
Priority Date: 20100927
Inventors: GALOR MICHA
OR OFIR
LITVAK SHAI
SALI EREZ
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/0233", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F40/274", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/011", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/011", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F40/274", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0233", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F17/276", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0233", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/011", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 45871519