PATENT DOCUMENT

Publication Number: US-9122311-B2
Application Number: US-201213592369-A
Country: US
Kind Code: B2

Title: Visual feedback for tactile and non-tactile user interfaces

Abstract:
A method, including presenting, by a computer, a scrollable list of interactive items on a display driven by the computer, and receiving an input from a user of the computer. The list is scrolled at a speed indicated by the input, and the list is zoomed in response to the speed of the scrolling.

Claims:
I claim: 
     
       1. A method, comprising:
 presenting, by a computer, multiple interactive items on a display driven by the computer; 
 receiving, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer; 
 defining an interaction surface in space in proximity to the user; 
 detecting in the maps a transverse motion of the hand in space, and positioning, on the display, a cursor in response to the transverse motion; 
 detecting, in the maps, a select gesture performed by the hand and directed toward one of the multiple interactive items while the hand engages the interaction surface; and 
 distorting a region on the display containing the one of the multiple interactive items in response to the select gesture and engagement of the interaction surface by the hand, by presenting the region surrounding the cursor as a recessed vertex. 
 
     
     
       2. The method according to  claim 1 , wherein the select gesture comprises the user moving the hand toward the one of the interactive items. 
     
     
       3. The method according to  claim 1 , and comprising detecting, in the maps a transverse motion of the hand, and positioning, on the display, a cursor in response to a transverse motion of the hand. 
     
     
       4. The method according to  claim 1 , and comprising presenting, on the display, a reflection of the cursor at an offset directly proportional to a distance between the hand and the interaction surface. 
     
     
       5. The method according to  claim 4 , and comprising presenting the reflection at a size inversely proportional to the distance between the hand and the interaction surface. 
     
     
       6. An apparatus, comprising:
 a depth sensor; and 
 a computer configured to present multiple interactive items on a display driven by the computer, to receive, from the depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to define an interaction surface in space in proximity to the user, to detect in the maps a transverse motion of the hand in space, and to position, on the display, a cursor in response to the transverse motion, to detect, in the maps, a select gesture performed by the hand and directed toward one of the multiple interactive items while the hand engages the interaction surface, and to distort a region on the display containing the one of the multiple interactive items in response to the select gesture and engagement of the interaction surface by the hand, by presenting the region surrounding the cursor as a recessed vertex. 
 
     
     
       7. The apparatus according to  claim 6 , wherein the select gesture comprises the user moving the hand toward the one of the interactive items. 
     
     
       8. The apparatus according to  claim 6 , wherein the computer is configured to detect, in the maps a transverse motion of the hand, and to position, on the display, a cursor in response to a transverse motion of the hand. 
     
     
       9. The apparatus according to  claim 6 , wherein the computer is configured to present, on the display, a reflection of the cursor at an offset directly proportional to a distance between the hand and the interaction surface. 
     
     
       10. The apparatus according to  claim 9 , wherein the computer is configured to present the reflection at a size inversely proportional to the distance between the hand and the interaction surface. 
     
     
       11. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface, cause the computer to present a scrollable list of items on a display driven by the computer, to receive, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to define an interaction surface in space in proximity to the user, to detect in the maps a transverse motion of the hand in space, and to position, on the display, a cursor in response to the transverse motion, to detect, in the maps, a select gesture performed by the hand and directed toward one of the multiple interactive items while the hand engages the interaction surface, and to distort a region on the display containing the one of the multiple interactive items in response to the select gesture and engagement of the interaction surface by the hand, by presenting the region surrounding the cursor as a recessed vertex.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Patent Application 61/526,696 filed on Aug. 24, 2011, which is incorporated herein by reference. 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to user interfaces for computerized systems, and specifically to user interfaces that are based on three-dimensional sensing. 
     BACKGROUND OF THE INVENTION 
     Many different types of user interface devices and methods are currently available. Common tactile interface devices include the computer keyboard, mouse and joystick. Touch screens detect the presence and location of a touch by a finger or other object within the display area. Infrared remote controls are widely used, and “wearable” hardware devices have been developed, as well, for purposes of remote control. 
     Computer interfaces based on three-dimensional (3D) sensing of parts of the user&#39;s body have also been proposed. For example, PCT International Publication WO 03/071410, whose disclosure is incorporated herein by reference, describes a gesture recognition system using depth-perceptive sensors. A 3D sensor provides position information, which is used to identify gestures created by a body part of interest. The gestures are recognized based on a shape of a body part and its position and orientation over an interval. The gesture is classified for determining an input into a related electronic device. 
     As another example, U.S. Pat. No. 7,348,963, whose disclosure is incorporated herein by reference, describes an interactive video display system, in which a display screen displays a visual image, and a camera captures 3D information regarding an object in an interactive area located in front of the display screen. A computer system directs the display screen to change the visual image in response to changes in the object. 
     Three-dimensional human interface systems may identify not only the user&#39;s hands, but also other parts of the body, including the head, torso and limbs. For example, U.S. Patent Application Publication 2010/0034457, whose disclosure is incorporated herein by reference, describes a method for modeling humanoid forms from depth maps. The depth map is segmented so as to find a contour of the body. The contour is processed in order to identify a torso and one or more limbs of the subject. An input is generated to control an application program running on a computer by analyzing a disposition of at least one of the identified limbs in the depth map. 
     Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered. 
     The description above is presented as a general overview of related art in this field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application. 
     SUMMARY OF THE INVENTION 
     There is provided, in accordance with an embodiment of the present invention a method including presenting, by a computer, a scrollable list of interactive items on a display driven by the computer, receiving an input from a user of the computer, scrolling the list at a speed indicated by the input, and zooming the list in response to the speed of the scrolling. 
     There is also provided, in accordance with an embodiment of the present invention a method including presenting, by a computer, a scrollable list of items on a display driven by the computer, receiving, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, defining a swipe region in space in proximity to the user, detecting, in the sequence of maps, a swipe gesture performed by the hand, in which the hand engages the swipe region at a first location in space and then disengages from the swipe region at a second location in space, initiating a scrolling of the list at a speed set in response to the engagement of the swipe region by the hand performing the swipe gesture, and after the disengagement of the swipe region by the hand performing the swipe gesture, gradually decelerating the speed of the scrolling of the list. 
     There is additionally provided, in accordance with an embodiment of the present invention a method including presenting, by a computer, multiple interactive items on a display driven by the computer, receiving, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, detecting, in the maps, a select gesture hand performed by the hand and directed toward one of the multiple interactive items, and distorting a region on the display containing the one of the multiple interactive items in response to the select gesture. 
     There is further provided, in accordance with an embodiment of the present invention an apparatus including a display, and a computer configured to present a scrollable list of interactive items on the display, to receive an input from a user of the computer, to scroll the list at a speed indicated by the input, and to zoom the list in response to the speed of the scrolling. 
     There is also provided, in accordance with an embodiment of the present invention an apparatus including a depth sensor, and a computer configured to present a scrollable list of items on a display driven by the computer, to receive, from the depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to define a swipe region in space in proximity to the user, to detect, in the sequence of maps, a swipe gesture performed by the hand, in which the hand engages the swipe region at a first location in space and then disengages from the swipe region at a second location in space, to initiate a scrolling of the list at a speed set in response to the engagement of the swipe region by the hand performing the swipe gesture, and after the disengagement of the swipe region by the hand performing the swipe gesture, to gradually decelerate the speed of the scrolling of the list. 
     There is additionally provided, in accordance with an embodiment of the present invention an apparatus including a depth sensor, and a computer configured to present multiple interactive items on a display driven by the computer, to receive, from the depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to detect, in the maps, a select gesture hand performed by the hand and directed toward one of the multiple interactive items, and to distort a region on the display containing the one of the multiple interactive items in response to the select gesture. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to present a scrollable list of interactive items on a display driven by the computer, to receive an input from a user of the computer, to scroll the list at a speed indicated by the input, and to zoom the list in response to the speed of the scrolling. 
     There is also provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to present a scrollable list of items on a display driven by the computer, to receive, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to define a swipe region in space in proximity to the user, to detect, in the sequence of maps, a swipe gesture performed by the hand, in which the hand engages the swipe region at a first location in space and then disengages from the swipe region at a second location in space, to initiate a scrolling of the list at a speed set in response to the engagement of the swipe region by the hand performing the swipe gesture, and after the disengagement of the swipe region by the hand performing the swipe gesture, to gradually decelerate the speed of the scrolling of the list. 
     There is further provided, in accordance with an embodiment of the present invention a computer software product including a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer to present a scrollable list of items on a display driven by the computer, to receive, from a depth sensor, a sequence of three-dimensional (3D) maps containing at least a hand of a user of the computer, to detect, in the maps, a select gesture hand performed by the hand and directed toward one of the multiple interactive items, and to distort a region on a display containing the one of the multiple interactive items in response to the select gesture. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosure is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
         FIG. 1  is a schematic, pictorial illustration of a computer system implementing a gesture-based three dimensional (3D) user interface with visual feedback, in accordance with an embodiment of the present invention; 
         FIG. 2  is a schematic, pictorial illustration showing a user interacting with visualization and interaction regions associated with the 3D user interface, in accordance with an embodiment of the present invention; 
         FIGS. 3A ,  3 B and  3 C are schematic pictorial illustrations of the computer presenting a variable sized scrollable list on a display, in accordance with an embodiment of the present invention; 
         FIG. 4  is a flow diagram that schematically illustrates a method of conveying zoom-based feedback for the scrollable list, in accordance with an embodiment of the present invention; 
         FIG. 5  is a schematic pictorial illustration of the computer presenting scrolling controls for the variable sized scrollable list, in accordance with an embodiment of the present invention; 
         FIG. 6  is a flow diagram that schematically illustrates a method of manipulating the scrollable list via a swipe gesture, in accordance with an embodiment of the present invention; 
         FIG. 7  is a schematic pictorial illustration of the user performing the swipe gesture, in accordance with an embodiment of the present invention; 
         FIG. 8  is a flow diagram that schematically illustrates a method of conveying visual distortion in response to the user performing a select gesture, in accordance with an embodiment of the present invention; and 
         FIG. 9  is a schematic pictorial illustration of the computer presenting the distortion on the display, in accordance with an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     When using a tactile input device such as a mouse, the user typically manipulates the physical device in a two-dimensional plane comprising a horizontal X-axis and a vertical Y-axis. However, when interacting with a non-tactile 3D user interface (also referred to herein as a 3D user interface), the user may perform gestures in mid-air, and perform the gestures from different positions within a field of view of a 3D sensor coupled to the interface. 
     Embodiments of the present invention provide methods and systems for a computer executing a 3D user interface to convey “friction-like” feedback in response to a gesture performed by a user. In some embodiments, the computer may present, on a display, a list of items that can be scrolled horizontally or vertically. The list can be presented on the display as a circular “wheel”, where the items are positioned along the wheel&#39;s outer rim, and the first element of the list appears immediately after the last element. 
     Using gestures such as Swipe, Grab and Release gestures described hereinbelow, the user can scroll the list in a direction of the user&#39;s hand performing the gesture. In some embodiments, the computer can determine a speed of the hand performing the gesture, and adjust the scrolling speed of the list accordingly. In additional embodiments, the computer can present the scrollable list at a size corresponding to the scrolling speed. For example, the computer can reduce the size of the presented scrollable list at higher scrolling speeds, and increase the size of the presented scrollable list at lower scrolling speeds, thereby enhancing the readability of the scrollable list. 
     Embodiments of the present invention also provide methods and systems for the computer to convey visual feedback as the user moves his hand along a depth axis toward the display. As explained in detail hereinbelow, an interaction surface can be defined in space in proximity to the user, and the computer can be configured to responsively execute control instructions upon a part of the user&#39;s body crossing the interaction surface. In some embodiments, a hand-shaped cursor can be presented as a reflection on the display, and the presented size of the hand cursor can be inversely proportional to a distance between the user&#39;s hand and the interaction surface. In other words, the computer can increase the size of the hand cursor as the user&#39;s hand moves along the depth axis toward the interaction surface. 
     In additional embodiments, the computer may present visual distortion as the user&#39;s hand moves forward and crosses the interaction surface. In some embodiments, the distortion may comprise presenting a region surrounding the hand-shaped cursor with an “elastic-like” effect, and the region on the display can be presented as a recessed vertex upon the user&#39;s hand crossing the interaction region. 
     System Description 
       FIG. 1  is a schematic, pictorial illustration of a non-tactile three dimensional (3D) user interface  20  for operation by a user  22  of a computer  26 , in accordance with an embodiment of the present invention. The 3D user interface is based on a 3D sensing device  24  (also referred to herein as a depth sensor) coupled to the computer, which captures 3D scene information of a scene that includes the body (or at least a body part, such as one or more of hands  30 ) of the user. Device  24  or a separate camera (not shown in the figures) may also capture video images of the scene. The information captured by device  24  is processed by computer  26 , which drives a display  28  so as to present and manipulate on-screen interactive items  38 . Details of the operation of 3D sensing device  24  are described in U.S. Patent Application 2010/0007717, filed on Mar. 4, 2009, whose disclosure is incorporated herein by reference. 
     Computer  26 , executing 3D user interface  20 , processes data generated by device  24  in order to reconstruct a 3D map of user  22 . The term “3D map” refers to a set of 3D coordinates measured, by way of example, with reference to a generally horizontal X-axis  32  in space, a generally vertical Y-axis  34  in space and a depth Z-axis  36  in space, based on device  24 . The 3D coordinates represent the surface of a given object, in this case the user&#39;s body. In one embodiment, device  24  projects a pattern of spots onto the object and captures an image of the projected pattern. Computer  26  then computes the 3D coordinates of points on the surface of the user&#39;s body by triangulation, based on transverse shifts of the spots in the pattern. Methods and devices for this sort of triangulation-based 3D mapping using a projected pattern are described, for example, in PCT International Publications WO 2007/043036, WO 2007/105205 and WO 2008/120217, whose disclosures are incorporated herein by reference. Alternatively, interface  20  may use other methods of 3D mapping, using single or multiple cameras or other types of sensors, as are known in the art. 
     Computer  26  typically comprises a general-purpose computer processor, which is programmed in software to carry out the functions described hereinbelow. The software may be downloaded to the processor in electronic form, over a network, for example, or it may alternatively be provided on non-transitory tangible media, such as optical, magnetic, or electronic memory media. Alternatively or additionally, some or all of the functions of the image processor may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although computer  26  is shown in  FIG. 1 , by way of example, as a separate unit from sensing device  24 , some or all of the processing functions of the computer may be performed by suitable dedicated circuitry within the housing of the sensing device or otherwise associated with the sensing device. 
     As another alternative, these processing functions may be carried out by a suitable processor that is integrated with display  28  (in a television set, for example) or with any other suitable sort of computerized device, such as a game console or media player. The sensing functions of device  24  may likewise be integrated into the computer or other computerized apparatus that is to be controlled by the sensor output. 
     Various techniques may be used to reconstruct the 3D map of the body of user  22 . In one embodiment, computer  26  extracts 3D connected components corresponding to the parts of the body from the depth data generated by device  24 . Techniques that may be used for this purpose are described, for example, in U.S. patent application Ser. No. 12/854,187, filed Aug. 11, 2010, whose disclosure is incorporated herein by reference. The computer analyzes these extracted components in order to reconstruct a “skeleton” of the user&#39;s body, as described in the above-mentioned U.S. Patent Application Publication 2010/0034457, or in U.S. patent application Ser. No. 12/854,188, filed Aug. 11, 2010, whose disclosure is also incorporated herein by reference. In alternative embodiments, other techniques may be used to identify certain parts of the user&#39;s body, and there is no need for the entire body to be visible to device  24  or for the skeleton to be reconstructed, in whole or even in part. 
     Using the reconstructed skeleton, computer  26  can assume a position of a body part such as a tip of a finger of hand  30 , even though the body part (e.g., the fingertip) may not be detected by the depth map due to issues such as minimal object size and reduced resolution at greater distances from device  24 . In some embodiments, computer  26  can auto-complete a body part based on an expected shape of the human part from an earlier detection of the body part, or from tracking the body part along several (previously) received depth maps. 
     In some embodiments, the information generated by computer  26  as a result of this skeleton reconstruction includes the location and direction of the user&#39;s head, as well as of the arms, torso, and possibly legs, hands and other features, as well. Changes in these features from frame to frame (i.e. depth maps) or in postures of the user can provide an indication of gestures and other motions made by the user. User posture, gestures and other motions may provide a control input for user interaction with interface  20 . These body motions may be combined with other interaction modalities that are sensed by device  24 , including user eye movements, as described above, as well as voice commands and other sounds. Interface  20  thus enables user  22  to perform various remote control functions and to interact with applications, interfaces, video programs, images, games and other multimedia content appearing on display  28 . 
       FIG. 2  is a schematic, pictorial illustration showing how user  22  may interact with visualization and interaction regions associated with 3D user interface  20 , in accordance with an embodiment of the present invention. For the purpose of this illustration, an X-Y plane  40  is taken to be parallel to the plane of display screen  28 , with distance (depth) perpendicular to this plane corresponding to the Z-axis, and the origin located at device  24 . Computer  26  creates a depth map of objects within a field of view  42  of device  24 , including the parts of the user&#39;s body that are in the field of view. 
     The operation of 3D user interface  20  is based on an artificial division of the space within field of view  42  into a number of regions:
         A visualization surface  44  defines the outer limit of a visualization region. Objects beyond this limit (such as the user&#39;s head in  FIG. 2 ) are ignored by user interface  20 . When a body part of the user is located within the visualization surface, the user interface detects it and provides visual feedback to the user regarding the location of that body part, typically in the form of an image or icon on display screen  28 . In  FIG. 2 , both of the user&#39;s hands are in the visualization region.   An interaction surface  46 , which is typically located within the visualization region, defines the outer limit of the interaction region. When a part of the user&#39;s body crosses the interaction surface, computer  26  can responsively execute control instructions, as would occur, for instance, if the user made physical contact with an actual touch screen. In this case, however, no physical contact is required to trigger the action. In the example shown in  FIG. 2 , the user&#39;s left hand has crossed the interaction surface and may thus interact with application objects.       

     The interaction and visualization surfaces may have any suitable shapes. In some embodiments, the inventors have found spherical surfaces to be convenient, as shown in  FIG. 2 . In alternative embodiments, one or both of the surfaces may be planar. 
     Various methods may be used to determine when a body part has crossed interaction surface  46  and where it is located. For simple tasks, static analysis of the 3D locations of points in the depth map of the body part may be sufficient. Alternatively, dynamic, velocity-based detection may provide more timely, reliable results, including prediction of and adaptation to user gestures as they occur. Thus, when a part of the user&#39;s body moves toward the interaction surface for a sufficiently long time, it is assumed to be located within the interaction region and may, in turn, result in objects being moved, resized or rotated, or otherwise controlled depending on the motion of the body part. 
     In operation, upon computer  26  detecting a given hand  30  between visualization surface  44  and display  28 , the computer may responsively highlight a given interactive item  38  in response to the given hand&#39;s location. In instances where user  22  positions both hands  30  between visualization surface  44  and display  28 , computer  26  may select the hand that is closer to the display as the given hand. 
     If user  22  wants to highlight a different interactive item  38 , the user can perform a Find gesture, by moving hand  30  along X-Y plane  40 . In some embodiments, computer  26  may highlight a given interactive item  38  by presenting a visual effect. For example, computer  26  can highlight the given interactive item by increasing the brightness of the given interactive item on display  28  or changing the color of the given interactive item. In alternative embodiments, computer  26  may highlight a given interactive item by presenting a cursor in proximity to the given interactive item. The Find gesture is described in U.S. patent application Ser. No. 13/314,207, filed on Dec. 8, 2011, whose disclosure is incorporated herein by reference. 
     Variable-Sized Scrollable Lists 
       FIGS. 3A ,  3 B and  3 C, referred to collectively as  FIG. 3 , are schematic pictorial illustrations of computer  26  presenting a variable-sized scrollable list  50  on display  28 , in accordance with an embodiment of the present invention. In the description of  FIG. 3  herein, some of interactive items  38  may be differentiated by appending a letter to the identifying numeral, so that list  50  comprises non-highlighted interactive items  38 A and a highlighted interactive item  38 B. In the example shown in  FIG. 3 , list  50  comprises 26 items “A”-“Z”. 
     In some configurations, scrollable list  50  behaves like a wheel spinning on a (vertical) axle, where the non-highlighted and the highlighted items are positioned along the wheel&#39;s outer rim. As computer  26  scrolls list  50  in a direction indicated by arrows  52 , the contents of highlighted item  38 B changes, but the location of the highlighted item can remain constant. In the example shown in  FIG. 3 , if highlighted item  38 B comprises “C”, and computer  26  scrolls list  50  three items to the left, then the highlighted item changes to “F”. 
     In some embodiments, computer  26  can present highlighted item  38 B larger than non-highlighted items  38 A. In alternative embodiments, computer  26  can convey other visual feedback in order to differentiate highlighted item  38 B from the non-highlighted items. For example, computer  26  can present highlighted item  38 B with different colors and/or with a greater luminance than the non-highlighted items. 
       FIG. 4  is a flow diagram that schematically illustrates a method of conveying zoom-based feedback for scrollable list  50 , in accordance with an embodiment of the present invention. In an initial step  60 , computer  26  presents scrollable list  50  on display  28 , and in a receive step  62 , the computer receives an input from user  22 . In some embodiments, the input may comprise a sequence of 3D maps received from sensing device  24 , and containing at least hand  30 . For example, computer  26  may receive a sequence of maps and detect, in the maps, hand  30  performing a Swipe gesture (described in further detail hereinbelow). In alternative embodiments, the input may comprise a signal received from a tactile input device such as a keyboard or a mouse (not shown). 
     In a scroll step  64 , computer  26  starts scrolling list  50  at a speed and in a direction indicated by the input. In embodiments where user  22  is interacting with computer  26  via 3D user interface  20 , computer  26  can scroll the list (left or right) at different scrolling speeds corresponding to the speed and the direction of the hand performing a Swipe gesture. 
     In a zoom step  66 , computer  26  zooms, i.e., either enlarges or reduces, scrollable list  50  in response to the scrolling speed, and the method ends. In some embodiments, computer  26  can present scrollable list  50  at a size inversely proportional to the scrolling speed. In other words, computer  26  can reduce the presented size of scrollable list  50  at higher scrolling speeds, and increase the presented size of the scrollable list at lower scrolling speeds. 
     In the example shown in  FIG. 3 ,  FIG. 3A  shows computer  26  presenting list  50  at a large size, when the list has a low scrolling speed or is stationary,  FIG. 3B  shows the computer presenting the list at a medium size for a moderate scrolling speed, and  FIG. 3C  shows the computer presenting the list at a small size for a high scrolling speed. 
     Adjusting the presented size can enhance the readability of scrollable list  50  at higher scrolling speeds, thereby enhancing the user&#39;s experience. For example, computer  26  can present list  50  scrolling one item per second at the low scrolling speed, two items per second at the moderate scrolling speed, and three items per second at the high scrolling speed. Adjusting the size of scrollable list  50  can help enhance the readability of the list at the different scrolling speeds, since scrolling the list presented in  FIG. 3A  at three items per second can be difficult to read (i.e., “too fast”) and scrolling the list presented in  FIG. 3C  at one frame per second can be slow and cumbersome (i.e., “too slow”). 
     In some embodiments, as computer  26  scrolls list  50  at a scrolling speed equaling or surpassing a threshold and reduces the size of scrollable list  50 , the computer may present an additional layer of information (not shown) either behind (i.e., the computer can present the list as slightly transparent), above or below the list. The additional layer may include information related to items  38 A and  38 B such as:
         If list  50  is sorted alphabetically, then the additional layer may comprise the first letter associated with items  38 A and  38 B. For example, if list  50  comprises a list of movies, the additional layer may comprise the first letters of the movies.   If list  50  comprises a list of digital photographs, then the additional layer can comprise dates that the digital photographs were taken.       

     In some embodiments, computer  26  may present the additional layer either in proximity (i.e., behind, above or below, as described supra) to each item  38 , or in proximity to every few items (e.g., behind every fourth item). Additionally or alternatively, computer  26  may present the additional layer scrolling at a slower speed than the scrolling speed of list  50 . Computer  26  can create a visual effect known as “motion parallax” if the computer presents the additional layer behind list  50 , and scrolls the additional layer at a slower speed (i.e., slower than the list). 
       FIG. 5  is a schematic pictorial illustration of computer  26  presenting scrolling controls for list  50 , in accordance with an embodiment of the present invention. In the configuration shown in  FIG. 5 , computer  26  presents a scroll bar  70  that comprises a scroll box  72 , a left scroll arrow  74 , a right scroll arrow  76 , a left tab region  78  and a right tab region  80 . 
     Computer  26  can be configured to scroll list  50  in response to the input received from the user (i.e., in step  62  described supra). In some embodiments, the input may comprise a sequence of 3D maps received from sensing device  24 , and computer  26  can scroll list  50  in response to detecting, in the maps, gestures directed toward scroll bar  70 . Examples of gestures that user  22  can perform toward scroll bar  70  include, but are not limited to Touch and Grab gestures. The Touch gesture is described in U.S. patent application Ser. No. 13/423,314, filed on Mar. 19, 2012, whose disclosure is incorporated herein by reference, and comprises user  22  moving hand  30  toward display  28  and crossing interaction surface  46 . The Grab gesture is also described in U.S. patent application Ser. No. 13/423,314, and comprises user  22  closing at least some fingers of hand  30 . 
     In operation, computer  26  can be configured to:
         Scroll list  50  one item to the left upon detecting, in the maps, a Touch gesture directed toward left scroll arrow  74 .   Scroll list  50  one item to the right upon detecting, in the maps, a Touch gesture directed toward right scroll arrow  76 .   Scroll list  50  a specified number (e.g., four) of items to the left upon detecting, in the maps, a Touch gesture toward left tab region  78 .   Scroll list  50  a specified number of items to the right upon detecting, in the maps, a Touch gesture toward right tab region  80 .   Scroll list  50  in a direction indicated by a direction user  22  is moving hand  30  upon detecting in the maps, a Grab gesture directed toward scroll box  72  and hand  30  subsequently moving side-to-side along interaction surface  46 .       

     In some embodiments, upon the 3D maps indicating user  22  performing a Grab gesture in order to manipulate scroll box  72  (i.e., by moving closed hand  30  side-to-side along interaction surface  46 ), computer  26  can scroll list  50  at a speed corresponding to a speed of the hand. As described supra, computer  26  can zoom list  50  in response to the scrolling speed (e.g., reduce the size of the list at faster scrolling speeds and increase the size of the list at slower scrolling speeds). 
     Swipe Gesture and Visual Friction Feedback 
       FIG. 6  is a flow diagram that schematically illustrates a method of manipulating scrollable list  50 , and  FIG. 7  is a schematic pictorial illustration of user  22  performing a Swipe gesture to manipulate the scrollable list, in accordance with an embodiment of the present invention. In an initial step  90 , computer  26  presents scrollable list  50  on display  28 , and in a receive step  92 , the computer receives, from sensing device  24 , a sequence of 3D maps containing at least hand  30 . 
     In a definition step  94 , computer  26  defines a swipe region  110  that comprises a region on interaction surface  46  between a left reference point  114  and a right reference point  116 . In some embodiments, computer  26  can define the left and the right reference points as locations along interaction surface  46  that are at angles  58  with respect to Z-axis  36 . 
     In a detect step  96 , computer  26  detects a Swipe gesture in which the sequence of 3D maps indicate hand  30  first engaging swipe region  110  at a first location in space, and then disengaging from the swipe region at a second region in space. To perform the Swipe gesture, user  22  can engage swipe region  110  by moving hand  30  toward display  28  and crossing the swipe region, and then moving the hand side-to-side along the swipe region. While performing a Swipe gesture, user  22  may pivot hand  30  around an associated wrist  112 . 
     In a scrolling step  98 , computer  26  initiates scrolling list  50  in response to hand  30  engaging swipe region  110 . Computer  26  typically scrolls list  50  at a scrolling speed and at a direction corresponding to a speed and a direction of hand  30  performing the Swipe gesture. In other words, as user  22  moves hand  30  along swipe region  110 , the hand “grasps” (i.e., directly manipulates) scrollable list  50 , and computer  26  scrolls the list accordingly (i.e., left or right) at a speed indicated by the hand&#39;s speed while performing the gesture. Similarly, if user  22  stops moving hand  30  while the hand is positioned within swipe region  110 , then computer  26  can stop scrolling list  50 . 
     In a deceleration step  100 , computer  26  decelerates the scrolling of list  50  upon hand  30  disengaging from swipe region  110 . To disengage from swipe region  110 , user  22  continues moving hand  30  along swipe region  110  past the left or the right reference points, wherein the passed reference point comprises a release point. For example, if user  22  moves hand  30  past reference point  116  while performing a left-to-right Swipe gesture, then reference point  116  comprises the release point. Likewise, if user  22  moves hand  30  past reference point  114  while performing a right-to-left Swipe gesture, then reference point  114  comprises the release point. 
     Upon detecting, in the maps, user  22  disengaging from swipe region  110  while performing a side-to-side Swipe gesture, computer can convey “friction-like” feedback to user  22  by scrolling list  50  at an initial scrolling speed corresponding to the speed of hand  30  as the hand moved along swipe region  110 , gradually slowing down the scrolling speed, and stopping the scrolling after a specific period of time. The friction-like feedback thus simulates the behavior of the rim of a physical wheel after the wheel has been pushed, and corresponds to a slowing down of the wheel. The push starts the wheel spinning, and the rim of the wheel gradually slows down once the wheel is no longer pushed, due to friction. The rate of slow down of the physical wheel depends on the friction, and on the inertia or mass of the wheel. 
     As discussed supra, computer  26  can adjust the presented size of scrollable list  50  based on the scrolling speed. Therefore, if user  22  performs a fast Swipe gesture, and continues the gesture past one of the reference points, then computer  26  can initially present list  50  as shown in  FIG. 3C . Assuming user  22  does not reposition hand  30  to re-engage swipe region  110 , computer  26  can gradually slow down the scrolling speed and present scrollable list  50  as shown in  FIG. 3B . Finally (again assuming that user  22  does not re-engage swipe region  110 ), computer  26  can present scrollable list  50  as shown in  FIG. 3A  as the computer further slows down and stops scrolling the list. 
     In some embodiments, the Swipe gesture may comprise user  22  positioning hand  30  along swipe region  110 , performing a Grab gesture, and then moving the hand side-to-side along the swipe region. Computer  26  can stop scrolling list  50  upon detecting, in the maps, user  22  stopping hand  30  while the hand is still closed (i.e., performing the Grab gesture) and positioned within swipe region  110 . 
     In alternative embodiments, computer  26  can convey the friction-like feedback described supra, upon detecting in the maps, user  22  positioning hand  30  on swipe region  110 , performing a Grab gesture, moving hand side-to-side along the swipe region, and then performing a Release gesture. The Release gesture, described in U.S. patent application Ser. No. 13/423,314 (referenced above), comprises user relaxing hand  30  so as to open the hand from a closed or folded state. Upon detecting the Release gesture, computer  26  can convey friction-like feedback to user  22  by scrolling list  50  at an initial scrolling speed corresponding to the speed of hand  30  as the hand performs the Release gesture, then gradually slowing down the scrolling speed by applying a simulated friction model to simulate friction behavior, and eventually stopping the scrolling after a specific period of time. 
     Visual and Audio Feedback 
     While interacting with traditional two-dimensional (2D) user interfaces via physical input devices such as a keyboard, the physical input devices typically convey tactile feedback to the user. However, while interacting with a 3D user interface, the user may perform gestures without engaging any physical device, and therefore not receive any tactile feedback. Embodiments of the present invention provide methods and systems for conveying visual and/or audio feedback to the user, thereby compensating for the lack of tactile feedback. 
     In some embodiments, computer  26  may convey visual and/or audio feedback to user  22  upon detecting, in the maps, hand  30  engaging and disengaging from scrollable list  50  while performing a Swipe gesture. For example, user  20  can initially position hand  30  on interaction surface  46  to the left of reference point  114 . As user  22  moves hand  30  along interaction surface  46  and passes reference point  114 , the user starts performing a Swipe gesture, and engages scrollable list  50 . Upon detecting, in the maps, user  22  engaging scrollable list  50 , computer  26  can convey audio feedback (e.g., a click or a beep) and/or visual feedback (e.g., increasing the luminance and/or change the colors used to present scrollable list  50 ). 
     As described supra, user  22  can disengage from scrollable list  50  by continuing the Swipe gesture past one of the reference points. Continuing the example from the previous paragraph, upon detecting, in the maps, user  22  continuing the side-to-side motion of hand  30  past one of the reference points, computer  26  can convey audio feedback (e.g., a click or a beep) and/or visual feedback (e.g., decreasing the luminance and/or changing the colors user to present scrollable list  50 ). 
     Z-Axis Visual Feedback 
       FIG. 8  is a flow diagram that schematically illustrates a method of conveying visual distortion, and  FIG. 9  is a schematic pictorial illustration of computer  26  conveying the distortion as a reflection and an elastic feedback on display  28 , in accordance with an embodiment of the present invention. In an initial step  120 , computer  26  presents multiple interactive items  38  on display  28 . In the configuration shown in  FIG. 9 , computer  26  presents interactive items  38  as a content grid  130 , wherein the interactive items may comprise media content choices. 
     In a receive step  122 , computer  26  receives a sequence of 3D maps containing at least hand  30 , and in a detect step  124 , the computer detects, in the maps, a select gesture toward a given interactive item  38 . A select gesture comprises a physical gesture performed by user  22  in order to select a given interactive item  38  presented on display  28 . The select gesture may comprise the Touch and Grab gestures described supra. 
     As described supra, computer  26  can define interaction surface  46  in proximity to user  22 . In some embodiments, computer  26  may detect, in the 3D maps, a transverse motion of hand  30  (i.e., along X-Y plane  40 ), and position a hand cursor  132  on display  28  in response to the transverse motion. Additionally, computer  26  may present a “reflection”  134  slightly offset (i.e., horizontally and/or vertically) from hand cursor  132 , by considering surface  46  as a “mirror.” Upon detecting user  22  moving hand  30  forward along Z-axis  36 , computer  26  can present hand cursor  132  converging with reflection  134 , and upon the hand reaching interaction surface  46  (the mirror), the hand cursor and the reflection “meet”. In other words, the offset between hand cursor  132  and reflection  134  can be directly proportional to a distance between hand  30  and interaction surface  46 . 
     Additional visual effects that computer  26  can convey when presenting hand cursor  132  and reflection  134  include:
         Upon detecting user  22  performing a Find gesture, the computer can present hand cursor  132  and reflection  134  at an initial small size. As user  22  moves hand  30  forward along Z-axis  36  towards interaction surface  46 , computer  26  may increase the size of the hand cursor and the reflection. In other words, the size of hand cursor  132  and reflection  134  can be inversely proportional to the distance between hand  30  and interaction surface  46 .   Upon detecting user  22  moving hand  30  away from interaction surface  46 , computer  26  can decrease the intensity of reflection  134  (i.e., “fading” the reflection away), and as the user moves the hand closer to the interaction surface, the computer can increase the intensity of the reflection.   Upon detecting user  22  moving hand  30  away from interaction surface  46 , computer  26  can blur reflection  134 , and as the user moves the hand closer to the interaction surface, the computer can sharpen the reflection.       

     Returning to the flow diagram, in a distort step  126 , computer  26  can distort a region  136  on display  28  that contains the given interactive item, and the method ends. In some embodiments, computer  26  may distort content region  136  when the sequence of 3D maps indicate that hand  30  reached and/or moved forward past interaction surface  46 . In some embodiments, computer  26  may create a distortion in the content choice on content grid  110  where hand cursor  132  is positioned. Computer  26  can present the distortion as an “elastic-like” effect, where it appears that hand cursor  132  is pushing against a flexible surface, thereby creating a recessed vertex. 
     It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features, including the transformations and the manipulations, described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Metadata:
Filing Date: 20120823
Publication Date: 20150901
Grant Date: 20150901
Priority Date: 20110824
Inventors: GALOR MICHA
Assignee: APPLE INC
CPC Classifications: [{"code": "G09G5/34", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G5/00", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/045", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/045", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/04", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0485", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}, {"code": "G09G2340/04", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G5/00", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G2340/14", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F2203/04806", "inventive": false, "first": false, "tree": "[]"}, {"code": "G09G5/34", "inventive": true, "first": false, "tree": "[]"}, {"code": "G09G2340/14", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 47745526