PATENT DOCUMENT

Publication Number: US-11397469-B2
Application Number: US-202117306618-A
Country: US
Kind Code: B2

Title: Motion and gesture input from a wearable device

Abstract:
This disclosure relates to detecting hand gesture input using an electronic device, such as a wearable device strapped to a wrist. The device can have multiple photodiodes, each sensing light at a different position on a surface of the device that faces skin of a user. Examples of the disclosure detect hand gestures by recognizing patterns in sensor data that are characteristic of each hand gesture, as the tissue expands and contracts and anatomical features in the tissue move during the gesture.

Claims:
What is claimed is: 
     
       1. A method for training gesture detection, the method comprising:
 displaying a user interface on a display of a wearable device; 
 prompting a user of the wearable device, via the user interface, to perform a first type of gesture; 
 after prompting the user to perform the first type of gesture and during a first time period:
 emitting light toward a user form a portion of the wearable device contacting the user; and 
 collecting first sensor data from a set of photodiodes of the wearable device, the first sensor data based on a portion of the emitted light reflected from the user; 
 
 prompting the user, via the user interface, to perform a second type of gesture; 
 after prompting the user to perform the second type of gesture and during a second time period after the first time period, collecting second sensor data from the set of photodiodes; 
 extracting first characteristics from the first sensor data; 
 assigning at least some of the first characteristics to a first cluster of signal characteristics; 
 extracting second characteristics from the second sensor data; 
 assigning at least some of the second characteristics to a second cluster of signal characteristics; 
 detecting the first type of gesture when subsequently collected sensor data corresponds to the first cluster of signal characteristics; and 
 detecting the second type of gesture when the subsequently collected sensor data corresponds to the second cluster of signal characteristics. 
 
     
     
       2. The method of  claim 1 , further comprising:
 comparing the first cluster of signal characteristics to the second cluster of signal characteristics; and 
 in response to determining that more of the first characteristics are assigned to the first cluster of signal characteristics than to the second cluster of signal characteristics, assigning the first cluster of signal characteristics to the first type of gesture. 
 
     
     
       3. The method of  claim 2 , further comprising, in response to determining that more of the second characteristics are assigned to the second cluster of signal characteristics than to the first cluster of signal characteristics, assigning the second cluster of signal characteristics to the second type of gesture. 
     
     
       4. The method of  claim 1 , further comprising:
 during a third time period after the second time period, collecting third sensor data from the set of photodiodes; 
 determining that the third sensor data does not correspond to the first cluster of signal characteristics or the second cluster of signal characteristics; 
 extracting third characteristics from the third sensor data; and 
 assigning the third characteristics to a third cluster of signal characteristics. 
 
     
     
       5. The method of  claim 1 , further comprising:
 generating a first template associated with the first type of gesture, the first template used to assign the at least some of the first characteristics to the first type of gesture; and 
 generating a second template associated with the second type of gesture, the second template used to assign the at least some of the second characteristics to the second type of gesture. 
 
     
     
       6. The method of  claim 1 , wherein:
 prompting the user to perform the first type of gesture comprises displaying a first image depicting the first type of gesture; and 
 prompting the user to perform the second type of gesture comprises displaying a second image depicting the second type of gesture. 
 
     
     
       7. The method of  claim 6 , wherein the second image depicting the second type of gesture is displayed after the first sensor data is collected from the set of photodiodes. 
     
     
       8. The method of  claim 1 , further comprising performing an operation with respect to the wearable device in response to detecting at least one of the first type of gesture or the second type of gesture. 
     
     
       9. A method for training gesture detection, the method comprising:
 displaying a training interface on a display of a wearable device; 
 displaying, via the training interface, a representation of a gesture; 
 emitting light toward a user from a portion of the wearable device contacting the user; 
 collecting sensor data from a set of photodiodes of the wearable device within a defined time period following displaying the representation of the gesture, the sensor data based on a portion of the emitted light reflected from the user; 
 determining characteristics of the sensor data; 
 assigning at least some of the characteristics to a cluster of characteristics; and 
 assigning the cluster of characteristics to the gesture. 
 
     
     
       10. The method of  claim 9 , wherein emitting light toward the user comprises emitting light from a set of light emitters of the wearable device toward a wrist of the user, wherein collecting sensor data from the set of photodiodes comprises receiving at least a portion of the light reflected from the wrist of the user. 
     
     
       11. The method of  claim 10 , wherein the sensor data is collected for a set of channels, each channel comprising a respective pair comprising a light emitter of the set of light emitters and a photodiode of the set of photodiodes, with each respective channel corresponding to a respective light wavelength. 
     
     
       12. The method of  claim 9 , wherein the representation of the gesture is an image corresponding to the gesture. 
     
     
       13. The method of  claim 9 , wherein:
 the gesture is a first gesture, the sensor data is first sensor data, the characteristics are first characteristics, and the cluster of characteristics is a first cluster of characteristics; and 
 the method further comprises:
 collecting second sensor data from the set of photodiodes; 
 determining second characteristics of the second sensor data; 
 assigning at least some of the second characteristics to a second cluster of characteristics; and 
 assigning the second cluster of characteristics to a second gesture. 
 
 
     
     
       14. The method of  claim 13 , further comprising:
 comparing the first cluster of characteristics to the second cluster of characteristics; 
 determining that more of the first characteristics are assigned to the first cluster of characteristics than to the second cluster of characteristics; 
 assigning the first cluster of characteristics to the first gesture; and 
 assigning the second cluster of characteristics to the second gesture. 
 
     
     
       15. The method of  claim 9 , further comprising:
 determining a mean signal characteristic of the cluster of characteristics; and 
 assigning a template to the cluster of characteristics based on the mean signal characteristic. 
 
     
     
       16. A wearable electronic device comprising:
 a housing; 
 a set of photodiodes disposed within the housing; 
 a display within the housing; and 
 a processor configured to:
 cause display of a training interface on the display; 
 cause the training interface to display a representation of a gesture; 
 emitting light toward a user from a portion of the wearable electronic device that contacts the user; 
 collect sensor data from the set of photodiodes of the wearable electronic device within a defined time period following display of the representation of the gesture, the sensor data determined from a portion of the emitted light reflected from the user; 
 determine characteristics of the sensor data; and 
 assign at least some of the characteristics to a cluster of characteristics. 
 
 
     
     
       17. The wearable electronic device of  claim 16 , further comprising:
 a set of light emitters; and 
 a strap coupling the housing to a wrist of the user. 
 
     
     
       18. The wearable electronic device of  claim 17 , wherein light emitted from the set of light emitters reflect from at least one sub-dermal feature of the wrist of the user and is received by at least one photodiode of the set of photodiodes. 
     
     
       19. The wearable electronic device of  claim 16 , wherein, based on detecting subsequent sensor data comprising at least one subsequent characteristic corresponding to the cluster of characteristics, the processor is further configured to perform an operation of the wearable electronic device when the gesture is performed by the user. 
     
     
       20. The wearable electronic device of  claim 19 , wherein the operation is at least one of opening an application, closing the application, returning to a home screen, or messaging a contact.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 16/706,353, filed Dec. 6, 2019, which is a continuation of U.S. patent application Ser. No. 15/914,838, filed Mar. 7, 2018, now U.S. Pat. No. 10,503,254, which is a continuation of U.S. patent application Ser. No. 14/973,573, filed Dec. 17, 2015, now U.S. Pat. No. 9,939,899, which claims the benefit of U.S. Provisional Patent Application No. 62/233,295, filed Sep. 25, 2015, the contents of which are incorporated by reference as if fully disclosed herein. 
    
    
     FIELD OF THE DISCLOSURE 
     This relates generally to detecting a user&#39;s motion and gesture input to provide commands to one or more devices. In particular, a device can use one or more sensors to determine a user&#39;s motion and gesture input based on movements of the user&#39;s hand, arm, wrist, and fingers. 
     BACKGROUND OF THE DISCLOSURE 
     Some existing portable electronic devices accept voice or touch input to control functionality of the devices. For example, a voice command system can map specific verbal commands to operations such as initiating a voice call with a particular contact in response to speaking the contact&#39;s name. In another example, a touch input system can map specific touch gestures to operations such as zooming out in response to a pinch gesture on a touch sensitive surface. However, there may be situations where the user&#39;s ability to speak a verbal command or perform a touch gesture may be limited. 
     SUMMARY OF THE DISCLOSURE 
     This disclosure relates to detecting hand gesture input using an electronic device, such as a wearable device strapped to a wrist. The device can have multiple photodiodes, each sensing light at a different position on a surface of the device that faces skin of a user. Due to this positioning, the sensor data from the photodiodes can capture movement of anatomical features in the tissue of the user during a hand gesture. Further, different light emitters on the device can emit light at different wavelengths (e.g., infrared light, green light, etc.), which penetrate to different depths in the tissue of the user before reflecting back to the photodiodes on the device. Accordingly, sensor data from the photodiodes can capture expansion and contraction in the tissue of the user during a hand gesture. Examples of the disclosure detect hand gestures by recognizing patterns in sensor data that are characteristic of each hand gesture, as the tissue expands and contracts and anatomical features in the tissue move during the gesture. 
     In one example, the device can be trained on sensor data as the user performs a plurality of hand gestures. For example, during a first period, a user can perform a hand flap gesture and sensor data can be collected as the gesture is performed. During a second period, a user can perform a hand clench gesture and further sensor data can be collected as the gesture is performed. The sensor data can then be processed to calculate signal characteristics (e.g., peak/trough extraction, phase detection, etc., as described below) based on the sensor data for each period. The signal characteristics can then be clustered (e.g., using a clustering algorithm such as k-means clustering), including assigning some or all of the signal characteristics from the first period to a first cluster and some or all of the signal characteristics from the second period to a second cluster. The first cluster can be considered a pattern that is characteristic of the hand flap gesture that was performed during the first period, and the second cluster can be considered a pattern that is characteristic of the hand clench gesture that was performed during the second period. 
     Gesture detection can then be performed based, in part, on the clusters formed during training. Further sensor data can be collected during a third period, and signal characteristics can be calculated from that sensor data. After calculating the signal characteristics from the third period, a hand gesture can be detected based on the cluster that the signal characteristics belong to. For example, if most of the signal characteristics from the third period belong to the first cluster, then a hand flap gesture can be detected. If most of the signal characteristics from the third period belong to the second cluster, then a hand clench gesture can be detected. If most of the signal characteristics from the third period belong to a third cluster, then it can be determined that the user has not performed the first hand gesture or the second hand gesture. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       For a better understanding of the various described examples, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures. 
         FIGS. 1A-1C  illustrate an exemplary electronic device with a plurality of sensors in accordance with examples of the disclosure. 
         FIGS. 2A-2D  illustrate exemplary hand gestures in accordance with examples of the disclosure. 
         FIG. 3  illustrates exemplary charts of sensor data in accordance with examples of the disclosure. 
         FIGS. 4A-4D  illustrate two-dimensional clustering examples in accordance with examples of the disclosure. 
         FIGS. 5A-5B  illustrate training user interfaces in accordance with examples of the disclosure. 
         FIG. 6  illustrates an exemplary method of training for gesture detection in accordance with examples of the disclosure. 
         FIG. 7  illustrates an exemplary method of gesture detection in accordance with examples of the disclosure. 
         FIG. 8  is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some examples. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description, reference is made to the accompanying drawings which form a part hereof, and in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the disclosed examples. 
     This disclosure relates to detecting hand gesture input using an electronic device, such as a wearable device strapped to a wrist. The device can have multiple photodiodes, each sensing light at a different position on a surface of the device that faces skin of a user. Due to this positioning, the sensor data from the photodiodes can capture movement of anatomical features in the tissue of the user during a hand gesture. Further, different light emitters on the device can emit light at different wavelengths (e.g., infrared light, green light, etc.), which penetrate to different depths in the tissue of the user before reflecting back to the photodiodes on the device. Accordingly, sensor data from the photodiodes can capture expansion and contraction in the tissue of the user during a hand gesture. Examples of the disclosure detect hand gestures by recognizing patterns in sensor data that are characteristic of each hand gesture, as the tissue expands and contracts and anatomical features in the tissue move during the gesture. 
     In one example, the device can be trained on sensor data as the user performs a plurality of hand gestures. For example, during a first period, a user can perform a hand flap gesture and sensor data can be collected as the gesture is performed. During a second period, a user can perform a hand clench gesture and further sensor data can be collected as the gesture is performed. The sensor data can then be processed to calculate signal characteristics (e.g., peak/trough extraction, phase detection, etc., as described below) based on the sensor data for each period. The signal characteristics can then be clustered (e.g., using a clustering algorithm such as k-means clustering), including assigning some or all of the signal characteristics from the first period to a first cluster and some or all of the signal characteristics from the second period to a second cluster. The first cluster can be considered a pattern that is characteristic of the hand flap gesture that was performed during the first period, and the second cluster can be considered a pattern that is characteristic of the hand clench gesture that was performed during the second period. 
     Gesture detection can then be performed based, in part, on the clusters formed during training. Further sensor data can be collected during a third period, and signal characteristics can be calculated from that sensor data. After calculating the signal characteristics from the third period, a hand gesture can be detected based on the cluster that the signal characteristics belong to. For example, if most of the signal characteristics from the third period belong to the first cluster, then a hand flap gesture can be detected. If most of the signal characteristics from the third period belong to the second cluster, then a hand clench gesture can be detected. If most of the signal characteristics from the third period belong to a third cluster, then it can be determined that the user has not performed the first hand gesture or the second hand gesture. 
     Although examples of the disclosure may be described herein primarily in terms of wearable devices strapped to a wrist and hand gestures, particularly hand flap gestures and hand clench gestures, it should be understood that examples of the disclosure are not so limited, but include wearable devices attached to other body parts, such as upper arms or legs, and gestures that can result therefrom. 
     Further, although examples of the disclosure may be described herein primarily in terms of devices with a plurality of photodiodes, it should be understood that examples of the disclosure are not so limited, but include devices with only a single photodiode. A channel of sensor data can correspond to each unique light sensor/emitter pair, whether there is one or multiple sensors, one or multiple emitters, etc. 
       FIGS. 1A-1C  illustrate an exemplary electronic device  100  with a plurality of sensors in accordance with examples of the disclosure. The electronic device  100  can include a plurality of photodiodes  101  (or any other light sensors) and a plurality of light emitters  105  (e.g., light-emitting diodes, etc.). When the electronic device is in use, the photodiodes  101  and the light emitters  105  face the tissue  200  of a user. 
     As illustrated in  FIG. 1B , each photodiode  101  can sense light at a different position on a surface of the device  100  that faces the tissue  200  of a user. Due to this positioning, the sensor data from the photodiodes  101  can capture movement of anatomical features in the tissue  200  of the user during a hand gesture. Further, as illustrated in  FIG. 1C , different light emitters  105  on the device  100  can emit light at different wavelengths (e.g., infrared light  600 , green light  602 , etc.), which penetrate to different depths in the tissue  200  of the user before reflecting back to the photodiodes on the device. Accordingly, sensor data from the photodiodes  101  can capture expansion and contraction in the tissue  200  of the user during a hand gesture. 
     In some examples, each possible photodiode emitter combination can be considered a separate channel of light sensor data. For example, in a device with two green light emitters, two infrared light emitters, and two photodiodes, there can be eight channels of light sensor data. When the first green light emitter emits light, the first and second photodiodes sense first and second channels of light sensor data, respectively. When the second green light emitter emits light, the first and second photodiodes sense third and fourth channels of light sensor data, respectively. When the first infrared light emitter emits light, the first and second photodiodes sense fifth and sixth channels of light sensor data, respectively. When the second infrared light emitter emits light, the first and second photodiodes sense seventh and eight channels of light sensor data, respectively. 
     In some examples, the device  100  can further include sensors that provide additional channels of sensor data. For example, a device that includes an accelerometer and a force sensor can provide four additional channels of sensor data. A force sensor that detects force of the wrist against the device can provide a first additional channel of sensor data. An accelerometer that senses acceleration in X, Y, and Z directions can provide second, third, and fourth additional channels of sensor data, respectively. In some examples, additional channels of sensor data can include data from a barometer, a magnetometer, a Global Positioning System (GPS) receiver, and/or numerous other possibilities. In some examples, other light sensors are used in place of or in addition to photodiodes. In some examples, a force sensor can be spatially discretized, sensing force independently at multiple positions of the surface of the device that contacts the wrist, in which case the force sensor can provide multiple (e.g.,  4 ) channels of pressure information. 
       FIGS. 2A-2D  illustrate exemplary hand gestures in accordance with examples of the disclosure. In some examples, a hand  202  can perform a hand flap gesture, as illustrated in  FIGS. 2A-2C . In  FIG. 2A , the fingers of the hand  202  are fully extended. In  FIG. 2B , the fingers come down toward the palm of the hand  202 . In  FIG. 2C , the fingers return to a fully extended position. 
     In some examples, a hand  202  can perform a hand clench gesture, as illustrated in  FIG. 2D .  FIG. 2D  illustrates the hand  202  in a first formation. The hand clench gesture can be performed by clenching the hand  202  when in the first formation. 
     The  FIGS. 2A-2D  are examples of hand clench and hand flap gestures. 
     Variations of these gestures and other gestures entirely may be trained on and detected in accordance with examples of the disclosure. 
       FIG. 3  illustrates exemplary charts of sensor data in accordance with examples of the disclosure. For example, the sensor data  300 ,  302 ,  304 , and  306  can represent different channels of sensor data during a hand clench gesture, and the sensor data  308 ,  310 ,  312 , and  314  can represent different channels of sensor data during a hand flap gesture. In one example,  300  and  308  can represent sensor data from the first light emitter/first photodiode pair (channel L 1 P 1 ),  302  and  310  can represent sensor data from the first light emitter/second photodiode pair (channel L 1 P 2 ),  304  and  312  can represent sensor data from the second light emitter/first photodiode pair (channel L 2 P 1 ), and  306  and  314  can represent sensor data from the second light emitter/second photodiode pair (channel L 2 P 2 ). 
     As can be observed in  FIG. 3 , during the clench gesture, channels L 1 P 1 , L 2 P 1 , and L 2 P 2  each exhibit a signal characteristic wherein the trough occurs just before the peak of the signal. In contrast, during the flap gesture, channels L 1 P 2 , L 2 P 1 , and L 2 P 2  each exhibit a signal characteristic wherein the peak occurs just before the trough of the signal. Accordingly, the two gestures can be distinguished based on signal characteristics of the L 2 P 1  and L 2 P 2  channels. 
     Whereas the visual signal characteristics can be easily observed in the exemplary sensor data of  FIG. 3 , a number of quantitative signal characteristics may be calculated based on the sensor data before clustering. For example, an amplitude difference can be calculated between a peak and a trough of the sensor data, with sign indicating whether the peak comes before the trough or vice versa, a time difference can be calculated between a peak and a trough of the sensor data, a maximum amplitude can be calculated, a period between peaks of the sensor data can be calculated, and/or a phase can be detected in the sensor data, among other possibilities. In some examples, signal characteristics can be observed in a frequency domain. For example, one or more frames of sensor data may be analyzed (e.g., by a Fourier transform) to extract frequency information as additional signal characteristics. These and other signal characteristics can be extracted from any or all of the channels of sensor data, including light sensors, force sensors, an accelerometer, and/or other sensors. 
       FIGS. 4A-4D  illustrate two-dimensional clustering examples in accordance with examples of the disclosure. In some examples, each frame in sensor data collection can be considered a point in multi-dimensional space, with each calculated signal characteristic for that frame being a coordinate in the multi-dimensional space. For example, sensor data can be collected during a first period in which a first gesture is performed by the user. The sensor data can be divided into a plurality of frames, and each frame can correspond to a set of coordinates defined by the signal characteristics calculated for that time frame. The data illustrated in  FIGS. 4A-4D  represent data collected with two signal characteristics: amplitude difference between peak and trough for the L 2 P 1  and L 2 P 2  channels, as discussed above with respect to  FIG. 3 . Although these figures only show two signal characteristics, examples of the disclosure are not so limited and contemplate using multiple kinds of signal characteristics from multiple channels, including light sensor data, force sensor data, and/or accelerometer data, among other possibilities. 
       FIG. 4A  illustrates sensor data collected during a first period in which a hand flap gesture is being performed as illustrated in  FIGS. 2A-2C , possibly multiple times in succession.  FIG. 4B  illustrates sensor data collected during a second period in which a hand clench gesture is being performed as illustrated in  FIG. 2D , possibly multiple times in succession.  FIG. 4C  illustrates sensor data collected during a third period in which no gesture is being performed.  FIG. 4D  illustrates the sensor data collected during all three periods, clustered into three clusters: first cluster  400 , second cluster  402 , and third cluster  404 . 
     As shown in  FIGS. 4A-4D  together, most of the points corresponding to the first period belong to the first cluster, most of the points corresponding to the second period belong to the second cluster, and most of the points corresponding to the third period belong to the third cluster. Accordingly, it may be inferred that any point that belongs to the first cluster  400  was collected during performance of a hand flap gesture, and any point that belongs to the second cluster  402  was collected during performance of a hand clench gesture. Further, it may be inferred that any point that does not belong to the first cluster  400  or the second cluster  402  was not collected during performance of a hand clench or hand flap gesture. 
       FIGS. 5A-5B  illustrate training user interfaces in accordance with examples of the disclosure.  FIG. 5A  illustrates a user interface prompting a user to perform a hand flap gesture. Such a user interface may be displayed during a first period and sensor data may be collected during the first period while the user interface is displayed.  FIG. 5B  illustrates a user interface prompting a user to perform a hand clench gesture. Such a user interface may be displayed during a second period and sensor data may be collected during the second period while the user interface is displayed. Additional user interfaces may be displayed prompting a user to perform additional gestures to train for detection of the additional gestures. 
       FIG. 6  illustrates an exemplary method of training for gesture detection in accordance with examples of the disclosure. In some examples, sensor data can be collected while the user performs various hand gestures to train the detection algorithm. During a first period in which the user performs a first hand gesture (e.g., when prompted by a user interface as illustrated in  FIG. 5A ), the electronic device can collect ( 600 ) first sensor data from the plurality of photodiodes. During a second period in which the user performs a second hand gesture (e.g., when prompted by a user interface as illustrated in  FIG. 5B ), the electronic device can collect ( 602 ) second sensor data from the plurality of photodiodes. In some examples, sensor data is further collected from other sensors, such as a force sensor and/or an accelerometer, among other possibilities. 
     After sensor data collection, signal characteristics can be extracted from the sensor data, as discussed above with respect to  FIG. 3 . The electronic device can calculate first signal characteristics ( 604 ) based on the first sensor data and calculate second signal characteristics ( 606 ) based on the second sensor data. 
     After calculating the signal characteristics, the electronic device can perform ( 608 ) clustering (e.g., a k-means clustering algorithm or other clustering algorithm) on the first and second signal characteristics. The clustering algorithm may assign ( 610 ) some or all of the first signal characteristics to a first cluster of signal characteristics, and assign ( 612 ) some or all of the second signal characteristics to a second cluster of signal characteristics. 
     In some examples, the electronic device can assign each cluster to one of the hand gestures as part of the training process. For example, the electronic device can compare the first cluster to the second cluster. Then, based on comparing the first cluster to the second cluster, the electronic device can determine there are more of the first signal characteristics assigned to the first cluster than to the second cluster. In accordance with the determination that there are more of the first signal characteristics assigned to the first cluster than to the second cluster, the electronic device can assign the first cluster to the first hand gesture. Similarly, based on comparing the first cluster to the second cluster, the electronic device can determine there are more of the second signal characteristics assigned to the second cluster than to the first cluster. In accordance with the determination that there are more of the second signal characteristics assigned to the second cluster than to the first cluster, the electronic device can assign the second cluster to the second hand gesture. 
     In some examples, the clustering process can be seeded by initially clustering the signal characteristics based on the time period in which the data was collected. For example, the first cluster can be initially assigned all the signal characteristics corresponding to the first period during which the first hand gesture was performed, and the second cluster can be initially assigned all the signal characteristics corresponding to the second period during which the second hand gesture was performed. Following this initial assignment, a clustering algorithm (e.g., k-means clustering) can be performed to optimize the clusters, potentially moving some points from the first cluster to the second cluster, moving some points from the second cluster to the first cluster, and/or moving some points from the first and second clusters to other clusters. 
     In some examples, the electronic device can generate a template for each of the first and second hand gestures to aid in the gesture detection process. For example, the electronic device can calculate first mean signal characteristics for the first cluster (e.g., as part of the k-means clustering process), and the first mean signal characteristics may be used as a template for the first cluster. Similarly, the first electronic device can calculate second mean signal characteristics for the second cluster (e.g., as part of the k-means clustering process), and the second mean signal characteristics may be used as a template for the second cluster. In another example, some or all of the first sensor data may be stored as the first template for the first cluster, and some or all of the second sensor data may be stored as the second template for the second cluster. In some examples, a generic template for each gesture may be stored and used as a starting point for the training process before any user-specific data has been collected. Then, each template can be adjusted based on user-specific data collected during training. Detection using templates is discussed further below. 
     In some examples, additional training can be conducted to train the device to detect when the user is not performing either the first or second gesture. The electronic device can collect additional sensor data during a period in which the user does not perform the first or second hand gesture. Signal characteristics can be calculated based on the additional sensor data, and these signal characteristics can be assigned to a third cluster. The third cluster can be a cluster that is associated with some third gesture (e.g., if the user performed a third gesture during the training period) or it can be a cluster that is not associated with any gesture. 
       FIG. 7  illustrates an exemplary method of gesture detection in accordance with examples of the disclosure. After training, gestures can be detected by collecting new sensor data and then using the clusters associated with each gesture to determine if one of the gestures has been performed. During a third period (e.g., during use of the device after training has concluded), the electronic device can collect ( 700 ) third sensor data from the plurality of sensors. In some examples, sensor data is further collected from sensors other than photodiodes, such as a force sensor and/or an accelerometer, among other possibilities. 
     Signal characteristics can be again extracted from the sensor data, as described above with respect to  FIG. 3 . The electronic device can calculate ( 702 ) the third signal characteristics based on the third sensor data. 
     To perform gesture detection, the electronic device can determine ( 704 ) whether the third signal characteristics belong to the first cluster, the second cluster, or a third cluster. The third cluster can be a cluster that is associated with some third gesture, different from the first or second, or it can be a cluster that is not associated with any gesture. 
     In some examples, determining whether the third signal characteristics belong to the first cluster, the second cluster, or the third cluster includes performing clustering (e.g., a k-means clustering algorithm, or other clustering algorithm) on the third signal characteristics with respect to the first, second, and third clusters. The cluster membership of the third signal characteristics may be determined by the results of the clustering. 
     In some examples, determining whether the third signal characteristics belong to the first cluster, the second cluster, or the third cluster includes comparing the third signal characteristics to first, second, and/or third templates corresponding to the first, second, and third clusters, respectively. The electronic device can thereby determine whether the third signal characteristics are closer to the first cluster or the second cluster based on the templates. For example, if each template includes mean signal characteristics, then the electronic device can calculate a first distance from the third signal characteristics to the first template (e.g., the first mean signal characteristics of the first cluster) and calculate a second distance from the third signal characteristics to the second template (e.g., the second mean signal characteristics of the second cluster). The distance calculation can be a Euclidean distance calculation between two points in multi-dimensional space. 
     In accordance with a determination that the first distance is shorter than the second distance, the electronic device can determine that the third signal characteristics belong to the first cluster. In accordance with a determination that the second distance is shorter than the first distance, the electronic device can determine that the third signal characteristics belong to the second cluster. In some examples, the electronic device can also compare the third signal characteristics to a third template in the same manner, or, if both the first and second distances are longer than a predetermined threshold distance, the electronic device can determine that the third signal characteristics belong to a third cluster by default. 
     Based on determining which cluster the third signal characteristics belong to, the electronic device can detect the first gesture, the second gesture, or no gesture. In accordance with a determination that the third signal characteristics belong to the first cluster (e.g., the cluster associated with the first hand gesture), the electronic device can determine ( 706 ) that the user has performed the first hand gesture. In accordance with a determination that the third signal characteristics belong to the second cluster (e.g., the cluster associated with the second hand gesture), the electronic device can determine ( 708 ) that the user has performed the second hand gesture. In accordance with a determination that the third signal characteristics belong to the third cluster (e.g., a cluster associated with some third gesture, or no gesture whatsoever), the electronic device can determine ( 710 ) that the user has not performed the first hand gesture or the second hand gesture. 
     After detecting the first gesture or the second gesture, the electronic device can perform an operation associated with the detected gesture. For example, if the electronic device detects the first gesture, the electronic device can perform an operation in response, such as opening an application, closing an application, returning to a home screen, messaging a contact, etc. 
     In some examples, sensor data (e.g., the first, second, or third sensor data described above) can be further processed before extracting signal characteristics (e.g., the first, second, or third signal characteristics described above). For example, a band pass filter may be applied to sensor data to filter out heart rate frequencies from the sensor data. As light sensor data may vary according to the periodic motion of blood through human tissue, it may be beneficial to filter out these frequencies to better isolate the contribution of hand gesture motion to the signal characteristics. 
     Attention is now directed toward examples of portable devices with touch-sensitive displays.  FIG. 8  is a block diagram illustrating portable multifunction device  100  with touch-sensitive display system  112  in accordance with some examples. Touch-sensitive display  112  is sometimes called a “touch screen” for convenience and is sometimes known as or called a “touch-sensitive display system.” Device  100  includes memory  102  (which optionally includes one or more computer-readable storage mediums), memory controller  122 , one or more processing units (CPUs)  120 , peripherals interface  118 , RF circuitry  108 , audio circuitry  110 , speaker  111 , microphone  113 , input/output (I/O) subsystem  106 , other input control devices  116 , and external port  124 . Device  100  optionally includes one or more optical sensors  164 . Device  100  optionally includes one or more contact intensity sensors  165  for detecting intensity of contacts on device  100  (e.g., a touch-sensitive surface such as touch-sensitive display system  112  of device  100 ). Device  100  optionally includes one or more tactile output generators  167  for generating tactile outputs on device  100  (e.g., generating tactile outputs on a touch-sensitive surface such as touch-sensitive display system  112  of device  100  or touchpad  355  of device  300 ). These components optionally communicate over one or more communication buses or signal lines  103 . 
     As used in the specification and claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact) on the touch-sensitive surface, or to a substitute (proxy) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four distinct values and more typically includes hundreds of distinct values (e.g., at least 256). Intensity of a contact is, optionally, determined (or measured) using various approaches and various sensors or combinations of sensors. For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., a weighted average) to determine an estimated force of a contact. Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and/or changes thereto, the capacitance of the touch-sensitive surface proximate to the contact and/or changes thereto, and/or the resistance of the touch-sensitive surface proximate to the contact and/or changes thereto are, optionally, used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the substitute measurements for contact force or pressure are used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurements). In some implementations, the substitute measurements for contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure). Using the intensity of a contact as an attribute of a user input allows for user access to additional device functionality that may otherwise not be accessible by the user on a reduced-size device with limited real estate for displaying affordances (e.g., on a touch-sensitive display) and/or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical/mechanical control such as a knob or a button). 
     As used in the specification and claims, the term “tactile output” refers to physical displacement of a device relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component relative to a center of mass of the device that will be detected by a user with the user&#39;s sense of touch. For example, in situations where the device or the component of the device is in contact with a surface of a user that is sensitive to touch (e.g., a finger, palm, or other part of a user&#39;s hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in physical characteristics of the device or the component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is, optionally, interpreted by the user as a “down click” or “up click” of a physical actuator button. In some cases, a user will feel a tactile sensation such as an “down click” or “up click” even when there is no movement of a physical actuator button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user&#39;s movements. As another example, movement of the touch-sensitive surface is, optionally, interpreted or sensed by the user as “roughness” of the touch-sensitive surface, even when there is no change in smoothness of the touch-sensitive surface. While such interpretations of touch by a user will be subject to the individualized sensory perceptions of the user, there are many sensory perceptions of touch that are common to a large majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., an “up click,” a “down click,” “roughness”), unless otherwise stated, the generated tactile output corresponds to physical displacement of the device or a component thereof that will generate the described sensory perception for a typical (or average) user. 
     It should be appreciated that device  100  is only one example of a portable multifunction device, and that device  100  optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. The various components shown in  FIG. 8  are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and/or application-specific integrated circuits. 
     Memory  102  may include one or more computer-readable storage mediums. The computer-readable storage mediums may be tangible and non-transitory. Memory  102  may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller  122  may control access to memory  102  by other components of device  100 . 
     Peripherals interface  118  can be used to couple input and output peripherals of the device to CPU  120  and memory  102 . The one or more processors  120  run or execute various software programs and/or sets of instructions stored in memory  102  to perform various functions for device  100  and to process data. In some examples, peripherals interface  118 , CPU  120 , and memory controller  122  may be implemented on a single chip, such as chip  104 . In some other examples, they may be implemented on separate chips. 
     RF (radio frequency) circuitry  108  receives and sends RF signals, also called electromagnetic signals. RF circuitry  108  converts electrical signals to/from electromagnetic signals and communicates with communications networks and other communications devices via the electromagnetic signals. RF circuitry  108  optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, and so forth. RF circuitry  108  optionally communicates with networks, such as the Internet, also referred to as the World Wide Web (WWW), an intranet and/or a wireless network, such as a cellular telephone network, a wireless local area network (LAN) and/or a metropolitan area network (MAN), and other devices by wireless communication. The RF circuitry  108  optionally includes well-known circuitry for detecting near field communication (NFC) fields, such as by a short-range communication radio. The wireless communication optionally uses any of a plurality of communications standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet message access protocol (IMAP) and/or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and/or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document. 
     Audio circuitry  110 , speaker  111 , and microphone  113  provide an audio interface between a user and device  100 . Audio circuitry  110  receives audio data from peripherals interface  118 , converts the audio data to an electrical signal, and transmits the electrical signal to speaker  111 . Speaker  111  converts the electrical signal to human-audible sound waves. Audio circuitry  110  also receives electrical signals converted by microphone  113  from sound waves. Audio circuitry  110  converts the electrical signal to audio data and transmits the audio data to peripherals interface  118  for processing. Audio data may be retrieved from and/or transmitted to memory  102  and/or RF circuitry  108  by peripherals interface  118 . In some examples, audio circuitry  110  also includes a headset jack (e.g.,  212 ,  FIG. 2 ). The headset jack provides an interface between audio circuitry  110  and removable audio input/output peripherals, such as output-only headphones or a headset with both output (e.g., a headphone for one or both ears) and input (e.g., a microphone). 
     I/O subsystem  106  couples input/output peripherals on device  100 , such as touch screen  112  and other input control devices  116 , to peripherals interface  118 . I/O subsystem  106  optionally includes display controller  156 , optical sensor controller  158 , intensity sensor controller  159 , haptic feedback controller  161 , and one or more input controllers  160  for other input or control devices. The one or more input controllers  160  receive/send electrical signals from/to other input control devices  116 . The other input control devices  116  optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate examples, input controller(s)  160  are, optionally, coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. The one or more buttons (e.g.,  208 ,  FIG. 2 ) optionally include an up/down button for volume control of speaker  111  and/or microphone  113 . The one or more buttons optionally include a push button (e.g.,  206 ,  FIG. 2 ). 
     A quick press of the push button may disengage a lock of touch screen  112  or begin a process that uses gestures on the touch screen to unlock the device, as described in U.S. Patent Application Pub. No. 2007/0150842, “Unlocking a Device by Performing Gestures on an Unlock Image,” published Jun. 28, 2007, U.S. Pat. No. 7,657,849, which is hereby incorporated by reference in its entirety. A longer press of the push button (e.g.,  206 ) may turn power to device  100  on or off. The user may be able to customize a functionality of one or more of the buttons. Touch screen  112  is used to implement virtual or soft buttons and one or more soft keyboards. 
     Touch-sensitive display  112  provides an input interface and an output interface between the device and a user. Display controller  156  receives and/or sends electrical signals from/to touch screen  112 . Touch screen  112  displays visual output to the user. The visual output may include graphics, text, icons, video, and any combination thereof (collectively termed “graphics”). In some examples, some or all of the visual output may correspond to user-interface objects. 
     Touch screen  112  has a touch-sensitive surface, sensor, or set of sensors that accepts input from the user based on haptic and/or tactile contact. Touch screen  112  and display controller  156  (along with any associated modules and/or sets of instructions in memory  102 ) detect contact (and any movement or breaking of the contact) on touch screen  112  and convert the detected contact into interaction with user-interface objects (e.g., one or more soft keys, icons, web pages, or images) that are displayed on touch screen  112 . In one example, a point of contact between touch screen  112  and the user corresponds to a finger of the user. 
     Touch screen  112  may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other examples. Touch screen  112  and display controller  156  may detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen  112 . In one example, projected mutual capacitance sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, Calif. 
     A touch-sensitive display in some examples of touch screen  112  may be analogous to the multi-touch sensitive touchpads described in the following U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat. No. 6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932 (Westerman), and/or U.S. Patent Publication 2002/0015024A1, each of which is hereby incorporated by reference in its entirety. However, touch screen  112  displays visual output from device  100 , whereas touch-sensitive touchpads do not provide visual output. 
     A touch-sensitive display in some examples of touch screen  112  may be as described in the following applications: (1) U.S. Patent Application Pub. No. 2007/0257890, “Multipoint Touch Surface Controller,” published Nov. 8, 2007; (2) U.S. Patent Application Pub. No. 2006/0097991, “Multipoint Touchscreen,” published May 11, 2006; (3) U.S. Patent Application Pub. No. 2006/0026521, “Gestures For Touch Sensitive Input Devices,” published Feb. 2, 2006; (4) U.S. Patent Application Pub. No. 2006/0026536, “Gestures For Touch Sensitive Input Devices,” published Feb. 2, 2006; (5) U.S. Patent Application Pub. No. 2006/0026535, “Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices,” published Feb. 2, 2006; (6) U.S. Patent Application Pub. No. 2006/0033724, “Virtual Input Device Placement On A Touch Screen User Interface,” published Feb. 16, 2006; (7) U.S. Patent Application Pub. No. 2006/0053387, “Operation Of A Computer With A Touch Screen Interface,” published Mar. 9, 2006; (8) U.S. Patent Application Pub. No. 2006/0085757, “Activating Virtual Keys Of A Touch-Screen Virtual Keyboard,” published Apr. 20, 2006; and (9) U.S. Patent Application Pub. No. 2006/0197753, “Multi-Functional Hand-Held Device,” published Sep. 7, 2006. All of these applications are incorporated by reference herein in their entirety. 
     Touch screen  112  may have a video resolution in excess of 100 dpi. In some examples, the touch screen has a video resolution of 160 dpi. The user may make contact with touch screen  112  using any suitable object or appendage, such as a stylus, a finger, and so forth. In some examples, the user interface is designed to work primarily with finger-based contacts and gestures, which can be less precise than stylus-based input due to the larger area of contact of a finger on the touch screen. In some examples, the device translates the rough finger-based input into a precise pointer/cursor position or command for performing the actions desired by the user. 
     In some examples, in addition to the touch screen, device  100  may include a touchpad (not shown) for activating or deactivating particular functions. In some examples, the touchpad is a touch-sensitive area of the device that, unlike the touch screen, does not display visual output. The touchpad may be a touch-sensitive surface that is separate from touch screen  112  or an extension of the touch-sensitive surface formed by the touch screen. 
     Device  100  also includes power system  162  for powering the various components. Power system  162  may include a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)) and any other components associated with the generation, management and distribution of power in portable devices. 
     Device  100  may also include one or more optical sensors  164 .  FIG. 8  shows an optical sensor coupled to optical sensor controller  158  in I/O subsystem  106 . Optical sensor  164  may include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Optical sensor  164  receives light from the environment, projected through one or more lenses, and converts the light to data representing an image. In conjunction with imaging module  143  (also called a camera module), optical sensor  164  may capture still images or video. In some examples, an optical sensor is located on the back of device  100 , opposite touch screen display  112  on the front of the device so that the touch screen display may be used as a viewfinder for still and/or video image acquisition. In some examples, an optical sensor is located on the front of the device so that the user&#39;s image may be obtained for video conferencing while the user views the other video conference participants on the touch screen display. In some examples, the position of optical sensor  164  can be changed by the user (e.g., by rotating the lens and the sensor in the device housing) so that a single optical sensor  164  may be used along with the touch screen display for both video conferencing and still and/or video image acquisition. 
     Device  100  optionally also includes one or more contact intensity sensors  165 .  FIG. 8  shows a contact intensity sensor coupled to intensity sensor controller  159  in I/O subsystem  106 . Contact intensity sensor  165  optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor  165  receives contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some examples, at least one contact intensity sensor is collocated with, or proximate to, a touch-sensitive surface (e.g., touch-sensitive display system  112 ). In some examples, at least one contact intensity sensor is located on the back of device  100 , opposite touch screen display  112 , which is located on the front of device  100 . 
     Device  100  may also include one or more proximity sensors  166 .  FIG. 6  shows proximity sensor  166  coupled to peripherals interface  118 . Alternately, proximity sensor  166  may be coupled to input controller  160  in I/O subsystem  106 . Proximity sensor  166  may perform as described in U.S. Patent Application Pub. Nos. 2006/0161871, “Proximity Detector In Handheld Device”; 2006/0161870, “Proximity Detector In Handheld Device”; 2008/0167834, “Using Ambient Light Sensor To Augment Proximity Sensor Output”; 2007/0075965, “Automated Response To And Sensing Of User Activity In Portable Devices”; and 2008/0140868, “Methods And Systems For Automatic Configuration Of Peripherals,” which are hereby incorporated by reference in their entirety. In some examples, the proximity sensor turns off and disables touch screen  112  when the multifunction device is placed near the user&#39;s ear (e.g., when the user is making a phone call). 
     Device  100  optionally also includes one or more tactile output generators  167 .  FIG. 8  shows a tactile output generator coupled to haptic feedback controller  161  in I/O subsystem  106 . Tactile output generator  167  optionally includes one or more electroacoustic devices such as speakers or other audio components and/or electromechanical devices that convert energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts electrical signals into tactile outputs on the device). Contact intensity sensor  165  receives tactile feedback generation instructions from haptic feedback module  133  and generates tactile outputs on device  100  that are capable of being sensed by a user of device  100 . In some examples, at least one tactile output generator is collocated with, or proximate to, a touch-sensitive surface (e.g., touch-sensitive display system  112 ) and, optionally, generates a tactile output by moving the touch-sensitive surface vertically (e.g., in/out of a surface of device  100 ) or laterally (e.g., back and forth in the same plane as a surface of device  100 ). In some examples, at least one tactile output generator sensor is located on the back of device  100 , opposite touch screen display  112 , which is located on the front of device  100 . 
     Device  100  may also include one or more accelerometers  168 .  FIG. 8  shows accelerometer  168  coupled to peripherals interface  118 . Alternately, accelerometer  168  may be coupled to an input controller  160  in I/O subsystem  106 . Accelerometer  168  may perform as described in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated by reference herein in their entirety. In some examples, information is displayed on the touch screen display in a portrait view or a landscape view based on an analysis of data received from the one or more accelerometers. Device  100  optionally includes, in addition to accelerometer(s)  168 , a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information concerning the location and orientation (e.g., portrait or landscape) of device  100 . 
     In some examples, the software components stored in memory  102  include operating system  126 , communication module (or set of instructions)  128 , contact/motion module (or set of instructions)  130 , graphics module (or set of instructions)  132 , text input module (or set of instructions)  134 , Global Positioning System (GPS) module (or set of instructions)  135 , and applications (or sets of instructions)  136 . Furthermore, in some examples, memory  102  ( FIG. 8 ) or  370  ( FIG. 3 ) stores device/global internal state  157 , as shown in  FIGS. 1A and 3 . Device/global internal state  157  includes one or more of: active application state, indicating which applications, if any, are currently active; display state, indicating what applications, views or other information occupy various regions of touch screen display  112 ; sensor state, including information obtained from the device&#39;s various sensors and input control devices  116 ; and location information concerning the device&#39;s location and/or attitude. 
     Operating system  126  (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components. 
     Communication module  128  facilitates communication with other devices over one or more external ports  124  and also includes various software components for handling data received by RF circuitry  108  and/or external port  124 . External port  124  (e.g., Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for coupling directly to other devices or indirectly over a network (e.g., the Internet, wireless LAN, etc.). In some examples, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, or similar to and/or compatible with, the 30-pin connector used on iPod® (trademark of Apple Inc.) devices. 
     Contact/motion module  130  optionally detects contact with touch screen  112  (in conjunction with display controller  156 ) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module  130  includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining an intensity of the contact (e.g., the force or pressure of the contact or a substitute for the force or pressure of the contact), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module  130  receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, optionally includes determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations are, optionally, applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., “multitouch”/multiple finger contacts). In some examples, contact/motion module  130  and display controller  156  detect contact on a touchpad. 
     In some examples, contact/motion module  130  uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has “clicked” on an icon). In some examples, at least a subset of the intensity thresholds are determined in accordance with software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of particular physical actuators and can be adjusted without changing the physical hardware of device  100 ). For example, a mouse “click” threshold of a trackpad or touch screen display can be set to any of a large range of predefined threshold values without changing the trackpad or touch screen display hardware. Additionally, in some implementations, a user of the device is provided with software settings for adjusting one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and/or by adjusting a plurality of intensity thresholds at once with a system-level click “intensity” parameter). 
     Contact/motion module  130  optionally detects a gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and/or intensities of detected contacts). Thus, a gesture is, optionally, detected by detecting a particular contact pattern. For example, detecting a finger tap gesture includes detecting a finger-down event followed by detecting a finger-up (liftoff) event at the same position as the finger-down event (e.g., at the position of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger-down event followed by detecting one or more finger-dragging events, and subsequently followed by detecting a finger-up (liftoff) event. 
     Graphics module  132  includes various known software components for rendering and displaying graphics on touch screen  112  or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual property) of graphics that are displayed. As used herein, the term “graphics” includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user-interface objects including soft keys), digital images, videos, animations, and the like. 
     In some examples, graphics module  132  stores data representing graphics to be used. Each graphic is, optionally, assigned a corresponding code. Graphics module  132  receives, from applications etc., one or more codes specifying graphics to be displayed along with, if necessary, coordinate data and other graphic property data, and then generates screen image data to output to display controller  156 . 
     Haptic feedback module  133  includes various software components for generating instructions used by tactile output generator(s)  167  to produce tactile outputs at one or more locations on device  100  in response to user interactions with device  100 . 
     Text input module  134 , which may be a component of graphics module  132 , provides soft keyboards for entering text in various applications (e.g., contacts  137 , e-mail  140 , IM  141 , browser  147 , and any other application that needs text input). 
     GPS module  135  determines the location of the device and provides this information for use in various applications (e.g., to telephone  138  for use in location-based dialing; to camera  143  as picture/video metadata; and to applications that provide location-based services such as weather widgets, local yellow page widgets, and map/navigation widgets). 
     Applications  136  may include the following modules (or sets of instructions), or a subset or superset thereof:
         Contacts module  137  (sometimes called an address book or contact list);   Telephone module  138 ;   Video conference module  139 ;   E-mail client module  140 ;   Instant messaging (IM) module  141 ;   Workout support module  142 ;   Camera module  143  for still and/or video images;   Image management module  144 ;   Video player module;   Music player module;   Browser module  147 ;   Calendar module  148 ;   Widget modules  149 , which may include one or more of: weather widget  149 - 1 , stocks widget  149 - 2 , calculator widget  149 - 3 , alarm clock widget  149 - 4 , dictionary widget  149 - 5 , and other widgets obtained by the user, as well as user-created widgets  149 - 6 ;   Widget creator module  150  for making user-created widgets  149 - 6 ;   Search module  151 ;   Video and music player module  152 , which merges video player module and music player module;   Notes module  153 ;   Map module  154 ; and/or   Online video module  155 .       

     Examples of other applications  136  that may be stored in memory  102  include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication. 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , contacts module  137  may be used to manage an address book or contact list (e.g., stored in application internal state  192  of contacts module  137  in memory  102  or memory  370 ), including: adding name(s) to the address book; deleting name(s) from the address book; associating telephone number(s), e-mail address(es), physical address(es) or other information with a name; associating an image with a name; categorizing and sorting names; providing telephone numbers or e-mail addresses to initiate and/or facilitate communications by telephone  138 , video conference module  139 , e-mail  140 , or IM  141 ; and so forth. 
     In conjunction with RF circuitry  108 , audio circuitry  110 , speaker  111 , microphone  113 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , telephone module  138  may be used to enter a sequence of characters corresponding to a telephone number, access one or more telephone numbers in contacts module  137 , modify a telephone number that has been entered, dial a respective telephone number, conduct a conversation, and disconnect or hang up when the conversation is completed. As noted above, the wireless communication may use any of a plurality of communications standards, protocols, and technologies. 
     In conjunction with RF circuitry  108 , audio circuitry  110 , speaker  111 , microphone  113 , touch screen  112 , display controller  156 , optical sensor  164 , optical sensor controller  158 , contact/motion module  130 , graphics module  132 , text input module  134 , contacts module  137 , and telephone module  138 , video conference module  139  includes executable instructions to initiate, conduct, and terminate a video conference between a user and one or more other participants in accordance with user instructions. 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , e-mail client module  140  includes executable instructions to create, send, receive, and manage e-mail in response to user instructions. In conjunction with image management module  144 , e-mail client module  140  makes it very easy to create and send e-mails with still or video images taken with camera module  143 . 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , the instant messaging module  141  includes executable instructions to enter a sequence of characters corresponding to an instant message, to modify previously entered characters, to transmit a respective instant message (for example, using a Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephony-based instant messages or using XMPP, SIMPLE, or IMPS for Internet-based instant messages), to receive instant messages, and to view received instant messages. In some examples, transmitted and/or received instant messages may include graphics, photos, audio files, video files and/or other attachments as are supported in an MMS and/or an Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephony-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, or IMPS). 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , GPS module  135 , map module  154 , and music player module, workout support module  142  includes executable instructions to create workouts (e.g., with time, distance, and/or calorie burning goals); communicate with workout sensors (sports devices); receive workout sensor data; calibrate sensors used to monitor a workout; select and play music for a workout; and display, store, and transmit workout data. 
     In conjunction with touch screen  112 , display controller  156 , optical sensor(s)  164 , optical sensor controller  158 , contact/motion module  130 , graphics module  132 , and image management module  144 , camera module  143  includes executable instructions to capture still images or video (including a video stream) and store them into memory  102 , modify characteristics of a still image or video, or delete a still image or video from memory  102 . 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , and camera module  143 , image management module  144  includes executable instructions to arrange, modify (e.g., edit), or otherwise manipulate, label, delete, present (e.g., in a digital slide show or album), and store still and/or video images. 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , browser module  147  includes executable instructions to browse the Internet in accordance with user instructions, including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages. 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , e-mail client module  140 , and browser module  147 , calendar module  148  includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) in accordance with user instructions. 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , and browser module  147 , widget modules  149  are mini-applications that may be downloaded and used by a user (e.g., weather widget  149 - 1 , stocks widget  149 - 2 , calculator widget  149 - 3 , alarm clock widget  149 - 4 , and dictionary widget  149 - 5 ) or created by the user (e.g., user-created widget  149 - 6 ). In some examples, a widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some examples, a widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widgets). 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , and browser module  147 , the widget creator module  150  may be used by a user to create widgets (e.g., turning a user-specified portion of a web page into a widget). 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , search module  151  includes executable instructions to search for text, music, sound, image, video, and/or other files in memory  102  that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions. 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , audio circuitry  110 , speaker  111 , RF circuitry  108 , and browser module  147 , video and music player module  152  includes executable instructions that allow the user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions to display, present, or otherwise play back videos (e.g., on touch screen  112  or on an external, connected display via external port  124 ). In some examples, device  100  optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.). 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , and text input module  134 , notes module  153  includes executable instructions to create and manage notes, to-do lists, and the like in accordance with user instructions. 
     In conjunction with RF circuitry  108 , touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , text input module  134 , GPS module  135 , and browser module  147 , map module  154  may be used to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data on stores and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions. 
     In conjunction with touch screen  112 , display controller  156 , contact/motion module  130 , graphics module  132 , audio circuitry  110 , speaker  111 , RF circuitry  108 , text input module  134 , e-mail client module  140 , and browser module  147 , online video module  155  includes instructions that allow the user to access, browse, receive (e.g., by streaming and/or download), play back (e.g., on the touch screen or on an external, connected display via external port  124 ), send an e-mail with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some examples, instant messaging module  141 , rather than e-mail client module  140 , is used to send a link to a particular online video. Additional description of the online video application can be found in U.S. Provisional Patent Application No. 60/936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed Jun. 20, 2007, and U.S. Patent Application Pub. No. 2008/0320391, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” published Dec. 25, 2008, the contents of which are hereby incorporated by reference in their entirety. 
     Each of the above-identified modules and applications corresponds to a set of executable instructions for performing one or more functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various examples. For example, video player module may be combined with music player module into a single module (e.g., video and music player module  152 ,  FIG. 8 ). In some examples, memory  102  may store a subset of the modules and data structures identified above. Furthermore, memory  102  may store additional modules and data structures not described above. 
     In some examples, device  100  is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and/or a touchpad. By using a touch screen and/or a touchpad as the primary input control device for operation of device  100 , the number of physical input control devices (such as push buttons, dials, and the like) on device  100  may be reduced. 
     The predefined set of functions that are performed exclusively through a touch screen and/or a touchpad optionally include navigation between user interfaces. In some examples, the touchpad, when touched by the user, navigates device  100  to a main, home, or root menu from any user interface that is displayed on device  100 . In such examples, a “menu button” is implemented using a touchpad. In some other examples, the menu button is a physical push button or other physical input control device instead of a touchpad. 
     In some examples, a method of detecting hand gestures performed by a user of an electronic device including a plurality of photodiodes is disclosed. The method may include: during a first period in which the user performs a first hand gesture, collecting first sensor data from the plurality of photodiodes; during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes; calculating first signal characteristics based on the first sensor data and second signal characteristics based on the second sensor data; assigning some or all of the first signal characteristics to a first cluster of signal characteristics; assigning some or all of the second signal characteristics to a second cluster of signal characteristics; during a third period, collecting third sensor data from the plurality of photodiodes; calculating third signal characteristics based on the third sensor data; determining whether the third signal characteristics belong to the first cluster, the second cluster, or a third cluster; in accordance with a determination that the third signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture; in accordance with a determination that the third signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and in accordance with a determination that the third signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture. 
     Additionally or alternatively to the other disclosed examples, the method may further include: comparing the first cluster to the second cluster; based on comparing the first cluster to the second cluster, determining there are more of the first signal characteristics assigned to the first cluster than to the second cluster; and in accordance with the determination that there are more of the first signal characteristics assigned to the first cluster than to the second cluster, assigning the first cluster to the first hand gesture. Additionally or alternatively to the other disclosed examples, the method may further include: based on comparing the first cluster to the second cluster, determining there are more of the second signal characteristics assigned to the second cluster than to the first cluster; and in accordance with the determination that there are more of the second signal characteristics assigned to the second cluster than to the first cluster, assigning the second cluster to the second hand gesture. Additionally or alternatively to the other disclosed examples, the method may further include: generating a first template for the first hand gesture and a second template for the second hand gesture; and comparing the third signal characteristics to the first and second templates. Additionally or alternatively to the other disclosed examples, the method may further include: calculating first mean signal characteristics for the first cluster, wherein the first template is generated based on the first mean signal characteristics for the first cluster; and calculating second mean signal characteristics for the second cluster, wherein the second template is generated based on the second mean signal characteristics for the second cluster. Additionally or alternatively to the other disclosed examples, comparing the third signal characteristics to the first and second templates may include: calculating a first distance from the third signal characteristics to the first template and calculating a second distance from the third signal characteristics to the second template; in accordance with a determination that the first distance is shorter than the second distance, determining that the third signal characteristics belong to the first cluster; and in accordance with a determination that the second distance is shorter than the first distance, determining that the third signal characteristics belong to the second cluster. Additionally or alternatively to the other disclosed examples, generating the first template for the first hand gesture may include storing some or all of the first sensor data as the first template, and generating the second template for the second hand gesture may include storing some or all of the second sensor data as the second template. Additionally or alternatively to the other disclosed examples, calculating the first signal characteristics may include calculating at least one of: an amplitude difference between a peak and a trough of the first sensor data, a time difference between a peak and a trough of the first sensor data, a maximum amplitude of the first sensor data, a period between peaks of the first sensor data, and a phase of the first sensor data. Additionally or alternatively to the other disclosed examples, the method may further include: filtering heart rate frequencies from the first sensor data before calculating the first signal characteristics based on the first sensor data. Additionally or alternatively to the other disclosed examples, the method may further include: during the first period in which the user performs the first hand gesture, further collecting the first sensor data from at least one of a force sensor and an accelerometer. Additionally or alternatively to the other disclosed examples, collecting the first sensor data from the plurality of photodiodes may include collecting a first channel of infrared light and collecting a second channel of green light. Additionally or alternatively to the other disclosed examples, the method may further include: during a fourth period in which the user does not perform the first or second hand gesture, collecting fourth sensor data from the plurality of photodiodes; calculating fourth signal characteristics based on the fourth sensor data; and assigning some or all of the fourth signal characteristics to the third cluster of signal characteristics. Additionally or alternatively to the other disclosed examples, assigning the first and second signal characteristics to the first and second clusters may be performed using a k-means clustering algorithm. Additionally or alternatively to the other disclosed examples, the k-means clustering algorithm may also be applied to the third signal characteristics and determining whether the third signal characteristics belong to the first cluster, the second cluster, or the third cluster may be based on the k-means clustering algorithm. 
     In some examples, a non-transitory computer readable medium is disclosed, the computer readable medium containing instructions, that, when executed, perform a method. The method may include: The method may include: during a first period in which the user performs a first hand gesture, collecting first sensor data from the plurality of photodiodes; during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes; calculating first signal characteristics based on the first sensor data and second signal characteristics based on the second sensor data; assigning some or all of the first signal characteristics to a first cluster of signal characteristics; assigning some or all of the second signal characteristics to a second cluster of signal characteristics; during a third period, collecting third sensor data from the plurality of photodiodes; calculating third signal characteristics based on the third sensor data; determining whether the third signal characteristics belong to the first cluster, the second cluster, or a third cluster; in accordance with a determination that the third signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture; in accordance with a determination that the third signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and in accordance with a determination that the third signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture. 
     In some examples, an electronic device is disclosed. The electronic device may include: one or more processors; memory; a plurality of photodiodes; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, which when executed by the one or more processors, cause the electronic device to perform a method. The method may include: during a first period in which the user performs a first hand gesture, collecting first sensor data from the plurality of photodiodes; during a second period in which the user performs a second hand gesture, collecting second sensor data from the plurality of photodiodes; calculating first signal characteristics based on the first sensor data and second signal characteristics based on the second sensor data; assigning some or all of the first signal characteristics to a first cluster of signal characteristics; assigning some or all of the second signal characteristics to a second cluster of signal characteristics; during a third period, collecting third sensor data from the plurality of photodiodes; calculating third signal characteristics based on the third sensor data; determining whether the third signal characteristics belong to the first cluster, the second cluster, or a third cluster; in accordance with a determination that the third signal characteristics belong to the first cluster, determining that the user has performed the first hand gesture; in accordance with a determination that the third signal characteristics belong to the second cluster, determining that the user has performed the second hand gesture; and in accordance with a determination that the third signal characteristics belong to the third cluster, determining that the user has not performed the first hand gesture or the second hand gesture. 
     Although the disclosed examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the disclosed examples as defined by the appended claims.

Metadata:
Filing Date: 20210503
Publication Date: 20220726
Grant Date: 20220726
Priority Date: 20150925
Inventors: ALLEC, NICHOLAS PAUL JOSEPH
MU, Xiaoyi
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F2218/12", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F18/23", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2218/12", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F18/23", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/015", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/014", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0308", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V40/28", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/014", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06K9/6218", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/014", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/017", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06K9/00536", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0304", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V40/28", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V10/751", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/015", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06V10/143", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0308", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 56851734