PATENT DOCUMENT

Publication Number: US-9024884-B2
Application Number: US-1597804-A
Country: US
Kind Code: B2

Title: Touch-sensitive electronic apparatus for media applications, and methods therefor

Abstract:
An electronic apparatus, such as an electronic mixing apparatus and an electronic keyboard apparatus, and associated methods are disclosed. The electronic mixing apparatus or the electronic keyboard apparatus is provided on a touch screen that provides user input and display capabilities. In one embodiment, the touch screen is a multipoint touch screen so that multiple user touch inputs can be simultaneously acquired. In another embodiment, surface guides can be provided on the touch screen to assist with user input.

Claims:
What is claimed is: 
     
       1. A method comprising:
 at a computing device having a multi-point touch screen and a processor:
 executing a media mixing software application that provides media mixing functionality; 
 while the media mixing software application is executing, displaying a mixing console Graphical User Interface (GUI) on the multi-point touch screen, the mixing console GUI having a plurality of graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials; 
 while displaying the mixing console GUI a plurality of having graphically depicted media mixing controls, detecting a plurality of simultaneous touch inputs on the multi-point touch screen; 
 identifying a plurality of distinct graphically depicted media mixing controls that are associated with the plurality of simultaneous touch inputs on the multi-point touch screen, wherein each respective touch input in the plurality of simultaneous touch inputs on the multi-point touch screen corresponds to a different graphically depicted media mixing control in the plurality of distinct graphically depicted media mixing controls; 
 determining modifications to the identified plurality of distinct graphically depicted media mixing controls based on the plurality of simultaneous touch inputs, wherein a respective modification to a respective graphically depicted media mixing control in the identified plurality of distinct graphically depicted media mixing controls is based on the respective touch input that corresponds to the respective graphically depicted media mixing control; 
 updating the identified plurality of distinct graphically depicted media mixing controls of the mixing console GUI to reflect the determined modifications; and 
 determining input data for media mixing operations in the media mixing software application based on the detected plurality of simultaneous touch inputs on the multi-point touch screen and the identified plurality of distinct graphically depicted media mixing controls that are associated with the detected plurality of simultaneous touch inputs. 
 
 
     
     
       2. The method of  claim 1 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a pressure applied by a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       3. The method of  claim 1 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a velocity of a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       4. The method of  claim 1 , further comprising:
 detecting an input corresponding to a request to change configuration of the mixing console GUI; 
 in response to detecting the input corresponding to a request to change the configuration of the mixing console GUI, changing the display of at least one of the graphically depicted media mixing controls. 
 
     
     
       5. The method of  claim 4 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the type of at least one of the graphically depicted media mixing controls. 
     
     
       6. The method of  claim 4 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the number of the graphically depicted media mixing controls that are displayed on the multi-point touch screen. 
     
     
       7. The method of  claim 4 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the position of the at least one of the graphically depicted media mixing controls with respect the mixing console GUI. 
     
     
       8. The method of  claim 1 , further comprising:
 detecting a touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI; and 
 in response to detecting the touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI, replacing the display of the mixing console GUI with display of a keyboard GUI comprising an array of a plurality of graphically depicted keys on the multi-point touch screen. 
 
     
     
       9. The method of  claim 1 , wherein one or more surface guides are provided on the multi-point touch screen to assist with the plurality of simultaneous touch inputs. 
     
     
       10. The method of  claim 9 , wherein configurations of the one or more surface guides correspond to configurations of the graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials. 
     
     
       11. The method of  claim 9 , wherein an overlay sheet is provided on the multi-point touch screen, the one or more surface guides are provided on a front surface of the overlay sheet, and a back surface of the overlay sheet is affixed to the multi-point touch screen. 
     
     
       12. An electronic device, comprising:
 a multi-point touch screen; 
 a processor; 
 memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the one or more programs including instructions for:
 executing a media mixing software application that provides media mixing functionality; 
 while the media mixing software application is executing, displaying a mixing console Graphical User Interface (GUI) on the multi-point touch screen, the mixing console GUI having a plurality of graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials; 
 while displaying the mixing console GUI having a plurality of graphically depicted media mixing controls, detecting a plurality of simultaneous touch inputs on the multi-point touch screen; 
 identifying a plurality of distinct graphically depicted media mixing controls that are associated with the plurality of simultaneous touch inputs on the multi-point touch screen, wherein each respective touch input in the plurality of simultaneous touch inputs on the multi-point touch screen corresponds to a different graphically depicted media mixing control in the plurality of distinct graphically depicted media mixing controls; 
 determining modifications to the identified plurality of distinct graphically depicted media mixing controls based on the plurality of simultaneous touch inputs, wherein a respective modification to a respective graphically depicted media mixing control in the identified plurality of distinct graphically depicted media mixing controls is based on the respective touch input that corresponds to the respective graphically depicted media mixing control; 
 updating the identified plurality of distinct graphically depicted media mixing controls of the mixing console GUI to reflect the determined modifications; and 
 determining input data for media mixing operations in the media mixing software application based on the detected plurality of simultaneous touch inputs on the multi-point touch screen and the identified plurality of distinct graphically depicted media mixing controls that are associated with the detected plurality of simultaneous touch inputs. 
 
 
     
     
       13. The device of  claim 12 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a pressure applied by a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       14. The device of  claim 12 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a velocity of a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       15. The device of  claim 12 , wherein the one or more programs further comprise instructions for:
 detecting an input corresponding to a request to change configuration of the mixing console GUI; 
 in response to detecting the input corresponding to a request to change the configuration of the mixing console GUI, changing the display of at least one of the graphically depicted media mixing controls. 
 
     
     
       16. The device of  claim 15 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the type of at least one of the graphically depicted media mixing controls. 
     
     
       17. The device of  claim 15 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the number of the graphically depicted media mixing controls that are displayed on the multi-point touch screen. 
     
     
       18. The device of  claim 15 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the position of the at least one of the graphically depicted media mixing controls with respect the mixing console GUI. 
     
     
       19. The device of  claim 12 , wherein the one or more programs further comprise instructions for:
 detecting a touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI; and 
 in response to detecting the touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI, replacing the display of the mixing console GUI with display of a keyboard GUI comprising an array of a plurality of graphically depicted keys on the multi-point touch screen. 
 
     
     
       20. The device of  claim 12 , wherein one or more surface guides are provided on the multi-point touch screen to assist with the plurality of simultaneous touch inputs. 
     
     
       21. The device of  claim 20 , wherein configurations of the one or more surface guides correspond to configurations of the graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials. 
     
     
       22. The device of  claim 20 , wherein an overlay sheet is provided on the multi-point touch screen, the one or more surface guides are provided on a front surface of the overlay sheet, and a back surface of the overlay sheet is affixed to the multi-point touch screen. 
     
     
       23. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a multi-point touch screen and a processor, cause the device to:
 execute a media mixing software application that provides media mixing functionality; 
 while the media mixing software application is executing, display a mixing console Graphical User Interface (GUI) on the multi-point touch screen, the mixing console GUI having a plurality of graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials; 
 while displaying the mixing console GUI having a plurality of graphically depicted media mixing controls, detect a plurality of simultaneous touch inputs on the multi-point touch screen; 
 identify a plurality of distinct graphically depicted media mixing controls that are associated with the plurality of simultaneous touch inputs on the multi-point touch screen, wherein each respective touch input in the plurality of simultaneous touch inputs on the multi-point touch screen corresponds to a different graphically depicted media mixing control in the plurality of distinct graphically depicted media mixing controls; 
 determine modifications to the identified plurality of distinct graphically depicted media mixing controls based on the plurality of simultaneous touch inputs, wherein a respective modification to a respective graphically depicted media mixing control in the identified plurality of distinct graphically depicted media mixing controls is based on the respective touch input that corresponds to the respective graphically depicted media mixing control; 
 update the identified plurality of distinct graphically depicted media mixing controls of the mixing console GUI to reflect the determined modifications; and 
 determine input data for media mixing operations in the media mixing software application based on the detected plurality of simultaneous touch inputs on the multi-point touch screen and the identified plurality of distinct graphically depicted media mixing controls that are associated with the detected plurality of simultaneous touch inputs. 
 
     
     
       24. The non-transitory computer readable storage medium of  claim 23 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a pressure applied by a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       25. The non-transitory computer readable storage medium of  claim 23 , wherein determining input data for media mixing operations in the media mixing software application is based in part on a velocity of a touch input in the detected plurality of simultaneous touch inputs on the multi-point touch screen. 
     
     
       26. The medium of  claim 23 , wherein the one or more programs further comprise instructions, which when executed by the device, cause the device to:
 detect an input corresponding to a request to change configuration of the mixing console GUI; 
 in response to detecting the input corresponding to a request to change the configuration of the mixing console GUI, change the display of at least one of the graphically depicted media mixing controls. 
 
     
     
       27. The medium of  claim 26 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the type of at least one of the graphically depicted media mixing controls. 
     
     
       28. The medium of  claim 26 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the number of the graphically depicted media mixing controls that are displayed on the multi-point touch screen. 
     
     
       29. The medium of  claim 26 , wherein changing the display of at least one of the graphically depicted media mixing controls comprises changing the position of the at least one of the graphically depicted media mixing controls with respect the mixing console GUI. 
     
     
       30. The medium of  claim 23 , wherein the one or more programs further comprise instructions, which when executed by the device, cause the device to:
 detect a touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI; and 
 in response to detecting the touch input on the multi-point touch screen corresponding to a request to replace the display of the mixing console GUI with display of a keyboard GUI, replacing the display of the mixing console GUI with display of a keyboard GUI comprising an array of a plurality of graphically depicted keys on the multi-point touch screen. 
 
     
     
       31. The medium of  claim 23 , wherein one or more surface guides are provided on the multi-point touch screen to assist with the plurality of simultaneous touch inputs. 
     
     
       32. The medium of  claim 31 , wherein configurations of the one or more surface guides correspond to configurations of the graphically depicted media mixing controls including graphically depicted sliders, buttons, and/or dials. 
     
     
       33. The medium of  claim 31 , wherein an overlay sheet is provided on the multi-point touch screen, the one or more surface guides are provided on a front surface of the overlay sheet, and a back surface of the overlay sheet is affixed to the multi-point touch screen.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims benefit of U.S. Provisional Patent Application No. 60/592,483, filed Jul. 30, 2004, entitled “TOUCH SENSITIVE TECHNIQUES AND INPUT DEVICES,” which is hereby incorporated herein by reference. 
     This application is also related to: (i) U.S. patent application Ser. No. 10/840,862, filed May 6, 2004, entitled “MULTIPOINT TOUCHSCREEN,” which is hereby incorporated herein by reference; (ii) U.S. patent application Ser. No. 10/903,964, filed Jul. 30, 2004, entitled “GESTURES FOR TOUCH SENSITIVE INPUT DEVICES,” which is hereby incorporated herein by reference; and (iii) U.S. patent application Ser. No. 10/654,108, filed Sep. 2, 2003, entitled “AMBIDEXTROUS MOUSE,” which is hereby incorporated herein by reference. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to electronic devices and, more particularly, to touch-sensitive electronic devices. 
     2. Description of the Related Art 
     Mixing boards or consoles are known and used in sound production. The mixing boards or consoles are dedicated devices that primarily amplify and equalize signals on various channels. A mixing board or console can, for example, support a large number of channels, such as 24 or 32 channels. Users are often intimidated by the many knobs, switches and sliders that are compactly provided on the mixing board or console. 
     Traditionally, the knobs, switches and sliders are manually manipulated by the user of the mixing board or console. However, more recently, mixing boards or consoles have permitted some of its controls (e.g., faders, equalizers) to be automatically adjusted, such as during a playback. Furthermore, in recent years, mixing board or consoles have become more digital, such as by including a small display screen in the mixing board or console. Also, in recent years, software programs that execute on general purpose computer, such as GarageBand™ and Logic Pro 6™ from Apple Computer, Inc., permit mixing capabilities for sound productions. 
     Unfortunately, however, these conventional approaches do not fully satisfy user needs. For example, the traditional mixing consoles tend to be fixed and inflexible. As another example, the software applications tend to provide greater flexibility but have limitations with respect to user interaction. Thus, there is a need for improved approaches to providing digital mixing boards or consoles. 
     SUMMARY OF THE INVENTION 
     Broadly speaking, the invention relates to an electronic apparatus for media applications, such as an electronic mixing apparatus or an electronic keyboard apparatus, as well as associated methods for use of the electronic apparatus. 
     According to one aspect of the invention, the electronic mixing apparatus is provided on a touch screen that provides user input and display capabilities. In one embodiment, the touch screen is a multipoint touch screen so that multiple user touch inputs can be simultaneously acquired. The touch screen can display a Graphical User Interface (GUI) having mixing controls that can be selected and manipulated through user touch inputs with the touch screen. In one embodiment, the mixing controls being displayed with the GUI can be modified in real time as a user provides touch inputs with the touch screen. 
     Another aspect of the invention pertains to surface guides that are provided on the touch screen to assist with user input. The surface guides can be a permanent part of the touch screen or a removable part of the touch screen. In one embodiment, the surface guides can be part of an overlay sheet that can be affixed to the touch screen. 
     Still another aspect of the invention pertains to an electronic keyboard apparatus. The electronic keyboard apparatus (e.g., virtual keyboard) is provided on a touch-sensitive apparatus capable of simultaneously acquiring multiple user touch inputs. 
     The invention can be implemented in numerous ways, including as a method, system, device, apparatus, or computer readable medium. Several embodiments of the invention are discussed below. 
     As a touch-sensitive apparatus operating as a media mixer, one embodiment of the invention includes at least: a touch screen having a display area that also operates as a touch input area; and a plurality of media mixing controls being displayed in the display area of the touch screen and being interactive with a user through interaction with the touch input area of the touch screen. 
     As a method for operating a computing device having a touch screen, one embodiment of the invention includes at least the acts of: displaying a mixing console Graphical User Interface (GUI) having a plurality of GUI objects on the touch screen; determining whether at least one touch input has been detected; identifying the one or more GUI objects that are associated with the at least one touch input; determining modifications to the one or more identified GUI objects based on the at least one touch input; updating the one or more identified GUI objects of the mixing console GUI to reflect the modifications; and determining input data based on the at least one touch input and the one or more identified GUI objects. 
     As an electronic media mixer, one embodiment of the invention includes at least: a multi-touch touch screen capable of concurrently receiving multiple touch inputs; and a computing device operatively connected to the multi-touch touch screen. The computing device is configured to: display a mixing console Graphical User Interface (GUI) having a plurality of GUI objects on the multi-touch touch screen; determine whether at least one touch input has been detected; identify the one or more GUI objects that are associated with the at least one touch input; determine modifications to the one or more identified GUI objects based on the at least one touch input; update the one or more identified GUI objects of the mixing console GUI to reflect the modifications; and determine input data based on the at least one touch input and the one or more identified GUI objects. 
     As a touch-sensitive apparatus operating as an electronic keyboard, one embodiment of the invention includes at least: a touch screen having a display area that also operates as a touch input area; and a plurality of keyboard keys being displayed in the display area of the touch screen and being interactive with a user through interaction with the touch input area of the touch screen. 
     As a method for operating a computing device having a touch screen, one embodiment of the invention includes at least the acts of: displaying a keyboard Graphical User Interface (GUI) having a plurality of keys on the touch screen; determining whether at least one touch input has been detected; identifying the one or more keys that are associated with the at least one touch input; determining modifications to the one or more identified keys based on the at least one touch input; updating the one or more identified keys of the keyboard GUI to reflect the modifications; and determining input data based on the at least one touch input and the one or more identified keys. 
     Other aspects and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings which illustrate, by way of example, the principles of the invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which: 
         FIG. 1  is a diagram of a computing device according to one embodiment of the invention. 
         FIG. 2A  is a diagram of an electronic mixing console according to one embodiment of the invention. 
         FIGS. 2B-2D  are diagrams illustrating operations of a slider according to one embodiment of the invention. 
         FIGS. 2E-2G  are diagrams illustrating operations of a dial according to one embodiment of the invention. 
         FIG. 3A  is a flow diagram of a mixing console process according to one embodiment of the invention. 
         FIG. 3B  is a flow diagram of a keyboard process according to one embodiment of the invention. 
         FIG. 4  is a multipoint touch method according to one embodiment of the invention. 
         FIG. 5  is a block diagram of a computer system according to one embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The invention relates to an electronic apparatus for media applications, such as an electronic mixing apparatus or an electronic keyboard apparatus, as well as associated methods for use of the electronic apparatus. 
     According to one aspect of the invention, the electronic mixing apparatus is provided on a touch screen that provides user input and display capabilities. In one embodiment, the touch screen is a multipoint touch screen so that multiple user touch inputs can be simultaneously acquired. The touch screen can display a Graphical User Interface (GUI) having mixing controls that can be selected and manipulated through user touch inputs with the touch screen. In one embodiment, the mixing controls being displayed with the GUI can be modified in real time as a user provides touch inputs with the touch screen. 
     Another aspect of the invention pertains to surface guides that are provided on the touch screen to assist with user input. The surface guides can be a permanent part of the touch screen or a removable part of the touch screen. In one embodiment, the surface guides can be part of an overlay sheet that can be affixed to the touch screen. 
     Still another aspect of the invention pertains to an electronic keyboard apparatus. The electronic keyboard apparatus (e.g., virtual keyboard) is provided on a touch-sensitive apparatus capable of simultaneously acquiring multiple user touch inputs. 
     Embodiments of the invention are discussed below with reference to  FIGS. 1-5 . However, those skilled in the art will readily appreciate that the detailed description given herein with respect to these figures is for explanatory purposes as the invention extends beyond these limited embodiments. 
       FIG. 1  is a diagram of a computing device  100  according to one embodiment of the invention. The computing device  100  has a housing  102  that contains components and circuitry for computer processing by the computing device  100 . In addition, the housing  102  has a touch screen  104 . The touch screen  104  not only serves as an output device, such as a display screen, but also serves as an input device by way of a touch-sensitive surface provided over the display device. In one embodiment, the touch screen  104  is a multi-point touch screen that is capable of simultaneously receiving multiple touch inputs. 
     The computing device  100  can be a special purpose computing device or a general purpose computing device. The size and configuration of the computing device  100  can also vary. As an example, the computing device  100  can pertain to a tablet computer which integrally combines computer components with a touch screen into a tablet-like form factor. 
     In any case, when the computing device  100  is operating, the touch screen  104  can display media mixing controls. For example, the computing device  100  can execute a media mixing software application that provides media mixing functionality to the computing device  100 . The media mixing controls are utilized to facilitate user input of media mixing operations that are supported by the media mixing software application. Hence, the computing device  100  can be a central component of a media mixing system. The media mixing system would provide media inputs to the computing device  100  and would receive media outputs provided by the computing device  100 . The touch screen  104  of the computing device  100  displays the media mixing controls on the touch screen  104 . Hence, the media mixing controls are virtual controls that are graphically displayed by the touch screen  104  but are not physically present. Nevertheless, a user of the computing device  100  can interact with the media mixing controls to effectuate media mixing operations. 
     More particularly, as shown in  FIG. 1 , the media mixing controls displayed on the touch screen  104  can include a plurality of dials  106 , a plurality of buttons  108 , a plurality of status indicators  110 , and sliders  112 . Using the touch screen  104 , a user could interact with one of the dials  106  so as to effectively rotate the dial to provide a media mixing operation. For example, the dial  106  being rotated could pertain to an equalizer level, a filter cut frequency, a trim amount, a pan amount, etc. Additionally, the user can interact with the touch screen  104  to select or deselect one of the buttons  108 . For example, the buttons  108  can correspond to channel selection, mute, polarity, etc. The status indicators  110  can indicate status of an aspect of media mixing. For example, the status indicators  110  can indicate power-on, button selected, channel utilized, etc. The sliders  112  can be selected by the user and then slid up or down. For example, the sliders  112  can be used as faders, such as for volume or signal level control. 
     As the media mixing controls are manipulated by the user through interaction with the touch screen  104 , the media mixing controls being displayed can be modified in real time. For example, as the user interacts with the dial  106  via the touch screen  104 , the graphics associated with the dial  106  can be modified to indicate that the dial is effectively rotating. Similarly, by selection of one of the buttons  108 , the graphics associated with the button  108  on the touch screen  104  can indicate its selection. As another example, the user manipulates the slider  112  and the graphics associated with the slider  112  can be modified as the slider  112  is manipulated. Hence, as the media mixing controls displayed by the touch screen  104  are manipulated a user, the user is essentially given immediate media feedback as to the manner by which the media mixing controls are being manipulated. 
     Still further, since the touch screen  104  is capable of multi-point touch sensing, a user (or a plurality of users) can simultaneously interact with more than one of the media mixing controls at any given point in time. For example, the user may concurrently manipulate one or more dials  108 , buttons  108  or sliders  112 , or any combination thereof. The multi-point touch screen  104  can simultaneously sense these different interactions with different media mixing controls so that the appropriate media mixing controls are able to not only be concurrently sensed but also have their graphical display modified. 
       FIG. 2A  is a diagram of an electronic mixing console  200  according to one embodiment of the invention. The electronic mixing console  200  includes a computing device  202  that has a housing  204 . The housing  204  contains the circuitry and components for the computing device  202 . The housing  204  also includes a touch screen  206  that provides a screen for output information as well as a touch-sensitive area for data input. The touch screen  206  of the computing device  202  can display media mixing controls, such as sliders  208  (e.g., faders), buttons  210  and dials  212 . These media mixing controls are not physically presented on the computing device  202  but are graphically depicted by the touch screen  206 . 
     In addition, the electronic mixing console  200  includes an overlay  214 . The overlay  214  is designed to be placed over the touch screen  206  of the computing device  202 . The overlay  214  is provided on the computing device  202  so as to render the computing device  202  more applicable to use as a mixing console. In this regard, the overlay  214  includes various surface guides. For example, as shown in  FIG. 2A , the surface guides can include slider guides  216 , button guides  218  and dial guides  220 . The slider guides  216  assist the user with interacting with the sliders  208 . The button guides  218  assist the user with interaction with the buttons  210 . The dial guides  220  assist the user with interacting with the dials  212 . 
     The overlay  214  can be formed from a variety of different materials. In one embodiment, the overlay  214  is a translucent, thin plastic sheet having the surface guides on an upper surface thereof. A back surface of the overlay  214  can then be affixed adjacent to the touch screen  206  of the computing device. The overlay  214  can be attached or held against the computing device  202  in a variety of different ways. For example, the overlay  214  can be attached or held against the computing device  202  by clips, pins, tabs, adhesive, static attraction, vacuum (e.g., suction cups). As other examples, the computing device  202  could include grooves or slots for receiving the overlay  214  and holding the same in position. In another embodiment, the overlay  214  can be permanently affixed to the touch screen  206  of the computing device  202 . 
     The user&#39;s interaction with the touch screen  206  via the overlay  214  can be such that a user performs a gesture. For example, the user could utilize the dial guide  220  and perform a rotate gesture. The touch screen  206  of the computing device  202  would understand the touch gesture and cause the corresponding dial control  212  to be manipulated as well as to provide input data for processing by the computing device  202 . Additional details on gesture recognition are provided in U.S. patent application Ser. No. 10/840,862, which has been incorporated herein by reference. 
       FIGS. 2B-2D  are diagrams illustrating operations of a slider according to one embodiment of the invention. The slider is, for example, suitable for use as one of the sliders shown in  FIG. 1  or  FIG. 2A . In  FIG. 2B , a slider includes a slider bar  250  depicted on a touch screen  252 . A slider control  254  is provided on the slider bar  250 . The user can interact and manipulate the position of the slider control  254  with respect to the slider bar  250 .  FIG. 2C  illustrates the slider control  254  towards the top of the slider bar  250 .  FIG. 2D  illustrates a portion of an overlay  256  having a guide groove  258  aligned over the slide bar  250 . The guide groove  258  assists a user with interacting with the slider bar  250  by maintaining alignment with respect to the slider bar  250 . Hence, the user can manipulate the slider bar  250  without having to visually coordinate one&#39;s finger or stylus position over the slider bar  250 . 
       FIGS. 2E-2G  are diagrams illustrating operations of a dial  270  according to one embodiment of the invention. The dial  270  is, for example, suitable for use as one of the dials shown in  FIG. 1  or  FIG. 2A . In  FIG. 2E , the dial  270  is depicted on a touch screen  272 . A position indicator  274  is provided on the dial  270 . The user can interact and manipulate the position of the position indicator  274  with respect to the dial  270  by rotating the dial  270  through a rotational action or gesture with respect to the touch screen  272  at the position of the dial  270 .  FIG. 2E  illustrates the position indicator  274  at an upward position, and  FIG. 2F  illustrates the position indicator  274  at a side position. Hence, the user can rotate the dial  270 , through a rotational action or gesture with respect to the touch screen  272 , such that the position indicator  274  moves from the upward position to the side position.  FIG. 2G  illustrates a portion of an overlay  276  having a dial guide  278  aligned over the dial  270 . The dial guide  278  assists a user with interacting with the dial  270  by maintaining alignment with respect to the dial  270 . Hence, the user can manipulate the dial  270  (e.g., a rotational action or gesture) without having to visually coordinate one&#39;s finger or stylus position over the dial  270 . 
       FIG. 3A  is a flow diagram of a mixing console process  300  according to one embodiment of the invention. The mixing console process  300  is, for example, performed by the computing device  100  illustrated in  FIG. 1  or the computing device  202  illustrated in  FIG. 2A . 
     The mixing console process  300  initially displays  302  a mixing console Graphical User Interface (GUI) on a touch screen of a computing device. For example, in  FIG. 1 , various media mixing controls associated with a mixing console GUI are displayed on the touch screen  104  of the computing device  100 . In one embodiment, a user can configure (e.g., move, arrange, or re-arrange) the various media mixing controls on the mixing console GUI. For example, the configuration adjustments can change the number of particular media mixing controls and their position with respect to the mixing console GUI. 
     After the mixing console GUI is displayed  302 , a decision  304  determines whether one or more touch inputs have been detected. When the decision  304  determines that no touch inputs have been detected, the mixing console process  300  awaits such input. 
     Once the decision  304  determines that one or more touch inputs have been detected, the mixing console process  300  continues. Namely, the GUI object or objects associated with the one or more touch inputs are identified  306 . Then, modifications to the identified GUI objects are determined  308  based on the touch inputs. Then, the identified GUI objects of the mixing console GUI are updated  310  to reflect the modifications. In addition, input data is determined  312  based on the touch inputs and the identified GUI objects. The operations  306 - 312  are rapidly performed at approximately the same time such that once a touch input is detected, the mixing console process  300  is able to not only determine  312  the input data but also update  310  the identified GUI objects being modified. Hence, the user is given the impression that the system is responsive to their input in an essentially real time manner. 
     It should also be noted that after the input data has been determined  312 , the mixing console process  300  returns to repeat the decision  304  and subsequent blocks so that additional touch inputs can be similarly processed. However, it should be noted that once the input data has been determined  312 , the computing device will operate to process the input data. For example, the input data might indicate that the rotation of a dial by the user is to cause the output volume on a given channel to be increased by 10%. 
       FIG. 3B  is a flow diagram of a keyboard process  300  according to one embodiment of the invention. The keyboard process  300  is, for example, performed by the computing device  100  illustrated in  FIG. 1  or the computing device  202  illustrated in  FIG. 2A . 
     The keyboard process  300  initially displays  352  a keyboard graphical user interface (GUI) on a touch screen of a computing device. For example, the touch screen  104  of the computing device  100  can display a virtual keyboard having a plurality of keys, instead of the various media mixing controls associated with a mixing console GUI as shown in  FIG. 1 . After the keyboard GUI is displayed  352 , a decision  354  determines whether one or more touch inputs have been detected. When the decision  354  determines that no touch inputs have been detected, the keyboard process  350  awaits such input. 
     Once the decision  354  determines that one or more touch inputs have been detected, the keyboard process  350  continues. Namely, the one or more keys associated with the one or more touch inputs are identified  356 . Then, modifications to the identified keys are determined  358  based on the touch inputs. Then, the identified keys of the keyboard GUI are updated  360  to reflect the modifications. In addition, input data is determined  362  based on the touch inputs and the identified keys. The operations  356 - 362  are rapidly performed at approximately the same time such that once a touch input is detected, the keyboard process  350  is able to not only determine  362  the input data but also update  360  the identified keys being modified. Hence, the user is given the impression that the system is responsive to their input in an essentially real time manner. 
     It should also be noted that after the input data has been determined  362 , the keyboard process  350  returns to repeat the decision  354  and subsequent blocks so that additional touch inputs can be similarly processed. However, it should be noted that once the input data has been determined  362 , the computing device will operate to process the input data. For example, the input data might indicate that a user has interacted with the keyboard GUI to play a chord and thus cause the computing device  100  to recognize such and perhaps output an audio sound pertaining to the chord. 
     In one embodiment, the input data being determined  362  can indicate which of the keys of the virtual keyboard are selected (e.g., pressed) by the user. Optionally, the input data can further indicate characteristics of the one or more touch inputs associated with the selection of such keys. For example, the characteristics can pertain to pressure and velocity. These characteristics can be used to characterize the one or more touch inputs, such as “hard” or “soft,” for example. These characteristics can then in turn affect the audio sound being output be the computing device  100 . In one embodiment, for a touch screen, pressure and velocity can be monitored through examination area and timing of touch inputs. For additional details, see U.S. patent application Ser. No. 10/840,862, which has been incorporated herein by reference; and U.S. patent application Ser. No. 10/654,108, which has been incorporated herein by reference. 
     In another embodiment, the virtual keyboard can also make use of surface guides to assist the user. Surface guides were discussed above with respect to a mixing console but can also assist users of virtual keyboards in understanding the position of the keys without having to look at the keyboard GUI. 
       FIG. 4  is a multipoint touch method  400  according to one embodiment of the invention. The method  400  generally begins at block  402  where multiple touches are received on the surface of the touch screen at the same time. This may, for example, be accomplished by placing multiple fingers on the surface of the touch screen. Following block  402 , the process flow proceeds to block  404  where each of the multiple touches is separately recognized by the touch screen. This may, for example, be accomplished by multipoint capacitance sensors located within the touch screen. Following block  404 , the process flow proceeds to block  406  where the touch data based on multiple touches is reported. The touch data may, for example, be reported to a host device such as a general purpose computer. 
       FIG. 5  is a block diagram of a computer system  500  according to one embodiment of the invention. The computer system  500  may correspond to personal computer systems such as desktops, laptops, tablets or handhelds. By way of example, the computer system  500  may correspond to any Apple or PC based computer system. The computer system may also correspond to public computer systems such as information kiosks, automated teller machines (ATM), point of sale machines (POS), industrial machines, gaming machines, arcade machines, vending machines, airline e-ticket terminals, restaurant reservation terminals, customer service stations, library terminals, learning devices, and the like. 
     As shown, the computer system  500  includes a processor  502  configured to execute instructions and to carry out operations associated with the computer system  500 . For example, using instructions retrieved from memory, the processor  502  may control the reception and manipulation of input and output data between components of the computing system  500 . The processor  502  can be a single-chip processor or can be implemented with multiple components. 
     In most cases, the processor  502  together with an operating system operates to execute computer code and produce and use data. The computer code and data may reside within a program storage block  504  that is operatively coupled to the processor  502 . Program storage block  504  generally provides a place to hold data that is being used by the computer system  500 . By way of example, the program storage block may include Read-Only Memory (ROM)  506 , Random-Access Memory (RAM)  508 , hard disk drive  510  and/or the like. The computer code and data could also reside on a removable storage medium and be loaded or installed onto the computer system when needed. Removable storage mediums include, for example, CD-ROM, PC-CARD, floppy disk, magnetic tape, and a network component. 
     The computer system  500  also includes an input/output (I/O) controller  512  that is operatively coupled to the processor  502 . The (I/O) controller  512  may be integrated with the processor  502  or it may be a separate component as shown. The I/O controller  512  is generally configured to control interactions with one or more I/O devices. The I/O controller  512  generally operates by exchanging data between the processor and the I/O devices that desire to communicate with the processor  502 . The I/O devices and the I/O controller  512  typically communicate through a data link  514 . The data link  514  may be a one way link or two way link. In some cases, the I/O devices may be connected to the I/O controller  512  through wired connections. In other cases, the I/O devices may be connected to the I/O controller  512  through wireless connections. By way of example, the data link  514  may correspond to PS/2, USB, FIREWIRE, IR, RF, Bluetooth or the like. 
     The computer system  500  also includes a display device  516  that is operatively coupled to the processor  502 . The processor  502  can drive the display device  516  or a separate display driver  525  can be used. The display device  516  may be a separate component (peripheral device) or it may be integrated with a base computer system to form a desktop computer (all in one machine), a laptop, handheld or tablet or the like. The display device  516  is configured to display a graphical user interface (GUI) including perhaps a pointer or cursor as well as other information to the user. By way of example, the display device  516  may be a monochrome display, color graphics adapter (CGA) display, enhanced graphics adapter (EGA) display, variable-graphics-array (VGA) display, super VGA display, liquid crystal display (e.g., active matrix, passive matrix and the like), cathode ray tube (CRT), plasma displays and the like. 
     The computer system  500  also includes a touch screen  518  that is operatively coupled to the processor  502 . The touch screen  518  is a transparent panel that is positioned in front of the display device  516 . The touch screen  518  may be integrated with the display device  516  or it may be a separate component. The touch screen  518  is configured to receive input from a user&#39;s touch and to send this information to the processor  502 . In most cases, the touch screen  518  recognizes touches and the position and magnitude of touches on its surface. The touch screen  518  reports the touches to the processor  502  and the processor  502  interprets the touches in accordance with its programming. For example, the processor  502  may initiate a task in accordance with a particular touch. 
     In accordance with one embodiment, the touch screen  518  is capable of tracking multiple objects, which rest on, tap on, or move across the touch sensitive surface of the touch screen at the same time. The multiple objects may for example correspond to fingers and palms. Because the touch screen is capable of tracking multiple objects, a user may perform several touch-initiated tasks at the same time. For example, the user may select an onscreen button with one finger, while moving a cursor with another finger. In addition, a user may move a scroll bar with one finger while selecting an item from a menu with another finger. Furthermore, a first object may be dragged with one finger while a second object may be dragged with another finger. Moreover, gesturing may be performed with more than one finger. 
     To elaborate, the touch screen  518  generally includes a sensing device  520  configured to detect an object in close proximity thereto and/or the pressure exerted thereon. The sensing device  520  may be widely varied. In one particular embodiment, the sensing device  520  is divided into several independent and spatially distinct sensing points, nodes or regions  522  that are positioned throughout the touch screen  518 . The sensing points  522 , which are typically hidden from view, are dispersed about the touch screen  518  with each sensing point  520  representing a different position on the surface of the touch screen  518  (or touch screen plane). The sensing points  522  may be positioned in a grid or a pixel array where each pixilated sensing point  522  is capable of generating a signal at the same time. In the simplest case, a signal is produced each time an object is positioned over a sensing point  522 . When an object is placed over multiple sensing points  522  or when the object is moved between or over multiple sensing points  522 , multiple signals are generated. 
     The number and configuration of the sensing points  522  may be widely varied. The number of sensing points  522  generally depends on the desired sensitivity as well as the desired transparency of the touch screen  518 . More nodes or sensing points generally increases sensitivity, but reduces transparency (and vice versa). With regard to configuration, the sensing points  522  generally map the touch screen plane into a coordinate system such as a Cartesian coordinate system, a Polar coordinate system, or some other coordinate system. When a Cartesian coordinate system is used (as shown), the sensing points  522  typically correspond to x and y coordinates. When a Polar coordinate system is used, the sensing points typically correspond to radial (r) and angular coordinates (θ). 
     The touch screen  518  may include a sensing circuit  524  that acquires the data from the sensing device  520  and that supplies the acquired data to the processor  502 . Alternatively, the processor  502  or a separate touch screen driver/interface  525  may include this functionality. In one embodiment, the sensing circuit  524  is configured to send raw data to the processor  502  so that the processor  502  processes the raw data. For example, the processor  502  receives data from the sensing circuit  524  and then determines how the data is to be used within the computer system  500 . The data may include the coordinates of each sensing point  522  as well as the pressure exerted on each sensing point  522 . In another embodiment, the sensing circuit  524  is configured to process the raw data itself. That is, the sensing circuit  524  reads the pulses from the sensing points  522  and turns them into data that the processor  502  can understand. The sensing circuit  524  may perform filtering and/or conversion processes. Filtering processes are typically implemented to reduce a busy data stream so that the processor  502  is not overloaded with redundant or non-essential data. The conversion processes may be implemented to adjust the raw data before sending or reporting them to the processor  502 . The conversions may include determining the center point for each touch region (e.g., centroid). 
     The sensing circuit  524  may include a storage element for storing a touch screen program, which is capable of controlling different aspects of the touch screen  518 . For example, the touch screen program may contain what value(s) to output based on the sensing points  522  selected (e.g., coordinates). In fact, the sensing circuit in conjunction with the touch screen program may follow a predetermined communication protocol. As is generally well known, communication protocols are a set of rules and procedures for exchanging data between two devices. Communication protocols typically transmit information in data blocks or packets that contain the data to be transmitted, the data required to direct the packet to its destination, and the data that corrects errors that occur along the way. By way of example, the sensing circuit may place the data in a HID format (Human Interface Device). 
     The sensing circuit  524  generally includes one or more microcontrollers, each of which monitors one or more sensing points  522 . The microcontrollers may, for example, correspond to an Application Specific Integrated Circuit (ASIC), which works with firmware to monitor the signals from the sensing device  520  and to process the monitored signals and to report this information to the processor  502 . 
     In accordance with one embodiment, the sensing device  524  is based on capacitance. As should be appreciated, whenever two electrically conductive members come close to one another without actually touching, their electric fields interact to form capacitance. In most cases, the first electrically conductive member is a sensing point  522  and the second electrically conductive member is an object  526  such as a finger. As the object  526  approaches the surface of the touch screen  518 , a tiny capacitance forms between the object  526  and the sensing points  522  in close proximity to the object  526 . By detecting changes in capacitance at each of the sensing points  522  and noting the position of the sensing points, the sensing circuit can recognize multiple objects, and determine the location, pressure, direction, speed and acceleration of the objects  80  as they are moved across the touch screen  70 . For example, the sensing circuit can determine when and where each of the fingers and palm of one or more hands are touching as well as the pressure being exerted by the finger and palm of the hand(s) at the same time. 
     The simplicity of capacitance allows for a great deal of flexibility in design and construction of the sensing device  520 . By way of example, the sensing device  520  may be based on self capacitance or mutual capacitance. In self capacitance, each of the sensing points  522  is provided by an individually charged electrode. As an object approaches the surface of the touch screen  518 , the object capacitive couples to those electrodes in close proximity to the object thereby stealing charge away from the electrodes. The amount of charge in each of the electrodes is measured by the sensing circuit  524  to determine the positions of multiple objects when they touch the touch screen  518 . In mutual capacitance, the sensing device  520  includes a two layer grid of spatially separated lines or wires. In the simplest case, the upper layer includes lines in rows while the lower layer includes lines in columns (e.g., orthogonal). The sensing points  522  are provided at the intersections of the rows and columns. During operation, the rows are charged and the charge capacitively couples to the columns at the intersection. As an object approaches the surface of the touch screen, the object capacitive couples to the rows at the intersections in close proximity to the object thereby stealing charge away from the rows and therefore the columns as well. The amount of charge in each of the columns is measured by the sensing circuit  524  to determine the positions of multiple objects when they touch the touch screen  518 . 
     The various aspects, embodiments, implementations or features of the invention can be used separately or in any combination. 
     The invention is preferably implemented by hardware, software or a combination of hardware and software. The software can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of the computer readable medium include read-only memory, random-access memory, CD-ROMs, DVDs, magnetic tape, optical data storage devices, and carrier waves. The computer readable medium can also be distributed over network-coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. 
     The advantages of the invention are numerous. Different aspects, embodiments or implementations may yield one or more of the following advantages. One advantage of the invention is that an electronic mixer for media items can be provided using a computing device having a touch screen. Another advantage of the invention is that an electronic mixer with complete capabilities can be provided without having any mechanical moving parts. Still another advantage of the invention is that an electronic keyboard for media items can be provided using a computing device having a touch screen. Yet still another advantage of the invention is that a computing device having a touch screen can be configured for multiple purposes, one of which can be as an electronic mixer and another of which can be as an electronic keyboard. 
     The many features and advantages of the present invention are apparent from the written description and, thus, it is intended by the appended claims to cover all such features and advantages of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, the invention should not be limited to the exact construction and operation as illustrated and described. Hence, all suitable modifications and equivalents may be resorted to as falling within the scope of the invention.

Metadata:
Filing Date: 20041217
Publication Date: 20150505
Grant Date: 20150505
Priority Date: 20030902
Inventors: LENGELING GERHARD
RUBINSTEIN JONATHAN JAKE
Assignee: APPLE INC
CPC Classifications: [{"code": "H04H60/04", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04886", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04847", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04847", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0393", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04808", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/03548", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04847", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F3/0393", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04845", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04809", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F2203/04809", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F2203/04808", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/04842", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04H60/04", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04H60/04", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04104", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04886", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04886", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/04883", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 35731595