PATENT DOCUMENT

Publication Number: US-9921684-B2
Application Number: US-201615144615-A
Country: US
Kind Code: B2

Title: Intelligent stylus

Abstract:
An intelligent stylus is disclosed. The stylus can provide a stylus condition in addition to a touch input. The stylus architecture can include multiple sensors to sense information indicative of the stylus condition, a microcontroller to determine the stylus condition based on the sensed information, and a transmitter to transmit the determined condition to a corresponding touch sensitive device so as to cause some action based on the condition.

Claims:
What is claimed is: 
     
       1. An input device comprising:
 a camera configured to capture one or more images; 
 a motion sensor; 
 an orientation sensor; and 
 a controller configured to:
 receive the one or more captured images, the motion data, and the orientation data; 
 detect that the input device is not proximate to a touch screen of an associated host device; and 
 in response to detecting that the input device is not proximate to the touch screen of the associated host device, transmit information associated with one or more of the captured one or more images, the motion data, and the orientation data so as to cause a writing action at the associated host device. 
 
 
     
     
       2. The input device of  claim 1 , further comprising an identification sensor configured to detect a code associated with the input device, wherein the transmitter is further configured to transmit the code so as to cause an authentication of the input device. 
     
     
       3. The input device of  claim 1 , further comprising a light emitter configured to emit light, and transmit to the controller an indication the light emitter emits light. 
     
     
       4. The input device of  claim 1 , wherein the one or more captured images comprise one of video footage and one or more still images. 
     
     
       5. A method of operating an input device, the method comprising:
 capturing, with a camera of the input device, one or more images; 
 capturing, with a motion sensor of the input device, motion data; 
 capturing, with an orientation sensor of the input device, orientation data; 
 receiving the one or more captured images, the motion data, and the orientation data at a controller; 
 detecting that the input device is not proximate to a touch screen of an associated host device; and 
 in response to detecting that the input device is not proximate to the touch screen of the associated host device, transmitting information associated with one or more of the captured one or more images, the motion data, and the orientation data so as to cause a writing action at the associated host device. 
 
     
     
       6. The method of  claim 5 , wherein the input device further comprises an identification sensor configured to detect a code associated with the input device, and the transmitter is further configured to transmit the code so as to cause an authentication of the input device. 
     
     
       7. The method of  claim 5 , further comprising:
 emitting light with a light emitter; and 
 transmitting to the controller an indication that the light emitter emits light. 
 
     
     
       8. The method of  claim 5 , wherein the one or more captured images comprise one of video footage and one or more still images. 
     
     
       9. A non-transitory computer readable storage medium storing executable program instructions, which, when executed by an input device with a controller, cause the input device to:
 capture, with a camera of the input device, one or more images; 
 capture, with a motion sensor of the input device, motion data; 
 capture, with an orientation sensor of the input device, orientation data; 
 receive the one or more captured images, the motion data, and the orientation data at the controller; 
 detect that the input device is not proximate to a touch screen of an associated host device; and 
 in response to detecting that the input device is not proximate to the touch screen of the associated host device, transmit information associated with one or more of the captured one or more images, the motion data, and the orientation data so as to cause a writing cation at the associated host device. 
 
     
     
       10. The non-transitory computer readable storage medium of  claim 9 , wherein the input device further comprises an identification sensor configured to detect a code associated with the input device, and the transmitter is further configured to transmit the code so as to cause an authentication of the input device. 
     
     
       11. The non-transitory computer readable storage medium of  claim 9 , wherein the program instructions further cause the input device to:
 emit light with a light emitter; and 
 transmit to the controller an indication that the light emitter emits light. 
 
     
     
       12. The non-transitory computer readable storage medium of  claim 9 , wherein the one or more captured images comprise one of video footage and one or more still images. 
     
     
       13. The input device of  claim 1 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises detecting that a tip of the input device is proximate to piece of paper. 
     
     
       14. The input device of  claim 1 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises extracting one or more of the input device&#39;s location, the input device&#39;s environment, the input device&#39;s motion, the input device&#39;s orientation, the host device&#39;s displayed information, the host device&#39;s position, the host device&#39;s motion, and the host device&#39;s orientation from the one or more captured images. 
     
     
       15. The method of  claim 5 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises detecting that a tip of the input device is proximate to piece of paper. 
     
     
       16. The method of  claim 5 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises extracting one or more of the input device&#39;s location, the input device&#39;s environment, the input device&#39;s motion, the input device&#39;s orientation, the host device&#39;s displayed information, the host device&#39;s position, the host device&#39;s motion, and the host device&#39;s orientation from the one or more captured images. 
     
     
       17. The non-transitory computer readable storage medium of  claim 9 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises detecting that a tip of the input device is proximate to piece of paper. 
     
     
       18. The non-transitory computer readable storage medium of  claim 9 , wherein detecting that the input device is not proximate to the touch screen of the associated host device comprises detecting that a tip of the input device is proximate to piece of paper.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 13/166,711, filed Jun. 22, 2011 and published on Dec. 27, 2012 as U.S. Publication No. 2012/0331546, the contents of which are incorporated by reference herein in their entirety for all intended purposes. 
    
    
     FIELD 
     This relates generally to touch sensing and more particularly, to providing an intelligent stylus for use with a touch sensitive device. 
     BACKGROUND 
     Many types of input devices are available for performing operations in a computing system, such as buttons or keys, mice, trackballs, touch sensor panels, joysticks, touch pads, touch screens, and the like. Touch sensitive devices, and touch screens, in particular, are becoming increasingly popular because of their ease and versatility of operation as well as their declining price. Touch sensitive devices can include a touch sensor panel, which can be a clear panel with a touch sensitive surface, and a display device such as a liquid crystal display (LCD) that can be positioned partially or fully behind the panel, or integrated with the panel, so that the touch sensitive surface can substantially cover the viewable area of the display device. Touch sensitive devices can generally allow a user to perform various functions by touching or hovering over the touch sensor panel using one or more fingers, a stylus or other object at a location often dictated by a user interface (UI) including virtual buttons, keys, bars, displays, and other elements, being displayed by the display device. In general, touch screens can recognize a touch event and the position of the touch event on the touch sensor panel or a hover event and the position of the hover event on the touch sensor panel, and the computing system can then interpret the touch or hover event in accordance with the display appearing at the time of the event, and thereafter can perform one or more operations based on the event. 
     When a stylus has been used as an input device, the stylus has traditionally provided simply a touch input to a touch sensitive device without additional information that can be helpful to operation of the touch sensitive device. 
     SUMMARY 
     This relates to an intelligent stylus that can provide a stylus condition in addition to a touch input. The stylus sensing circuitry can include multiple sensors to sense information indicative of the stylus condition, a microcontroller to determine the stylus condition based on the sensed information, and a transmitter to transmit the determined condition to a corresponding touch sensitive device to cause some action based on the condition. The stylus&#39;s ability to determine its condition and provide that condition to the touch sensitive device advantageously improves touch and hover sensing and increases device capabilities. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an exemplary intelligent stylus for use with a touch sensitive device according to various embodiments. 
         FIG. 2  illustrates exemplary sensing circuitry of an intelligent stylus according to various embodiments. 
         FIG. 3  illustrates an exemplary intelligent stylus with a replaceable tip according to various embodiments. 
         FIG. 4  illustrates an exemplary intelligent stylus with sensing circuitry including a pressure sensor according to various embodiments. 
         FIG. 5  illustrates an exemplary intelligent stylus with sensing circuitry including a surrounding touch sensor according to various embodiments. 
         FIG. 6  illustrates an exemplary intelligent stylus with sensing circuitry including a rotation sensor according to various embodiments. 
         FIG. 7  illustrates an exemplary intelligent stylus with sensing circuitry including a pushbutton according to various embodiments. 
         FIG. 8  illustrates an exemplary intelligent stylus with sensing circuitry including a camera according to various embodiments. 
         FIG. 9  illustrates an exemplary intelligent stylus with sensing circuitry including a light emitter according to various embodiments. 
         FIG. 10  illustrates an exemplary intelligent stylus with sensing circuitry including contact sensors disposed on the stylus and on a removable stylus cap according to various embodiments. 
         FIG. 11  illustrates an exemplary bristle-tipped intelligent stylus with sensing circuitry including bristle sensors according to various embodiments. 
         FIG. 12  illustrates an exemplary intelligent stylus with sensing circuitry including a color sensor according to various embodiments. 
         FIG. 13  illustrates an exemplary lead-tipped intelligent stylus with sensing circuitry according to various embodiments. 
         FIG. 14  illustrates an exemplary intelligent stylus with sensing circuitry including a microcontroller for selecting multiple sensors according to various embodiments. 
         FIG. 15  illustrates an exemplary method for detecting a condition of an intelligent stylus according to various embodiments. 
         FIG. 16  illustrates another exemplary method for detecting a condition of an intelligent stylus according to various embodiments. 
         FIG. 17  illustrates an exemplary computing system for use with an intelligent stylus according to various embodiments. 
         FIG. 18  illustrates an exemplary mobile telephone for use with an intelligent stylus according to various embodiments. 
         FIG. 19  illustrates an exemplary digital media player for use with an intelligent stylus according to various embodiments. 
         FIG. 20  illustrates an exemplary personal computer for use with an intelligent stylus according to various embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description of example embodiments, reference is made to the accompanying drawings in which it is shown by way of illustration specific embodiments that can be practiced. It is to be understood that other embodiments can be used and structural changes can be made without departing from the scope of the various embodiments. 
     This relates to an intelligent stylus that includes sensing circuitry to provide the stylus condition in addition to a touch input. The sensing circuitry can include multiple sensors to sense information indicative of the stylus condition, a microcontroller to determine the stylus condition based on the sensed information, and a transmitter to transmit the determined condition to a corresponding touch sensitive device to cause some action based on the condition. The stylus&#39;s ability to determine its condition and provide that condition to the touch sensitive device advantageously improves touch and hover sensing and increases panel capabilities. Additionally, the stylus can be used with the touch sensitive device to provide these advantages without making design modifications to the panel. 
     Although some embodiments are described herein in terms of a stylus, it is to be understood that other input devices and/or pointing devices can be used according to various embodiments. 
     Although some embodiments are described herein in terms of a touch sensitive device, it is to be understood that any device capable of sensing a touch or hover event thereat and/or processing the sensed event can be used according to various embodiments. The touch sensitive device (or any other suitable device) can include touch sensing circuitry for touch and hover sensing and, in some instances, a processor and memory for touch and hover data processing. 
       FIG. 1  illustrates an exemplary intelligent stylus for use with a touch sensitive device according to various embodiments. In the example of  FIG. 1 , touch sensitive device  120  can include an array of pixels  106  formed at the crossing points of conductive rows  101  and columns  102 . Though  FIG. 1  depicts the conductive elements  101 ,  102  in rows and columns, other configurations of conductive elements are also possible according to various embodiments. 
     When stylus  110  touches or hovers over a surface of the touch sensitive device  120 , the stylus can form a capacitance with one or more of the conductive rows  101  and/or columns  102  that can be detected by device sensing circuitry (not shown). The stylus touch can be represented in an image captured at the touch sensitive device  120  and processed for touch input information, e.g., the location on the touch sensitive device that the stylus touched or hovered over. 
     In addition to providing the touch input information, the stylus  110  can provide information sensed by the stylus regarding its condition, which can be used by the touch sensitive device  120  to perform some action. In some embodiments, the information can be used by the stylus to perform some action. The stylus  110  can include multiple sensors to provide information about its condition, where the sensors can be selectively used alone or in various combinations. The ability to provide information beyond simply touch input information, particularly information about the stylus&#39;s condition that the stylus determines itself, gives this stylus an intelligence absent from a traditional stylus. 
       FIG. 2  illustrates a cross-sectional view of an exemplary intelligent stylus having sensing circuitry according to various embodiments. In the example of  FIG. 2 , stylus  210  can include contact/proximity sensor  212 , motion/orientation sensor  214 , microcontroller (MCU)  222 , and transmitter  232 . The contact/proximity sensor  212  can sense a touch at the stylus tip by an object or a proximity of the stylus to an object. In some embodiments, the contact/proximity sensor  212  can be a capacitive sensor, a resistive sensor, a force sensor, or any other suitable sensor capable of sensing a touch at or proximity to the stylus. The motion/orientation sensor  214  can sense the motion and/or orientation of the stylus. In some embodiments, the motion/orientation sensor  214  can be an accelerometer, a gyroscope, an magnetometer, or any other suitable six degree-of-freedom sensor capable of sensing a motion and/or orientation of the stylus. 
     In some embodiments, an additional contact/proximity sensor can be disposed at the stylus non-tip end to sense a touch at that end of the stylus by an object or a proximity of that end of the stylus to an object. 
     The MCU  222  can receive, select, and process stylus sensor measurements to determine a condition of the stylus. For example, the MCU  222  can receive a contact/proximity measurement from the contact/proximity sensor  212  and determine the stylus&#39;s condition as touching (contacting) or hovering over (proximate to) a surface. Similarly, the MCU  222  can receive a motion/orientation measurement from the motion/orientation sensor  214  and determine the stylus&#39;s condition as moving in a particular direction at a particular pace. The MCU  222  can also receive a motion/orientation measurement and determine the stylus&#39;s condition as having a particular orientation. Similarly, the MCU  222  can receive both contact/proximity and motion/orientation measurements and determine the stylus&#39;s condition as moving on a surface. In some embodiments, the MCU  222  can be a single application specific integrated circuit (ASIC) that can include one or more programmable processors, random access memory (RAM), and input-output (I/O) ports. In some embodiments, the MCU  222  can also include data compression software and/or hardware. 
     For the additional contact/proximity sensor at the stylus non-tip end, the MCU can determine the stylus&#39;s condition at the non-tip end as touching or hovering over a surface, e.g., to emulate an eraser. 
     The transmitter  232  can transmit stylus information, e.g., the stylus&#39;s condition, from the MCU  222  to a touch sensitive device (or some other device in communication with the stylus) and cause the device to perform some action based on the transmitted information. For example, the transmitter  232  can transmit that the stylus is touching or hovering over the touch sensitive device, causing the device to perform some touch operation, such as displaying writing or drawings or executing a program corresponding to the location the stylus touched. The transmitter  232  can transmit that the stylus is moving at an angle along the touch sensitive device, causing the device to display a thicker line corresponding to the stylus angle and the stylus motion. The transmitter  232  can transmit that the stylus has rotated at least 180 degrees so that the eraser end of the stylus is touching the touch sensitive device, causing the device to erase the displayed information. The transmitter  232  can be wireless or wired. Transmission between the transmitter  232  and the touch sensitive device can be via WiFi, Bluetooth, zigbee, RFID, NFC, ultrasound, and the like. 
     In addition to a stylus, other input devices can transmit input information and/or condition information to the touch sensitive device, thereby causing an action based on the information, according to various embodiments. For example, a bar code scanner can transmit bar code data and/or a condition of the scanner to the touch sensitive device to cause the device and/or the scanner to perform some action based on the transmitted information. A camera can transmit images and videos and/or a camera condition to the touch sensitive device to cause the device and/or the camera to perform some action based on the transmitted information. An ultrasound device can transmit ultrasound data and/or a condition of the device to the touch sensitive device to cause the touch sensitive device and/or the ultrasound device to perform some action based on the transmitted information. An NFC/RFID reader can transmit product identification codes and/or a condition of the reader to the touch sensitive device to cause the device and/or the reader to perform some action based on the transmitted information. Other input devices performing similarly can be used according to various embodiments. Transmission between the input devices&#39; transmitters and the touch sensitive device can be via WiFi, Bluetooth, zigbee, RFID, NFC, ultrasound, and the like. 
     In some embodiments, the transmitter  232  can be replaced with a transceiver, such that the stylus can both send signals to the touch sensitive device and receive signals from the touch sensitive device. For example, after the touch sensitive device performs its action based on the information sent from the stylus, the device can send a signal back to the stylus indicating that the action is complete or requesting some action by the stylus in return. 
     In some embodiments, the motion/orientation sensor  214  can be used to sense a wake-up motion of the stylus  210 . For example, a user can move the stylus  210  in a predetermined motion to power up the stylus. Note that powering up can be from a low power mode to a full power mode and powering down can be the reverse. In some embodiments, the predetermined motion can be programmed into the stylus to be unique to the user. Such a motion can in effect be a stylus password. The motion/orientation sensor  214  can sense the motion and transmit it to the MCU  222  for processing. The MCU  222  can recognize the motion and power up the stylus. A similar process can be done for powering down the stylus  210 . The transmitter  232  can transmit that the stylus is powered either up or down to the touch sensitive device. 
     Although only two sensors are illustrated in  FIG. 2 , it is to be understood that additional and/or other sensors can be used according to the requirements of the stylus. Examples of additional sensors will be described below. The sensors can be selectively used in the stylus alone or in various combinations. Also, the sensing circuitry is not limited to the sensors, the MCU, and the transmitter illustrated here, but can include additional and/or other components capable of determining a condition of an intelligent stylus according to various embodiments. 
       FIG. 3  illustrates a cross-sectional view of an exemplary intelligent stylus with a replaceable tip according to various embodiments. In the example of  FIG. 3 , stylus  310  can include replaceable tip  302  having connector  342  embedded therein to connect to connector  343  in stylus body  304 . The tip connector  342  can have a unique identification code that identifies the particular tip  302 . The body connector  343  can have an identification sensor  313  to sense the code upon connecting with the tip connector  342 . The identification sensor  313  can transmit the code to the MCU for processing. The code can be used by the MCU to determine how the stylus will be used and can be used by the touch sensitive device to perform some action based on that use. For example, if the code indicates that the replaceable tip is a stylus brush, the MCU can determine that the stylus will be used as a brush and the touch sensitive device can perform an action according to brush motions and touches. 
     The body connector  343  can also have a unique identification code that identifies the stylus owner. The identification sensor  313  can transmit the owner code to the MCU for processing. The owner code can be used by the MCU to identify the owner and by the touch sensitive device to perform some action based on that owner. For example, if two stylus users interact with the touch sensitive device, the panel can differentiate between the two users&#39; inputs and use different colors, symbols, fonts, etc., or otherwise distinguish between the users and/or interact differently with each user according to the owner code. In some embodiments, the body connector  343  can be programmable, so that the current user can program in his/her unique identification code when using the stylus and delete the code when done. The MCU and the touch sensitive device can then operate based on that particular user&#39;s code. 
     The body connector  343  can also have a unique identification code that identifies the stylus itself. The identification sensor  313  can transmit the stylus code to the MCU, which can then transmit the stylus code to the touch sensitive device. The touch sensitive device can then perform some action based on the stylus code. For example, the touch sensitive device can authenticate the stylus as being an acceptable input device for the panel and can accept stylus input if the stylus is authenticated and deny stylus input if the stylus is not. The touch sensitive device can also display or speak the authentication result to the user. 
       FIG. 4  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a pressure sensor according to various embodiments. In the example of  FIG. 4 , stylus  410  can include pressure sensor  414 . The pressure sensor  414  can sense a force being applied by the stylus to a surface and can transmit a force measurement to the MCU for processing. The MCU can determine that the stylus is applying force to a surface and/or the amount of force being applied and can transmit that determination to the touch sensitive device. The touch sensitive device can then perform some action based on the applied force. For example, the touch sensitive device can display a darker, thicker image for a heavier force and a lighter, thinner image for a lighter force. 
     In some embodiments, the pressure sensor  414  can act in combination with the motion/orientation sensor  214  of  FIG. 2  to provide motion and pressure measurements to the MCU. The MCU can then determine the cadence of the stylus as a user signs with the stylus. The touch sensitive device can use the determined cadence to verify the user&#39;s signature. 
       FIG. 5  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a surrounding touch sensor according to various embodiments. In the example of  FIG. 5 , stylus  510  can include surrounding touch sensor  515 , which can encircle the stylus body at the stylus handheld position. The touch sensor  515  can sense a position of a hand holding the stylus and can transmit a touch measurement to the MCU for processing. The MCU can determine how the hand is holding the stylus and can transmit that determination to the touch sensitive device. The touch sensitive device can then perform some action based on the hand position. For example, the touch sensitive device can display or speak a message to the user that the hand position is correct, too low on the stylus, too light on the stylus and so on and how the hand position can be adjusted or changed. 
     Alternatively, the MCU can extract fingerprints from the touch measurement and can identify the user of the stylus. The MCU can transmit the user identification to the touch sensitive device. The touch sensitive device can then perform some action based on the identification. For example, the touch sensitive device can authenticate the user of the stylus and can accept stylus input if the user is authenticated and deny stylus input if the user is not. The touch sensitive device can also display or speak the authentication result to the user. 
       FIG. 6  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a rotation sensor according to various embodiments. In the example of  FIG. 6 , stylus  610  can include rotation sensor  616 . The rotation sensor  616  can sense a rotation of a portion of the stylus. Here the stylus body can be in two parts separated by a break  672 , so that the two parts can rotate independently. The rotation sensor  616  can sense when one or both parts rotate away from a default position and can transmit a rotation measurement to the MCU for processing. The MCU can determine a stylus mode or setting based on the rotation measurement and can transmit that determination to the touch sensitive device. The touch sensitive device can then perform some action based on the stylus mode or setting. For example, the touch sensitive device can know from the stylus that the stylus is in a color mode and can display any stylus inputs in color, or that the stylus is in an erase mode and can erase any displayed information where the stylus touches, and so on. 
     In some embodiments, the MCU can power up the stylus upon receipt of a rotation indication from the rotation sensor  616 . The MCU can then send a determination that the stylus is powered up to the touch sensitive device. Similar actions can be done for a power down of the stylus. Alternatively, the MCU can send the stylus&#39;s condition as rotated to the touch sensitive device and the touch sensitive device can then send a signal back to the stylus to either power up or power down. 
       FIG. 7  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a pushbutton according to various embodiments. In the example of  FIG. 7 , stylus  710  can include pushbutton  717 . The pushbutton  717  can sense a push thereon and can transmit the push indication to the MCU for processing. The MCU can determine a stylus setting change based on the push indication and can transmit that determination to the touch sensitive device. The touch sensitive device can then perform some action based on the stylus setting change. For example, the touch sensitive device can change the color of stylus writing displayed at the panel when the pushbutton is pushed. 
     In some embodiments, the MCU can power up the stylus upon receipt of a push indication from the pushbutton  717 . The MCU can then send a determination that the stylus is powered up to the touch sensitive device. Similar actions can be done for a power down of the stylus. Alternatively, the MCU can send the stylus&#39;s push indication to the touch sensitive device and the touch sensitive device can then send a signal back to the stylus to either power up or power down. 
       FIG. 8  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a camera according to various embodiments. In the example of  FIG. 8 , stylus  810  can include camera  852 . The camera  852  can capture images and/or video and transmit the captured images and/or video to the MCU for processing. The MCU can extract relevant information from the captured images and/or video about the stylus&#39;s location, environment, motion, orientation, and the like or about the touch sensitive device&#39;s displayed information, position, motion, orientation, and the like. The MCU can transmit the extracted information to the touch sensitive device, which can then perform some action based on that information. For example, the touch sensitive device can confirm its orientation and location based on information sent from the stylus. 
     The camera  852  can be mounted in a replaceable tip with a unique identification code that can be transmitted to the MCU to identify the camera, as described previously. The camera identification code can indicate to the MCU that the incoming information includes captured images and/or video for processing. 
       FIG. 9  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry that includes a light emitter according to various embodiments. In the example of  FIG. 9 , stylus  910  can include light emitter  962 . The light emitter  962  can emit light onto a proximate surface and can transmit an indication that light is being emitted to the MCU for processing. The MCU can determine that the stylus is acting as a light source and can transmit that determination to the touch sensitive device, which can then perform some action based on that determination. For example, suppose the touch sensitive device has optical sensors. The determination that the stylus is emitting light can be used by the touch sensitive device to detect the emission and perform some action based on the location relative to the panel, the duration, etc., of the detected emission. Alternatively, the touch sensitive device can know that the stylus is currently acting as a light source (e.g., a flashlight) and await the stylus&#39;s return to being an input device for the panel. 
     The light emitter  962  can be mounted in a replaceable tip with a unique identification code that can be transmitted to the MCU to identify the light emitter. The light emitter identification code can indicate to the MCU that the incoming information includes a light indication for processing. 
       FIG. 10  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry including contact sensors according to various embodiments. In the example of  FIG. 10 , stylus  1010  can include removable stylus cap  1071  with contact  1018  embedded therein. The stylus body can have corresponding contact  1018  embedded therein. When the cap  1071  covers the stylus tip, the contacts  1018  can be engaged and can transmit an indication to the MCU for processing. The MCU can determine that the cap covers the stylus and can transmit that determination to the touch sensitive device. The touch sensitive device can then perform some action based on that determination. For example, the touch sensitive device can know that the stylus is not in usage and can await an indication that the stylus cap has been removed so as to receive stylus input. 
     In some embodiments, the MCU can power up the stylus upon receipt of a indication from the contacts  1018  that the stylus cap has been removed. Conversely, the MCU can power down the stylus upon receipt of an indication from the contacts  1018  that the stylus cap has been placed on the stylus. The MCU can send a determination that the stylus is powered either up or down to the touch sensitive device. Alternatively, the MCU can send the stylus&#39;s cap condition to the touch sensitive device and the touch sensitive device can send a signal back to the stylus to either power up or power down. 
     In some embodiments, the contacts  1018  can act in combination with the motion/orientation sensor  214  of  FIG. 2  to provide contact and motion measurements to the MCU. The MCU can then use the measurements to determine whether the stylus cap covers the stylus and whether the stylus is moving. If the cap have been off the stylus for a particular amount of time and the stylus has not been moving, the MCU can determine that the stylus is not in use and can power down the stylus. Conversely, if the stylus is powered down and begins to move with the cap off, the MCU can power up the stylus. The MCU can send a determination that the stylus is powered either up or down to the touch sensitive device. Alternatively, the MCU can send the stylus&#39; cap and motion conditions to the touch sensitive device and the touch sensitive device can send a signal back to the stylus to either power up or power down. 
       FIG. 11  illustrates a cross-sectional view of an exemplary intelligent brush-tipped stylus with sensing circuitry including bristle sensors according to various embodiments. In the example of  FIG. 11 , stylus  1110  can include a brush tip having multiple bristles  1183 , each bristle having a bristle sensor  1119 .  FIG. 11  shows a plan view of the bristle sensors  1119 . The bristles sensors  1119  can detect force applied to a surface by the bristles  1183  and the deflection of the bristles on the surface. The bristle sensors  1119  can transmit force and deflection measurements to the MCU for processing. The MCU can determine that the stylus bristles are applying force to a surface and/or the amounts of force applied and that the stylus bristles are deflected on the surface and/or the amounts of deflection. The MCU can transmit the determination to the touch sensitive device, which can then perform some action based on the determination. For example, the touch sensitive device can display a pattern corresponding to the deflection pattern, where the thickness or darkness of the pattern corresponds to the amounts of force applied. 
     In some embodiments, the bristle sensors  1119  can act in combination with the motion/orientation sensors  214  of  FIG. 2  to provide force, deflection, and motion measurements. The MCU can then determine the brushing motion of the stylus. The touch sensitive device can use the determined brushing motion to fill in a displayed area, for example. 
     The brush can be mounted in a replaceable tip with a unique identification code that can be transmitted to the MCU to identify the brush. The brush identification code can indicate to the MCU that the incoming information includes bristle force and deflection measurements for processing 
       FIG. 12  illustrates a cross-sectional view of an exemplary intelligent stylus with sensing circuitry including a color sensor according to various embodiments. In the example of  FIG. 12 , stylus  1210  can include color sensor  1294 . The color sensor  1294  can detect a color at a proximate surface and transmit the detected color measurement to the MCU for processing. In some embodiments, the color sensor  1294  can be a colorimeter. The MCU can determine the color components from the measurement and transmit the determined components to the touch sensitive device, which can perform some action based on the components. For example, the touch sensitive device can display the detected color using the transmitted components. The touch sensitive device can also change the font colors or some other display feature to that of the transmitted components. The touch sensitive device can additionally store the color information for later use. 
     The color sensor  1294  can be mounted in a replaceable tip with a unique identification code that can be transmitted to the MCU to identify the sensor. The sensor identification code can indicate to the MCU that the incoming information includes color measurements for processing. 
       FIG. 13  illustrates a cross-sectional view of an exemplary intelligent lead-tipped stylus with sensing circuitry according to various embodiments. In the example of  FIG. 13 , stylus  1310  can include lead tip  1385  for writing on real paper. The lead tip  1385  can act in combination with motion/orientation sensor  1314  (same as sensor  214  of  FIG. 2 ) to provide motion measurements of the lead tip  1385  as it writes on the real paper. The motion measurements can be transmitted to the MCU for processing. The MCU can determine that the stylus is writing on real paper, based on the lead tip&#39;s identification code, and can transfer the determination along with the writing motions to the touch sensitive device, which can perform some action based on the measurements. For example, the touch sensitive device can reproduce the writing from the real paper electronically on the touch sensitive device. In some embodiments, the motion/orientation sensor  1314  can also provide orientation measurements of the lead tip  1385  as it writes on the real paper and transmit the orientation measurements to the MCU for processing. The MCU can determine a thickness of the writing based on the orientation and can transfer that determined thickness to the touch sensitive device. The touch sensitive device can then reproduce the writing from the real paper, including the thickness of the writing strokes, electronically on the touch sensitive device. 
     The lead can be mounted in a replaceable tip with a unique identification code that can be transmitted to the MCU to identify the lead tip. The lead tip identification code can indicate to the MCU that the incoming information includes writing for processing. 
     As described previously, the intelligent stylus can include multiple sensors that can be used alone or in various combinations to determine the stylus&#39;s condition. The stylus sensing circuitry can be scalable, such that the stylus can use none of the sensors and act as simply a touch input device for a minimal touch sensitive device or the stylus can use one or more of the sensors and act as an intelligent device for a maximal touch sensitive device having logic to interpret and process all the various combinations of stylus sensor information.  FIG. 14  illustrates exemplary scalable stylus sensing circuitry within a stylus according to various embodiments. In the example of  FIG. 14 , stylus  1410  can include multiplexer (MUX)  1498  (or any suitable switch) coupled to MCU  1422  and multiple sensors  1411  for selecting which sensor(s) to enable and transmit a measurement to the MCU for processing. The MCU  1422  can send control signal  1421  to the MUX  1498  to indicate which sensors  1411  to select. The selection can be made according to the capabilities of the touch sensitive device in communication with the stylus via transmitter  1432 . The selection can also be made according to the purpose of the stylus. 
       FIG. 15  illustrates an exemplary method for detecting a condition of an intelligent stylus according to various embodiments. In the example of  FIG. 15 , one or more sensors can selectively make a stylus measurement ( 1510 ). A stylus condition can be determined from the sensor measurements ( 1520 ). Examples of a stylus condition can include the stylus being in use, the stylus having a particular orientation, the stylus moving, the stylus making a brushing motion, the stylus being held, the stylus being used by a particular user, and so on. The determined condition can cause an action therefrom ( 1530 ). In some embodiments, the condition can be transmitted to a touch sensitive device, for example, causing the touch sensitive device to perform an action. The touch sensitive device can then display some information, update the display with new information, execute a function, or perform some other appropriate action based on the condition. Alternatively, the condition can be transmitted to a component of the stylus, such as the power supply, for example, causing the stylus component to perform an action. The stylus can then switch between low and full power or perform some other appropriate action based on the condition. 
       FIG. 16  illustrates another exemplary method for detecting a condition of an intelligent stylus according to various embodiments. In the example of  FIG. 16 , one or more stylus measurements can be provided ( 1610 ). The stylus measurements can be provided by the stylus itself and transmitted to a touch sensitive device for processing. One or more sensors can selectively make measurements corresponding to the stylus measurements ( 1620 ). The measurements can be made by the touch sensitive device, for example, in communication with the stylus. The measurements can be made coincident with the stylus measurements and based on the same operation(s) by the stylus and/or the touch sensitive device. The two sets of measurements can be correlated ( 1630 ). The correlation can include comparing the two sets of measurements and determining how close the two sets are. If close, the measurements can be considered correct. If not close, the measurements can be considered in error and can be discarded. If the correlation is favorable ( 1640 ), a stylus condition can be determined from the measurements ( 1650 ). The fact that the correlation is favorable can confirm that the subsequently determined stylus condition is valid. Examples of a stylus condition are described above regarding  FIG. 15 . An action can be performed based on the determined stylus condition ( 1660 ). The action can be performed by the touch sensitive device or by the stylus, for example. Example actions are described above regarding  FIG. 15 . In some embodiments, as part of the performed action, a signal can be transmitted from the touch sensitive device to the stylus to cause the stylus to perform an action based on the determined stylus condition. 
     In alternate embodiments, a stylus can use its stylus measurements to determine its condition and then provide the determined condition. A touch sensitive device can use its measurements to also determine the stylus condition. The touch sensitive device can then correlate the provided stylus condition with its determined stylus condition to confirm the condition. If the condition is confirmed, the touch sensitive device can then perform an associated action. The stylus can also perform an associated action. 
       FIG. 17  illustrates an exemplary computing system that can use an intelligent stylus according to various embodiments. In the example of  FIG. 17 , computing system  1700  can include touch controller  1706 . The touch controller  1706  can be a single application specific integrated circuit (ASIC) that can include one or more processor subsystems  1702 , which can include one or more main processors, such as ARM968 processors or other processors with similar functionality and capabilities. However, in other embodiments, the processor functionality can be implemented instead by dedicated logic, such as a state machine. The processor subsystems  1702  can also include peripherals (not shown) such as random access memory (RAM) or other types of memory or storage, watchdog timers and the like. The touch controller  1706  can also include receive section  1707  for receiving signals, such as touch (or sense) signals  1703  of one or more sense channels (not shown), other signals from other sensors such as sensor  1711 , etc. The touch controller  1706  can also include demodulation section  1709  such as a multistage vector demodulation engine, panel scan logic  1710 , and transmit section  1714  for transmitting stimulation signals  1716  to touch panel  1724  to drive the panel. The scan logic  1710  can access RAM  1712 , autonomously read data from the sense channels, and provide control for the sense channels. In addition, the scan logic  1710  can control the transmit section  1714  to generate the stimulation signals  1716  at various frequencies and phases that can be selectively applied to rows of the touch panel  1724 . 
     The touch controller  1706  can also include charge pump  1715 , which can be used to generate the supply voltage for the transmit section  1714 . The stimulation signals  1716  can have amplitudes higher than the maximum voltage by cascading two charge store devices, e.g., capacitors, together to form the charge pump  1715 . Therefore, the stimulus voltage can be higher (e.g., 6V) than the voltage level a single capacitor can handle (e.g., 3.6 V). Although  FIG. 17  shows the charge pump  1715  separate from the transmit section  1714 , the charge pump can be part of the transmit section. 
     Computing system  1700  can include host processor  1728  for receiving outputs from the processor subsystems  1702  and performing actions based on the outputs that can include, but are not limited to, moving an object such as a cursor or pointer, scrolling or panning, adjusting control settings, opening a file or document, viewing a menu, making a selection, executing instructions, operating a peripheral device coupled to the host device, answering a telephone call, placing a telephone call, terminating a telephone call, changing the volume or audio settings, storing information related to telephone communications such as addresses, frequently dialed numbers, received calls, missed calls, logging onto a computer or a computer network, permitting authorized individuals access to restricted areas of the computer or computer network, loading a user profile associated with a user&#39;s preferred arrangement of the computer desktop, permitting access to web content, launching a particular program, encrypting or decoding a message, and/or the like. The host processor  1728  can also perform additional functions that may not be related to touch processing, and can be connected to program storage  1732  and display device  1730  such as an LCD for providing a UI to a user of the device. Display device  1730  together with touch panel  1724 , when located partially or entirely under the touch panel, can form a touch screen. 
     Touch panel  1724  can include a capacitive sensing medium having drive lines and sense lines. It should be noted that the term “lines” can sometimes be used herein to mean simply conductive pathways, as one skilled in the art can readily understand, and is not limited to structures that can be strictly linear, but can include pathways that change direction, and can include pathways of different size, shape, materials, etc. Drive lines can be driven by stimulation signals  1716  and resulting touch signals  1703  generated in sense lines can be transmitted to receive section  1707  in touch controller  1706 . In this way, drive lines and sense lines can be part of the touch and hover sensing circuitry that can interact to form capacitive sensing nodes, which can be thought of as touch picture elements (touch pixels), such as touch pixels  1726 . This way of understanding can be particularly useful when touch panel  1724  can be viewed as capturing an “image” of touch. In other words, after touch controller  1706  has determined whether a touch or hover has been detected at each touch pixel in the touch panel, the pattern of touch pixels in the touch panel at which a touch or hover occurred can be thought of as an “image” of touch (e.g. a pattern of fingers touching or hovering over the touch panel). 
     An intelligent stylus according to various embodiments can touch or hover over the touch panel  1724  to provide touch input information. The intelligent stylus can transmit additional information about the stylus condition to the processor subsystem  1702  or to the host processor  1728  for processing. The processor subsystem  1702  or the host processor  1728  can include logic to interpret and process the additional information from the intelligent stylus. 
     Note that one or more of the functions described above, can be performed, for example, by firmware stored in memory (e.g., one of the peripherals) and executed by the processor subsystem  1702 , or stored in the program storage  1732  and executed by the host processor  1728 . The firmware can also be stored and/or transported within any non-transitory computer readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “non-transitory computer readable storage medium” can be any non-transitory medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device. The non-transitory computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disc such a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as compact flash cards, secured digital cards, USB memory devices, memory sticks, and the like. 
     The firmware can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “transport medium” can be any medium that can communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The transport readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic or infrared wired or wireless propagation medium. 
     It is to be understood that the touch panel, as described in  FIG. 17 , can sense touch and hover according to various embodiments. In addition, the touch panel described herein can be either single- or multi-touch. 
       FIG. 18  illustrates an exemplary mobile telephone  1800  that can include touch panel  1824 , display device  1836 , and other computing system blocks for use with an intelligent stylus according to various embodiments. 
       FIG. 19  illustrates an exemplary digital media player  1900  that can include touch panel  1924 , display device  1936 , and other computing system blocks for use with an intelligent stylus according to various embodiments. 
       FIG. 20  illustrates an exemplary personal computer  2000  that can include touch pad  2024 , display  2036 , and other computing system blocks for use with an intelligent stylus according to various embodiments. 
     The mobile telephone, media player, and personal computer of  FIGS. 18 through 20  can improve touch and hover sensing and improve device capabilities by utilizing a stylus according to various embodiments. 
     Although embodiments have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the various embodiments as defined by the appended claims.

Metadata:
Filing Date: 20160502
Publication Date: 20180320
Grant Date: 20180320
Priority Date: 20110622
Inventors: FALKENBURG DAVID R.
SIMON DAVID I.
HARLEY JONAH A.
MUCIGNAT ANDREA
LAND BRIAN R.
MULLENS CHRISTOPHER TENZIN
HOTELLING STEVEN P.
Assignee: APPLE INC
CPC Classifications: [{"code": "G06F3/0416", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0412", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/03545", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04105", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/044", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0414", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F21/83", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F21/36", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0418", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06T11/001", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04108", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/0416", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F21/36", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0416", "inventive": true, "first": true, "tree": "[]"}, {"code": "G06F21/83", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/03545", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0412", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06T11/001", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/0346", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04108", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F3/03545", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F2203/04105", "inventive": false, "first": false, "tree": "[]"}, {"code": "G06F21/36", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F21/83", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 46465290