PATENT DOCUMENT

Publication Number: US-9569169-B2
Application Number: US-201514930601-A
Country: US
Kind Code: B2

Title: Using a plurality of buffers to provide audio for synchronized playback to multiple audio devices having separate device clocks

Abstract:
According to one embodiment, a media system communicates with an aggregate device that includes multiple media output devices. When providing media data for presentation, the system adjusts for device clock drift by resampling the media data provided to a media output device based at least in part on a device clock rate difference between a device clock of one of the media output devices and a device clock of another of the media output devices.

Claims:
The invention claimed is: 
     
       1. A method comprising:
 reading, by a device-side audio handling input/output unit (DIO) of an audio playback device, audio data from a buffer at a DIO position of the buffer during playing back audio from the buffer; 
 wrapping the DIO position to a start of the buffer upon reaching an end of the buffer; 
 generating timestamp information usable by a system-side audio handling input/output unit (SIO) to estimate the DIO position, the timestamp information including a clock time of a central processing unit (CPU) clock of the SIO and an indication of which audio samples have been processed by the DIO; and 
 providing the timestamp information by the DIO to the SIO. 
 
     
     
       2. The method of  claim 1 , wherein the buffer is a ring buffer. 
     
     
       3. The method of  claim 1 , wherein the SIO is an audio hardware abstraction layer provided in a core audio framework of an operating system. 
     
     
       4. The method of  claim 1 , wherein generating the timestamp information occurs at time the DIO performs the wrapping. 
     
     
       5. The method of  claim 1 , wherein the timestamp information is further usable by the SIO to determine an insertion point, the insertion point including a point at which to insert audio data with the buffer. 
     
     
       6. An audio playback device comprising:
 a device-side audio handling input/output unit (DIO); and 
 a non-transitory computer-readable medium storing instructions operable to cause the DIO to perform operations comprising:
 reading audio data from a buffer at a DIO position of the buffer during playing back audio from the buffer; 
 wrapping the DIO position to a start of the buffer upon reaching an end of the buffer; 
 generating timestamp information usable by a system-side audio handling input/output unit (SIO) to estimate the DIO position, the timestamp information including a clock time of a central processing unit (CPU) clock of the SIO and an indication of which audio samples have been processed by the DIO; and 
 providing the timestamp information by the DIO to the SIO. 
 
 
     
     
       7. The audio playback device of  claim 6 , wherein the buffer is a ring buffer. 
     
     
       8. The audio playback device of  claim 6 , wherein the SIO is an audio hardware abstraction layer provided in a core audio framework of an operating system. 
     
     
       9. The audio playback device of  claim 6 , wherein generating the timestamp information occurs at time the DIO performs the wrapping. 
     
     
       10. The audio playback device of  claim 6 , wherein the timestamp information is further usable by the SIO to determine an insertion point, the insertion point including a point at which to insert audio data with the buffer. 
     
     
       11. A non-transitory computer-readable medium storing instructions operable to cause a device-side audio handling input/output unit (DIO) to perform operations comprising:
 reading, by a device-side audio handling input/output unit (DIO) of an audio playback device, audio data from a buffer at a DIO position of the buffer during playing back audio from the buffer; 
 wrapping the DIO position to a start of the buffer upon reaching an end of the buffer; 
 generating timestamp information usable by a system-side audio handling input/output unit (SIO) to estimate the DIO position, the timestamp information including a clock time of a central processing unit (CPU) clock of the SIO and an indication of which audio samples have been processed by the DIO; and 
 providing the timestamp information by the DIO to the SIO. 
 
     
     
       12. The non-transitory computer-readable medium of  claim 11 , wherein the buffer is a ring buffer. 
     
     
       13. The non-transitory computer-readable medium of  claim 11 , wherein the SIO is an audio hardware abstraction layer provided in a core audio framework of an operating system. 
     
     
       14. The non-transitory computer-readable medium of  claim 11 , wherein generating the timestamp information occurs at time the DIO performs the wrapping. 
     
     
       15. The non-transitory computer-readable medium of  claim 11 , wherein the timestamp information is further usable by the SIO to determine an insertion point, the insertion point including a point at which to insert audio data with the buffer.

Description:
BENEFIT CLAIM 
     This application claims benefit as a Continuation of U.S. patent application Ser. No. 14/557,349, filed Dec. 1, 2014, which is a Continuation of U.S. patent application Ser. No. 13/477, 681, filed May 22, 2012, which is a Continuation of application Ser. No. 12/605,137, filed Oct. 23, 2009, which is a Divisional of application Ser. No. 10/877,762, filed Jun. 25, 2004, the entire contents of each of which is hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. §120. The applicant(s) hereby rescind any disclaimer of claim scope in the parent application(s) or the prosecution history thereof and advise the USPTO that the claims in this application may be broader than any claim in the parent application(s). 
    
    
     TECHNICAL FIELD 
     The present invention relates to synchronizing audio with multiple devices and, more specifically, to handling the communication of audio information between applications and multiple devices. 
     BACKGROUND 
     Electronic devices, such as computer systems, typically communicate with audio devices through device drivers associated with the audio devices. For example, a computer system plays sound on a speaker by sending audio information to a device driver designed for the speaker. Similarly, a computer system captures audio information from a microphone by receiving audio data from the device driver associated with the microphone. 
     Various approaches may be used to handle the communication of audio information between client applications (“clients”) that use audio information and the device drivers of audio devices. For example, according to an interrupt-based approach, a direct connection is established between the driver and its client application. Any time the driver wants audio data from the client, the driver raises an interrupt, suspending other activity on the Central Processing Unit (“CPU”) of the computer system on which the client is running Therefore, for a certain period of time, the driver is in control of the computer system. 
     To avoid problems associated with the interrupt approach, a “ring buffer” approach has been developed. According to the ring buffer approach, execution of a device driver is decoupled from execution of its client application by using a ring buffer as a rendezvous point for communicating audio data between the device driver and the client application. Specifically, the device driver for a playback device reads audio data from the ring buffer without directly communicating with the client application. The client application, in turn, feeds audio data into the ring buffer ahead of the device driver so that the device driver continues to have new audio data to read. Similarly, the device driver for a recording device writes audio data into the ring buffer without directly communicating with the client application. The client application, in turn, reads audio data from the ring buffer at a rate to ensure that the device driver does not overwrite unread audio data. 
     As audio systems become more sophisticated, the type and number of audio devices used by a system has increased. Consequently, there is a need to be able to send audio to multiple playback devices connected to a system in a manner that results in synchronized playback by the multiple devices. Similarly, there is a need to be able to consume audio from multiple audio sources connected to a system, to allow a single client application to concurrently consume audio from the various audio sources. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: 
         FIG. 1  is a block diagram of a system in which audio is communicated between an system-side audio handling I/O unit and a device-side audio handling I/O unit through the use of a ring buffer; 
         FIG. 2  is a block diagram of a system in which audio is communicated between an system-side audio handling I/O unit and multiple device-side audio handling I/O units through the use of multiple ring buffers; 
         FIG. 3  illustrates how an system-side audio handling I/O unit determines an offset into ring buffers for some devices based on offset latencies of other devices; and 
         FIG. 4  is a block diagram illustrating a computer system upon which embodiments of the invention may be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention. 
     Audio I/O System 
     As mentioned above, using the ring buffer approach, an audio I/O system communicates audio data to and from devices by using a ring buffer as a rendezvous point for the audio data. Such an audio I/O system typically includes (1) a ring buffer, where audio data is written to and read from, (2) a device-side audio handling I/O unit (“DIO”) (typically a device driver), and (3) an system-side audio handing I/O unit (“SIO”). A specific example of such a system is illustrated in  FIG. 1 . 
     Referring to  FIG. 1 , it illustrates an audio I/O system  100  that includes a DIO  105 , an SIO  110 , and a ring buffer  101 . For the purpose of illustration, it shall be assumed that DIO  105  is a device driver for a playback device, such as a speaker. Thus, DIO  105  reads audio data from ring buffer  101 . 
     SIO  110  generally represents a system-side entity through which applications communicate with audio devices. For example, the operating system of a computer system may provide an interface through which applications make calls to communicate with audio devices. In response to such calls, the operating system executes a routine to perform the requested audio transfer operation. SIO  110  may be, for example, an Audio Hardware Abstraction Layer (HAL) provided in the core audio framework of MAC OS X, generally available from Apple Computer, Inc. 
     While playing back audio from ring buffer  101 , DIO  105  reads audio data from ring buffer  101 . The specific position, within ring buffer  101 , from which DIO  105  is reading audio data at any given point in time is referred to herein as the “DIO position” within ring buffer  101 . The DIO position wraps around to the start  130  of ring buffer  101  upon reaching the end  140  of ring buffer  101 . 
     It is the responsibility of SIO  110  to place audio data into ring buffer  101  in front of the DIO position of DIO  105  so that it may be subsequently consumed by DIO  105 . To place audio data in the appropriate location within ring buffer  101 , SIO  110  needs to know the current DIO position. However, it would be impractical for DIO  105  to continuously communicate its position to the SIO  110 . Therefore, according to one embodiment, DIO  105  periodically generates information that SIO  110  can use to estimate the current DIO position. 
     In one embodiment, the information used by SIO  110  to estimate the current DIO location is timestamp information generated by DIO  105  when DIO  105  performs a wrap around operation. Specifically, when DIO  105  reaches the end  140  of ring buffer  101 , DIO  105  generates timestamp information that indicates (1) the current clock time of the CPU clock  112 , and (2) an indication of which audio samples have been processed by DIO  105 . From this timestamp information, and the current time of CPU clock  112 , SIO  110  is able to estimate the current DIO position at any given point in time, and thereby determine the appropriate point at which to insert audio data within ring buffer  101  (the “insertion point”). 
     Once SIO  110  has determined the current DIO position, SIO  110  must determine how far ahead of the DIO position to insert audio data. In particular, to ensure proper playback, SIO  110  may add a safety offset  115  to the current DIO position. The specific safety offset  115  used by SIO  110  may vary based on the nature of the audio device associated with the DIO  105 . Essentially, a safety offset is a limitation on how close the SIO  110  can write data in Buffer  101  ahead of the current DIO position  107 . 
     Moreover, in one embodiment, the SIO  110  may allow the application that is providing the audio data up to a full buffer unit in addition to the safety offset  115  to deliver audio data to the SIO  110  for writing. Hence, when SIO  110  wakes up to write data to buffer  101 , the application providing the audio data may take potentially as much real time to compute the data as it is going to take to play it, and no more. Thus, based on a prediction of where the current DIO position  107  is, the SIO  110  may decide that the appropriate insertion point is at point  120 . 
     Aggregate Devices 
     An aggregate device is a collection of devices that may be treated by an application as if it were a single device. According to one embodiment, a single SIO is used for synchronized playback and/or synchronized receipt of audio from an aggregate device.  FIG. 2  is a block diagram illustrating a system for synchronizing communications with an aggregate device, according to an embodiment of the invention. 
     Referring to  FIG. 2 , Aggregate Device  200  includes multiple devices  201  and  251 , all of which are communicating with a single SIO  210 . For the purpose of explanation, it shall be assumed that devices  201  and  251  are audio playback devices, and that SIO  210  is receiving audio from an application for synchronized playback on devices  201  and  251 . 
     As illustrated in  FIG. 2 , each device  201  and  251  has its own ring buffer (buffers  202  and  252 , respectively). Each device  201  and  251  may have its own buffer size and safety offset, e.g., safety offset  215  and safety offset  265 . In addition, each device  201  and  251  may have its own device clock that affects the rate at which the device consumes audio data. Consequently, at any given time, the current DIO positions  207  and  257  of the devices may be at different points in their respective ring buffers  202  and  252 . 
     According to one embodiment, each DIO  205  and  255  generates timestamp information every time the DIO performs a wrap-around operation. The SIO  210  uses the timestamp information from each DIO  205  and  255  to independently determine the current DIO position of each device. However, if the insertion positions selected by the SIO  210  are made without consideration to the fact that the device belong to an aggregate device, the devices will inevitably get out of synchronization, and the quality of the audio experience will degrade. According to one embodiment, SIO  210  is designed to compensate for factors that result in such degradation, including differences presentation latency and device clock rates. 
     Correcting for Differences in Presentation Latency 
     The presentation latency of a device is the actual amount of time between when the device&#39;s DIO reads data from a ring buffer and when the device actually plays the audio data. If the devices that belong to an aggregate device have different presentation latencies, then the sound played by some of the devices will lag behind the sound played by other devices, unless some form of adjustment is made for the differences. 
     According to one embodiment, the SIO  210  receives presentation latency data about the presentation latency of each device  201  and  251  in aggregate device  200 . The SIO  210  adjusts the insertion points within buffers  202  and  252  so that each device plays the same audio data at the same time. 
     Techniques for correcting for differences in presentation latency shall now be described with reference to  FIG. 3 . Referring to  FIG. 3 , it illustrates a scenario in which an aggregate device includes 3 devices  310 ,  320  and  330 . Each device  310 ,  320  and  330  has a corresponding presentation latency. The presentation latency of each device may be determined experimentally by each device and supplied to the Aggregate SIO, as previously discussed. 
     According to one embodiment, how far ahead the SIO writes ahead of each DIO position of each device takes into account a latency offset ( 350  and  360 ), as well as the safety offset of the device. While the safety offset of a particular device is dictated by the characteristics of that particular device, the latency offset of a particular device is dictated by how the characteristics of the other devices that belong to the same aggregate device differ from the characteristics of that particular device. Specifically, according to one embodiment, the latency offset for a given device is based on the difference between the composite latency of the device (safety offset+presentation latency) and the maximum composite latency of any device in the aggregate device. For example, device  310  has the highest composite latency (presentation latency  315  +safety offset  318 ) of devices  310 ,  320  and  330 . Therefore, the latency offset of each of devices  310 ,  320  and  330  is determined based on the difference between each of their composite latencies, and the composite latency of device  310 . 
     In the illustrated example, the SIO compensates for the difference in the composite latency of device  310  and the composite latency of device  320  by using a latency offset  350  with device  320 . Similarly, the SIO compensates for the difference in composite latency of device  330  with device  310  by using a latency offset  360  with device  330 . Since device  310  has the highest composite latency, no latency offset is needed for device  310 . 
     Referring again to  FIG. 2 , the gap between DIO position  257  and the insertion point  270  used by SIO  210  for device  251  takes into account both the safety offset  265  of device  251  and a latency offset  230 . As explained above, latency offset  230  compensates for the difference between the composite latency (safety offset+presentation latency) of device  201  and the composite latency (safety offset+presentation latency) of device  251 . 
     Correcting Clock Drift for Multiple Devices 
     Each audio device in an aggregate device may have its own device clock. The rate at which a device consumes and/or produces audio is driven by the device clock. Due to differences in device clocks, device clocks may “tick” at slightly different rates. Hence, although both device  201  and  251  are theoretically running at the same rate, device  201  may actually be consuming samples from buffer  202  faster than device  251 . Over short periods of time, such differences may not be detectable. However, over time, the distortion caused by such “clock drift” increases. If not addressed, clock drift between the devices that make up an aggregate device will lead to a situation where the synchronized data provided by the SIO is not being played by the devices in synchronization. 
     According to one embodiment, the SIO compensates for rate discrepancies between clocks on different devices of the same aggregate device. Specifically, based on the timestamp information supplied by each DIO, the SIO determines how fast the device clocks are actually running, and performs dynamic resampling on audio data prior to inserting the audio data into the ring buffer of one or more of the devices. 
     According to one embodiment, SIO designates one device to be the “master device” against which to compare the clock rates of other devices. Preferably, the device designated by the SIO to be the master device would be the device with the most stable device clock. In one embodiment, the device that reports the most latency is designated as the master device. In another embodiment, the master designation may be dynamically changed based on a variety of factors. For example, during playback the SIO may determine that one device in the aggregate device would be a better master than the currently designated master. In response to such a designation, the SIO may change the master designation, and begin compensating for clock drift based on the clock speed of the new master. 
     Once a master device has been designated, the SIO resamples the audio data provided to the other devices to correct/makeup for the discrepancy between their device clock rates and the device clock rate of the master device. Specifically, if a device has a slower clock rate than the master device, then the audio data for a given segment of audio is reduced. When the slower device consumes the reduced-size audio data, the slower device effectively “catches up” relative to the master device. Conversely, if a device has a faster clock rate than the master device, then the audio data for a given segment of audio is increased. When the faster device consumes the increased-size audio data, the faster device is effectively “slowed down” relative to the master device. 
     Any one of numerous well-known techniques for resampling audio data may be used to perform the resample. The present invention in not limited to any particular resampling technique. 
     Although SIO  210  is designed to compensate for clock drift among members of an aggregate device, it is also able to handle scenarios in which a mechanism is used to prevent clock drift. For example, if all of the devices that belong to an aggregate device are being driven by the same clock, then no clock drift will occur. In this scenario, the SIO determines that no resampling is required. 
     Synchronizing the Receipt of Audio from Multiple Devices 
     While embodiments described above have used examples in which audio is synchronously provided to an aggregate device, the same techniques may be used to provide synchronous receipt of audio data from multiple audio source devices. Thus, rather writing audio data ahead of the DIO position of multiple audio-consuming devices, the SIO reads audio data behind the DIO position of the audio-producing devices. Similar to the scenarios described above, the offset between the SIO reading-position and the DIO insertion positions includes a safety offset, and a latency offset that compensates for the difference between (1) safety offsets, and (2) time lapse between (a) when each device receives audio data and (b) when the device places the data into its corresponding ring buffer. 
     Multiple SIO Systems 
     In the examples given above, a single SIO is providing audio data to, or receiving audio data from, an aggregate audio device. However, multiple SIOs can concurrently communicate with the same aggregate device. Thus, while SIO  210  is writing audio data for one application into ring buffers  202  and  252  for playback on devices  201  and  251 , a second SIO may be doing the same for a different application. Under these circumstances, the second SIO would compensate for clock drift, presentation latency differences, and safety offset differences, in the same manner as SIO  210 . However, in adjusting for clock drift, the second SIO need not select the same device to be the master device. 
     The aggregate devices with which SIOs interact may not have identical memberships. For example, a first SIO may be providing audio for synchronized playback on a first aggregate device that includes devices A, B and C. At the same time, a second SIO may be providing audio for synchronized playback on a second aggregate device that includes devices B, C, and D. In this scenario, both SIOs are inserting audio data for devices B and C into the ring buffers for devices B and C. However, they determine latency offsets and resampling adjustments based on the specific membership of their aggregate devices, and their independent selection of a master device. 
     Single-thread of Execution 
     According to one embodiment, each SIO is run as a single thread of execution, regardless of how many devices belong to the aggregate device with which the SIO exchanges audio data. Consequently, audio applications are able to provide synchronized audio to multiple devices without having to include the complexities inherent in managing multiple execution threads. The single thread of execution determines appropriate insertion points into each of the multiple ring buffers using the techniques described above, thereby insulating the audio applications from the complexity of dealing with multiple audio devices. 
     Hardware Overview 
       FIG. 4  is a block diagram that illustrates a computer system  400  upon which an embodiment of the invention may be implemented. Computer system  400  includes a bus  402  or other communication mechanism for communicating information, and a processor  404  coupled with bus  402  for processing information. Computer system  400  also includes a main memory  406 , such as a random access memory (RAM) or other dynamic storage device, coupled to bus  402  for storing information and instructions to be executed by processor  404 . Main memory  406  also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor  404 . Computer system  400  further includes a read only memory (ROM)  408  or other static storage device coupled to bus  402  for storing static information and instructions for processor  404 . A storage device  410 , such as a magnetic disk or optical disk, is provided and coupled to bus  402  for storing information and instructions. 
     Computer system  400  may be coupled via bus  402  to a display  412 , such as a cathode ray tube (CRT), for displaying information to a computer user. An input device  414 , including alphanumeric and other keys, is coupled to bus  402  for communicating information and command selections to processor  404 . Another type of user input device is cursor control  416 , such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor  404  and for controlling cursor movement on display  412 . This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. 
     The invention is related to the use of computer system  400  for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system  400  in response to processor  404  executing one or more sequences of one or more instructions contained in main memory  406 . Such instructions may be read into main memory  406  from another machine-readable medium, such as storage device  410 . Execution of the sequences of instructions contained in main memory  406  causes processor  404  to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software. 
     The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operation in a specific fashion. In an embodiment implemented using computer system  400 , various machine-readable media are involved, for example, in providing instructions to processor  404  for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device  410 . Volatile media includes dynamic memory, such as main memory  406 . Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus  402 . 
     Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, or any other non-transitory medium from which a computer can read. 
     Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor  404  for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system  400  can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus  402 . Bus  402  carries the data to main memory  406 , from which processor  404  retrieves and executes the instructions. The instructions received by main memory  406  may optionally be stored on storage device  410  either before or after execution by processor  404 . 
     Computer system  400  also includes a communication interface  418  coupled to bus  402 . Communication interface  418  provides a two-way data communication coupling to a network link  420  that is connected to a local network  422 . For example, communication interface  418  may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface  418  may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface  418  sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information. 
     Network link  420  typically provides data communication through one or more networks to other data devices. For example, network link  420  may provide a connection through local network  422  to a host computer  424  or to data equipment operated by an Internet Service Provider (ISP)  426 . ISP  426  in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”  428 . Local network  422  and Internet  428  both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link  420  and through communication interface  418 , which carry the digital data to and from computer system  400 , are exemplary forms of carrier waves transporting the information. 
     Computer system  400  can send messages and receive data, including program code, through the network(s), network link  420  and communication interface  418 . In the Internet example, a server  430  might transmit a requested code for an application program through Internet  428 , ISP  426 , local network  422  and communication interface  418 . 
     The received code may be executed by processor  404  as it is received, and/or stored in storage device  410 , or other non-volatile storage for later execution. In this manner, computer system  400  may obtain application code in the form of a carrier wave. 
     In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Metadata:
Filing Date: 20151102
Publication Date: 20170214
Grant Date: 20170214
Priority Date: 20040625
Inventors: MOORE JEFFREY C.
STEWART WILLIAM GEORGE
LENGELING GERHARD
Assignee: APPLE INC
CPC Classifications: [{"code": "H04N21/44004", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/8106", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04L67/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/4143", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/00", "inventive": false, "first": false, "tree": "[]"}, {"code": "H04L67/1095", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/44004", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/165", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N21/8106", "inventive": true, "first": false, "tree": "[]"}, {"code": "G11B20/10527", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/162", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/4143", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04L67/10", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/4143", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/4307", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/162", "inventive": true, "first": false, "tree": "[]"}, {"code": "G06F3/165", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N21/8106", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/44004", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N21/43076", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N21/43076", "inventive": true, "first": false, "tree": "[]"}]
Family ID: 34980227