PATENT DOCUMENT

Publication Number: US-7817180-B2
Application Number: US-11855305-A
Country: US
Kind Code: B2

Title: Video processing in a multi-participant video conference

Abstract:
Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant&#39;s image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved.

Claims:
1. A method of distributing video images by a central distributor in a video conference, said conference having a plurality of participants, the method comprising:
 at one particular video-conference participant&#39;s video-conference device that houses a central focus distributor for distributing video images to a plurality of non-focus video conference devices of a plurality of other participants during the video conference:
 receiving directly video images from each of the other plurality of participants through the non-focus video conference device of each of the other participants; 
 generating composite images from (i) the received video images and (ii) video images captured locally with the device that houses the central focus distributor; and 
 transmitting said composite images directly to each of the plurality of the other participants&#39; non-focus video-conference devices. 
 
 
     
     
       2. The method of  claim 1 , wherein the device housing the central focus distributor comprises a data storage, the method further comprising placing each received video image in said data storage, wherein each received video image is not interleaved with any other received video image. 
     
     
       3. The method of  claim 1  further comprising:
 creating a separate composite image for each particular participant; and 
 for each particular participant, removing the video image received from the particular participant from a particular composite image for the particular participant before transmitting the particular composite image to the particular participant. 
 
     
     
       4. The method of  claim 3  further comprising placing an indicator in place of each particular participant&#39;s removed image in the particular participant&#39;s separate composite image. 
     
     
       5. The method of  claim 3  further comprising supplying the particular composite image to the particular participant, wherein a participant image is (a) captured locally at the particular participant device and (b) rendered on a screen of the particular participant along with the particular composite image. 
     
     
       6. The method of  claim 1 , wherein a particular composite image comprises a plurality of pixel sets, each pixel set comprising an identifier that identifies a particular video image associated with the pixel set, wherein the identifiers of the pixel sets are used to identify the video images included within the transmitted composite image. 
     
     
       7. The method of  claim 1  further comprising resizing a particular received video image before generating the composite image. 
     
     
       8. The method of  claim 1  further comprising:
 saving said received video images before generating said composite images; 
 comparing a particular received video image of a particular participant with a previously saved video image of the particular participant; and 
 utilizing the received video image in generating the composite image when the received video image and the previously saved received video image are not identical. 
 
     
     
       9. The method of  claim 8  further comprising foregoing inclusion of a video image of the particular participant in generating the composite image when the received video image and the previously saved video image are identical. 
     
     
       10. The method of  claim 1  further comprising including video images of the central distributor in the generated composite images. 
     
     
       11. The method of  claim 1  further comprising receiving, at the device of said particular participant, the locally captured video images of the particular participant. 
     
     
       12. The method of  claim 1 , wherein receiving directly a video image from a non-focus video conference device at the focus video-conference device comprises receiving a video image that does not traverse any other non-focus video conference device. 
     
     
       13. The method of  claim 1 , wherein transmitting directly a composite image from the focus video-conference device to a non-focus video conference device comprises transmitting the composite image to the non-focus video conference device without the composite image traversing any other non-focus video conference device. 
     
     
       14. The method of  claim 1 , wherein the locally captured video images are received from a camera attached to the device housing the central focus distributor. 
     
     
       15. A central distributor module for a multi-participant video conference, said central distributor module comprising:
 a) a set of decoders for decoding video images received from a plurality of participants of the video conference, through the participants&#39; respective computers; 
 b) a compositing engine for generating composite video images from the decoded video images; and 
 c) at least one encoder for encoding said composite video images, 
 wherein the central distributor module is stored on a computer readable medium housed within a first particular participant&#39;s computer and is active on only the first particular participant&#39;s computer during the video conference. 
 
     
     
       16. The central distributor module of  claim 15  further comprising a set of intermediate buffers for storing a set of decoded video images before the compositing engine generates a composite video image based on the set of decoded video images. 
     
     
       17. The central distributor module of  claim 15  further comprising a frame rate controller for adjusting a rate at which the compositing engine generates composite video images. 
     
     
       18. The central distributor module of  claim 15  further comprising a redundancy remover to remove a second particular participant&#39;s image from a composite video image that is to be sent to the second particular participant. 
     
     
       19. The central distributor module of  claim 15  further comprising a resizer for receiving a decoded video image and resizing said decoded video image before the compositing engine generates an encoded composite image from the decoded video image. 
     
     
       20. The method of  claim 1 , wherein the video conference device that serves as the central focus distributor is a computer. 
     
     
       21. The method of  claim 20 , wherein the plurality of non-focus video conference devices are also computers. 
     
     
       22. For receiving and transmitting video content in a multi-participant video conference, a system comprising:
 a) a plurality of non-focus point modules, each non-focus point module stored on a computer readable medium housed within a different participant device; and 
 b) a focus point module stored on a computer readable medium housed within a participant device different from the participant devices with the non-focus point modules, the focus point module for directly receiving video images from and directly transmitting composite video images to said plurality of non-focus point modules, wherein each composite video image is a composite of a plurality of video images received (i) from at least one non-focus point module and (ii) locally at the focus point module, wherein the directly receiving and directly transmitting requires the focus point module and the non-focus point module to communicate to each other without using another non-focus point module. 
 
     
     
       23. The system of  claim 22 , wherein the focus point module is further for eliminating a particular participant&#39;s image from a composite video image that is sent to the particular participant. 
     
     
       24. The system of  claim 22 , wherein a directly received video image from a non-focus point module at the focus point module and a directly transmitted composite image from the focus point module to a non-focus point module do not traverse any other non-focus point module. 
     
     
       25. A computer readable medium housed within a computing device, the computer readable medium storing a computer program for distributing video content in a multi-participant video conference, the computer program for execution by at least one processor, the computer program comprising sets of instructions for:
 at the device, receiving, through a network, a separate stream of video images from each of a plurality of participants; 
 at the device, locally capturing video images from another participant that is using the device to participate in the conference; 
 with the device, generating composite images from the received and captured video images; and 
 from the device, transmitting, through the network, separate streams of said composite images to each of the plurality of participants. 
 
     
     
       26. The computer readable medium of  claim 25 , wherein the device includes a data storage, the computer program further comprising a set of instructions for placing each received video image in said data storage, wherein each received video image is not interleaved with any other received video image. 
     
     
       27. The computer readable medium of  claim 25 , wherein the computer program further comprises a set of instructions for:
 creating a separate composite image for each particular participant; and 
 removing each particular participant&#39;s received video image from a particular composite image for the particular participant before transmitting the separate composite image to the particular participant. 
 
     
     
       28. The computer readable medium of  claim 25 , wherein the computer program further comprises a set of instructions for resizing a particular received video image before generating the composite image. 
     
     
       29. A method of providing a participant module for a video conference having a plurality of participants at a plurality of different devices, wherein a central distributor module at one of the different participant devices is connected to all other participant devices and the other participant devices are connected only to the central distributor module, said method comprising:
 providing a decoder for:
 receiving, from the central distributor module, an encoded composite image comprising a plurality of independent sub-images of at least two different participants, wherein image information for one sub-image does not rely on image information for any other sub-image; and 
 decoding the encoded composite image in order to extract each independent sub-image as a separate image; and 
 
 providing a local image controller for displaying the extracted sub-images. 
 
     
     
       30. The method of  claim 29  further comprising providing a perspective adjuster for performing a transformation on a particular sub-image prior to displaying the particular sub-image, wherein the transformation is at least one of a skew and a resize operation. 
     
     
       31. The method of  claim 29 , wherein the provided participant module is not a central distributor of video content. 
     
     
       32. The method of  claim 29 , wherein the local image controller is further for displaying a placeholder for a particular participant when the sub-image for the particular participant is not available. 
     
     
       33. A method of providing a central distributor module for distributing video content from a first participant device for a video conference having a plurality of participants, the method comprising:
 providing a set of decoders for receiving and decoding video images, at the first participant device, from a set of other participant devices of the video conference, wherein said set of other participant devices are not directly connected to each other; 
 providing a compositing engine for generating a plurality of different composite video images from the decoded video images at the participant device; and 
 providing an encoder for encoding said composite video images at the participant device. 
 
     
     
       34. The method of  claim 33  further comprising providing a local image capture module for capturing video images at the participant device, wherein the compositing engine generates the plurality of different composite video images from (i) the decoded video images and (ii) the locally captured video images.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This Application is related to the following applications: U.S. patent application Ser. No. 11/118,931, filed Apr. 28, 2005; U.S. patent application Ser. No. 11/118,554, filed Apr. 28, 2005, now issued as U.S. Pat. No. 7,653,250; U.S. patent application Ser. No. 11/118,932, filed Apr. 28, 2005; U.S. patent application Ser. No. 11/118,555, filed Apr. 28, 2005; U.S. patent application Ser. No. 11/118,297, filed Apr. 28, 2005; and U.S. patent application Ser. No. 11/118,615, filed Apr. 28, 2005, now issued as U.S. Pat. No. 7,692,682. 
     FIELD OF THE INVENTION 
     The present invention relates to video processing in a multi-participant video conference. 
     BACKGROUND OF THE INVENTION 
     With proliferation of general-purpose computers, there has been an increase in demand for performing video conferencing through personal or business computers. Establishing such a conference, however, creates significant challenges in how to transfer data between participants. Prior solutions require large amount of data to be exchanged, which consumes a lot of computing resources as well as a lot of bandwidth. 
     Due to these resources and bandwidth limitations, general-purpose computers that are readily available for use in home and offices have not been able to perform video conferencing effectively and inexpensively. Therefore, there is a need in the art for a video conferencing architecture that uses an efficient method for transmitting data between video-conference participants. Such an approach would allow the video conference to be conducted thru commonly available network connections. 
     SUMMARY OF THE INVENTION 
     Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant&#39;s image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments are set forth in the following figures. 
         FIG. 1  illustrates an example of the video conference architecture of some embodiments of the invention. 
         FIGS. 2 and 3  illustrate how some embodiments exchange video content in a multi-participant video conference. 
         FIG. 4  shows an exemplary arrangement of participants&#39; images on one of the participants&#39; display. 
         FIG. 5  shows the software components of the video conferencing application of some embodiments of the invention. 
         FIG. 6  illustrates the focus-point module of some embodiments of the invention. 
         FIG. 7  shows the composite video data and video streams that the focus-point module of some embodiments produces. 
         FIG. 8  is a flow chart showing focus point video processing of some of the embodiments. 
         FIG. 9  illustrates the focus-point module with difference comparator in some embodiments of the invention. 
         FIG. 10  is an example of new frames arriving at different times at the focus point encoder. 
         FIG. 11  illustrates the application of avoiding redundant frames in a non-video conference application. 
         FIG. 12  illustrates the non-focus point module of some embodiments. 
         FIG. 13  illustrates the decoding operation of a non-focus point module of some embodiments. 
         FIG. 14  illustrates the non-focus point module with difference comparator in some embodiments. 
         FIG. 15  shows the focus point&#39;s 2×2 positioning of participants&#39; sub frames in some embodiments. 
         FIG. 16  illustrates how in some embodiments the focus point uses several encoders to eliminate each participant&#39;s own image from the composite image the participant receives. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     In the following description, numerous details are set forth for the purpose of explanation. However, one of ordinary skill in the art will realize that the invention may be practiced without the use of these specific details. In other instances, well-known structures and devices are shown in block diagram form in order not to obscure the description of the invention with unnecessary detail. 
     Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant&#39;s image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved. 
     Several detailed embodiments of the invention are described below. In these embodiments, the central distributor is the computer of one of the participants of the video conference, and the images that are transmitted to and from the central distributor are video frames. One of ordinary skill in the art will realize that other embodiments are implemented differently. For instance, the central distributor in some embodiments is not the computer of any of the participants of the video conference. Also, in some embodiments, the images transmitted to and from the central distributor are not video frames (e.g., the images might be fields that comprise a frame, or some other video image representation). 
     I. Overview 
       FIG. 1  illustrates an example of the video conference architecture  100  of some embodiments of the invention. This architecture allows multiple participants to engage in a video conference. In the example illustrated in  FIG. 1 , four participants A, B, C, and D are engaged in the video conference through their four computers  105 - 120  and a network (not shown) that connects these computers. The network that connects these computers can be any network, such as a local area network, a wide area network, a network of networks (e.g., the Internet), etc. 
     During the video conference, the computer  105  of one of the participants (participant D in this example) serves as a central distributor of audio/video content, as shown in  FIG. 1 . This central distributor  125  will be referred to below as the focus point of the multi-participant conference. The computers of the other participants will be referred to below as non-focus machines or non-focus computers. 
     Also, the discussion below focuses on the video operations of the focus and non-focus computers. The audio operations of these computers are further described in U.S. patent application Ser. No. 11/118,555, entitled “Audio Processing in a Multi-Participant Conference”, filed concurrently with this application. In addition, U.S. patent application Ser. No. 11/118,931, entitled “Multi-Participant Conference Setup”, filed concurrently with this application, describes how some embodiments set up a multi-participant video conference through a focus-point architecture, such as the one illustrated in  FIG. 1 . Both these applications are incorporated herein by reference. 
     As the central distributor of audio/video content, the focus point  125  receives video images from each participant, composites and encodes these images, and then transmits the composite images to each of the non-focus machines.  FIG. 2  shows an example of such an image exchange for the four participant example of  FIG. 1 . Specifically,  FIG. 2  illustrates the focus point  125  receiving one video frame  215 - 225  from each participant. From the received images  215 - 225 , the focus point  125  generates a composite frame  230  that includes each of the received frames and a frame  235  of the focus-point participant D. The focus point  125  then transmits the composite frame  230  to each non-focus machine  110 ,  115 , and  120 . 
     In the example illustrated in  FIG. 2 , the composite frame  230  that is transmitted to each particular non-focus participant also includes the video frame of the particular non-focus participant. In some embodiments, however, the focus point removes a particular non-focus participant&#39;s image from the composite image that the focus point transmits to the particular non-focus participant. The focus point  125  removes each participant&#39;s own image from its corresponding composite image in order to save bandwidth. 
       FIG. 3  illustrates an example of this removal for the example illustrated in  FIG. 2 . Specifically,  FIG. 3  illustrates (1) for participant A, a composite image  330  that does not have participant A&#39;s own image  215 , (2) for participant B, a composite image  335  that does not have participant B&#39;s own image  220 , and (3) for participant C, a composite image  340  that does not have participant C&#39;s own image  225 . 
       FIG. 3  also illustrates that instead of a non-focus participant&#39;s own image, some embodiments insert an empty-field flag  345  in the location of the non-focus participant&#39;s image in the composite image  330 . This flag indicates the absence of the sub-image corresponding to the non-participant&#39;s own image. Since the size of this indicator  345  is much smaller than the size of non-focus participant&#39;s image, the removal of a particular non-focus participant&#39;s image from this participant&#39;s composite image reduces the amount of bandwidth used by the composite image. Some embodiments do not use the empty-field flag  345 , and instead simply remove the image of a particular non-focus participant from the composite image sent to the particular non-focus participant. For instance, in some embodiments, the non-focus point module&#39;s decoder determines how the video image is composed with remaining encoded sub images because each encoded macroblock in each image has an identifier that represents its location. 
     Once each non-focus machine receives its encoded composite image, the non-focus machine decodes the composite image, extracts each of the sub-images in the composite image and then displays the decoded, extracted images on its display.  FIG. 4  illustrates an example of such a display for the case of four participants in a video conference. As shown in  FIG. 4 , the images  405 - 415  of the three other participants are displayed horizontally on the participant&#39;s computer display  400 . The local participant&#39;s own image  420  is displayed with a smaller size relative to the other participants&#39; images  405 - 415  and is located on the bottom of the display  400 . 
     Some embodiments are implemented by a video conference application that can perform both focus and non-focus point operations.  FIG. 5  illustrates the software architecture for one such application. Specifically, this figure illustrates a video conference application  505  that has two modules, a focus-point module  510  and a non-focus point module  515 . Both these modules  510  and  515 , and the video conference application  505 , run on top of an operating system  520  of a video-conference participant&#39;s computer. 
     During a multi-participant video conference, the video conference application  505  uses the focus-point module  510  when this application is serving as the focus point of the conference, or uses the non-focus point module when it is not serving as the focus point. The focus-point module  510  performs focus-point video-processing operations when the video conference application  505  is the focus point of a multi-participant video conference. On the other hand, the non-focus point module  515  performs non-focus point, video-processing operations when the application  505  is not the focus point of the conference. In some embodiments, the focus and non-focus point modules  510  and  515  share certain resources. 
     The focus-point module  510  is described in Section II of this document, while the non-focus-point module  515  is described in Section III. 
     II. The Focus-Point Module 
       FIG. 6  illustrates the focus-point module  510  of some embodiments of the invention. In this example, the focus-point module  510  is shown during a video conferencing with four participants, three non-focus participants and one focus participant. Specifically,  FIG. 6  shows focus-point module  510  as utilizing three decoders  620 - 630 , three intermediate buffers  635 - 645 , three resizers  647 - 649 , a local image capture module  651 , a frame rate controller  652 , a composite image buffer  655 , an encoder  660 , a redundancy remover  665 , a perspective adjuster  675 , and a local image controller  685 . 
     The decoders  620 - 630 , the intermediate buffers  635 - 645 , and the resizers  647 - 649  form three video decoding pipelines into three sections  657 - 659  of the composite image buffer  655 . These three video decoding pipelines allow the focus-point module  510  to decode and composite video signals from up to three participants during a video conference. 
     Specifically, each decoder  620 ,  625 , or  630  is responsible for decoding video signals from one non-focus computer during a video conference. For the example illustrated in  FIG. 1 ,  FIG. 6  illustrates the decoder  620  receiving video signals from participant A&#39;s computer, the decoder  625  receiving video signals from participant B&#39;s computer, and the decoder  630  receiving video signals from participant C&#39;s computer. After decoding a received frame, each decoder  620 ,  625 , or  630  stores the decoded frame in an intermediate buffer  635 ,  640 , or  645 . In some embodiments, each intermediate buffer is a location in the memory of the focus-point computer. 
     At a particular frame sampling rate, each resizer  647 ,  648 , or  649  (1) retrieves a frame that is stored in its corresponding intermediate buffer, (2) resizes this frame, if such resizing is necessary, and (3) stores the frame in its corresponding section in the composite image buffer  655 . For instance, the resizer  648  retrieves a decoded frame of the participant B from the intermediate buffer  640 , resizes this retrieved frame if necessary, and stores this frame in the composite-buffer section  658 . 
     The frame rate controller  652  defines the frame sampling rate at which the resizers  647 - 649  retrieve frames from the intermediate buffers  635 - 645 . The frame rate controller  652  determines this rate based on a variety of factors, which may include the system bandwidth, the computational resources of the focus-point computer, the number of participants in the video conference, etc. At the frame sampling rate that the controller  652  supplies to the resizers  647 - 649 , the frame rate controller  652  also directs the local image capture module  651  to store frames in section  656  of the composite image buffer  655 . These stored frames are the images of the video-conference participant who is using the focus-point computer during the video conference. These images are captured by the camera  650  and the local image capture module  651  at the focus-point computer. In some embodiments, the frame rate controller  652  changes the particular frame rate during a video conference, as the conditions of the video conference change. 
     As mentioned above, the resizers  647 - 649  retrieve frames from the buffers  635 - 645  based on the frame rate they receive from the controller  652 . Before storing a retrieved frame in the composite image buffer, a resizer resizes the retrieved frame when the non-focus computer that supplied this frame supplied it at a different size than the size of the composite-buffer section for this frame. For instance, to save bandwidth or computational resources during the encoding, a non-focus computer might encode and transmit smaller frames (i.e., encode frames at coarser level of granularity and transmit packets with less encoded content for each frame). 
     Also, as mentioned above, the resizers  647 - 649  store potentially-resized frames in their corresponding sections  657 - 659  of the composite image buffer  655 . In some embodiments, the composite image buffer  655  is a location in the memory of the focus-point computer, and each section  656 - 659  in this buffer is a contiguous logical section at this location in the memory. 
     At the sampling rate that the controller  652  defines, the encoder  660  encodes the composite frame that is stored in the composite image buffer. The encoder encodes the sub-frame that is stored in each section  656 ,  657 ,  658 , or  659  independently of the sub-frames that are stored in the other sections of the composite image buffer  655 . 
     To illustrate this,  FIG. 7  presents a diagram that summarizes the encoding of a composite frame. Specifically, this figure illustrates the four sections  656 - 659  of the composite image buffer  655 . For the encoding, the sub-frames in the section  656 - 59  are divided into sets of pixels. For instance, in some embodiments, each sub-frame is divided into a set of macroblocks, which are then intra-encoded without reference to any other frame&#39;s macroblock, or are inter-encoded by reference to another frame&#39;s macroblock.  FIG. 7  illustrates the dividing of the sub-frame in section  656  into several macroblocks  705 - 710 , the dividing of the sub-frame in section  657  into several macroblocks  715 - 720 , the dividing of the sub-frame in section  658  into several macroblocks  725 - 730 , and the dividing of the sub-frame in section  659  into several macroblocks  735 - 740 . 
     As indicated above, the encoder  660  decouples the encoding of each sub-frame in each section  656 ,  657 ,  658 , or  659  so that the encoding of each sub-frame does not depend on any other sub-frame (i.e., the encoding of one section does not use video data beyond the boundaries of each section). For example, the encoding of the macroblocks in the sub-frame of participant A in section  657  does not depend on the encoding of the macroblocks in the sub-frame of participant B in the section  658 . This encoding is further described in U.S. patent application Ser. No. 11/118,615, entitled “Video Encoding in a video Conference”, filed concurrently with the present application, now issued as U.S. Pat. No. 7,692,682. This application is incorporated in the present application by reference. 
     After encoding a composite frame, the encoder  660  supplies the redundancy remover with an encoded video stream that contains each participant&#39;s encoded video data in a separate section (i.e., contains different participants encoded video data in separate, non-interleaved sections). For instance,  FIG. 7  illustrates an encoded video stream  750  that includes encoding data for each intra-encoded or inter-encoded macroblock of the composite frame. In this composite stream  750 , the encoded data for each sub-frame is not interleaved with the encoded data of any other sub-frame, as shown in  FIG. 7 . 
     This non-interleaved structure of the encoded stream allows the redundancy remover to remove quickly a particular non-focus participant&#39;s video data from the video stream that is to be transmitted to the particular non-focus participant. For instance,  FIG. 7  illustrates (1) for participant A, a video stream  755  that has participant A&#39;s encoded video data replaced with a empty-field flag  760 , (2) for participant B, a video stream  765  that has participant B&#39;s encoded video data replaced with a empty-field flag  770 , and (3) for participant C, a video stream  775  that has participant A&#39;s encoded video data replaced with a empty-field flag  780 . As mentioned above, some embodiments do not employ such empty-field flags, and instead rely on the non-focus module decoders to automatically identify the empty sub-frames. 
     Once the redundancy remover removes each participant&#39;s redundant image data from the participant&#39;s video stream, the redundancy remover transmits the participant&#39;s video stream to the participant. Accordingly,  FIG. 6  illustrates the redundancy remover  665  sending three video streams to the three non-focus participants A, B. and C. 
       FIG. 6  also illustrates that the perspective adjuster  675  of the focus-point module  510  retrieves the focus point participant&#39;s image from focus point sub image  656  in the composite image buffer  655 . The perspective adjuster  675  also retrieves non-focus participants sub images from the intermediate buffers  635 ,  640 , and  645 . The adjuster  675  adjusts each non-focus participant&#39;s sub-frame in the composite image for the perspective view illustrated in  FIG. 4 . It then supplies the adjusted composite frame to the local image controller, which then renders the final composite image for display on the display device  695  of the focus-point computer. 
     During a video conference, each of the components of the focus-point module  510  iteratively performs the above-described operations.  FIG. 8  presents a process  800  that conceptually shows these iterative operations of the focus-point module  510  in some embodiments. In some embodiments, the focus-point module  510  performs this process  800  for a video conference with n participants, where n can be any number more than two. 
     As shown in  FIG. 8 , the focus-point process  800  initially defines (at  805 ) one or more parameters relating to the video conference. Different embodiments define different parameters at  805 . Examples of such parameters include the type of encoding used by the encoder of each non-focus computer, the size of frames transmitted by each non-focus computer, the type of encoding used by the focus-point computer, the size of the composite image buffer, etc. The above-incorporated U.S. patent application Ser. No. 11/118,931, entitled “Multi-Participant Conference Setup”, describes how these parameters are set and possibly reset during a video conference. 
     Next, the focus-point module receives (at  810 ) a frame from each non-focus computer. A decoder (e.g., a decoder  620 ,  625 , or  630 ) of the focus-point module  510  then decodes (at  815 ) the received frame and stores the received frame in an intermediate image buffer (e.g., a buffer  635 ,  640 , or  645 ). To decode frames from a particular non-focus computer, a decoder uses decoding algorithms that are appropriate for the encoding that the particular non-focus computer uses. These encoding and/or decoding algorithms are specified during the initialization operation  805  of the process  800 . In some embodiments, these algorithms might be re-specified during a video conference as the conditions of the video conference change (e.g., as new participants join or leave the video conference), as mentioned in the above-incorporated application. 
     After  815 , the focus-point module  510  determines (at  817 ) whether it is time for the resizers to sample the intermediate buffers (i.e., to retrieve decoded frames from the intermediate buffers, e.g., buffers  635 - 645  in case of three non-focus participants). As mentioned above, the sampling rate is set by the frame rate controller  652 . 
     When the process  800  determines (at  817 ) that it is not time for the resizers to sample the intermediate buffers, the process transitions to  818 . At  818 , the process determines whether any new frame has been received from a non-focus participant. If so, the process transitions to  815  to decode the received frame and store the decoded frame in the appropriate intermediate image buffer. On the other hand, when the process determines (at  818 ) that it is not receiving any frame, it transitions back to  817 , to determine whether it is time for the resizers to sample the intermediate buffers. 
     When the process determines (at  817 ) that it is time for the resizers to sample the intermediate buffers, the resizers (e.g., resizers  647 - 649 ) retrieve (at  820 ) decoded frames from the intermediate buffers (e.g., buffers  635 - 645 ), resize these retrieved frames if necessary, and store these frames in the composite image buffer  655 . 
     Next, at  825 , the local image capture  651  stores in composite-buffer section  656  a frame that the camera  650  captures of the participant that is using the focus-point computer. At  830 , the focus point sub image  656  from the composite image buffer and non-focus participants&#39; sub images from the intermediate buffers  635 ,  640 , and  645  are supplied to the perspective adjuster  675 , which then adjusts each non-focus participant&#39;s sub-frame in the composite image for the perspective view illustrated in  FIG. 4 . The adjuster  675  supplies the adjusted composite frame to the local image controller, which then renders the final composite image for display on the display device  695  of the focus-point computer. 
     The composite frame is also supplied (at  835 ) to the encoder  660 , which encodes each sub-frame in the composite image independently of the other sub-frames. The redundancy remover  665  then generates (at  840 ) a video stream for each non-focus participant by removing the non-focus participant&#39;s video content from the encoded video stream produced by the encoder. The redundancy remover transmits (at  845 ) each participant&#39;s video stream to the participant. After  845 , the focus-point process  800  determines (at  850 ) whether the multi-participant video conference has terminated. If so, the process  800  terminates. Otherwise, the process transitions back to  810  to receive another frame. 
     As mentioned above,  FIG. 8  is a conceptual illustration of the focus point&#39;s decoding and encoding operations in some embodiments. In some embodiments, the focus point module performs one or more decoding and encoding operations concurrently and independently in a multi-threaded processing scheme. 
     Also, in some embodiments, the focus point module includes other software modules. For instance,  FIG. 9  illustrates an alternative implementation of the focus point module. This implementation is similar to the implementation illustrated in  FIG. 6 , except that the focus point module  900  (1) has a frame rate controller with update time tracker  905 , (2) a difference comparator  910 , and (3) intermediate buffers  915 - 925  with timestamp fields. 
     The additional features of the frame rate controller and the intermediate buffer allow the focus-point module to avoid encoding the same frame from a non-focus point computer more than once. Specifically, when one of the decoders  620 - 630  writes a new frame into one of the intermediate buffers  915 - 925 , the decoder records the time in the timestamp field of the intermediate buffer. 
     At a particular frame rate, the frame rate controller  905  checks the timestamp field of each intermediate buffer. If the frame rate controller detects that an intermediate buffer&#39;s timestamp is later than the previous time that this buffer&#39;s content was supplied to its corresponding resizer, the frame rate controller directs the buffer&#39;s corresponding resizer to retrieve the buffer&#39;s content. Alternatively, when the frame rate controller detects no difference between the buffer&#39;s timestamp and the last time that the buffer was read by its corresponding resizer, the frame controller forgoes calling the resizer to read the content of the buffer. This is because in this case the intermediate buffer has not received any new frames since the last time that it was read by its corresponding resizer. Foregoing read out of the same frame multiple times eliminates unnecessary encoding of duplicate frames, and thereby save computational and bandwidth resources of the computers involved in the video conference. 
     To illustrate this benefit,  FIG. 10  presents an example where delays in signal transmission between the focus and non-focus computers cause the focus point computer to skip the encoding of certain duplicate frames. Specifically, this figure illustrates the storage of frames in intermediate buffers  915 - 925  of the focus point module. In this figure, passage of time is shown from left to right, while the storage of each participant&#39;s frame in a focus-point intermediate buffer is shown by dots placed on the horizontal lines. 
     For example, at time  0 , the focus-point intermediate buffers have new frames from the focus point and all non-focus participants. These frames are labeled as F 1 , A 1 , B 1 , and C 1  in  FIG. 10 . Each of these frames is encoded and transmitted to the non-focus participants. By time  1 , the only new frame that is stored is the focus participant&#39;s image, which is identified as frame F 2  in  FIG. 10 . The focus-point module, therefore, skips the retrieval and the encoding of the non-focus point participants&#39; frames since they have not changed since time  0 . 
     Between times  1  and  2 , the focus-point computer receives one new frame from the focus point camera, two new frames from the participant A, and one new frame from the participant B. The newly arrived frames are identified as frames F 3 , A 2 , A 3 , and B 2  respectively in  FIG. 10 . Hence, by time  2 , the focus-point module stores frames F 3 , A 3 , and B 2  as sub-frames in the composite image buffer, and then encodes and transmits composite images based on these stored sub-frames. In this case, frame A 2  is lost to the other participants in the conference as it was overwritten in the intermediate buffer before it could be retrieved by the focus-point module. Also, at time  2 , the resizer  649  (for participant C&#39;s data) does not retrieve the content of the intermediate buffer  925  since the content of this buffer at time  2  has not change from time  0 , which was the last time that this intermediate buffer  925  was read by the resizer  649 . 
     Due to a variety of reasons, the camera  650  at the focus-point computer might produce identical frames during two or more successive instances when the local image capture module  651  receives frames from it. Accordingly, to avoid duplicate processing of images of the local focus-point participant D, the focus-point module  900  utilizes the difference comparator  910 . Specifically, at a particular frame rate, the frame rate controller  905  directs the local image capture  651  to capture a frame from the local camera  650 . The local image capture module  651  of  FIG. 9  supplies the captured frame to difference comparator  910 , unlike the capture module  651  of  FIG. 6 , which would directly store this frame in the section  656  of the composite image buffer  655 . The difference comparator  910  then stores the received frame in section  656  of the composite image buffer so long as it determines that the last frame it stored in this section was not identical or very similar to the received frame. In some embodiments, the difference comparator  910  computes a metric that expresses the difference between the received frame and the last frame that the comparator stored in the composite image buffer. If this metric is lower than a particular threshold, the comparator  910  will detect that the received frame is identical or very similar to the last frame that it stored in the composite image buffer. 
     If the comparator determines that the received frame is identical or very similar to the last frame it stored in the composite image buffer, it discards the received frame, in order to conserve the computational and bandwidth resources of the computers involved in the video conference. Otherwise, the comparator stores the received frame in section  656  of the composite image buffer and maintains a copy of this frame for its difference comparison the next time that it receives a frame from the local image capture. 
     As mentioned above, the update tracking of the frame rate controller and the difference comparison of the comparator  910  may cause one or more sections of the composite image buffer  655  to be empty at a particular time that the focus-point module  900  is generating and encoding composite frames. Accordingly, the composite frames that are generated at such instances by the focus-point module will have one or more empty sub-frames. Such empty sub-frames may be identified by flags, or may be automatically identified by the decoders of the non-focus computers, as mentioned above. 
     Avoiding processing and encoding of duplicate frames is useful in many contexts other than a multi-participant video conference. For instance, in peer-to-peer video conferences, it is useful for avoiding duplicate decoding of a frame from the other participant, or duplicate encoding of a frame that is locally captured. 
     Avoiding duplicate processing and encoding is also useful in on-video conference settings.  FIG. 11  illustrates one such setting. Specifically, this figure illustrates a case where a computer  1100  captures frames from a camera, encodes these frames, and then locally stores these frames. As shown in this figure, the computer  1100  has a local capture module that captures frames from a video camera  1105  at a particular rate. Due to a variety of reasons, the camera  1105  might produce identical frames during two or more successive instances when the local image capture module  1115  receives frames from it. 
     The local image capture module  1115  supplies each captured frame to the difference comparator  1120 , which then may or may not forward the captured frame to the encoder  1125 . In particular, the comparator  1120  compares the frame that it receives from the capture module  1115  with the last frame that the comparator supplied to the encoder  1125 . If the two frames are identical or very similar, the difference comparator foregoes supplying the received frame to the encoder. Alternatively, when the two frames are not identical or very similar, the difference comparator forwards the received frame to the encoder for encoding. The encoder encodes any frames that it receives and then stores them on the storage, which could be computer memory, hard disk, DVD, or similar media. 
     III. Non-Focus Point Module 
       FIG. 12  illustrates the non-focus point module  515  of some embodiments of the invention. The non-focus point module performs encoding and decoding operations. For its encoding operation, the non-focus point module  515  utilizes a local image capture module  1240 , a frame rate controller  1242 , and an encoder  1250 . For its decoding operation, the non-focus point module  515  utilizes a decoder  1210 , an intermediate buffer  1215 , a perspective adjuster  1220 , and a local image controller  1230 . 
     During the video conference, a camera  1225  attached to the non-focus computer films the video-conference participant who is using the non-focus point computer. During the encoding operation, the local image capture module  1240  receives and captures video frames that are produced by the camera. At a particular sampling rate that is specified by the frame rate controller  1242 , the local image capture module  1240  directs the captured frames to the encoder  1250 , which then encodes and transmits the frames to focus-point computer. In some embodiments, the frame rate controller  1242  changes the particular frame rate during a video conference as the conditions of the video conference change. 
     During its decoding operation, the non-focus module  515  receives composite frames from the focus point module  510  and decodes them for display on the display device  1235  of the non-focus computer. This decoding operation is further described by reference to  FIG. 13 , which conceptually illustrates a decoding process  1300  of the non-focus point module. 
     As shown in  FIG. 13 , the decoding process  1300  of the non-focus point module starts when the non-focus point module receives (at  1305 ) a video stream that contains a composite frame from the focus point module. Next, the decoder  1210  of the non-focus module  515  decodes (at  1310 ) the received composite frame. 
     In some embodiments, the decoder  1210  decodes the composite frame without first extracting the sub-frame that make up the composite frame. In some embodiments, the decoder uses any empty-field flag to identify any sub-frame that is left empty. In other embodiments, the decoder does not need to rely on the empty-field flag, because it can identify the sub-frames through some other approach (e.g., it can identify the sub-frame associated with each macroblock in the composite frame). 
     Each decoded sub-frame represents a frame of one of the other participants in the video conference. After decoding a composite frame, the decoder stores (at  1315 ) the sub-frames in the decoded composite frame in an intermediate buffer  1215 . In some embodiments, the intermediate buffer  1215  is formed by three smaller intermediate buffers, one for storing each potential sub-frame of the decoded composite frame. At  1320 , the perspective adjuster then retrieves the decoded sub-frame from the intermediate buffer, and adjusts the perspective view of these images of the other conference participant. As mentioned above,  FIG. 4  illustrates one example of such perspective views. The adjuster  1220  supplies the adjusted composite frame to the local image controller  1230 . The local image controller also receives a video frame of the local non-focus point video-conference participant from the local image capture  1240 . From the adjusted frames of the other conference participants and the local participant&#39;s captured frame, the local image controller  1230  then renders (at  1325 ) the video-conference display presentation for display on the display device  1235  of the non-focus computer. 
     After  1330 , the non-focus decoding process  1300  determines (at  1335 ) whether the multi-participant video conference has terminated. If so, the process  1300  terminates. Otherwise, the process returns to  1305  to receive another encoded composite image. 
     As mentioned above,  FIG. 13  is a simplified conceptual illustration of the non-focus point module&#39;s decoding and encoding operations. In some embodiments, the non-focus point module includes other software modules. For instance,  FIG. 14  illustrates an alternative implementation of the non-focus point module. This implementation is identical to the implementation illustrated in  FIG. 12 , except that the non-focus point module  1400  has a difference comparator  1410 . 
     This difference comparator serves the same role as the difference comparator  910  in  FIG. 9 . Specifically, it prevents duplicate encoding of identical or very similar locally captured frames. When the local image capture module  1240  captures a frame, it supplies the captured frame to the difference comparator  1410  instead of sending it directly to encoder  1250 . 
     The difference comparator  1410  then supplies the captured frame to the encoder  1250  so long as it determines that the last frame that it supplied to the encoder was not identical or very similar to the captured frame. In some embodiments, the difference comparator  1410  computes a metric that expresses the difference between the received frame and the last frame that the comparator supplied to the encoder  1250 . If this metric is lower than a particular threshold, the comparator  1410  will detect that the received frame is identical or very similar to the last frame that it supplied to the encoder. 
     If the comparator determines that the received frame is identical or very similar to the last frame supplied to the encoder, it discards the received frame, in order to conserve the computational and bandwidth resources of the computers involved in the video conference. Otherwise, the comparator supplies the received frame to the encoder  1250  and maintains a copy of this frame for its difference comparison the next time that it receives a frame from the local image capture. 
     In some embodiments, the non-focus point module also includes a frame rate controller with update time tracker and an intermediate buffer with timestamp. Similar to the discussion for the focus point module in the previous section, in these embodiments the non-focus point module can save computing resources by avoiding sending duplicate decoded images from the intermediate buffer  1215  to perspective adjuster  1220 . 
     One of ordinary skill in the art will realize that the above-described video encoding, decoding, and distribution techniques have numerous advantages. For instance, the video compositing approach of the focus-point module  510  simplifies the removal of redundant video data, which, in turn, conserves much needed network bandwidth during the video conference. 
     To illustrate these benefits,  FIG. 15  illustrates another approach for compositing the video data in a 4-way video conference. In this approach, the focus-point module composites the frames of the participants in a 2×2 format  1500 . The focus point  510  then encodes and sends the resulting encoded image to each participant.  FIG. 15  illustrates that the encoding of the composite 2×2 frame  1500  results in the video stream  1550 , which has the video data from the different participants interleaved. Specifically, the initial portion of the video stream  1550 , which is illustrated in  FIG. 15 , shows the video data from the focus-point participant F and the non-focus point participant A interleaved. 
     This interleaving makes it difficult to remove each participant&#39;s own video data from the video stream that the focus-point module supplies to the participant. To simplify this removal, one solution would be (1) to generate three different composite video frames for three different non-focus participants, as illustrated in  FIG. 16 , and then (2) to use three different encoders to encode the three different composite video frames. This approach does eliminate each participant&#39;s video data from the composite video stream that the participant receives, but it requires more processing resources to support the extra encoders. 
     While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In other places, various changes may be made, and equivalents may be substituted for elements described without departing from the true scope of the present invention. For instance, instead of encoding after producing a composite frame, the focus-point module  510  of some embodiments might encode the frames of the non-focus participant before packaging the encoded frames into one composite frame for transmission. Thus, on of ordinary skill in the art would understand that the invention is not limited by the foregoing illustrative details, but rather is to be defined by the appended claims.

Metadata:
Filing Date: 20050428
Publication Date: 20101019
Grant Date: 20101019
Priority Date: 20050428
Inventors: JEONG HYEONKUK
ABUAN JOE
NORMILE JIM
Assignee: APPLE INC
CPC Classifications: [{"code": "H04N7/147", "inventive": true, "first": true, "tree": "[]"}, {"code": "H04N7/147", "inventive": true, "first": false, "tree": "[]"}, {"code": "H04N7/152", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 37234039