Patent Document

BACKGROUND OF THE INVENTION  
       [0001]     1. Field of the Invention  
         [0002]     This invention relates generally to the field of motion capture. More particularly, the invention relates to an improved apparatus and method for performing motion capture using shutter synchronization and/or using phosphorescent paint.  
         [0003]     2. Description of the Related Art  
         [0004]     “Motion capture” refers generally to the tracking and recording of human and animal motion. Motion capture systems are used for a variety of applications including, for example, video games and computer-generated movies. In a typical motion capture session, the motion of a “performer” is captured and translated to a computer-generated character.  
         [0005]     As illustrated in  FIG. 1  in a motion capture system, a plurality of motion tracking “markers” (e.g., markers  101 ,  102 ) are attached at various points on a performer&#39;s  100 &#39;s body. The points are selected based on the known limitations of the human skeleton. Different types of motion capture markers are used for different motion capture systems. For example, in a “magnetic” motion capture system, the motion markers attached to the performer are active coils which generate measurable disruptions x, y, z and yaw, pitch, roll in a magnetic field.  
         [0006]     By contrast, in an optical motion capture system, such as that illustrated in  FIG. 1 , the markers  101 ,  102  are passive spheres comprised of retro-reflective material, i.e., a material which reflects light back in the direction from which it came, ideally over a wide range of angles of incidence. A plurality of cameras  120 ,  121 ,  122 , each with a ring of LEDs  130 , 131 ,  132  around its lens, are positioned to capture the LED light reflected back from the retro-reflective markers  101 ,  102  and other markers on the performer. Ideally, the retro-reflected LED light is much brighter than any other light source in the room. Typically, a thresholding function is applied by the cameras  120 ,  121 ,  122  to reject all light below a specified level of brightness which, ideally, isolates the light reflected off of the reflective markers from any other light in the room and the cameras  120 ,  121 ,  122  only capture the light from the markers  101 ,  102  and other markers on the performer.  
         [0007]     A motion tracking unit  150  coupled to the cameras is programmed with the relative position of each of the markers  101 ,  102  and/or the known limitations of the performer&#39;s body. Using this information and the visual data provided from the cameras  120 - 122 , the motion tracking unit  150  generates artificial motion data representing the movement of the performer during the motion capture session.  
         [0008]     A graphics processing unit  152  renders an animated representation of the performer on a computer display  160  (or similar display device) using the motion data. For example, the graphics processing unit  152  may apply the captured motion of the performer to different animated characters and/or to include the animated characters in different computer-generated scenes. In one implementation, the motion tracking unit  150  and the graphics processing unit  152  are programmable cards coupled to the bus of a computer (e.g., such as the PCI and AGP buses found in many personal computers). One well known company which produces motion capture systems is Motion Analysis Corporation (see, e.g., www.motionanalysis.com).  
       SUMMARY  
       [0009]     A method is described comprising: applying phosphorescent paint to specified regions of a performer&#39;s face and/or body; strobing a light source on and off, the light source charging the phosphorescent paint when on; and strobing the shutters of a first plurality of cameras synchronously with the strobing of the light source to capture images of the phosphorescent paint, wherein the shutters are open when the light source is off and the shutters are closed when the light source is open.  
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]     A better understanding of the present invention can be obtained from the following detailed description in conjunction with the drawings, in which:  
         [0011]      FIG. 1  illustrates a prior art motion tracking system for tracking the motion of a performer using retro-reflective markers and cameras.  
         [0012]      FIG. 2  illustrates one embodiment of the invention which employs a curve pattern to track facial expression.  
         [0013]      FIG. 3  illustrates one embodiment of the invention which synchronizes light panels and camera shutters.  
         [0014]      FIG. 4  is a timing diagram illustrating the synchronization between the light panels and the shutters according to one embodiment of the invention.  
         [0015]      FIG. 5  is a schematic representation of an exemplary LED array and the connectors for the synchronization signals.  
         [0016]      FIG. 6   a  illustrates a set of exemplary illuminated curves painted on a performer&#39;s face during a lit frame.  
         [0017]      FIG. 6   b  illustrates a set of exemplary illuminated curves painted on a performer&#39;s face during a “glow” frame.  
         [0018]      FIG. 7  is a timing diagram illustrating the synchronization between the light panels and the camera shutters in an embodiment for capturing both lit frames and glow frames.  
         [0019]      FIG. 8  is a timing diagram illustrating the synchronization between the light panels and the camera shutters in another embodiment for capturing both lit frames and glow frames.  
         [0020]      FIG. 9  illustrates one embodiment of a system for capturing both lit frames and glow frames.  
         [0021]      FIG. 10  illustrates a timing diagram associated with the system shown in  FIG. 9 .  
     
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS  
       [0022]     Described below is an improved apparatus and method for performing motion capture using shutter synchronization and/or phosphorescent paint. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid obscuring the underlying principles of the invention.  
         [0023]     The assignee of the present application previously developed a system for performing color-coded motion capture and a system for performing motion capture using a series of reflective curves painted on a performer&#39;s face. These systems are described in the co-pending applications entitled “A PPARATUS AND  M ETHOD FOR  C APTURING THE  M OTION AND/OR  E XPRESSION OF A  P ERFORMER ,” Ser. No. 10/942,609, and Ser. No. 10/942,413, Filed Sep. 15, 2004. These applications are assigned to the assignee of the present application and are incorporated herein by reference.  
         [0024]     As described in these co-pending applications, by analyzing curves rather than discrete data points on a performer&#39;s face, the motion capture system is able to generate significantly more surface data than traditional marker-based tracking systems.  FIG. 2  illustrates an exemplary motion capture system described in the co-pending applications in which a predefined facial curve pattern  201  is adjusted to fit the topology of each performer&#39;s face  202 . In one embodiment, the three-dimensional (3-D) curve pattern is adjusted based on a 3-D map of the topology of the performer&#39;s face captured using a 3-D scanning system.  
         [0025]     The curves defined by the curve pattern  201  are painted on the face of the performer using retro-reflective, non-toxic paint or theatrical makeup. As described in detail below, in one embodiment of the invention, non-toxic phosphorescent paint is used to create the curves.  
         [0026]     As described in the co-pending applications, each curve painted on the performer&#39;s face has a unique identifying name and/or number (to support systematic data processing) and potentially a color that can be easily identified by the optical capture system. Once the curve pattern is applied, in one embodiment, the curve pattern is tracked by a motion capture processing system  210  comprised of one or more camera controllers  205  and a central motion capture controller  206  during the course of a performance. In one embodiment, each of the camera controllers  205  and central motion capture controller  206  is implemented using a separate computer system. Alternatively, the cameral controllers and motion capture controller may be implemented as software executed on a single computer system or as any combination of hardware and software.  
         [0027]     In one embodiment, each of the camera controllers  205  and/or the motion capture controller  206  is programmed with data  203  representing the curve pattern  201 . The motion capture system  210  uses this information to trace the movement of each curve within the curve pattern during a performance. For example, the performer&#39;s facial expressions provided by each of the cameras  204  (e.g., as bitmap images) are analyzed and the curves identified using the defined curve pattern.  
         [0028]     In one embodiment, the curve data  203  is provided to the motion capture system in the form of a “connectivity map,” which is a text file representation of the curve pattern  201  which includes a list of all curves in the pattern and a list of all surface patches in the pattern, with each patch defined by its bounding curves. It is used by the camera controllers  205  and/or the central motion capture controller  206  to identify curves and intersections in the optically captured data. This, in turn, allows point data from the curves to be organized into surface patches and ultimately the triangulated mesh of a final 3-D geometry  207 .  
         [0029]     In one embodiment of the invention, the efficiency of the motion capture system is improved by using phosphorescent paint and/or by precisely controlling synchronization between the cameras&#39; shutters and the illumination of the painted curves. More specifically, referring to  FIG. 3 , in one embodiment of the invention, the predefined facial curve pattern  301  is painted on the performer&#39;s face  202  using phosphorescent paint. In addition, light panels  308 - 309  (e.g., LED arrays) are precisely synchronized with the opening and closing of the shutters of the motion capture cameras  304 . In one embodiment, the synchronization between the light panels  308 - 309  and cameras  304  is controlled via synchronization signals  322  and  321 , respectively. As indicated in  FIG. 3 , in one embodiment, the synchronization signals are provided from a peripheral component interface (“PCI”) card  323  coupled to the PCI bus of a personal computer  320 . An exemplary PCI card is a PCI-6601 manufactured by National Instruments of Austin, Tex. However, the underlying principles of the invention are not limited to any particular mechanism for generating the synchronization signals.  
         [0030]     The synchronization between the light sources and the cameras employed in one embodiment of the invention is illustrated in  FIG. 4 . In this embodiment, the two synchronization signals  321 ,  322  are the same. In one embodiment, the synchronization signals cycle between 0 to 5 Volts. In response to the synchronization signals  321 ,  322 , the shutters of the cameras are periodically opened and closed and the light panels are periodically turned off and on, respectively. For example, on the rising edge  412  of the synchronization signals, the camera shutters are closed and the light panels are illuminated. The shutters remain closed and the light panels remain illuminated for a period of time  413 . Then, on the falling edge of the synchronization signals  414 , the shutters are opened and the light panels are turned off. The shutters and light panels are left in this state for another period of time  415 . The process then repeats on the rising edge  417  of the synchronization signals.  
         [0031]     As a result, during the first period of time  413 , no image is captured by the cameras, and the phosphorescent paint is illuminated with light from the light panels  308 - 309 . During the second period of time  415 , the light is turned off and the cameras capture an image of the glowing phosphorescent paint on the performer. Because the light panels are off during the second period of time  415 , the contrast between the phosphorescent paint and the rest of the room is extremely high (i.e., the rest of the room is pitch black), thereby improving the ability of the system to differentiate the various curves painted on the performer&#39;s face. In addition, because the light panels are on half of the time, the performer will be able to see around the room during the performance. The frequency  416  of the synchronization signals may be set at such a high rate that the performer will not even notice that the light panels are being turned on and off. For example, at a flashing rate of 75 Hz or above, most humans are unable to perceive that a light is flashing and the light appears to be continuously illuminate. In psychophysical parlance, when a high frequency flashing light is perceived by humans to be continuously illuminated, it is said that “fusion” has been achieved. In one embodiment, the light panels are cycled at 120 Hz; in another embodiment, the light panels are cycled at 140 Hz, both frequencies far above the fusion threshold of any human. However, the underlying principles of the invention are not limited to any particular frequency.  
         [0032]      FIG. 6   a  is an exemplary picture of the performer during the first time period  413  (i.e., when the light panels are illuminated) and  FIG. 6   b  shows the illuminated reflective curves captured by the cameras  304  during the second time period  415  (i.e., when the light panels are turned off). During the first time period, the phosphorescent paint is charged by the light from the light panels and, as illustrated in  FIG. 6   b , when the light panels are turned off, the only light captured by the cameras is the light emanating from the charged phosphorescent paint. Thus, the phosphorescent paint is constantly recharged by the strobing of the light panels, and therefore retains its glow throughout the motion capture session. In addition, because it retains its glow for a period of time, if a performer happens to move so that for a few frames some of the phosphorescent lines are in shadow and not illuminated by the light panels, even though the phosphorescent paint is not getting fully charged for those frames, the paint will still retain its glow from previous frame times (i.e., when the paint was not in shadow).  
         [0033]     As mentioned above, in one embodiment, the light panels  308 ,  309  are LED arrays. A schematic of an exemplary LED array  501  and associated connection circuitry is illustrated in  FIG. 5 . The synchronization signals are applied to the LED array  501  via connector J 2 - 1  illustrated to the left in  FIG. 5 . In one embodiment, the connectors are RJ-45 connectors. The synchronization signal is initially inverted by inverter IC 2 B and the inverted signal is applied to the base of transistor Q 2 , causing transistor Q 2  to turn on and off in response to the inverted signal. This causes current to flow through resistor R 3 , thereby causing transistor Q 1  to turn on and off. This, in turn, causes the LEDs within the LED array  501  to turn on and off. In one embodiment, the inverted signal from IC 2 B is applied to three additional LED arrays as indicated in  FIG. 5 . A plurality of additional connectors J 1 - 1 , J 1 - 2 , J 1 - 3 , and J 1 - 4  are provided for additional light panels (i.e., the light panels may be daisy-chained together via these connectors) using inverters IC 2 C, IC 2 D, IC 2 E and IC 2 F for buffering. If daisy-chaining without buffering is desired (e.g. due to critical timing requirements that would be hampered by the IC 2  propagation delays), then connector J 2 - 2  can be used. The voltage regulaor IC 1  used for the LED array (shown at the top of  FIG. 5 ) takes a 12V input and produces a 5V regulated output used by IC 2 . In one embodiment, transistors Q 1  is a MOSFET transistor. However, the underlying principles are not limited to any particular type of circuitry.  
         [0034]     In one embodiment of the invention, the cameras are configured to capture pictures of the performer&#39;s face (e.g.,  FIG. 6   a ) in addition to capturing the phosphorescent curves (e.g.,  FIG. 6   b ). The pictures of the performer&#39;s face may then be used, for example, by animators as a texture map to interpolate between the curves and render and more accurate representation of the performer.  
         [0035]     The signal timing illustrated in  FIG. 7  represents one such embodiment in which an asymmetric duty cycle is used for the synchronization signal for the cameras (in contrast to the 50% duty cycle shown in  FIG. 4 ). In this embodiment, synchronization signal  2  remains the same as in  FIG. 4 . The rising edge  722  of synchronization signal  2  illuminates the light panels; the panels remain on for a first time period  723 , turn off in response to the falling edge  724  of synchronization signal  2 , and remain off for a second time period  725 .  
         [0036]     By contrast, synchronization signal  1 , which is used to control the shutters, has an asymmetric duty cycle. In response to the rising edge  712  of synchronization signal  1 , the shutters are closed. The shutters remain closed for a first period of time  713  and are then opened in response to the falling edge  714  of synchronization signal  1 . The shutters remain open for a second period of time  715  and are again closed in response to the rising edge of synchronization signal  1 . The signals are synchronized so that the rising edge of synchronization signal  1  always coincides with both the rising and the falling edges of synchronization signal  2 . As a result, the cameras capture one lit frame during time period  715  (i.e., when the shutters are open the light panels are illuminated) and capture one “glow frame” during time period  716  (i.e., when the shutters are open and the light panels are off).  
         [0037]     In one embodiment, the data processing system  310  shown in  FIG. 3  separates the lit frames from the glow frames to generate two separate streams of image data, one containing the images of the performer&#39;s face and the other containing phosphorescent curve data. The glow frames may then be used to generate the mesh  307  of the performer&#39;s face and the lit frames may be used, for example, as a reference for animators (e.g., to interpolate between the curves) and/or as a texture map of the performer&#39;s face. The two separate video sequences may be synchronized and viewed next to one another on a computer or other type of image editing device.  
         [0038]     Given the significant difference in overall illumination between the lit frames and the glow frames, some cameras may become overdriven during the lit frames if their light sensitivity is turned up very high to accommodate glow frames. Accordingly, in one embodiment of the invention, the sensitivity of the cameras is cycled between lit frames and glow frames. That is, the sensitivity is set to a relatively high level for the glow frames and is then changed to a relatively low level for the lit frames.  
         [0039]     Alternatively, if the sensitivity of the cameras  304  cannot be changed on a frame-by-frame basis, one embodiment of the invention changes the amount of time that the shutters are open between the lit frames and the glow frames.  FIG. 8  illustrates the timing of one such embodiment in which synchronization signal  1  is adjusted to ensure that the cameras will not be overdriven by the lit frames. Specifically, in this embodiment, during the period of time that synchronization signal  2  is causing the light panels to be illuminated, synchronization signal  1  causes the shutter to be closed for a relatively longer period of time than when synchronization signal  2  is not illuminating the light panels. In  FIG. 8 , for example, synchronization signal  1  is high during time period  853 , thereby closing the shutter, and is low during period  855 , thereby opening the shutter. By contrast, during the glow frame, synchronization signal  1  is high for a relatively short period of time  813  and is low for a relatively longer period of time  815 .  
         [0040]     In one embodiment, illustrated in  FIG. 9 , both color and grayscale cameras are used and are synchronized using different synchronization signals. Specifically, in this embodiment, color cameras  914 - 915  are used to capture the lit frames and grayscale cameras  904 - 905  are used to capture the phosphorescent curves painted on the performer&#39;s face. One of the benefits of this configuration is that grayscale cameras typically have a relatively higher resolution and higher light sensitivity than comparable sensor resolution color cameras, and can therefore capture the phosphorescent curves more precisely. By contrast, color cameras are more well suited to capturing the color and texture of the performer&#39;s face. In addition, grayscale cameras may be adjusted to a relatively higher sensitivity than the color cameras.  
         [0041]     As illustrated in  FIG. 10 , in one embodiment, different synchronization signals,  1 A and  1 B are used to control the grayscale and color cameras, respectively. In  FIG. 10 , synchronization signals  1 A and  1 B are  180  degrees out of phase. As a result, the falling edge  1014  of synchronization signal  1 B occurs at the same time as the rising edge  1024  of synchronization signal  1 A, thereby opening the shutters for the color cameras  914 ,  915  and closing the shutters for the grayscale cameras  904 ,  905 . Similarly, the falling edge  1012  of synchronization signal  1 B occurs at the same time as the falling edge  1022  of synchronization signal  1 A, thereby closing the shutters for the color cameras  914 ,  915  and opening the shutters for the grayscale cameras  904 ,  905 . The synchronization signal  2  for the light panels is not illustrated in  FIG. 10  but, in one embodiment, is the same as it is in  FIG. 4 , turning the light panels on when the color camera shutters are opened and turning the light panels off when the grayscale camera shutters are opened.  
         [0042]     When the embodiments of the present invention described herein are implemented in the real world, the synchronization signals (e.g.,  321  and  322  of  FIG. 3 ) may require slight delays between respective edges to accommodate delays in the cameras and LED arrays. For example, on some video cameras, there is a slight delay after rising edge  412  of  FIG. 4  before the camera shutter closes. This can be easily accommodated by delaying signal  322  relative to signal  321 . Such delays are typically on the order of less than a millisecond. As such, when the system is started, the timing signals may initially need to be precisely calibrated by observing whether the video cameras  304  are capturing completely black frames and adjusting the timing signals  321  and  322  prior to the actual performance.  
         [0043]     Although the embodiments described above describe the use of a series of curves painted on the face of a performer, the underlying principles of the invention are not limited to this implementation. For example, instead of curves, one embodiment of the invention uses markers dipped in phosphorescent paint to capture the skeletal motion of the performer using the shutter and light panel synchronization techniques described above (either in lieu of or in addition to the curves on the performer&#39;s face, and either in lieu of or in addition to retroreflective markers). Moreover, curves may also be painted on the body and/or clothing of the performer while still complying with the underlying principles of the invention.  
         [0044]     In one embodiment, the phosphorescent paint applied to the performer&#39;s face is Fantasy F/XT Tube Makeup; Product #: FFX; Color Designation: GL; manufactured by Mehron Inc. of 100 Red Schoolhouse Rd. Chestnut Ridge, N.Y. 10977. In addition, in one embodiment, Basler A311f cameras  304  are used to capture the images of the performer. However, the underlying principles of the invention are not limited to any particular type of phosphorescent paint or camera.  
         [0045]     Embodiments of the invention may include various steps as set forth above. The steps may be embodied in machine-executable instructions which cause a general-purpose or special-purpose processor to perform certain steps. Various elements which are not relevant to the underlying principles of the invention such as computer memory, hard drive, input devices, have been left out of the figures to avoid obscuring the pertinent aspects of the invention.  
         [0046]     Alternatively, in one embodiment, the various functional modules illustrated herein and the associated steps may be performed by specific hardware components that contain hardwired logic for performing the steps, such as an application-specific integrated circuit (“ASIC”) or by any combination of programmed computer components and custom hardware components.  
         [0047]     Elements of the present invention may also be provided as a machine-readable medium for storing the machine-executable instructions. The machine-readable medium may include, but is not limited to, flash memory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, propagation media or other type of machine-readable media suitable for storing electronic instructions. For example, the present invention may be downloaded as a computer program which may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection).  
         [0048]     Throughout the foregoing description, for the purposes of explanation, numerous specific details were set forth in order to provide a thorough understanding of the present system and method. It will be apparent, however, to one skilled in the art that the system and method may be practiced without some of these specific details. Accordingly, the scope and spirit of the present invention should be judged in terms of the claims which follow.

Technology Category: g