Patent Document

BACKGROUND OF THE INVENTION 
     The present invention relates to an audio/video synchronous reproducer and a method of audio/video synchronous reproduction which carry out audio reproduction and video display synchronously based on audio data and video data. Techniques of this kind are described in, for example, JP-A-06-343065 and JP-A-09-149411. 
     In such an audio/video synchronous reproducer, audio data and video data digitally stored in a storage medium, such as DVD-ROM, a CD-ROM and a hard disk, are read out therefrom so as to carry out audio reproduction and video reproduction based on those data via a loudspeaker and a CRT monitor, respectively. The audio/video synchronous reproducer is proposed in Japanese Patent Application No. 09-001922 which is laying-opened as JP-A-10-200862 after the Convention date of the instant patent application. This audio/video synchronous reproducer will be explained as an earlier art in more detail with reference to FIG.  8 . 
     Referring to FIG. 8, the audio/video synchronous reproducer comprises a data storage  101  storing compressed and multiplexed digital audio and video data, an AV separating section  102  for reading out the data from the data storage  101  and separating them into audio data and video data, a video processing section  103  for decompressing or expanding the video data separated at the AV separating section  102 , an audio processing section  104  for expanding the audio data separated at the AV separating section  102 , an AV synchronous control section  105  for performing an audio/video synchronous control, a CRT monitor  106  for performing video display based on the expanded video data, and a loudspeaker  107  for performing audio reproduction based on the expanded audio data. 
     However, the audio/video synchronous reproducer shown in FIG. 8 does not consider the case wherein audio data and video data have their own reference times which are independent and different from each other. 
     Further, if there exists mismatching between time stamps included in video data and the number of video frames, the expected reproduction result can not be achieved. The reason for this is that time stamps included in audio data or video data are not used. 
     Further, there rises a case wherein the accuracy of a reference time to be used for audio/video synchronization is not sufficient. The reason for this is that a processed audio data amount if used as a reference time as it is. If the reference time accuracy is not sufficient, there rises a case wherein the accuracy of the processed audio data amount is lowered. As a result, the accuracy of the audio/video synchronization is also lowered. 
     SUMMARY OF THE INVENTION 
     It is therefore an object of the present invention to provide an improved audio/video synchronous reproducer that can eliminate one or more of the disadvantages inherent in the foregoing conventional technique. 
     It is another object of the present invention to provide an improved method of audio/video synchronous reproduction that can eliminate one or more of the disadvantages inherent in the foregoing conventional technique. 
     Other object of the present invention will become clear as the description proceeds. 
     According to one aspect of the present invention, there is provided an audio/video synchronous reproducer for carrying out audio reproduction and video display synchronously in response to audio data and video data. The audio/video synchronous reproducer comprises video processing means for processing the video data to produce a video display control signal and a video time stamp, video display means connected to the video processing means for performing the video display in accordance with the video display control signal, audio processing means for processing the audio data to produce an audio reproduction control signal and a processed audio data amount, audio reproducing means connected to the audio processing means for performing the audio reproduction in accordance with the audio reproduction control signal, video time stamp detecting means connected to the video processing means for detecting the video time stamp, audio time stamp deriving means connected to the audio processing means for deriving an audio time stamp equivalent value based on the processed audio data amount, time stamp comparing means connected to the video time stamp detecting means and the audio time stamp deriving means for comparing the video time stamp with the audio time stamp equivalent value to produce a comparison result signal indicative of a delayed/advanced relationship between the video data and the audio data, and video control means connected to the video processing means and the timestamp comparing means for controlling an operation of the video processing means in response to the comparison result signal. 
     According to another aspect of the present invention, there is provided an audio/video synchronous reproducer comprising an AV synchronous control section for performing comparison between a video time stamp assigned to video data and an audio time stamp equivalent value derived based on a processed amount of audio data, the AV synchronous control section controlling video display according to a result of the comparison so as to be synchronous with audio reproduction. 
     According to still another aspect of the present invention, there is provided a method of audio/video synchronous reproduction comprising the steps of detecting a video time stamp assigned to video data; deriving an audio time stamp equivalent value based on a processed amount of audio data; performing comparison between the video time stamp and the audio time stamp equivalent value; and controlling video display according to a result of the comparison so as to be synchronous with audio reproduction. 
    
    
     BRIEF DESCRIPTION OF THE DRAWING 
     FIG. 1 is a block diagram of an audio/video synchronous reproducer according to a first preferred embodiment of the present invention; 
     FIG. 2 is a flowchart for explaining an operation of the audio/video synchronous reproducer shown in FIG. 1; 
     FIG. 3 is a flowchart for explaining an operation of an audio processing section included in the audio/video synchronous reproducer shown in FIG. 1; 
     FIG. 4 is a flowchart for explaining an operation of an AV synchronous control section included in the audio/video synchronous reproducer shown in FIG. 1; 
     FIG. 5 is a block diagram of an audio/video synchronous reproducer according to a second preferred embodiment of the present invention; 
     FIG. 6 is a flowchart for explaining an operation of an audio processing section included in the audio/video synchronous reproducer shown in FIG. 5; 
     FIG. 7 is a flowchart for explaining an operation of an AV synchronous control section included in the audio/video synchronous reproducer shown in FIG. 5; and 
     FIG. 8 is a block diagram of a conventional audio/video synchronous reproducer. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring now to FIG. 1, an audio/video synchronous reproducer will be described according to the first preferred embodiment of the present invention. 
     The audio/video synchronous reproducer comprises a file device  1 , such as a CD-ROM drive, a DVD-ROM drive and a hard disk drive, an AV separating section  2  for reading out data from the file device  1  and separating them into video data, audio data and other data, a video processing section  3  for processing the video data separated at the AV separating section  2 , and a video display device  4  such as a CRT display for displaying the video data processed at the video processing section  3 . The reproducer further comprises an AV synchronous control section  5  for carrying out a synchronous reproduction control of the video data and the audio data, an audio processing section  6  for processing the audio data separated at the AV separating section  2 , and an audio reproducing device  7  for carrying out audio reproduction based on the audio data processed at the audio processing section  6 . In this embodiment, the file device  1  includes a plurality of file devices A, B, C, but there is no particular limitation to the number of file devices, which may also be one. 
     The AV separating section  2  comprises a file device choosing section  21  for choosing one or more of the file devices A to C to read out data therefrom, a video separating section  22  for separating only video data from the read-out data, and an audio separating section  23  for separating only audio data from the read-out data. 
     The video processing section  3  comprises a compressed video storage  31 , a video decoder  32 , a video frame storage  33  and a video display control section  34 . The compressed video storage  31  stores the video data separated at the video separating section  22 . The video decoder  32  carries out a decompressing or expanding process when the video data stored in the compressed video storage  31  is compressed, and stores expanded video data per frame into the video frame storage  33 . The video display control section  34  transfers the expanded video data per frame to the video display device  4 , using a video time stamp given to the expanded video data as a reference time. 
     The AV synchronous control section  5  compresses a video time stamp detecting section  51 , an audio time stamp deriving section  52 , a time stamp comparing section  53 , a video skip control section  54  and a video repeat control section  55 . The video time stamp detecting section  52  obtains from the video decoder  32  a video time stamp of video data being processed. The audio time stamp deriving section  52  obtains from an audio reproduction control section  64  an amount of audio data processed up to then for audio reproduction and converts it into an audio time stamp equivalent value. The time stamp comparing section  53  compares the video time stamp from the video time stamp detecting section  51  and the audio time stamp equivalent value from the audio time stamp deriving section  52  and produces a comparison result signal. Depending on the comparison result signal, the video skip control section  54  or the video repeat control section  55  controls a video frame display. The video skip control section  54  and the video repeat control section  55  cooperatively constitute a video display control means. 
     The audio processing section  6  comprises a compressed audio storage  61 , an audio decoder  62 , an audio storage  63  and an audio reproduction control section  64 . The compressed audio storage  61  stores the audio data separated at the audio separating section  23 . The audio decoder  62  carries out an expanding process when the audio data stored in the compressed audio storage  61  is compressed, and stores expanded audio data into the audio storage  63 . The audio reproduction control section  64  transfers the expanded audio data to the audio reproducing device  7 . 
     Now, referring also to FIG. 2, an operation of the audio/video synchronous reproducer shown in FIG. 1 will be described in detail. 
     The video processing section  3 , the AV synchronous control section  5  and the audio processing section  6  operate independently of each other, and may be realized by specific circuits or a software control based on time-division multitask processing or the like. 
     The file device choosing section  21  chooses one or more of the file devices A to C from the file device  1  designated by an operator or the like for video data and audio data, respectively, to read out compressed data therefrom and send the read-out compressed data to the video separating section  22  and the audio separating section  23 . The video separating section  22  separates compressed video data from the read-out compressed data and stores the compressed video data into the compressed video storage  31  (step A 1 ). At this time, if it is immediately after the start of the reproduction, the video separating section  22  retrieves a first video time stamp and sends it to the time stamp comparing section  53 . 
     The video display control section  34  checks whether a command of video repeat, i.e. a video repeat command signal, is received from the video repeat control section  55  (step A 2 ). If the video repeat command signal is received, the video display control section  34  controls the video display device  4  to continue displaying a video frame being displayed for a designated time (step A 4 ). 
     Then, the video display control section  34  checks whether a command of video skip, i.e. a video skip command signal, is received from the video skip control section  54  (step A 3 ). If the video skip command signal is received, the video display control section  34  commands the video decoder  32  to skip a video expanding process corresponding to a designated time period (step A 5 ). 
     The video decoder  32  expands the compressed video data stored in the compressed video storage  31  and stores expanded video data into the video frame storage  33  (step A 6 ). At this time, the video decoder  32  extracts a video time stamp from the video data and notifies it to the video time stamp detecting section  51  (step A 7 ). In response to the video stamp notification, the AV synchronous control section  5  starts an AV synchronous process. The video display control section  34  displays the expanded video data on the video display device  4  (step A 8 ). Execution of steps A 1  to A 8  is repeated until all the video data have been processed (step A 9 ). 
     Now, referring also to FIG. 3, an operation of the audio processing section  6  will be described. 
     The audio separating section  23  separates compressed audio data from the foregoing compressed data read out from the file device  1  and stores the compressed audio data into the compressed audio storage  61  (step B 1 ). At this time, if it is immediately after the start of the reproduction, the audio separating section  23  retrieves a first audio time stamp and feeds it to the time stamp comparing section  53 . 
     The audio decoder  62  expands the compressed audio data stored in the compressed audio storage  61  and stores expanded audio data into the audio storage  63  (step B 2 ). The audio reproduction control section  64  controls the audio reproducing device  7  to carry out audio reproduction based on the audio data stored in the audio storage  63  (step B 3 ). Execution of steps B 1  to B 3  is repeated until all the audio data have been processed (step B 4 ). 
     Now, referring also to FIG. 4, an operation of the AV synchronous control section  5  will be described. 
     The video time stamp detecting section  51  obtains a video time stamp of video data being processed from the video decoder  32  (step C 1 ). The audio time stamp deriving section  52  obtains from the audio reproduction control section  64  an amount of audio data processed up to then for audio reproduction (step C 2 ) and converts it into an audio time stamp equivalent value (step C 3 ). 
     The audio time stamp equivalent value can be derived by the following conversion equation: 
     
       
           ATSEV=PADA÷ ( SF×QB×C )  (1) 
       
     
     wherein ATSEV represents an audio time stamp equivalent value, PADA an audio data amount processed up to then for reproduction from the start of the reproduction, SF a sampling frequency, QB the number of quantization bits, and C the number of channels. 
     The sampling frequency, the number of quantization bits and the number of channels are obtained from a format of audio reproduced at the audio reproducing device  7 . 
     For example, assuming that, with respect to the audio data, the sampling frequency is 48 KHz, the number of quantization bits is 16, the number of channels is 2, the unit of the data amount is byte, and the unit of a time stamp is 90 KHz (in case that a video compression technique is MPEG), the conversion equation is given by: 
     
       
           ATSEV=PADA× 8×90000÷(48000×16×2).  (2) 
       
     
     Then, the time stamp comparing section  53  checks whether it is the first synchronous control from the start of the reproduction (step C 4 ). If positive, the audio time stamp deriving section  52  derives a correction value A (step C 6 ). The correction value A is given by: 
     
       
           A=FVTS−ATSEV   (3) 
       
     
     wherein FVTS represents a first time stamp in video data. 
     The correction value A is necessary when audio data is not processed for reproduction from the beginning, or when reference times of video data and audio data differ from each other. 
     On the other hand, if negative at step C 4 , i.e. if it is not the first synchronous control from the start of the reproduction, the time stamp comparing section  53  compares a last video time stamp and a current video time stamp (step C 5 ). If the last video time stamp is greater than the current video time stamp, the audio time stamp deriving section  52  derives a correction value B (step C 7 ). The correction value B is given by: 
       B=CVTS−ATSEV   (4) 
     wherein CVTS represents a current video time stamp. 
     The correction value B is necessary for achieving AV synchronization when processing data whose time stamp does not simply increase. 
     The time stamp comparing section  53  stores the current video time stamp (step C 8 ). This stored value is used as a last video time stamp in the next execution of step C 5 . 
     The time stamp comparing section  53  corrects the audio time stamp equivalent value using the correction values A and B (step C 9 ) to derive a corrected audio time stamp equivalent value. The corrected audio time stamp equivalent value is given by: 
     
       
           CATSEV=ATSEV+A−B   (5) 
       
     
     wherein CATSEV represents a corrected audio time stamp equivalent value. 
     Then, the time stamp comparing section  53  compares the corrected audio time stamp equivalent value and the video time stamp with each other (step C 10 ). If the video time stamp and the corrected audio time stamp equivalent value are equal to each other, nothing is done because synchronization is realized between the video and audio data. For preventing the synchronization process from being frequently carried out, it is arranged that the comparison process at step C 10  is executed with a certain time interval, for example, with a time interval corresponding to a reproducing time of several video frames. If the video time stamp is smaller than the corrected audio time stamp equivalent value, it means that the process of video data is delayed so that the video skip control section  54  commands the video decoder  32  and the video display control section  34  to skip the expanding process and the display updating process of the video data corresponding to a time delay in the process of video data (step C 11 ). On the other hand, if the video time stamp is greater than the corrected audio time stamp equivalent value, it means that the process of video data is advanced so that the video repeat control section  55  commands the video decoder  32  and the video display control section  34  to temporarily stop the expanding process and the display updating process of the video data corresponding to an advanced time (step C 12 ). 
     According to the audio/video synchronous reproducer shown in FIG. 1, the audio processing system and the video processing system can realize the exact AV synchronization on the fully independent data. This is achieved by provision of the audio time stamp deriving section  52  which converts a processed audio data amount into a value equivalent to a time stamp in the video processing system. 
     Further, even if intervals between the time stamps are not constant or the time stamp does not simply increase, the reproduction can be carried out continuously without intermission. This is achieved by provision of the time stamp comparing section  53 . 
     Referring now to FIG. 5, an audio/video synchronous reproducer according to the second preferred embodiment of the present invention will be described. In FIG. 5, those components corresponding to the components in FIG. 1 are assigned the same reference signs so as to omit explanation thereof. In the audio/video synchronous reproducer shown in FIG. 5, the audio processing section  6  further includes an audio reproduction termination notifying section  65  as compared with the audio/video synchronous reproducer shown in FIG.  1 . 
     Referring also to FIG. 6, an operation of the audio processing section  6  will be described. 
     First, the audio reproduction control section  64  derives an audio data reproducing size to be processed in one reproducing process (equivalent to a data amount corresponding to a reproducing time of one video frame) based on an audio reproducing format and a video frame rate, using the following equation (step D 1 ): 
     
       
           ADRS=SF×QB×C÷FR   (6) 
       
     
     wherein ADRS represents an audio data reproducing size, SF a sampling frequency, QB the number of quantization bits, C the number of channels, and FR a video frame rate. 
     For example, assuming that, with respect to the audio data, the sampling frequency is 48 KHz, the number of quantization bits is 16, the number of channels is 2, the unit of the data amount is byte, and the video frame rate is 30 frame/sec, the audio data reproducing size is given by: 
     
       
           ADRS= 48000×16÷8×2÷30=6400 (bytes)  (7) 
       
     
     The audio reproduction control section  64  secures a plurality of audio buffers on the audio storage  63 , each having the size thus derived and set as one block (step D 2 ). 
     The audio decoder  62  expands the compressed audio data corresponding to a data amount to occupy all the secured audio buffers, and thus stores them in the audio storage  63  (step D 3 ). The audio reproduction control section  64  sends the expanded audio data per block to the audio reproducing device  7  to start audio reproduction (step D 4 ). 
     The audio reproduction control section  64  waits for termination of the audio reproduction of one block (step D 5 ). The audio reproduction termination notifying section  65  notifies the audio time stamp deriving section  52  of the termination of audio reproduction of one block, i.e. corresponding to one video frame (step D 6 ). The audio reproduction control section  64  further judges whether all the audio data have been processed (step D 7 ). 
     If negative at step D 7 , the audio separating section  23  separates compressed audio data from compressed data read out from the file device  1 , and stores the compressed audio data into the compressed audio storage  61  (step D 8 ). 
     The audio decoder  62  expands the compressed audio data stored in the compressed audio storage  61  corresponding to one block and stores expanded audio data into the audio storage  63  (step D 9 ). The audio reproduction control section  64  controls the audio reproducing device  7  to carry out audio reproduction based on the audio data stored in the audio storage  63  (step D 10 ). Execution of steps D 5  to D 10  is repeated until all the audio data have been processed (step D 7 ). 
     Now, referring also to FIG. 7, an operation of the AV synchronous control section  5  will be described. 
     The AV synchronous control section  5  starts processing in response to the notification of the termination of audio reproduction of one block received from the audio reproduction termination notifying section  65 . The audio time stamp deriving section  52  obtains from the audio reproduction control section  64  an amount of audio data processed up to then for reproduction (step E 1 ), and converts it into an audio time stamp equivalent value (step E 2 ). The audio time stamp equivalent value is given by the foregoing equations (1) and (2). The audio time stamp equivalent value thus derived always becomes integer times a producing time of one video frame. Therefore, it is possible to carry out the exact AV synchronous process based on the audio reproducing time. 
     The video time stamp detecting section  51  obtains from the video decoder  32  a video time stamp of video data being processed (step E 3 ). 
     Operations at the following steps C 4  to C 12  are the same as those in the foregoing first preferred embodiment, so that explanation thereof is omitted. 
     Similarly, since an operation of the video processing section  3  is the same as that in the foregoing first preferred embodiment, explanation thereof is omitted. 
     In the audio/video synchronous reproducer according to the second preferred embodiment, as compared with that in the foregoing first preferred embodiment, the exact AV synchronization can be achieved even when the audio data amount processed for reproduction up to a given time can not be obtained or can be obtained but with poor accuracy. This is realized by provision of the audio reproduction termination notifying section which makes it possible to obtain an exact reference time being integer times a producing time of one video frame. 
     According to the foregoing first and second preferred embodiments, since the processed audio data amount is converted into a value equivalent to a time stamp in the video processing system, the AV synchronization can be achieved even with such data having different reference times for audio and video data.

Technology Category: 5