PATENT DOCUMENT

Publication Number: US-8355906-B2
Application Number: US-78503510-A
Country: US
Kind Code: B2

Title: Method and apparatus for extending the bandwidth of a speech signal

Abstract:
A bandwidth extension module, and an associated method and computer-readable medium, suitable for use in artificially extending the bandwidth of a lowband speech signal. The bandwidth extension module comprises a band-pass filter configured to produce a band-pass signal from the lowband speech signal; at least one carrier frequency modulator, each carrier frequency modulator configured to pitch-synchronously modulate the band-pass signal about a respective carrier frequency, the at least one carrier frequency modulator collectively producing a highband speech signal component; a synthesis filter configured to determine a highband speech signal based on the highband speech signal component; and a summation module configured to combine the lowband speech signal with the highband speech signal to obtain a bandwidth-extended speech signal.

Claims:
1. A method of extending the bandwidth of an audio signal, comprising:
 bandpass filtering the audio signal to obtain a bandpass signal; 
 generating at least one bandwidth extension signal by pitch-synchronously modulating the bandpass signal about at least one carrier frequency; and
 combining the audio signal and the at least one bandwidth extension signal. 
 
 
     
     
       2. The method defined in  claim 1 , further comprising classifying the audio signal as belonging to a harmonic mode, an unvoiced mode or a mixed mode. 
     
     
       3. The method defined in  claim 2 , wherein the step of generating a bandwidth extension signal by pitch-synchronously modulating the bandpass signal is performed only in response to the audio signal being classified as belonging to the harmonic mode. 
     
     
       4. The method defined in  claim 3 , further comprising multiplying an output of a noise generator with an output of an envelope operator applied to the bandpass signal to produce the at least one bandwidth extension signal in response to the audio signal being classified as belonging to the unvoiced mode or the mixed mode. 
     
     
       5. The method defined in  claim 3 , further comprising detecting a pitch of the audio signal. 
     
     
       6. The method defined in  claim 5 , further comprising determining the at least one carrier frequency based on the pitch and a passband of the bandpass filter. 
     
     
       7. The method defined in  claim 5 , further comprising determining a plurality of carrier frequencies based on the pitch and a passband of the bandpass filter; and wherein:
 the step of generating the at least one bandwidth extension signal comprises pitch-synchronously modulating the bandpass signal about each of the plurality of carrier frequencies to produce a plurality of bandwidth extension signal components; and 
 the step of combining the audio signal and the bandwidth extension signal comprises combining the audio signal and the plurality of bandwidth extension signal components. 
 
     
     
       8. The method defined in  claim 3 , further comprising inverse filtering the bandpass signal to flatten a spectrum of the bandpass signal before generating the at least one bandwidth extension signal. 
     
     
       9. The method defined in  claim 5 , wherein the step of generating the at least one bandwidth extension signal further comprises multiplying the pitch-synchronously modulated bandpass signal by an excitation gain to produce a scaled excitation signal. 
     
     
       10. The method defined in  claim 9 , further comprising determining the excitation gain based on the pitch and on a set of lowband linear spectral frequencies. 
     
     
       11. The method defined in  claim 9 , wherein the step of generating the at least one bandwidth extension signal comprises synthesizing the at least one bandwidth extension signal based on the scaled excitation signal and a set of highband linear spectral frequencies. 
     
     
       12. The method defined in  claim 11 , further comprising determining the highband linear spectral frequencies based on the pitch and on a set of lowband spectral frequencies. 
     
     
       13. The method defined in  claim 12 , further comprising determining the lowband linear spectral frequencies based on the audio signal. 
     
     
       14. The method defined in  claim 13 , further comprising inverse filtering the bandpass signal to compensate for amplitude variations in a spectrum of the bandpass signal before generating the at least one bandwidth extension signal, the amplitude variations being characterized by the lowband linear spectral frequencies. 
     
     
       15. The method defined in  claim 14 , wherein the step of combining the audio signal and the at least one bandwidth extension signal comprises combining the at least one bandwidth extension signal with a delayed version of the audio signal. 
     
     
       16. The method defined in  claim 3 , further comprising pre-filtering an input audio signal to produce the audio signal to cause partial extension of a frequency spectrum of the input audio signal into an intermediate frequency band. 
     
     
       17. The method defined in  claim 16 , wherein the pre-filtering comprises upsampling, lowpass filtering and spectral shaping. 
     
     
       18. Apparatus for extending the bandwidth of an audio signal, comprising:
 a bandpass filter operable to bandpass filter the audio signal to obtain a bandpass signal; 
 a bandwidth extension signal generator operable to pitch-synchronously modulate the bandpass signal about at least one carrier frequency to generate at least one bandwidth extension signal; and 
 a signal combiner operable to combine the audio signal and the at least one bandwidth extension signal. 
 
     
     
       19. The apparatus defined in  claim 18 , further comprising a signal classifier operable to classify the audio signal as belonging to a harmonic mode, an unvoiced mode or a mixed mode. 
     
     
       20. The apparatus defined in  claim 19 , wherein the bandwidth extension signal generator is operable to pitch-synchronously modulate the bandpass signal only in response to the audio signal being classified as belonging to the harmonic mode. 
     
     
       21. The apparatus defined in  claim 20 , further comprising a noise generator and an envelope operator, wherein the bandwidth extension signal generator is operable to multiply an output of the noise generator with an output of the envelope operator applied to the bandpass signal to produce the at least one bandwidth extension signal in response to the audio signal being classified as belonging to the unvoiced mode or the mixed mode. 
     
     
       22. A non-transitory computer-readable storage medium comprising computer-readable instructions which, when interpreted by a computing apparatus, causes the computing apparatus to execute a method of extending the bandwidth of an audio signal, the instructions comprising:
 instructions executable to bandpass filter the audio signal to obtain a bandpass signal; 
 instructions executable to pitch-synchronously modulate the bandpass signal about at least one carrier frequency to generate at least one bandwidth extension signal; and 
 instructions executable to combine the audio signal and the at least one bandwidth extension signal. 
 
     
     
       23. The computer-readable medium defined in  claim 22 , further comprising instructions executable to classify the audio signal as belonging to a harmonic mode, an unvoiced mode or a mixed mode. 
     
     
       24. The computer-readable medium defined in  claim 23 , wherein the instructions executable to pitch-synchronously modulate the bandpass signal are executable only in response to the audio signal being classified as belonging to the harmonic mode. 
     
     
       25. The computer-readable medium defined in  claim 24 , wherein the instructions for generating at least one bandwidth extension signal comprise instructions executable in response to the audio signal being classified as belonging to the unvoiced mode or the mixed mode, comprising:
 instructions executable to apply an envelope operator the bandpass signal to produce an envelope signal; 
 instructions executable to generate a noise signal; and 
 instructions to multiply the noise signal by the envelope signal to produce the at least one bandwidth extension signal.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     The present application is a CONTINUATION of U.S. patent application Ser. No. 11/469,705, filed on Sep. 1, 2006 now U.S. Pat. No. 7,734,462, hereby incorporated by reference herein. Benefit is claimed under 35 USC §120. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates generally to speech signal processing and, more particularly, to a method and apparatus for enhancing the perceived quality of a speech signal by artificially extending the bandwidth of the speech signal. 
     BACKGROUND OF THE INVENTION 
     Telephone speech transmitted in public wireline and wireless telephone networks is band-limited to 300-3400 Hz. The upper boundary is specified in order to reduce the bandwidth requirements for digitization at 8 kilosamples per second, while retaining sufficient intelligibility, though sacrificing naturalness. In particular, the absence of components in the range above 3400 Hz leads to muffled sounds. This renders it difficult to distinguish between unvoiced phonemes (e.g., /s/ and /f/), whose differentiating components are largely to be found in the missing highband range. 
     With the rapid evolution of telecommunications technology, devices capable of generating and processing wideband speech (hereinafter, “wideband-capable devices”) have been developed. Wideband speech refers to speech having a large bandwidth (e.g., up to 7000 Hz), which has the advantage of yielding high perceived voice quality. As wideband capable devices enter the marketplace, voice communications increasingly tend to involve such wideband-capable devices. While this allows for very high quality speech communication over private, high-bandwidth networks, the wideband capabilities of wideband-capable devices are largely wasted when the communication involves a public telephone network, since the speech transmitted in such networks is quite severely band-limited. 
     Nevertheless, the perceived speech quality at a wideband-capable device may be improved by enhancing the band-limited speech with artificially generated spectral content in the highband range. Based on a classical speech production model, artificial generation of the spectral content in the highband range comprises determining certain highband spectral parameters and a highband excitation signal. The highband excitation signal is passed through a linear prediction synthesis filter defined by the highband spectral parameters in order to generate the spectral content in the highband range. The combination of the artificially generated spectral content and the band-limited speech results in semi-artificial wideband speech. The wideband speech so created is considered to be of high quality when it sounds, perceptually, as if it had been issued directly from the source. 
     Two existing methods of generating the aforesaid highband excitation signal include (i) spectral-folding techniques and (ii) full-wave rectification of prediction residuals. However, these techniques tend to produce unsatisfactory results. For example, it has been found that the use of certain prior art techniques for generating the highband excitation signal cause artifacts in the resulting wideband speech when the band-limited speech contains nasal phonemes (e.g., /n/, /m/). 
     Against this background, there is a need in the industry for an improved technique of extending the bandwidth of a speech signal. 
     SUMMARY OF THE INVENTION 
     A first broad aspect of the present invention seeks to provide a method of artificially extending the bandwidth of a lowband speech signal. The method comprises band-pass filtering the lowband speech signal to obtain a band-pass signal; pitch-synchronously modulating said band-pass signal about at least one carrier frequency to obtain a highband speech signal component; determining a highband speech signal based on said highband speech signal component; and combining said lowband speech signal with said highband speech signal to obtain a bandwidth-extended speech signal. 
     A second broad aspect of the present invention seeks to provide a bandwidth extension module suitable for use in artificially extending the bandwidth of a lowband speech signal. The bandwidth extension module comprises means for band-pass filtering the lowband speech signal to obtain a band-pass signal; means for pitch-synchronously modulating said band-pass signal about at least one carrier frequency to obtain a highband speech signal component; means for determining a highband speech signal based on said highband speech signal component; and means for combining said lowband speech signal with said highband speech signal to obtain a bandwidth-extended speech signal. 
     A third broad aspect of the present invention seeks to provide a computer-readable medium comprising computer-readable program code which, when interpreted by a computing apparatus, causes the computing apparatus to execute a method of artificially extending the bandwidth of a lowband speech signal. The computer-readable program code comprises first computer-readable program code for causing the computing apparatus to obtain a band-pass signal by band-pass filtering the lowband speech signal; second computer-readable program code for causing the computing apparatus to obtain a highband speech signal component by pitch-synchronously modulating said band-pass signal about at least one carrier frequency; third computer-readable program code for causing the computing apparatus to determine a highband speech signal based on said highband speech signal component; and fourth computer-readable program code for causing the computing apparatus to obtain a bandwidth-extended speech signal by combining said lowband speech signal with said highband speech signal. 
     A fourth broad aspect of the present invention seeks to provide a bandwidth extension module suitable for use in artificially extending the bandwidth of a lowband speech signal. The bandwidth extension module comprises a band-pass filter configured to produce a band-pass signal from the lowband speech signal; at least one carrier frequency modulator, each said carrier frequency modulator configured to pitch-synchronously modulate said band-pass signal about a respective carrier frequency, the at least one carrier frequency modulator collectively producing a highband speech signal component; a synthesis filter configured to determine a highband speech signal based on said highband speech signal component; and a summation module configured to combine said lowband speech signal with said highband speech signal to obtain a bandwidth-extended speech signal. 
     A fifth broad aspect of the present invention seeks to provide an excitation signal generator. The excitation signal generator comprises a bandpass filter configured to produce a band-pass signal from the lowband speech signal; a modulator bank comprising a plurality of carrier frequency modulators, each of said carrier frequency modulators configured to frequency shift the band-pass signal to a respective carrier frequency associated with the respective carrier frequency modulator, thereby to produce a respective one of a plurality of modulated signals; and a summation module configured to combine the modulated signals into an excitation signal for use in generating a highband speech signal that complements the lowband speech signal in a highband frequency range. In accordance with this fifth broad aspect, the carrier frequency associated with a given one of the carrier frequency modulators is selected based on a pitch of the lowband speech signal to ensure pitch-synchronicity between the bandpass signal and the respective modulated signal produced by the given one of the carrier frequency modulators. 
     A sixth broad aspect of the present invention seeks to provide a bandwidth extension module. The bandwidth extension module comprises an input for receiving a first speech signal having first frequency content in a first frequency range; a processing entity; and an output for producing a second speech signal having second frequency content in a second frequency range that includes the first frequency range and an additional; frequency range outside the first frequency range. When the first frequency content contains harmonics in the first frequency range obeying a harmonic relationship, the processing entity is configured to cause the second frequency content to contain harmonics in the first frequency range and in the additional frequency range that collectively obey the same harmonic relationship. 
     These and other aspects and features of the present invention will now become apparent to those of ordinary skill in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       In the accompanying drawings: 
         FIGS. 1A-1C  depict various network scenarios that may benefit from usage of a bandwidth extension module in accordance with embodiments of the present invention; 
         FIG. 2  shows various functional components of a bandwidth extension module of any of  FIGS. 1A-1C , including an excitation signal generator, in accordance with an embodiment of the present invention; 
         FIG. 3  shows details of the excitation signal generator of  FIG. 2 , in accordance with an embodiment of the present invention; 
         FIGS. 4A-4D  illustrate the concept of pitch-synchronicity that is applicable to the excitation signal generator detailed in  FIG. 3 ; 
         FIG. 5A  shows an example frequency response of an particular type of anti-alisasing filter; 
         FIG. 5B  shows the inverse of the frequency response of  FIG. 5A ; 
     
    
    
     It is to be expressly understood that the description and drawings are only for the purpose of illustration of certain embodiments of the invention and are an aid for understanding. They are not intended to be a definition of the limits of the invention. 
     DETAILED DESCRIPTION OF EMBODIMENTS 
     With reference to  FIG. 1A , there is shown a first non-limiting example system, in which a telephony device  10  is in communication with a telephony device  12 A that is connected by an analog subscriber line  16 A to a central office  18 A of a telephony network  14 A. In the case of  FIG. 1A , the telephony device  12 A is an analog wideband-capable telephony device, meaning that it has the ability to reproduce analog speech signals having frequency content in a highband range as well as lower-frequency components. By way of non-limiting example, the telephony device  12 A may be a POTS phone. For the sake of simplicity, only one direction of communication is shown, namely, from the telephony device  10  to the telephony device  12 A, but it should be understood that in practice, communication will tend to be bidirectional. 
     The central office  18 A typically receives a circuit-switched digital speech signal  20 A from elsewhere in the telephony network  14 A. The circuit-switched digital speech signal  20 A represents the outcome of a sampling process performed on an audio signal captured by a microphone (not shown) at the telephony device  10 . An anti-aliasing filter (not shown) in the telephony network  14 A will have ensured that the sampling process can occur at a rate of 8 kilosamples per second (ksps). Typically, such anti-aliasing filter is responsible for ensuring that the circuit-switched digital speech signal  20 A is band-limited to 300-3400 Hz, and therefore it is inconsequential whether telephony device  10  is capable of generating frequency content in the highband range. 
     The central office  18 A is responsible for converting the circuit-switched digital speech signal  20 A into an analog speech signal  22  and for outputting the analog speech signal  22  onto the analog subscriber line  16 A. Conversion of the circuit-switched digital speech signal  20 A into the analog speech signal  22  is achieved by a digital-to-analog (D/A) converter  24  in tandem with a low-pass filter  26 . At the telephony device  12 A, the signal received along the analog subscriber line  16 A is converted by a transponder  28  (e.g., a loudspeaker) into an audio signal  30  that is ultimately perceived by a user  32 . 
     The present invention is useful in enhancing the perceived speech quality of the audio signal  30 , where such perception is from the point of view of the user  32 . Accordingly, a bandwidth extension module is provided at an appropriate point where it is desired to produce a bandwidth-extended speech signal from a band-limited speech signal. The bandwidth extension module serves to populate the highband range of the band-limited speech signal (e.g., digital speech signal  20 A) with frequency content so as to improve the perceived quality of the bandwidth-extended signal. In a non-limiting example embodiment, the highband range may span the frequency range of 4000-7000 Hz, but in other embodiments the highband range may span different frequency ranges such as 3400-7000 Hz, 4000-6000 Hz, and so on. In general, the extent of the highband range is not particularly limited by the present invention. 
     In one specific manifestation of the first non-limiting example system shown in  FIG. 1A , a bandwidth extension module (shown in solid outline at  34   1 ) acts on the circuit-switched digital speech signal  20 A and, as such, the bandwidth extension module  34   1  may be connected in front of the D/A converter  24 . The output of the bandwidth extension module  34   1  is a bandwidth-extended speech signal  36   1 , which is processed by the D/A converter  24  and then by the low-pass filter  26 , resulting in the analog speech signal  22 . Of note is the fact that the low-pass filter  26  should be designed to have a cut-off frequency that is sufficiently high so as not to remove valuable highband components of the bandwidth-extended speech signal  36   1  generated by the bandwidth extension module  34   1 . By “highband components” is meant frequency content in the highband range. 
     In another specific manifestation of the first non-limiting example system shown in  FIG. 1A , a bandwidth extension module (shown in dashed outline at  34   2 ) acts on the analog speech signal  22 . As such, the bandwidth extension module  34   2  may be connected in front of the telephony device  12 A. This may be achieved by providing an adapter that has a first connection to a wall jack and a second connection out to the telephony device  12 A; alternatively, the bandwidth extension module  34   2  may be integrated with the telephony device  12 A itself. In this case, the output of the bandwidth extension module  34   2  is a bandwidth-extended speech signal  36   2 , which is converted by the transponder  28  into the audio signal  30 . It is noted that in this manifestation, the bandwidth extension module  34   2  is preceded by an analog-to-digital input interface (shown in dashed outline at  52 ) and followed by a digital-to-analog output interface (shown in dashed outline at  54 ), to allow the bandwidth extension module  34   2  to operate in the digital domain. 
     With reference to  FIG. 1B , there is shown a second non-limiting example system, in which the aforesaid telephony device  10  is in communication with a mobile telephony device  12 B that is connected by a wireless link  16 B to a mobile switching center  18 B of a telephony network  14 B, possibly via one or more base stations (not shown). In the case of  FIG. 1B , the mobile telephony device  12 B is wideband-capable, meaning that it has the ability to process modulated wireless signals and reproduce digital speech signals carried therein, such digital speech signals having frequency content in the aforesaid highband range as well as lower-frequency components. By way of non-limiting example, the telephony device  12 B may be implemented as a wireless telephone phone, a telephony-enabled wireless personal digital assistant (PDA), etc. Again, for the sake of simplicity, only one direction of communication is shown, namely, from the telephony device  10  to the mobile telephony device  12 B, but it should be understood that in practice, communication will tend to be bidirectional. 
     The mobile switching center  18 B typically receives a digital speech signal  20 B from elsewhere in the telephony network  14 B. The digital speech signal  20 B represents the outcome of a sampling process performed on an audio signal captured by a microphone (not shown) at the telephony device  10 . The mobile switching center  18 B comprises a modulation unit  40  responsible for modulating the digital speech signal  20 B onto a carrier and for outputting the modulated signal  42  onto the wireless link  16 B. At the mobile telephony device  12 B, the signal received along the wireless link  16 B is demodulated by a demodulator  44 , whose output is converted into analog form by a D/A converter  46  and then processed by the aforesaid transponder  28  (e.g., a loudspeaker) into the aforesaid audio signal  30  that is ultimately perceived by the user  32 . 
     In accordance with an embodiment of the present invention, a bandwidth extension module is provided at an appropriate point where it is desired to produce a bandwidth-extended speech signal from a band-limited speech signal. The bandwidth extension module serves to populate the highband range of the band-limited speech signal (e.g., digital speech signal  20 B) with frequency content so as to improve the perceived quality of the bandwidth-extended signal. As stated earlier, the highband range may span the frequency range of 4000-7000 Hz, but in other embodiments the highband range may span different frequency ranges such as 3400-7000 Hz, 4000-6000 Hz, and so on. In general, the extent of the highband range is not particularly limited by the present invention. 
     In one specific manifestation of the second non-limiting example system shown in  FIG. 1B , a bandwidth extension module (shown in solid outline as  34   3 ) acts on the digital speech signal  20 B and, as such, the bandwidth extension module  34   3  may be connected in front of the modulation unit  40 . The output of the bandwidth extension module  34   3  is a bandwidth-extended speech signal  36   3 , which is modulated by the modulation unit  40 , resulting in the modulated signal  42 . Of note is the fact that the wireless link  16 B should be designed to allow the transmission of higher-bandwidth signals at a given carrier frequency. 
     In another specific manifestation of the second non-limiting example system shown in  FIG. 1B , a bandwidth extension module (shown in dashed outline at  34   4 ) acts on the output of the demodulator  44  at the telephony device  12 B, prior to the D/A converter  46 . In this case, the output of the bandwidth extension module  34   4  is a bandwidth-extended speech signal  36   4 , which is converted by the transponder  28  into the audio signal  30 . 
     With reference to  FIG. 1C , there is shown a third non-limiting example system, in which the aforesaid telephony device  10  in communication with a telephony device  12 C that is connected by a digital subscriber line  16 C to digital switching equipment  18 C of a telephony network  14 C. In the case of  FIG. 1C , the telephony device  12 C is a digital wideband-capable telephony device, meaning that it has the ability to process packets (e.g., IP packets transmitted over a LAN or over a public data network such as the Internet) and reproduce a digital speech signal carried therein, such digital speech signals having frequency content in the aforesaid highband range as well as lower-frequency components. By way of non-limiting example, the telephony device  12 C may be implemented as a Voice-over-IP phone (where the digital subscriber line  16 C is a LAN connection) or a computer executing a telephony software application (where the digital subscriber line  16 C is an xDSL connection providing Internet connectivity via an xDSL modem at the customer premises). Once again, for the sake of simplicity, only one direction of communication is shown, namely, from the telephony device  10  to the telephony device  12 C, but it should be understood that in practice, communication will tend to be bidirectional. 
     The digital switching equipment  18 C typically receives from elsewhere in the packet-switched network  14 C a packet data stream  60  that carries a digital speech signal. The digital speech signal carried in the packet data stream  60  represents the outcome of a sampling process performed on an audio signal captured by a microphone (not shown) at the telephony device  10 . The digital switching equipment  18 C is responsible for ensuring delivery of the packet data stream  60  to the telephony device  12 C over the digital subscriber line  16 C. Suitable hardware, software and/or control logic may be provided in the digital switching equipment  18 C for this purpose. At the telephony device  12 C, the signal received along the digital subscriber line  16 C is extracted from the packet data stream  60  by a de-packetizer  48 , converted into analog form by a D/A converter  50  and then processed by the aforesaid transponder  28  (e.g., a loudspeaker) into the aforesaid audio signal  30  that is ultimately perceived by the user  32 . 
     In accordance with an embodiment of the present invention, a bandwidth extension module is provided at an appropriate point where it is desired to produce a bandwidth-extended speech signal from a band-limited speech signal. The bandwidth extension module serves to populate the highband range of the band-limited speech signal (e.g., contained in the packet data stream  60 ) with frequency content so as to improve the perceived quality of the bandwidth-extended signal. As mentioned above, the highband range may span the frequency range of 4000-7000 Hz, but in other embodiments the highband range may span different frequency ranges such as 3400-7000 Hz, 4000-8000 Hz, and so on. In general, the extent of the highband range is not particularly limited by the present invention. 
     In one specific manifestation of the third non-limiting example system shown in  FIG. 1C , a bandwidth extension module (shown in solid outline at  34   5 ) acts on the digital speech signal carried in the packet data stream  60 . It is noted that in this embodiment, the bandwidth extension module  34   5  is preceded by a de-packetizer input interface  56  and followed by a re-packetizer output interface  58 , to allow the bandwidth extension module  34   5  to extract the digital speech signal, denoted  20 C, that is carried in the packet data stream  60 . 
     In another specific manifestation of the third non-limiting example system shown in  FIG. 1C , a bandwidth extension module (shown in dashed outline at  34   6 ) acts on the output of the de-packetizer  48  at the telephony device  12 C, prior to the D/A converter  50 . In this case, the output of the bandwidth extension module  34   6  is a bandwidth-extended speech signal  36   6 , which is converted by the transponder  28  into the audio signal  30 . 
     For ease of reference, the bandwidth extension module  34   1 ,  34   2 ,  34   3 ,  34   4 ,  34   5 ,  34   6  is referred to hereinafter by the single reference numeral  34 , and the bandwidth-extended speech signal  36   1 ,  36   2 ,  36   3 ,  36   4 ,  36   5 ,  36   6  is referred to hereinafter by the single reference numeral  36 . In addition, the digital speech signal  20 A,  20 B,  20 C is referred to hereinafter by the single reference numeral  20 .  FIG. 2  shows functional components of the bandwidth extension module  34 , which is configured to process the digital speech signal  20  and to produce the bandwidth-extended speech signal  36  as a result of this processing. The various functional components of the bandwidth extension module  34 , which may be implemented in hardware, software and/or control logic, as desired, are now described in further detail. 
     With reference therefore to  FIG. 2 , therefore, a pre-emphasis module  202  produces frames of a signal S 1  from frames of the digital speech signal  20 . It should be noted that the presence of the pre-emphasis module  202  is not required, but may be beneficial in some circumstances. The functionality of the pre-emphasis module  202 , which is optional, is to recover speech content in an intermediate frequency band, based on the digital speech signal  20 . For details about the design of a suitable non-limiting example of the pre-emphasis module  202 , the reader is referred to Y. Qian and P. Kabal, “Combining Equalization And Estimation For Bandwidth Extension Of Narrowband Speech”,  Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing  (Montreal, Canada), pp. I-713 to I-716, May 2004. This document is hereby incorporated by reference herein. 
     Of course, if one chooses to employ the pre-emphasis module  202 , one is free to select the intermediate frequency band in which one desires to recover speech content, and this intermediate frequency band may be dependent on the bandwidth of the digital speech signal. In a specific non-limiting example, assume that the digital speech signal  20  is band-limited to 300-3400 Hz. This does not mean that there is no signal strength outside this range, but rather that the signal strength is significantly suppressed. Thus, there may be some recoverable signal content in the range below 300 Hz and some recoverable signal content in the range above 3400 Hz. Assume for the moment that one wishes to perform a preliminary expansion of the frequency content to, say, 4000 Hz before performing linear predictive analysis and other functions. To this end, the pre-emphasis module  202  may consist of an interpolator (comprising an upsampler producing samples at, say, 16 kHz, followed by a low-pass filter having a steep response at 4000 Hz and significant attenuation at, say, 4800 Hz), combined with a spectral shaping filter. 
     One potential benefit of using the spectral shaping filter in the pre-emphasis module  202  is to reverse the effect, in the intermediate frequency band (in this case 3400-4000 Hz), of an anti-aliasing filter that was thought to have been used in the network  14 A,  14 B,  14 C to band-limit the digital speech signal  20 . In the case where the anti-aliasing filter used in the network  14 A,  14 B,  14 C was known to be an ITU-T G.712 channel filer (whose frequency response is shown in  FIG. 5A ), the frequency response of the spectral shaping filter in the pre-emphasis module  202  may resemble that shown in  FIG. 5B . Further non-limiting examples of anti-aliasing filters that may be used include ITU-T P.48 and ITU-T P.830, and the existence of yet others will be apparent to those skilled in the art. It should be understood, however, that one is generally free to select the shape of the spectral shaping filter used in the pre-emphasis module  202  to meet specific operational goals, which may be different from seeking to compensate for a specific type of anti-aliasing filter. 
     In addition, the spectral shaping filter in the pre-emphasis module  202  may also be used to perform equalization of the low frequency content of the digital speech signal  200 , e.g., in the range from 100 Hz to 300 Hz. This is manifested in  FIGS. 5A and 5B  as a “bump” at low frequencies. It should also be understood that the shape of the spectral shaping filter in the pre-emphasis module  202 , rather than being predetermined, may be determined adaptively to match the characteristics of the aforesaid anti-aliasing filter in the network  14 A,  14 B,  14 C. 
     Those skilled in the art will appreciate that the pre-emphasis module  202  may be preceded by a speech decompression module (not shown) in order to transform mu-law or A-law coded PCM samples into 16-bit PCM samples or raw sampled speech. In this way, the speech processing functions are executed on raw data rather than compressed data. It will also be appreciated that such a decompression module may be useful even in the absence of the pre-emphasis module  202 . 
     Continuing to refer to  FIG. 2 , the output of the pre-emphasis module  202 , i.e., signal S 1 , is fed to a zero-crossing module  204 , to a pitch analysis module  206 , to a linear predictive analysis module  208  and to an excitation signal generator  210 . The zero crossing module  204  produces a zero crossing result, denoted Z 0 , while the pitch analysis module  206  produces a fundamental frequency, denoted F 0 , and a pitch prediction gain, denoted B 0 . The pitch prediction gain B 0  is defined as a prediction coefficient which gives a minimum mean square error between a frame of input speech and a frame of past pitch-delayed values weighted by the pitch prediction coefficient B 0 . 
     The zero crossing result Z 0 , the fundamental frequency F 0  and the pitch prediction gain B 0  are fed to a classifier  212 , which produces a mode indicator M 0  for each frame of the signal S 1 . The mode indicator M 0  is indicative of whether the current frame of the signal S 1  (and therefore, the current frame of the digital speech signal  20 ) is in one or another of several modes that may include strong harmonic mode, unvoiced mode and/or mixed mode. For example, if the pitch prediction gain B 0  is larger than a certain threshold, and the fundamental frequency F 0  is less than another threshold, then the classifier  212  may conclude that the current frame of the signal S 1  is in the strong harmonic mode. If the pitch prediction gain B 0  is less than yet another threshold, the classifier  212  may conclude that the current frame of the signal S 1  is in the unvoiced mode. If neither conclusion has been reached, the classifier  212  may conclude that the current frame of the signal S 1  is in the mixed mode. Of course, other modes are conceivable, and the present invention does not particularly constrain the characteristics of individual modes or the total number of possible modes. Furthermore, different classification schemes and algorithms can be used, depending on operational requirements, and without departing from the spirit of the invention. 
     The linear predictive (LP) analysis module  208 , which can be a conventional functional module, calculates linear prediction coefficients (LPC) of each frame of the signal S 1 . Clearly, these LPCs will characterize the frequency content in a lower-frequency portion of the spectrum of the signal S 1  which, it is recalled, is missing frequency content in the highband range. For ease of reference, and in contrast to the expression “highband range”, the lower-frequency portion of the spectrum of the signal S 1  will hereinafter be referred to as a “lowband range”. In a non-limiting example, where the highband range extends from 4000 Hz to 7000 Hz, the lowband range may extend from 300 Hz to 4000 Hz. However, the present invention does not particularly constrain the demarcation point between the lowband range and the highband range. 
     In an example, fourteen (14) LPCs may be used to characterize the frequency content of the signal S 1  in the lowband range. The LP analysis module  208  further converts these fourteen (14) LPCs to a corresponding number of lowband line spectrum frequencies (LSFs), denoted L 0 . The lowband linear spectrum frequencies L 0  are provided to the excitation signal generator  210 , to an LSF estimator  214  and to an excitation gain estimator  216 . It should be understood that the present invention does not particularly limit the number of LPCs that need to be generated by the LP analysis module  208 , and therefore persons skilled in the art should appreciate that a greater or smaller number of LPCs may be adequate or appropriate, depending on such factors as the extent of the lowband frequency range and others. 
     The excitation signal generator  210  produces a highband excitation signal, denoted E 0 , based on the signal S 1 , the fundamental frequency F 0  and the lowband linear spectrum frequencies L 0 . The excitation signal generator  210  is now described in greater detail with reference to  FIG. 3 . Firstly, it is noted that the excitation signal generator  210  comprises a bandpass filter  306  that filters the signal S 1  around a passband to produce a bandpass filtered signal S 1 *. In addition, it is noted that the excitation signal generator  210  is capable of selectably operating in one of two potential operational states. Entry into one of the two operational states is implemented by a selector, which is in this case symbolized by a pair of switches  302 ,  304  located at the output of the bandpass filter  306  and at the output of the excitation signal generator  210 , respectively. Of course, the actual implementation of the selector may vary from one embodiment to another, and may involve various combinations of hardware, software and/or control logic. Such variations would be understood by persons skilled in the art and therefore require no further expansion here. 
     The first operational state is entered in response to the mode indicator M 0  being indicative of a strong harmonic mode. In this first operational state, the bandpass filtered signal S 1 * feeds an inverse filter  307 , whose coefficients are the lowband linear spectrum frequencies L 0  from the LP analysis module  208 . The effect of the inverse filter  307  is to flatten the spectrum of the bandpass filtered signal S 1 *, thereby to produce a residual signal denoted S*R. Such flattening may be effected by designing the inverse filter to compensate for amplitude variations that are characterized by the lowband linear spectrum frequencies L 0 . 
     The residual signal S*R is passed to a modulator bank  308 . The modulator bank  308  comprises a parallel arrangement of one or more carrier frequency modulators; in the illustrated non-limiting embodiment, the modulator bank  308  comprises three carrier frequency modulators  310 ,  312 ,  314 . Each of the carrier frequency modulators  310 ,  312 ,  314  is associated with a respective carrier frequency F 310 , F 312 , F 314  received from a carrier frequency selection module  326 . If only one carrier frequency modulator is used, then that carrier frequency modulator produces an output that is the highband excitation signal E 0  at the output of the switch  304 . On the other hand, if more than one carrier frequency modulator is used, the outputs of the plural carrier frequency modulators are combined into the highband excitation signal E 0 . In the illustrated non-limiting embodiment, the outputs of the three carrier frequency modulators  310 ,  312 ,  314  (referred to as “modulated signals” and denoted E 310 , E 312 , E 314 , respectively) are combined at a summation block  316  to yield the highband excitation signal E 0 . 
     As will be appreciated, each of the carrier frequency modulators  310 ,  312 ,  314  in the modulator bank  308  is operable to frequency shift the residual signal S 1 *R to around the respective carrier frequency F 310 , F 312 , F 314  received from the carrier frequency selection module  326 . The bandwidth and center frequency of the bandpass filter  306  are related to the portion of the frequency content of the signal S 1  from which valuable information will be extracted for the purposes of replication in the highband range. For example, if the signal S 1  contains frequency content up to 4000 Hz (e.g., when the pre-emphasis module  202  is used), then certain frequency content in the range extending from 3000 Hz to 4000 Hz may contain valuable information. As such, in a non-limiting example embodiment, the bandpass filter  306  may have a bandwidth of 1000 Hz centered around a frequency of 3500 Hz. However, it should be understood that the present invention does particularly limit the bandwidth or center frequency of the bandpass filter  306 . 
     In particular, the properties/configuration of the modulator bank  308  may be adjusted to match the user&#39;s preferences. For instance, the upper limit of bandwidth extension achieved by an embodiment of the present invention may be selectable by the user. 
     The number of carrier frequency modulators and their respective carrier frequencies are a function of the bandwidth of the bandpass filter  306 , as well as the bandwidth of the highband frequency range that one wishes to artificially generate. Generally speaking, when there are N carrier frequency modulators, N≧1, the carrier frequency of the n th  given carrier frequency modulator, N≧n≧1, is the sum of a respective nominal carrier frequency and a respective correction factor selected to ensure “pitch synchronicity”. It should be mentioned that the present invention does not particularly limit the number of carrier frequency modulators to be employed, or on their nominal carrier frequencies. Nevertheless, it may be useful to consider an example, not to be considered limiting, where it is assumed that the highband frequency range that one wishes to artificially generate extends from 4000 Hz to 7000 Hz, and where it is assumed that the bandwidth of the bandpass filter is 1000 Hz. In this non-limiting example, a total of three carrier frequency modulators are required to fill the desired highband frequency range. To cover as much of the desired highband frequency range as possible with minimal artifacts, the three carrier frequency modulators  310 ,  312  and  314  should have respective carrier frequencies F 310 , F 312  and F 314  corresponding to 4500+D 1  Hz, 5500+D 2  Hz and 6500+D 3  Hz, where 4500 Hz, 5500 Hz and 6500 Hz are the “nominal carrier frequencies” of the three carrier frequency modulators  310 ,  312 ,  314 , and where D 1 , D 2  and D 3  are the “correction factors” selected to ensure pitch synchronicity. 
     To better understand what is meant by “pitch synchronicity”, reference is made to  FIG. 4A , which shows the spectrum of the residual signal S 1 *R at the output of the inverse filter  307 . Since what is presently being described is the excitation signal generator  210 , it can be assumed that the mode indicator M 0  is indicative of the signal  51  being in strong harmonic mode. Accordingly, one will notice the presence of distinct frequency components  402  (also called “harmonics”) in the spectrum of the residual signal S 1 *R and, more particularly, in the portion of the spectrum of the residual signal S 1 *R corresponding to the frequency range admitted by the bandpass filter  306 . The frequency components  402  obey what is known as a harmonic relationship, i.e., adjacent ones of the harmonics are separated by the fundamental frequency F 0  (which was determined by the pitch analysis module  206 ). 
     One will also appreciate that for a naturally sounding signal containing harmonics both inside and outside the frequency range admitted by the bandpass filter  306 , such harmonics would all obey the same harmonic relationship (i.e., adjacent ones of the harmonics are separated by the same aforesaid fundamental frequency F 0 ). With this knowledge, it is possible to predict at which frequencies one should expect to find harmonics outside the frequency range admitted by the bandpass filter  306 , and more specifically inside the frequency ranges that are occupied by the outputs of the carrier frequency modulators  310 ,  312 ,  314 . Since the output of each carrier frequency modulator contains a shifted version of the residual signal S 1 *R whose harmonics, though frequency-shifted as a whole, remain mutually spaced by the fundamental frequency F 0 , one will appreciate that consistency with a naturally sounding signal can be obtained by ensuring that the frequency-shifted harmonics together with the frequency components  402  collectively obey the same harmonic relationship as the frequency components  402  obeyed on their own. This can be achieved by controlling the amount of frequency shift in order to achieve the situation where:
         the lowest-frequency harmonic of the modulated signal E 310  is separated by F 0  from the highest-frequency harmonic of the residual signal S 1 *R;   the lowest-frequency harmonic of the modulated signal E 312  is separated by F 0  from the highest-frequency harmonic of the modulated signal E 310 ; and   the lowest-frequency harmonic of the modulated signal E 314  is separated by F 0  from the highest-frequency harmonic of the modulated signal E 312 .       

     Controlling the amount of shift corresponds to adjusting the nominal carrier frequency of each carrier frequency modulator by the respective correction factor. For example, as illustrated in  FIG. 4B , when the correction factor D 310  is too low, the lowest-frequency harmonic of the modulated signal E 310  will be separated by less than F 0  from the highest-frequency harmonic of the residual signal S 1 *R.  FIG. 4C  shows the situation when the correction factor D 310  is correctly chosen, such that the lowest-frequency harmonic of the modulated signal E 310  will be separated by F 0  from the highest-frequency harmonic of the signal residual S 1 *R. Finally,  FIG. 4D  shows the situation when the correction factor D 310  is too high, such that the lowest-frequency harmonic of the modulated signal E 310  will be separated by more than F 0  from the highest-frequency harmonic of the residual signal S 1 *R. Thus, the correction factors determined (either implicitly or explicitly) by the carrier frequency selection module  326  are a function of the fundamental frequency F 0  and the bandwidth and center frequency of the bandpass filter  306 . One will note that individual correction factors are not expected to exceed the fundamental frequency F 0 , which typically ranges from about 65 Hz to about 400 Hz depending on the age and gender of the speaker, without being limited to this range. 
     Returning now to  FIG. 3 , the excitation signal generator  210  enters the second operational state in response to the mode indicator M 0  being indicative of either of the other two modes (i.e., unvoiced mode or mixed mode). In this second operational state, the signal S 1 * exiting the bandpass filter  306  feeds an envelope operator  318  without passing through the inverse filter  307 . The envelope operator  318  is configured to take the absolute value of the signal S 1 *, and the resulting envelope signal, denoted E 318 , is provided to a first input of a modulator  320 . A second input of the modulator  320  is provided with a noise signal E 322  emitted by, for example, a Gaussian noise generator  322  capable of producing a practical equivalent of a random variable with zero mean, unity variance and unity standard deviation. The output of the modulator  320  corresponds to the highband excitation signal E 0 , which is present at the output of the switch  304 . 
     Returning now to  FIG. 2 , the highband excitation signal E 0  is fed to a first input of a multiplication block  218 . A second input of the multiplication block  218  is provided by the output of the excitation gain estimator  216 , which is now described in further detail. In particular, based on the fundamental frequency F 0  and the lowband linear spectrum frequencies L 0 , as well as on the mode indicator M 0 , the excitation gain estimator  216  produces a highband excitation gain, denoted G 0 . The highband excitation gain G 0  can be defined as the square root of the energy ratio between (i) the highband components (i.e., including frequency components in the highband range that may, in a non-limiting example, extend between 4000 Hz and 7000 Hz) expected to have been present in the true wideband speech from which the signal S 1  was derived and (ii) an expected artificial highband speech signal which would be produced by the excitation signal E 0  from the excitation signal generator  210  is applied to a synthesis filter with a spectrum corresponding to estimated highband linear spectrum frequencies. 
     Various techniques can be used for producing the highband excitation gain G 0 . For example, one can employ three separate estimators, depending on the mode indicator M 0 . In a specific non-limiting example embodiment, each of the three estimators utilizes 256 entries of a respective fifteen- (15-) dimensional vector-quantized codebook, with fourteen (14) of the total number of dimensions being the lowband linear spectrum frequencies L 0  (as provided by the LP analysis module  208 ), and the fifteenth dimension being the highband excitation gain G 0 . The three codebooks can be trained by a typical Generalized Lloyd-Max method, whereby each VQ codevector is the centroid of 256 cells of training data and the cells are clustered using a minimum Euclidian distance criterion. In addition to aforementioned VQ estimation methods, other statistical methods, such as Gaussian Mixture Modeling (GMM) and hidden Markov Modeling (HMM) can also be utilized to estimate the highband excitation gain G 0 . 
     The multiplication block  218  multiplies the highband excitation signal E 0  by the highband excitation gain G 0  to produce a scaled highband excitation signal, denoted E 1 , which is fed to a first input of a highband linear prediction synthesis filter  220 . A second input of the highband linear prediction synthesis filter  220  is provided by the LSF estimator  214 , which is now described. 
     The LSF estimator  214  produces a set of highband linear spectrum frequencies, denoted L 1 , based on the fundamental frequency F 0 , the lowband linear spectrum frequencies L 0  and the mode indicator M 0 . Various techniques can be used for producing the highband linear spectrum frequencies L 1 . For example, one can employ three separate estimators, depending on the mode indicator M 0 . Each estimator could employ a known statistical method, such as vector quantization (VQ), Gaussian Mixture Model (GMM) and Hidden Markov Model (HMM). In a specific non-limiting example embodiment, each of the three estimators utilizes 256 entries of a respective twenty-four- (24-) dimensional vector-quantized codebook, with fourteen (14) of the total number of dimensions being the lowband linear spectrum frequencies L 0  (as provided by the LP analysis module  208 ), and the remaining ten (10) dimensions being the highband spectrum linear spectrum frequencies L 1 . The three codebooks can be trained by a typical Generalized Lloyd-Max method, whereby each VQ codevector is the centroid of 256 cells of training data and the cells are clustered using a minimum Euclidian distance criterion. 
     Based on the highband linear spectrum frequencies L 1  and the scaled highband excitation signal E 1 , the highband linear prediction synthesis filter  220  produces an artificial highband speech signal, denoted S 2 . In a specific non-limiting embodiment, the highband linear prediction synthesis filter  220  can be a tenth order all-pole filter, but the present invention does not particularly limit the number of poles or any other characteristic of the highband linear prediction synthesis filter  220 . In the case where the highband linear prediction synthesis filter  220  is indeed a ten-pole filter, each of the ten linear predictive coefficients representing the spectrum of the artificial highband speech signal S 2  is multiplied by a respective expansion factor, Gamma, to i power, where i is equal to 0, 1, . . . 10. Setting Gamma to 253/256 gives a fixed 60 Hz bandwidth expansion of each pole. 
     Finally, the signal S 1  is delayed by a delay block  224  that is configured to have the same delay as the time it took for the artificial highband speech signal S 2  to be generated from the signal S 1 . The artificial highband speech signal S 2  and the delayed version of the signal S 1  are combined together at a summation block  222  to form the bandwidth-extended speech signal  36 . In an example, the bandwidth of the signal S 1  will be approximately 100-4000 Hz, the bandwidth of the artificial highband signal S 2  will be approximately 4000-7000 Hz, and therefore the bandwidth extended speech signal  36  will have a bandwidth of approximately 100-7000 Hz. In another example, the bandwidth of the signal S 1  will be approximately 300-4000 Hz, the bandwidth of the artificial highband signal S 2  will be approximately 4000-6000 Hz, and therefore the bandwidth extended speech signal  36  will have a bandwidth of approximately 300-6000 Hz. Of course, other bandwidth combinations are within the scope of the present invention. 
     Those skilled in the art will appreciate that the present invention does not preclude the use of additional techniques, in conjunction with those described herein, to expand other (e.g., lower-frequency) portions of the spectrum of a band-limited signal. Thus, combining the teachings of the present invention with other expansion techniques may result in added benefits. 
     Those skilled in the art will appreciate that in some embodiments, the functionality of the bandwidth extension module  34  may be implemented using pre-programmed hardware or firmware elements (e.g., application specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), etc.), or other related components. In other embodiments, the functionality of the bandwidth extension module  34  may be achieved using a computing apparatus that has access to a code memory (not shown) which stores computer-readable program code for operation of the computing apparatus. The computer-readable program code could be stored on a medium which is fixed, tangible and readable directly by the bandwidth extension module  34 , (e.g., removable diskette, CD-ROM, ROM, fixed disk, USB drive), or the computer-readable program code could be stored remotely but transmittable to the bandwidth extension module  34  via a modem or other interface device (e.g., a communications adapter) connected to a network (including, without limitation, the Internet) over a transmission medium. The transmission medium may be either a non-wireless medium (e.g., optical or analog communications lines) or a wireless medium (e.g., microwave, infrared or other transmission schemes) or a combination thereof. 
     While specific embodiments of the present invention have been described and illustrated, it will be apparent to those skilled in the art that numerous modifications and variations can be made without departing from the scope of the invention as defined in the appended claims.

Metadata:
Filing Date: 20100521
Publication Date: 20130115
Grant Date: 20130115
Priority Date: 20050902
Inventors: KABAL PETER
RABIPOUR RAFI
QIAN YASHENG
Assignee: APPLE INC
CPC Classifications: [{"code": "G10L25/18", "inventive": false, "first": false, "tree": "[]"}, {"code": "G10L21/038", "inventive": true, "first": true, "tree": "[]"}, {"code": "G10L25/18", "inventive": false, "first": false, "tree": "[]"}, {"code": "G10L21/038", "inventive": true, "first": true, "tree": "[]"}]
Family ID: 42710598