Transformers.js documentation

utils/audio

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

utils/audio

Helper module for audio processing.

These functions and classes are only used internally, meaning an end-user shouldn’t need to access anything here.


utils/audio.read_audio(url, sampling_rate) β‡’ <code> Promise. < Float32Array > </code>

Helper function to read audio from a path/URL.

Kind: static method of utils/audio
Returns: Promise.<Float32Array> - The decoded audio as a Float32Array.

ParamTypeDescription
urlstring | URL

The path/URL to load the audio from.

sampling_ratenumber

The sampling rate to use when decoding the audio.


read_audio~audio : <code> Float32Array </code>

Kind: inner property of read_audio


utils/audio.hanning(M) β‡’ <code> Float64Array </code>

Generates a Hanning window of length M.

Kind: static method of utils/audio
Returns: Float64Array - The generated Hanning window.

ParamTypeDescription
Mnumber

The length of the Hanning window to generate.


utils/audio.mel_filter_bank(num_frequency_bins, num_mel_filters, min_frequency, max_frequency, sampling_rate, [norm], [mel_scale], [triangularize_in_mel_space]) β‡’ <code> Array. < Array < number > > </code>

Creates a frequency bin conversion matrix used to obtain a mel spectrogram. This is called a mel filter bank, and various implementation exist, which differ in the number of filters, the shape of the filters, the way the filters are spaced, the bandwidth of the filters, and the manner in which the spectrum is warped. The goal of these features is to approximate the non-linear human perception of the variation in pitch with respect to the frequency.

Kind: static method of utils/audio
Returns: Array.<Array<number>> - Triangular filter bank matrix, which is a 2D array of shape (num_frequency_bins, num_mel_filters). This is a projection matrix to go from a spectrogram to a mel spectrogram.

ParamTypeDescription
num_frequency_binsnumber

Number of frequencies used to compute the spectrogram (should be the same as in stft).

num_mel_filtersnumber

Number of mel filters to generate.

min_frequencynumber

Lowest frequency of interest in Hz.

max_frequencynumber

Highest frequency of interest in Hz. This should not exceed sampling_rate / 2.

sampling_ratenumber

Sample rate of the audio waveform.

[norm]string

If "slaney", divide the triangular mel weights by the width of the mel band (area normalization).

[mel_scale]string

The mel frequency scale to use, "htk" or "slaney".

[triangularize_in_mel_space]boolean

If this option is enabled, the triangular filter is applied in mel space rather than frequency space. This should be set to true in order to get the same results as torchaudio when computing mel filters.


utils/audio.spectrogram(waveform, window, frame_length, hop_length, options) β‡’ <code> Object </code>

Calculates a spectrogram over one waveform using the Short-Time Fourier Transform.

This function can create the following kinds of spectrograms:

  • amplitude spectrogram (power = 1.0)
  • power spectrogram (power = 2.0)
  • complex-valued spectrogram (power = None)
  • log spectrogram (use log_mel argument)
  • mel spectrogram (provide mel_filters)
  • log-mel spectrogram (provide mel_filters and log_mel)

In this implementation, the window is assumed to be zero-padded to have the same size as the analysis frame. A padded window can be obtained from window_function(). The FFT input buffer may be larger than the analysis frame, typically the next power of two.

Kind: static method of utils/audio
Returns: Object - Spectrogram of shape (num_frequency_bins, length) (regular spectrogram) or shape (num_mel_filters, length) (mel spectrogram).

ParamTypeDefaultDescription
waveformFloat32Array | Float64Array

The input waveform of shape (length,). This must be a single real-valued, mono waveform.

windowFloat32Array | Float64Array

The windowing function to apply of shape (frame_length,), including zero-padding if necessary. The actual window length may be shorter than frame_length, but we're assuming the array has already been zero-padded.

frame_lengthnumber

The length of the analysis frames in samples (a.k.a., fft_length).

hop_lengthnumber

The stride between successive analysis frames in samples.

optionsObject
[options.fft_length]number

The size of the FFT buffer in samples. This determines how many frequency bins the spectrogram will have. For optimal speed, this should be a power of two. If null, uses frame_length.

[options.power]number1.0

If 1.0, returns the amplitude spectrogram. If 2.0, returns the power spectrogram. If null, returns complex numbers.

[options.center]booleantrue

Whether to pad the waveform so that frame t is centered around time t * hop_length. If false, frame t will start at time t * hop_length.

[options.pad_mode]string""reflect""

Padding mode used when center is true. Possible values are: "constant" (pad with zeros), "edge" (pad with edge values), "reflect" (pads with mirrored values).

[options.onesided]booleantrue

If true, only computes the positive frequencies and returns a spectrogram containing fft_length // 2 + 1 frequency bins. If false, also computes the negative frequencies and returns fft_length frequency bins.

[options.preemphasis]number

Coefficient for a low-pass filter that applies pre-emphasis before the DFT.

[options.mel_filters]Array.<Array<number>>

The mel filter bank of shape (num_freq_bins, num_mel_filters). If supplied, applies this filter bank to create a mel spectrogram.

[options.mel_floor]number1e-10

Minimum value of mel frequency banks.

[options.log_mel]stringnull

How to convert the spectrogram to log scale. Possible options are: null (don't convert), "log" (take the natural logarithm) "log10" (take the base-10 logarithm), "dB" (convert to decibels). Can only be used when power is not null.

[options.reference]number1.0

Sets the input spectrogram value that corresponds to 0 dB. For example, use max(spectrogram)[0] to set the loudest part to 0 dB. Must be greater than zero.

[options.min_value]number1e-10

The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). For a power spectrogram, the default of 1e-10 corresponds to a minimum of -100 dB. For an amplitude spectrogram, the value 1e-5 corresponds to -100 dB. Must be greater than zero.

[options.db_range]number

Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.

[options.remove_dc_offset]boolean

Subtract mean from waveform on each frame, applied before pre-emphasis. This should be set to true in order to get the same results as torchaudio.compliance.kaldi.fbank when computing mel filters.

[options.max_num_frames]number

If provided, limits the number of frames to compute to this value.

[options.do_pad]booleantrue

If true, pads the output spectrogram to have max_num_frames frames.

[options.transpose]booleanfalse

If true, the returned spectrogram will have shape (num_frames, num_frequency_bins/num_mel_filters). If false, the returned spectrogram will have shape (num_frequency_bins/num_mel_filters, num_frames).


utils/audio.window_function(window_length, name, options) β‡’ <code> Float64Array </code>

Returns an array containing the specified window.

Kind: static method of utils/audio
Returns: Float64Array - The window of shape (window_length,) or (frame_length,).

ParamTypeDefaultDescription
window_lengthnumber

The length of the window in samples.

namestring

The name of the window function.

optionsObject

Additional options.

[options.periodic]booleantrue

Whether the window is periodic or symmetric.

[options.frame_length]number

The length of the analysis frames in samples. Provide a value for frame_length if the window is smaller than the frame length, so that it will be zero-padded.

[options.center]booleantrue

Whether to center the window inside the FFT buffer. Only used when frame_length is provided.


utils/audio~hertz_to_mel(freq, [mel_scale]) β‡’ <code> T </code>

Kind: inner method of utils/audio

ParamTypeDefault
freqT
[mel_scale]string"htk"

utils/audio~mel_to_hertz(mels, [mel_scale]) β‡’ <code> T </code>

Kind: inner method of utils/audio

ParamTypeDefault
melsT
[mel_scale]string"htk"

utils/audio~_create_triangular_filter_bank(fft_freqs, filter_freqs) β‡’ <code> Array. < Array < number > > </code>

Creates a triangular filter bank.

Adapted from torchaudio and librosa.

Kind: inner method of utils/audio
Returns: Array.<Array<number>> - of shape (num_frequency_bins, num_mel_filters).

ParamTypeDescription
fft_freqsFloat64Array

Discrete frequencies of the FFT bins in Hz, of shape (num_frequency_bins,).

filter_freqsFloat64Array

Center frequencies of the triangular filters to create, in Hz, of shape (num_mel_filters,).


utils/audio~linspace(start, end, num) β‡’

Return evenly spaced numbers over a specified interval.

Kind: inner method of utils/audio
Returns: num evenly spaced samples, calculated over the interval [start, stop].

ParamTypeDescription
startnumber

The starting value of the sequence.

endnumber

The end value of the sequence.

numnumber

Number of samples to generate.


utils/audio~padReflect(array, left, right) β‡’ <code> T </code>

Kind: inner method of utils/audio
Returns: T - The padded array.

ParamTypeDescription
arrayT

The array to pad.

leftnumber

The amount of padding to add to the left.

rightnumber

The amount of padding to add to the right.


utils/audio~_db_conversion_helper(spectrogram, factor, reference, min_value, db_range) β‡’ <code> T </code>

Helper function to compute amplitude_to_db and power_to_db.

Kind: inner method of utils/audio

ParamType
spectrogramT
factornumber
referencenumber
min_valuenumber
db_rangenumber

utils/audio~amplitude_to_db(spectrogram, [reference], [min_value], [db_range]) β‡’ <code> T </code>

Converts an amplitude spectrogram to the decibel scale. This computes 20 * log10(spectrogram / reference), using basic logarithm properties for numerical stability. NOTE: Operates in-place.

The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it. This means that large variations in energy may not sound all that different if the sound is loud to begin with. This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

Kind: inner method of utils/audio
Returns: T - The modified spectrogram in decibels.

ParamTypeDefaultDescription
spectrogramT

The input amplitude (mel) spectrogram.

[reference]number1.0

Sets the input spectrogram value that corresponds to 0 dB. For example, use np.max(spectrogram) to set the loudest part to 0 dB. Must be greater than zero.

[min_value]number1e-5

The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). The default of 1e-5 corresponds to a minimum of -100 dB. Must be greater than zero.

[db_range]number

Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.


utils/audio~power_to_db(spectrogram, [reference], [min_value], [db_range]) β‡’ <code> T </code>

Converts a power spectrogram to the decibel scale. This computes 10 * log10(spectrogram / reference), using basic logarithm properties for numerical stability. NOTE: Operates in-place.

The motivation behind applying the log function on the (mel) spectrogram is that humans do not hear loudness on a linear scale. Generally to double the perceived volume of a sound we need to put 8 times as much energy into it. This means that large variations in energy may not sound all that different if the sound is loud to begin with. This compression operation makes the (mel) spectrogram features match more closely what humans actually hear.

Based on the implementation of librosa.power_to_db.

Kind: inner method of utils/audio
Returns: T - The modified spectrogram in decibels.

ParamTypeDefaultDescription
spectrogramT

The input power (mel) spectrogram. Note that a power spectrogram has the amplitudes squared!

[reference]number1.0

Sets the input spectrogram value that corresponds to 0 dB. For example, use np.max(spectrogram) to set the loudest part to 0 dB. Must be greater than zero.

[min_value]number1e-10

The spectrogram will be clipped to this minimum value before conversion to decibels, to avoid taking log(0). The default of 1e-10 corresponds to a minimum of -100 dB. Must be greater than zero.

[db_range]number

Sets the maximum dynamic range in decibels. For example, if db_range = 80, the difference between the peak value and the smallest value will never be more than 80 dB. Must be greater than zero.


< > Update on GitHub