W AV 2T OK : D EEP S EQUENCE T OKENIZER FOR A UDIO
R ETRIEVAL
Anonymous authors
Paper under double-blind review
A BSTRACT
Search over audio sequences is a fundamental problem.
In this paper, we propose a method to extract concise discrete representations for audio that can be used for efficient retrieval.
Our motivation comes from orthography which represents speech of a given language in a concise and distinct discrete form.
The proposed method, Wav2Tok, learns such representations for any kind of audio, speech or non-speech, from pairs of similar audio.
Wav2Tok compresses the query and target sequences into shorter sequences of tokens that are faster to match.
The learning method makes use of CTC loss and expectation maximization algorithm, which are generally used for supervised automatic speech recognition and for learning discrete latent variables, respectively.
Experiments show a consistent performance of Wav2Tok across two audio retrieval tasks, namely, music search (query by humming) and speech search via audio query, outperforming state of the art baselines.
I NTRODUCTION
Sequence Retrieval aims at retrieving sequences similar to a query sequence, with the constraint that an ordered alignment exists between the query and the target sequence.
In this paper, we address the following problem: Can we extract discrete tokens from any continuous signal for the purpose of retrieval of similar signals?
This problem has deep connections with tasks such as child language acquisition, music cognition and learning languages without written forms.
Some direct applications of the proposed task include speech search, where the order of constituent units, such as phonemes, syllables or words, remains same; and music search – query by humming or query by example – where the order of constituent units, such as relative notes or phrases, remains same.
Apart from audio, the problem also extends to tasks such as handwritten word search and gesture search.
One can define similarity metrics over sequences using methods based on Dynamic Time
Warping (DTW) (M ¨ uller, 2007).
These methods are inefficient if the sequences are continuous valued and have high sampling rates.
Moreover, they depend on matching hand-made features, which are ineffective in the face of high variability of query sequences.
Problems such as spoken term detection involve detection of a query utterance in a long speech audio.
The search space is huge, and performing DTW based search of query takes long time (Rodriguez-Fuentes et al., 2014).
A more efficient way of sequence retrieval is by mapping them to sequences of discrete tokens.
Automatic speech recognition (ASR) can be employed for this purpose (Mamou et al., 2013).
However, ASR training requires knowledge of basic units of transcription.
The popularly used units are phonemes and graphemes.
This method thus becomes language dependent.
Non-linguistic sounds, such as cough and sneeze, could be mapped to certain tokens defined for them.
This approach could not be used when precise tokens are not defined, e.g., music search.
In query by humming based music search, audio is mapped to discrete melody-related tokens, such as notes, and these token sequence are matched for search (Unal et al., 2008).
However, several music traditions do not have precise transcription systems.
There, one can tell if two pieces, or motifs, are similar but cannot precisely transcribe them to tokens.
The embellishments used in music could be too dynamic to be transcribed precisely.
Moreover, when a musically untrained user sings a query, s/he cannot hit the right notes matching the target song.
So the matching could rely on several factors other than notes, such as phonemes of lyrics (Mesaros & Virtanen, 2010), onset times (rhythm) (Kosugi et al., 2000), and note transitions (Ranjan & Arora, 2020).
Hence, the tokens to be used may not be derived from notes alone.
In this way, each tokenizer - for speech, music or other signals, in general - uses a domain-specific hand-made tokens defined by a domain expert.
In this paper, we propose a tokenizer to map audio sequences to sequences of discrete tokens with an aim of retrieval.
The mapping is learnt only from pairs of similar audio sequences.
The tokens are not defined manually, but correspond to distinct semantic units learned from pairs of similar audio sequences.
The method is general and can be applied to signals other than audio.
In this paper, we apply the proposed method to speech and music audio search, for the problems of spoken term detection and query by humming, respectively.
The proposed method, named Wav2Tok, encodes audio via a BiLSTM (Schuster & Paliwal, 1997) network.
The encoder generated representations are then mapped to discrete tokens via a K -means vector quantizer network.
Each discrete token corresponds to a discrete representation in the vector quantizer’s codebook which are intialized and updated via offline K -means clustering only.
Wav2Tok is trained with pairs of similar audio sequences in an self-supervised fashion without any transcription using a novel training algorithm.
For each pair, we average the encoder generated representations, which map to the same token, by the K -means vector quantizer network to generate a prototype for that token.
We then perform a contrastive learning task to increase similarity between the generated prototype for a particular token and the quantizer codebook discrete representation corresponding to the same token.
We simultaneously minimize the edit distance between the token sequences generated from each sequence in the pair via Connectionist temporal classification (CTC) (Graves et al., 2006) framework to constrain both sequences to get mapped to same token sequence.
We compare Wav2Tok to state of the art methods for discrete representation learning , such as
Wav2Vec 2.0, and state of the art (SOTA) ASR models finetuned to perform phonetic tokenization.
We evaluate the generalization capability of the tokens generated by the models on search experiments, namely, query-by-humming and spoken term detection.
Wav2Tok outperforms the baselines in performance and uses much lesser trainable parameters, ensuring faster inference and deployment.
R ELATED W ORK
Sequence Labelling.
With expert-defined tokens, various methods are popularly used for mapping sequences to tokens.
In conventional methods, Hidden Markov Models (Rabiner & Juang, 1986) and Conditional Random Fields (Lafferty et al., 2001) have been popularly used for sequence labelling.
These methods involve a significant amount of domain knowledge and many assumptions to make tractable models, which are avoided by End-to-End learning models such as Recurrent Neural Networks (RNNs) using Connectionist Temporal Classification framework (Graves et al., 2006).
Sequence labeling can be used for sequence retrieval by converting the sequences to tokens, which are easy to search over.
But this approach inevitably depends upon expert-defined tokens.
Unsupervised Speech Representation Learning.
Automatic Speech Recognition systems are pretrained on large amounts of untranscribed speech data to generate SOTA continuous representations which encodes the slowly varying phoneme features in raw speech.
The representations are then mapped to phoneme tokens via Connectionist Temporal Classification (CTC) (Graves et al., 2006) finetuning on small amount of transcribed audio.
Works like Contrastive Predictive Coding (CPC) (van den Oord et al., 2018), Autoregressive Predictive Coding (APC) (Chung & Glass, 2020), and
Wav2Vec (Schneider et al., 2019) generate continuous representations with powerful autoregressive models pretrained to predict future time-step representations.
Further works started discretizing the continuous representations with VQ-VAE (van den Oord et al., 2017) to generate discrete representations for speech.
Works like VQ-Wav2Vec (Baevski et al., 2019) and VQ-APC (Chung et al., 2020) discretize the representations and perform the same prediction tasks as in Wav2Vec (Schneider et al., 2019) and APC (Chung & Glass, 2020) respectively but over discrete representations.
In VQ-Wav2Vec, the discrete representations are generated with either a K-Means Vector Quantizers (Baevski et al., 2019) or Gumbel-Softmax based Vector Quantizers (Baevski et al., 2019).
The learnt discrete representations are used to pretrain a BERT (Devlin et al., 2018) to generate stronger continuous representations much in correspondence to BERT pretraining in Natural Language Processing.
Wav2Vec 2.0 (Baevski et al., 2020) generates strong discrete representations via a Gumble Softmax based Vector Quantizer (Baevski et al., 2019) by pretraining huge transformer networks in identifying the correct discrete representation at a masked time step corresponding to the encoder generated latent representation at that time step.
The main motivation of these methods for discretizing raw audio in a latent space was to identify all possible acoustic units from speech than identify a unique representation for each phoneme.
The codes or tokens generated by the models aren’t constrained to be interpretable and hence initialised in large numbers ( ∼ 102 .
4 K codes).
Optimization of the pretraining loss function converges a subset of these codes or tokens to represent acoustic units.
CTC based finetuning over transcribed audio groups these acoustic units to
K distinct phonemes (linguistic units) as present in the transcriptions.
HuBERT (Hsu et al., 2021) learns continuous representations which are generated with huge transformer networks by utilizing a few iterations of offline clustering to provide aligned target labels for a BERT-like prediction loss.
The clustering is performed with an ensemble of K -means clusterers with K = { 100 , 500 } clusters on transformer-generated context representations.
Wav2Vec-Unsupervised (Baevski et al., 2021) presents an approach for completely unsupervised speech recognition.
Representations generated by a pretrained Wav2Vec 2.0 model are mapped to phoneme sequences using a generator network.
A discriminator then identifies if the phoneme sequence is real or fake based on the knowledge extracted from unlabeled text data which are also converted to phoneme sequences.
Note, all aforementioned approaches generate representations for speech using industry-grade models with large number of parameters pretrained on large amount of unlabeled audio and finetuned on transcribed audio.
Our work aims to perform phoneme level tokenization of input audio using models having a minute number of parameters compared to the aforementioned models.
Our learning approach can generate semantic tokens which outperform phonetic tokens generated by the baseline models with large number of parameters while training pairwise on small amount of unlabeled audio data.
Audio Representations for Retrieval.
Now Playing (Arcas et al., 2017) and (Chang et al., 2020) use a Neural Network Fingerprinter (NNFP) module outputting representations which are efficient for search in query-by-example tasks where the difference between query and actual song is pretty minute in comparison to humming where only the melody is sung.
Now Playing (Arcas et al., 2017) trains representations by optimizing the Triplet loss (Schroff et al., 2015) and (Chang et al., 2020) trains representations by simulating the Maximum Inner Product Search (MIPS) on mini-batches of representations.
For Query by Humming task, (Mostafa et al., 2016) and (Mostafa & Fung, 2017) use deep learning models like DNNs and CNNs to generate representations which they map to midinumbers or note tokens.
Such works require note transcribed data to train the models.
For Spoken
Term Detection task, approaches like (Zhang & Glass, 2009), (Rodriguez-Fuentes et al., 2014), (Lee et al., 2015), (Ram et al., 2018) convert audio to sequences of feature vectors and apply different variations of DTW based template matching to detect query in long utterances of speech which is time-consuming.
Cross Domain Alignment.
Given a pair of semantically similar inputs for training, tasks such as visual question answering (text and image) and machine translation (text) involve learning of an alignment.
The alignment here is not ordered and the inputs may be from different modalities.
Attention models have been used to find alignment between output entities and input regions (Yao et al., 2018).
(Chen et al., 2020) use Gromov-Wasserstein distance between output and input entities to match them.
However, there is no notion of tokens there, rather the salient entities in the input are represented as vectors in a graph.
Graph Matching.
Graph Neural Networks (Gori et al., 2005) are used to generate embeddings for graphs.
These embeddings are used to perform graph matching to find similarity of structured graphs (Li et al., 2019).
However, they perform the matching jointly on the pair of inputs, rather than representing each input independently.
This makes them unsuitable for the search problem at hand due to large run-time complexity.
The distance metrics used for graph matching are based on edit distance (Li et al., 2019) and Wasserstein distance (Chen et al., 2020).
P ROBLEM S TATEMENT
We aim to map X , a sequence of vectors, to ˜ T , a sequence of discrete tokens from a finite alphabet A , such that the similarity of sequences is preserved in the sense of edit distance.
The length of sequence ˜ T may be less than or equal to that of the sequence X .
In other words, given a pair of similar sequences ( X i , X j ) and sequence X k which is not similar to either sequences in the pair, we want to map them to token sequences such that ED ( ˜ T i , ˜ T j ) should be less than min { ED ( ˜ T i , ˜ T k ) , ED ( ˜ T j , ˜ T k ) } , where ED ( · , · ) is the edit distance between two sequences.
M ODEL A RCHITECTURE
Wav2Tok is comprised of an encoder f : X (cid:55)→ Z which takes as input a temporal sequence of audio features X =
[ x t ∈ R n ; t ∈ [ T ]] of length T , where x t is the feature vector at time step t , and outputs a sequence of L- 2 normalised representations Z =
[ z t = f ( x t ) ∈
R m ; t ∈ [ T ]] .
The encoder is implemented as a 2 -layer BiLSTM followed by a L- 2 normalization layer.
BiLSTMs summarise information in both directions and encodes surrounding context.
A K -means vector quantizer network g : Z (cid:55)→ T then labels sequence Z at each time-step with tokens belonging to a finite K -element alphabet
A =
[ K ] and generates sequence of tokens T =
[ τ t = g ( z t ) ∈ A ; t ∈ [ T ]] .
Network g vector quantizes input z t with a codebook E = { e k ∈ Z ; k ∈ [ K ] } comprised of | A | =
K discrete representations which are cluster centroids in representation space Z and outputs token τ t = arg max k z t · e k .
Note, here the dot product gives a cosine similarity score since both the vectors are L- 2 normalised, as a result, e k ∈ E closest to z t is chosen as it’s discrete representation and index k as it’s token τ t .
The K discrete representations in network g are trainable parameters.
A compressor C compresses sequence of tokens T to sequence ˜ T of length ˜ T ≤ T by deleting all consecutive repetitions of tokens.
C also generates the corresponding compressed sequence ˜ Z of length ˜ T by averaging representations
z t ∈
Z over the consecutive tokens and L- 2 normalising the averaged representation.
Figure 1 presents an illustration demonstrating our model architecture.
Audio is converted to spectrogram X and passed as input to Wav2Tok.
An encoder block generates a sequence of representations Z of length T .
A K-means Vector Quantizer
then maps each representation in the sequence to a token to generate a sequence of tokens T .
Compressor C compresses T to sequence ˜ T .
T RAINING
Wav2Tok is trained on pairs of sequences of audio features ( X , X ′ ) where the raw audio corresponding to X ′ is an augmented replica of that corresponding to X .
We apply either pitch shift or time stretch or both augmentations to raw audio to generate it’s augmented replica.
X and X ′ may differ in sources as well, i.e. a different person may sing the recording corresponding to X ′ .
The discrete representations in quantizer g codebook E are initialized as K centroids obtained via offline K -means clustering over freshly initialized encoder generated representations.
Given ( X , X ′ ) , encoder f generates sequence of representations Z from input X and Z ′ from X ′ .
Quantizer g generates sequence of tokens T from input Z and T ′ from Z ′ via cosine similarity based comparison with codebook vectors e ∈ E initialized via offline clustering over freshly initialized representation space Z .
Compressor C compresses sequence of tokens T to sequence ˜ T and T ′ to ˜ T ′ .
We average all encoder generated representations in pair ( Z , Z ′ ) which map to the same token, say τ , to generate a prototype for τ .
We then perform a contrastive task where we compare the prototype with each of of the K discrete representations in codebook E and increase it’s similarity with the discrete representation corresponding to τ .
We also increase the likelihood that Wav2Tok maps pair ( X , X ′ ) to same token sequence via CTC framework to minimize edit distance between ˜ T and ˜ T ′ .
Our loss function is defined as,
[Equation]
[Equation]
where L m is loss defined for contrastive task, L ctc is the loss maximising aforementioned likelihood, and α, β are positive constants.
We optimize this loss function in a manner similar to Expectation
Contrastive Loss.
Let the set of K ′ ≤ K unique tokens occurring in pair ( ˜ T , ˜ T ′ ) be U
= { τ i ;
i ∈ [ K ′ ] } .
We generate a list of token prototypes P = { p τ ; τ ∈ U} where p τ is L- 2 normalised mean of representations in { z ∈ {Z ; Z ′ } : g ( z ) = τ } .
Figure 2 presents an illustration demonstrating how we generate list of token prototypes P .
Given p τ ∈ P , we perform a contrastive task to increase its similarity with discrete representation e τ ∈ E tok .
To compare p τ with the codebook, metrics such as cosine similarity and Euclidean distance could be used.
However, we find that using the following parameterized score for this purpose gives better performance,
[Equation]
where sg ( x ) ≡ x, ddx sg ( x ) ≡ 0 is the stop-gradient operator, σ ( . ) is sigmoid function generating a score in the range [0 , 1] and W ∈ R 1 × d is a parameter matrix.
s τ,k acts as a parameterized similarity score between p τ and discrete representation e k ∈ E .
We define our contrastive loss L m as,
[Equation]
Input audio X is augmented in the Augmentation block to generate augmented replica
X ′ .
The augmentation block may also output
X ′ as the same audio corresponding to X but with different source i.e. uttered by a different speaker.
Wav2Tok maps X and X ′ to compressed sequences of tokens ˜ T and ˜ T ′ respectively.
We generate a token prototype for each of the unique
tokens in ˜ T and ˜ T ′ by averaging all representations generated by Wav2Tok encoder network from inputs X and X
′ mapping to the same token.
This results in a list of token prototypes P which we use to perform our contrastive task
Likelihood Loss.
We maximize the likelihood that sequence X maps to token sequence ˜ T ′ , which corresponds to X ′ , via the CTC framework.
It puts a constraint to generate the same token sequence for X and X ′ .
We calculate the probability of x t mapping to token τ t =
k as l t,k = exp( f ( x t ) · sg ( e k ))
(cid:80) Ki =1 exp( f ( x t ) · sg ( e i )) .
The likelihood P ( ˜ T ′ |X ) is then calculated as sum of probabilities of all T -length paths π over tokens
τ ∈
A such that C ( π ) =
˜ T ′ .
The loss is defined as,
[Equation]
where the path probabilities are calculated over token probability scores in sequence l = { l t ∈
R K ; t ∈ [ T ] } via CTC forward-backward framework (Graves et al., 2006) without the use of blanks.
We present the CTC forward and backward variables for our use case in Appendix B.
Clustering.
We perform offline K -means clustering on a subset of encoder representations during initialisation of our network and at regular intervals during training to set the discrete representations in codebook E of network g .
Initialising the clusters in this way prevents Wav2Tok from converging to a local optimum during the matching task, as is the case we found with random initialization of centroids.
The intermittent clustering during training iteratively refines the discrete representations and prevents codebook collapse.
We use the sklearn library to perform K -means clustering.
We train Wav2Tok using the ADAM (Kingma & Ba, 2017) optimizer and a linear learning schedule with a learning rate of 0 .
001 and 8% of the training steps
E XPERIMENTS
We test the performance of tokens and encoder generated continuous representations of Wav2Tok in audio retrieval.
We perform Query by Humming (QbH) and Spoken Term Detection experiments to evaluate performance of Wav2Tok in comparison to the baselines.
6. M USIC M ELODY S EARCH : Q UERY BY H UMMING
Task.
Given a test query audio, we are to find in the search audio database the audio with the most similar melody.
Experiment Details.
We use the MIR-QbSH dataset which is composed of 4431 humming audio recordings of 30 s duration corresponding to 48 songs.
Each song is sung by several individuals.
All individuals sing the same part of the song.
The recordings have variation in the environments they were recorded in, tonal qualities, voices, pitch and time stretch.
We train our models on hums of 40 songs in MIR-QbSH dataset and evaluate search performance on hums of the remaining 8 songs.
The training dataset has 1970 hums for training and 676 hums for validation.
The test dataset has 225 hums as search database and 659 query hums.
We evaluate the performance of our models in identifying which song a given query corresponds to via comparison with all sequences in search database.
All the audios in our test dataset are converted to sequences of tokens or representations by each of the models.
Each query sequence is compared to all sequences in search database via Edit
Distance (ED) (if tokens) or DTW (if representations).
The song id of the most similar sequence in the search database is then selected as query song id.
We calculate Mean Reciprocal Ranking (MRR) score with ground truth song id of the queries for evaluation.
The Reciprocal Ranking (RR) score is given as 1 /r if the r th most similar sequence in search database has same song id as query.
All the audio recordings are converted to Short Term Fourier Transform (STFT) matrices before being passed as inputs to our models.
The STFT matrices are computed with 513 frequency bins, a window length of 1024 samples (summarising 128 ms of audio), and hop length of 512 samples.
6. S POKEN T ERM D ETECTION
Task.
Given a test query audio, we are to detect it’s occurrence in a long utterance.
Experiment Details.
We use the TIMIT dataset which is composed of 6300 utterances of English speech with time aligned word transcriptions.
We choose 59 most-occurring words with more than 2 characters as keywords and all others as non-keywords.
We use utterances of random sentences formed with 6 words sampled from a subset of 25 keywords for training and evaluate on STD experiments for the detection of the remaining 34 keywords.
The test dataset is composed ofutterances corresponding to the 34 queries and 100 long utterances per query with half of them containing a single occurrence of query amongst non-keywords and other half containing only nonkeywords.
Given a query and a long utterance, we convert both to sequences of tokens using each of the audio tokenizers.
We perform approximate string matching (Hall & Dowling, 1980) for detection of query in the utterance.
The STFT matrix inputs to the models are computed with 185 frequency bins, a window length of 368 samples (summarising 23 ms of audio), and hop length of 92 samples.
6. B ASELINES
Triplet.
We train encoder f : X (cid:55)→ Z to generate L- 2 normalised continuous representations for retrieval.
Encoder f is trained via optimizing the Triplet Loss (Schroff et al., 2015) as done in training a NNFP in Now Playing (Arcas et al., 2017).
Given pair of similar sequences ( X , X ′ ) , encoder f generates sequences Z and Z ′ .
We form a mini-batch of size N of triplets { z , z + , z − } where representation z is sampled from sequence Z , z
+ and z − are positive and negative samples respectively for z sampled from sequence Z ′ .
The loss is defined as, L Triplet =
(cid:80) N i =1 max {|| z
i − z
+ i
||
−
||
z i − z − i
||
+ m, 0 }
, where m is a margin for similarity.
MIPS.
We train encoder f : X (cid:55)→ Z to generate L- 2 normalised continuous representations for retrieval.
Encoder f is trained via simulation of MIPS (Mussmann & Ermon, 2016) on mini-batches of representations as proposed by (Chang et al., 2020).
Given pair of similar sequences ( X , X ′ ) , encoder f generates sequences Z and Z ′ .
We form a mini-batch of size N of pairs of { z , z + } where encoder generated representation z is sampled from sequence Z and z + is a positive for z sampled from Z ′ .
The loss is defined as, L MIPS =
− (cid:80) Ni =1 log
[Equation]
Wav2Vec2.
We train our audio tokenizer via Wav2Vec 2.0 (Baevski et al., 2020) learning framework.
Quantizer g in our audio tokenizer is chosen to be a Gumbel Softmax based Vector Quantizer (See Appendix C for details) as used in (Baevski et al., 2020) but with a single codebook with K members.
Given sequence X , encoder f outputs sequence of L- 2 normalised representations Z of length T .
Quantizer g outputs sequence of discrete representations Q = { q t =
g ( z t ∈ Z ); t = 1 , ..., T } .
We mask spans of 10 time steps with random starting indices in sequence Z
and then pass the new sequence to a transformer network h : Z
(cid:55)→ O which generates a sequence of contextualised representations O = { o t = h ( z t ∈ Z )
; t = 1 , ..., T } .
For transformer output o t over masked time step t , we identify the true discrete representation q t from a set D t composed of q t and
D distractors which are discrete representations sampled from other time steps.
The loss is defined as, L w ( o t , D t )
= − log
[Equation] [Equation]
[Equation]
q ))
+ L d , where sim ( a, b ) =
a
[Equation]
b || a |||| b || is cosine similarity and L d is codebook diversity loss.
Wav2Vec2P.
We train Wav2Vec
audio tokenizer with our variation of Wav2Vec 2.0 (Baevski et al., 2020) learning framework which learns discrete representations from pairs of similar sequences.
Given pair ( X , X ′ ) , encoder f outputs sequences Z of length T and Z ′ of length T ′ respectively.
Assuming T ≤ T ′ , we generate sequence Z + of length T whose t time step element z
+ t is a positive for z t ∈ Z sampled from sequence Z ′ .
Gumbel Softmax based Vector Quantizer g quantizes each representation in sequence Z + to generate sequence Q + .
We mask sequence Z and Z + at the same time steps.
Transformer h inputs masked sequences and generates sequences O and O + .
For masked time step t , we use transformer output o t to identify q + t ∈ Q + from set D + t with distractors sampled from sequence Q + and transformer output o + t to identify q t ∈ Q from set D t with distractors sampled from sequence Q .
The loss is defined as, L wP =
L w ( o t , D + t ) + L w ( o + t , D t ) .
Wav2Vec2-O.
The original Wav2Vec 2.0 base model with 12 Transformer blocks and 95 M parameters as proposed by (Baevski et al., 2020).
It is pretrained on 960 hours of LibriSpeech data and fine tuned on TIMIT dataset.
It uses K = 32 tokens for tokenization.
Wav2Vec2-Multi.
A Wav2Vec 2.0 large model with 24 Transformer blocks and 317 M parameters pretrained on 53 languages as proposed by (Conneau et al., 2020).
It is finetuned on Common Voice to detect all possible phonemes in training languages with K = 392 tokens.
Triplet and MIPS use a 2 -layer BiLSTM as encoder with 3 .
6 M parameters.
We use the LAMB optimizer (You et al., 2020) and a Cosine Annealing Learning Schedule (Loshchilov & Hutter, 2017) with a learning rate restart of 0 . 0001 to train them.
Wav2Vec2 and Wav2Vec2P use a 2 -layer
BiLSTM encoder with 3 .
6 M parameters to generate latent representations and 3 Transformer blocks with 8 .
5 M parameters.
Both are trained using the ADAM (Kingma & Ba, 2017) optimizer and a linear learning schedule with a learning rate of 0 . 001 and 8% of the training steps as warm-up steps.
Proposed Wav2Tok uses only a 2 -layer BiLSTM as encoder with 3 .
6 M parameters.
R ESULTS 7. M USIC M ELODY S EARCH : Q UERY BY H UMMING
We present search performances for 3 settings of query namely-
Query with no augmentation or
Vanilla Query (V), Time Stretched Query (TS), and Pitch Shifted Query (PS).
Time stretch and pitch shift are the most common augmentations that may be faced in query by humming data.
No augmentations were applied to audio in search database.
Evaluations are performed on sequences corresponding to songs not seen during training.
The results present generalizability of the tokens or representations generated by the models.
We set the number of tokens as K = 25 for Wav2Tok ,
Wav2Vec2, and Wav2Vec2P (See Appendix A.2 for experiments to support our choices).
Quality of Tokenization.
Table 1 presents the performance of the sequence of tokens ˜ T generated by the audio tokenizers on ED based similarity search.
Tokens generated by Wav2Tok present good generalization capabilities in terms of MRR and outperforms all the baselines.
It generates time and pitch invariant tokens as we see no drop in performance when either augmentation is ap- plied to query.
Wav2Vec2-O is trained on english speech only.
The tokens generated by it do not contain much melodic information but are robust to augmentations.
The multilingual training of
Wav2Vec2-Multi infuse both melodic and phonetic information to its 392 tokens, thereby giving good performance.
Wav2Tok outperforms both Wav2Vec2-O and Wav2Vec2-Multi given its pairwise training which allows it to infuse more melodic information to the tokens while also being trained on a small amount of unlabelled data.
The gumbel softmax based quantizer in Wav2Vec and Wav2Vec2P isn’t ideal for infusing melodic information to tokens but it does infuse phonetic information as will be seen in Section 7.2.
We compare the tokens with representations learnt by MIPS and Triplet evaluated on DTW based similarity search.
The continuous representations present subpar generalization to unseen songs.
We compare Wav2Tok with SOTA melody extraction algorithm proposed in (Salamon & G´omez, 2012) which converts hums to midi sequences.
Wav2Tok generates token sequences much smaller than the respective midi sequences and outperforms the midi tokens in search performance, search time and robustness.
In addition, Wav2Tok outperforms the algorithm in inference time.
We further compare Wav2Tok with SOTA QbH system proposed in (Mostafa &
Fung, 2017).
In our implementation, we map audio to midi sequences using the aforementioned
SOTA melody extraction algorithm instead of a CNN.
Given midi sequence 53 , 53 , 58 , 50 with durations 0 s, 0 .
5 s, 1 s, 2 s , a relative note sequence is generated as (0 , 0) , (0 , 0 . 5) , (5 , 1) , ( − 8 , 2) over which DTW is performed for retrieval.
Wav2Tok tokens outperform the SOTA QbH system in both performance and robustness; the performance of the latter drops drastically with time stretch.
We present the performances of the uncompressed sequences T and Z and compressed sequence ˜ Z generated by the audio tokenizers in Appendix A.1.
We observe a drop in performance for all audio tokenizers when we apply sequence compression to sequences T and Z .
Wav2Tok outperforms all the baselines and generates superior quality of continuous representations and discrete tokens.
Search Time.
Table 1 presents the search time taken for similarity search over the tokens or representations generated by the models.
The search time taken per query is 2 order of magnitude lesser for ED based Search over compressed sequence of tokens ˜ T than standard DTW based Search over continuous representations Z .
The pretrained models being finetuned on transcribed audio give the best tokens in terms of compression and search time.
Wav2Tok gives comparable tokens but outperforms the pretrained models in inference time
[Table]
[Table] [Table]
Ablation Studies.
Query by humming involves similarity based on melody information, which is carried by the semantic pairing of the audio in training data.
We constrain this pairing to include sequences not semantically similar and call this model Wav2Tok+NoSim.
We optimize the contrastive loss L m to train the model.
The results are shown in Table 2 (full table in Appendix A.3).
There is a significant drop in token robustness and performance but the representations suffer a small drop (see
Appendix A.3).
Hence, although the representation space may be well clustered, Wav2Tok is able to add more semantics to the tokens as it is being trained with pairs of similar sequences in comparison to Wav2Tok+NoSim.
We train Wav2Tok with cosine similarity scores instead of a parameterized score (Wav2Tok+Cos).
The drop in performance validates the enhancement brought about by using a parameterized score.
We also train Wav2Tok with L ctc only (Wav2Tok+CTC).
Such a training gave comparable performance to Wav2Tok+NoSim.
The use of both the losses give the best tokens.
Some Variations.
In Wav2Tok+Trans, we use 3 Transformer blocks amounting to 8 .
5 M parameters as our encoder.
There is an overall drop in performance and robustness as shown in Table 2.
Possible cause for such drop may be overfitting.
In Wav2Tok+NewInit, we associate the discrete representations with K centroids in the input space X .
Such association does not initialize our tokenizer with optimal centroids which cluster the space Z perfectly.
This results in significant drop in performance and robustness as shown in Table 2.
We train Wav2Tok on MIR-1K dataset (Wav2Tok+MIR1K) which is composed of polyphonic music recordings of 1000 distinct songs.
The tokens generalise well to monophonic hums in MIR-QbSH dataset giving comparable performance to midi tokens.
This validates that Wav2Tok tokens do learn melodic information and is robust to variations in- curred in hums.
We further compare Wav2Tok with log-mel features and token sequences (with no compression) obtained via quantization of log-mel features.
Wav2Tok tokens outperforms both.
7. S POKEN T ERM D ETECTION
Quality of Tokenization.
Table 3 presents the quality of tokezination of the query keywords by the models evaluated in the Spoken Term Detection experiments.
We present the performances of Wav2Vec2 , Wav2Vec2P, Wav2Vec2-O, Wav2Vec2-Multi and proposed Wav2Tok.
We conduct search experiments on a test dataset composed of a search database of 337 utterances of thekeywords used as queries in the Spoken Term Detection experiments and 1289 query utterances.
We identify the keyword to which each query corresponds to via comparison to all the 337 utterances in the search database via ED based similarity score.
The word id of the most similar utterance is selected as the word to which the query corresponds to.
We set K = 40 equivalent to the number of phonemes in English for speech experiments.
Wav2Tok gives the best performance in terms of MRR scores.
It outperforms huge models like Wav2Vec2-O and Wav2Vec2-Multi which are designed specifically for the task of phonetic tokenization of speech audio by a large margin while using a minute number of parameters in comparison to them.
Wav2Vec2 and Wav2Vec2P also outperform
Wav2Vec2-Multi and Wav2Vec2-O
while using smaller number of parameters.
Wav2Vec2-O
and
Wav2Vec2-Multi use a blank token to handle consecutive occurrences of the same tokens and to label background noise.
The utterances of each keyword in the test dataset are very small in time duration.
This causes Wav2Vec2-O to confuse word utterances as background noise.
It ends up generating a sequence of blank tokens and performs poorly in search.
Wav2Vec2-Multi using a larger number of phonetic tokens does not suffer this issue.
Wav2Tok , Wav2Vec2, and Wav2Vec2P have no such blank token.
This brings a drop in search performance with sequence compression.
We further present the performance of Wav2Tok trained on much larger LibriSpeech 100 hours dataset (Wav2Tok+LibriSpeech).
It is able to outperform Wav2Vec2-O and give comparable performance to Wav2Vec2-Multi.
Using a stronger encoder network will further improve the performance.
[Table]
[Table]
Spoken Term Detection.
We convert the query word utterance and the long utterance where we are to detect the query in to sequences of tokens by all our models and detect the occurrence of the query via approximate string matching.
We use fuzzysearch library to perform approximate string matching.
It automatically chooses the fastest algorithm for matching.
Table 4 presents the performance of Wav2Vec2 , Wav2Vec2P , Wav2Vec2-O, Wav2Vec2-Multi, and proposed Wav2Tok in
STD.
All the models give comparable performance in terms of F 1 - score with Wav2Tok performing slightly better.
We also implement the STD system proposed in (Anguera & Ferrarons, 2013) which performs highly competitive STD via subsequence DTW
(S-DTW) over gaussian posterior features.
In our implementation, we extract the posterior features with SOTA ASR models like Wav2Vec2-O and Wav2Vec2-Multi.
The results are presented in the column named DTW in Table 4.
Note, the results for other models in same column are for STD via S-DTW over representations.
We observe
STD over tokens to give better F1-score with Wav2Tok outperforming all the baselines in all cases.
C ONCLUSION AND F UTURE W ORK
In this paper, we presented an audio sequence tokenizer Wav2Tok that generates semantically meaningful ordered representations (or tokens) that can be used for efficient retrieval by query sequences.
The model learns only from pairs of semantically similar sequences, and outperforms state of the art approaches for spoken term detection and query by humming.
One may apply more efficient search algorithms such as locality sensitive hashing and longest common subsequence search on the generated tokens to further speed up the search.
The proposed framework can also be extended to image and video retrieval, as they also have spatial ordering.
We would like to investigate the domain specific, i.e., linguistic or musicological, aspects of the tokens so extracted.
For instance, during retrieval, the matching algorithm assume all the tokens to be equidistant from each other.
One may study or use the metric space of these tokens.
R EPRODUCIBILITY
The codes required to reproduce the results of this paper will be made available after the review.
The experiments are performed using standard datasets.
A F URTHER S TUDIES A. S EQUENCE C OMPRESSION
We present the quality of sequence of tokens T and sequence of representations Z and their corresponding compressed versions sequences (cid:101) T and (cid:101) Z generated by the audio tokenizers in Table 5.
Wav2Tok outperformed the baselines and generated the best quality of sequences T , Z , (cid:101) T and (cid:101) Z .
Sequence compression brings an order of magnitude drop in search time for all the audio tokenizers with a trade-off in search performance.
Compression from T to (cid:101) T increases the robustness of the token sequences generated by Wav2Tok to various augmentations.
Wav2Vec2P learnt better tokens and representations than Wav2Vec2 because of it’s pairwise training on similar audio.
[Table]
A. V ARIATION IN NUMBER OF TOKENS K
The effect of varying the size of alphabet A is shown in Table 6.
We train Wav2Vec2, Wav2Vec2P, and proposed Wav2Tok with alphabets of size K ∈ { 15 , 25 , 40 } .
Out of the three settings for K ,
K = 25 gives the best performance for all models.
Wav2Tok gives best performance for all settings of K .
A. A BLATION S TUDIES AND S OME V ARIATIONS
We present the full version of Table 2 in table 7.
Note Wav2Tok+NoSim repsentations are well clustered.
Wav2Tok+Trans representations are also comparable with Wav2Tok but the tokens are of lesser quality.
This is due to model overfitting.
A. Q UALITY OF R EPRESENTATIONS
We present the performance of the continuous representations generated by Wav2Tok and the baselines in Table 8.
Wav2Tok generates the best representations for music outperforming representations generated by the large Wav2Vec 2.0 models.
Wav2Tok trained on MIR1K generates representations outperforming domain-specific QbH baselines.
Note, Wav2Vec2-O outperforms Wav2Vec2Multi as the hums in the dataset were all in english.
Wav2Vec2-O is pretrained and finetuned on
English speech only while Wav2Vec2-Multi is pretrained multilingually.
Hence , Wav2Vec2-O gave better results.
[Table] [Table]
Wav2Tok+Trans ED 0.
Wav2Tok+MIR1K ED 0.
Wav2Tok ED 0.
A. T RAINING ON LARGER S PEECH DATASET
We train Wav2Tok on 100 -hours subset of LibriSpeech (Panayotov et al., 2015) dataset.
We evaluate the quality of tokenization of word utterances done by Wav2Tok on TIMIT (Garofolo et al., 1993) dataset.
We use a 2 -layer BiLSTM network with 3 . 6 million parameters as encoder network which takes MFCC feature sequences as input.
We perform tokenization with K = 40 tokens.
Wav2Tok outperforms Wav2Vec2-O by a large margin and gives comparable performance to
Wav2Vec2-Multi in terms of MRR score.
Wav2Tok uses a minute number of parameters in comparison to 95 million parameters in Wav2Vec2-O and 317 million parameters in Wav2Vec2-Multi.
Note,
Wav2Vec2-O and Wav2Vec2-Multi were pretrained on large amount of unlabelled speech data and
[Table]
finetuned with transcription to perform tokenization of audio.
Moreover, Wav2Vec2-O was finetuned to perform tokenization on TIMIT (Garofolo et al., 1993) dataset.
Proposed Wav2Tok was trained on 100 hours of LibriSpeech dataset only.
The tokens learnt by Wav2Tok on LibriSpeech (Panayotov et al., 2015)
dataset generalised well to TIMIT (Garofolo et al., 1993)
[Table]
[Table]
B CTC WITHOUT B LANKS
We present the forward and backward variables used in calculating the gradients of the CTC loss
L ctc ( X , ˜ T ′ ) with no blank tokens.
The forward variable is defined as ,
[Equation]
where π corresponds to all T -length paths over tokens such that C ( π ) =
˜ T ′ .
Here, C is a compressor which compresses π a T -length sequence of tokens via de-duplication.
We initialise as follows,
[Equation] and recursively calculate α t ( s ) as, [Equation] We set α t ( s ) = 0 , ∀ s < 1 .
The backward variable is defined as,
[Equation]
We initialise as follows,
[Equation]
and recursively calculate β t ( s ) as,
[Equation]
We set β t ( s ) = 0 , ∀ s > | ˜ T ′ | .
C G UMBEL S OFTMAX BASED V ECTOR Q UANTIZER
The Gumbel Softmax based Vector Quantizer (Baevski et al., 2019) quantizes input latent representation z t ∈ R m with C codebooks containing K quantizers e ∈ R K × mC each.
For our experiments, we set C = 1 and K ∈ { 15 , 25 , 40 } .
Given z t , one of the K quantizers from each of the C codebooks are chosen resulting in vectors e 1 , ..., e C .
The codebook vectors are then concatenated and linearly transformed from R m to R d to output a discrete representation q t ∈ R d .
z t is mapped to l ∈ R C × K logits to give probability scores for the choice of codeword.
The probability p c,k of choosing k th quantizer in c th codebook is given as,
[Equation]
[Equation] where τ is a non-negative temperature, n = − log ( − log ( u )) and u are samples from the uniform distribution Unif (0 , 1) .
During forward pass, the codeword is chosen as κ = arg max j p c,j .
During backward pass, the loss is calculated over the gumble softmax distribution p .
We use the straight-through gradient estimator (Yin et al., 2019) to estimate the gradient.
Codebook Diversity Loss L d .
This loss promotes equal use of all the entries in each of the C codebooks.
Minimization of this loss maximizes the entropy of the averaged softmax distribution ˜ p over the K entries for each codebook ˜ p c across a batch of utterances.
[Equation]