Speech Recognition with Quaternion Neural Networks
Abstract
Neural network architectures are at the core of powerful automatic speech recognition systems (ASR).
However, while recent researches focus on novel model architectures, the acoustic input features remain almost unchanged.
Traditional
ASR systems rely on multidimensional acoustic features such as the Mel ﬁlter bank energies alongside with the ﬁrst, and second order derivatives to characterize time-frames that compose the signal sequence.
Considering that these components describe three different views of the same element, neural networks have to learn both the internal relations that exist within these features, and external or global dependencies that exist between the time-frames.
Quaternion-valued neural networks (QNN), recently received an important interest from researchers to process and learn such relations in multidimensional spaces.
Indeed, quaternion numbers and QNNs have shown their efﬁciency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with up to four times less learning parameters than real-valued models.
We propose to investigate modern quaternion-valued models such as convolutional and recurrent quaternion neural networks in the context of speech recognition with the TIMIT dataset.
The experiments show that QNNs always outperform real-valued equivalent models with way less free parameters, leading to a more efﬁcient, compact, and expressive representation of the relevant information.
Introduction
During the last decade, deep neural networks (DNN) have encountered a wide success in automatic speech recognition.
Many architectures such as recurrent (RNN) [34, 15, 1, 31, 13], time-delay (TDNN) [39, 28], or convolutional neural networks (CNN) [42] have been proposed and achieved better performances than traditional hidden Markov models (HMM) combined with gaussian mixtures models (GMM) in different speech recognition tasks.
However, despite such evolution of models and paradigms, the acoustic feature representation remains almost the same.
The acoustic signal is commonly split into time-frames, for which Mel-ﬁlter banks energies, or Mel frequency scaled cepstral coefﬁcients (MFCC) [8] are extracted, alongside with the ﬁrst and second order derivatives.
In fact, time-frames are characterized by 3 -dimensional features that are related by representing three different views of the same basic element.
Consequently an efﬁcient neural networks-based model has to learn both external dependencies between time-frames, and internal relations within the features.
Traditional real-valued architectures deal with both dependencies at the same level, due to the lack of a dedicated mechanism to learn the internal and external relations separately.
Quaternions are hypercomplex numbers that contain a real and three separate imaginary components, ﬁtting perfectly to three and four dimensional feature vectors, such as for image processing and robot kinematics [35, 29, 4].
The idea of bundling groups of numbers into separate entities is also exploited by the recent capsule network [33].
Contrary to traditional homogeneous representations, capsule and quaternion neural networks bundle sets of features together.
Thereby, quaternion numbers allow neural models to code latent inter-dependencies between groups of input features during the learning process with up to four times less parameters than real-valued neural networks, by taking advantage of the Hamilton product as the equivalent of the dot product between quaternions.
Early applications of quaternion-valued backpropagation algorithms [3, 2] have efﬁciently solved quaternion functions approximation tasks.
More recently, neural networks of complex and hypercomplex numbers have received an increasing attention [16, 38, 7, 40], and some efforts have shown promising results in different applications.
In particular, a deep quaternion network [24, 25, 37], and a deep quaternion convolutional network [5, 27] have been successfully employed for challenging tasks such as images and language processing.
Contributions: This paper proposes to evaluate previously investigated quaternion-valued models in two different realistic conditions of speech recognition, to see whether the quaternion encoding of the signal, alongside with the quaternion algebra and the important parameter reduction help to better capture the acoustic signal nature, leading to a more expressive representation of the information.
Based on the TIMIT [10] phoneme recognition task, a quaternion convolutional neural network (QCNN) is compared to a real-valued CNN in a end-to-end framework, and a quaternion recurrent neural network (QRNN) is compared to an RNN within a more traditional HMM-based system.
In the end-to-end approach, the experiments show that the QCNN outperforms the CNN with a phoneme error rate (PER) of 19 .
5% against the 20 .
6% achieved for CNNs.
Moreover, the QRNN outperforms the RNN with a PER of 18 . 5% against 19 . 0% for the RNN.
Furthermore, such results are observed with a maximum reduction factor of the number of neural network parameters of 3 .
96 times.
Motivations
A major challenge of current machine learning models is to obtain efﬁcient representations of relevant information for solving a speciﬁc task.
Consequently, a good model has to efﬁciently code both the relations that occur at the feature level, such as between the Mel ﬁlter energies, the ﬁrst, and second order derivatives values of a single time-frame, and at a global level, such as a phonemes or words described by a group of time-frames.
Moreover, to avoid overﬁtting, better generalize, and to be more efﬁcient, such models also have to be as small as possible.
Nonetheless, real-valued neural networks usually require a huge set of parameters to well-perform on speech recognition tasks, and hardly code internal dependencies within the features, since they are considered at the same level as global dependencies during the learning.
In the following, we detail the motivations to employ quaternion-valued neural networks instead of real-valued ones to code inter and intra features dependencies with less parameters.
First, a better representation of multidimensional data has to be explored to naturally capture internal relations within the input features.
For example, an efﬁcient way to represent the information composing an acoustic signal sequence is to consider each time-frame as being a whole entity of three strongly related elements, instead of a group of unidimensional elements that could be related to each others, as in traditional real-valued neural networks.
Indeed, with a real-valued NN, the latent relations between the Mel ﬁlter banks energies, and the ﬁrst and second order derivatives of a given time-frame are hardly coded in the latent space since the weight has to ﬁnd out these relations among all the time-frames composing the sequence.
Quaternions are fourth dimensional entities and allow one to build and process elements made of up to 4 elements, mitigating the above described problem.
Indeed, the quaternion algebra and more precisely the Hamilton product allows quaternion neural network to capture these internal latent relations within the features of a quaternion.
It has been shown that QNNs are able to restore the spatial relations within 3 D coordinates [20], and within color pixels [17], while real-valued NNs failed.
In fact, the quaternion-weight components are shared through multiple quaternion input parts during the Hamilton product , creating relations within the elements.
Indeed, Figure 1 shows that the multiple weights required to code latent relations within a feature are considered at the same level as for learning global relations between different features, while the quaternion weight w codes these internal relations within a unique quaternion Q out during the
Second, quaternion neural networks make it possible to deal with the same signal dimension than real-valued NN, but with 4 times less neural parameters.
Indeed, a 4 -number quaternion weight linking two 4-number quaternion units only has 4 degrees of freedom, whereas a standard neural net parametrization have 4 × 4 = 16 , i.e., a 4-fold saving in memory.
Therefore, the natural multidimensional representation of quaternions alongside with their ability to drastically reduce the number of parameters indicate that hyper-complex numbers are a better ﬁt than real numbers to create more efﬁcient models in multidimensional spaces such as speech recognition.
Indeed, modern automatic speech recognition systems usually employ input sequences composed of multidimensional acoustic features, such as log Mel features, that are often enriched with their ﬁrst, second and third time derivatives [8, 9], to integrate contextual information.
In standard NNs, static features are simply concatenated with their derivatives to form a large input vector, without effectively considering that signal derivatives represent different views of the same input.
Nonetheless, it is crucial to consider that these three descriptors represent a special state of a time-frame, and are thus correlated.
Following the above motivations and the results observed on previous works about quaternion neural networks, we hypothesize that for acoustic data, quaternion NNs naturally provide a more suitable representation of the input sequence, since these multiple views can be directly embedded in the multiple dimensions space of the quaternion, leading to smaller and more accurate models.
Quaternion Neural Networks
Real-valued neural networks architectures are extended to the quaternion domain to beneﬁt from its capacities.
Therefore, this section proposes to introduce the quaternion algebra (Section 3.1), the quaternion internal representation (section 3.2), a quaternion convolutional neural networks (QCNN,
Section 3.3) and a quaternion recurrent neural network (QRNN, Section 3.4).
3.1 Quaternion Algebra
The quaternion algebra H deﬁnes operations between quaternion numbers.
A quaternion Q is an extension of a complex number deﬁned in a four dimensional space as:
[Equation]
where r , x , y , and z are real numbers, and 1 , i , j , and k are the quaternion unit basis.
In a quaternion, r is the real part, while x i + y j + z k with i
=
j 2 = k 2 = ijk = − 1 is the imaginary part, or the vector part.
Such a deﬁnition can be used to describe spatial rotations.
The information embedded in the quaterion Q can be summarized into the following matrix of real numbers, that turns out to be more suitable for computations:
[Equation]
The conjugate Q ∗ of Q is deﬁned as:
[Equation]
Then, a normalized or unit quaternion Q (cid:47) is expressed as:
[Equation]
Finally, the Hamilton product ⊗ between two quaternions Q 1 and Q 2 is computed as follows:
[Equation]
The Hamilton product is used in QRNNs to perform transformations of vectors representing quaternions, as well as scaling and interpolation between two rotations following a geodesic over a sphere in the R 3 space as shown in [21].
3.2 Quaternion internal representation
In a quaternion layer, all parameters are quaternions, including inputs, outputs, weights, and biases.
The quaternion algebra is ensured by manipulating matrices of real numbers [27].
Consequently, for each input vector of size N , output vector of size M , dimensions are split into four parts: the ﬁrst one equals to r , the second is x i , the third one equals to y j , and the last one to z k to compose a quaternion Q = r
+ x i + y j
+ z k .
The inference process is based in the real-valued space on the dot product between input features and weights matrices.
In any quaternion-valued NN, this operation is replaced with the Hamilton product (eq. 5) with quaternion-valued matrices (i.e. each entry in the weight matrix is a quaternion).
3.3 Quaternion convolutional neural networks
Convolutional neural networks (CNN) [19] have been proposed to capture the high-level relations that occur between neighbours features, such as shape and edges on an image.
However, internal dependencies within the features are considered at the same level than these high-level relations by real-valued CNNs, and it is thus not guaranteed that they are well-captured.
In this extend, a quaternion convolutional neural network (QCNN) have been proposed by [5, 27] 1 .
Let γ lab and S lab , be the quaternion output and the pre-activation quaternion output at layer l and at the indexes ( a, b ) of the new feature map, and w the quaternion-valued weight ﬁlter map of size K × K .
A formal deﬁnition of the convolution process is:
[Equation]
6) with
[Equation]
where α is a quaternion split activation function [41] deﬁned as:
[Equation]
with f corresponding to any standard activation function.
The output layer of a quaternion neural network is commonly either quaternion-valued such as for quaternion approximation [2], or realvalued to obtains a posterior distribution based on a softmax function following the split approach of
Eq. 8.
Indeed, target classes are often expressed as real numbers.
Finally, the full derivation of the backpropagation algorithm for quaternion valued neural networks can be found in [23].
3.4 Quaternion recurrent neural networks
Despite the fact that CNNs are efﬁcient to detect and learn patterns in an input volume, recurrent neural networks (RNN) are more adapted to represent sequential data.
Indeed, recurrent neural networks obtained state-of-the-art results on many tasks related to speech recognition [32, 12].
Therefore, a quaternary version of the RNN called QRNN have been proposed by [26] 2 .
Let us deﬁne a QRNN with an hidden state composed of H neurons.
Then, let w hh , w hγ , and w γh be the hidden to hidden, input to hidden, and hidden to output weight matrices respectively, and b ln be the bias at neuron n and layer l .
Therefore, and with the same parameters as for the QCNN, the hidden state h t,l n of the neuron n at timestep t and layer l can be computed as:
[Equation]
with α any split activation function.
Finally, the output of the neuron n is computed following:
[Equation]
+
[Equation]
[Equation]
(10) with β any split activation function.
The full derivation of the backpropagation trough time of the
QRNN can be found in [26].
Experiments on TIMIT
Quaternion-valued models are compared to their real-valued equivalents on two different benchmarks with the TIMIT phoneme recognition task [10].
First, an end-to-end approach is investigated based on QCNNs compared to CNNs in Section 4.2.
Then, a more traditional and powerful method based on QRNNs compared to RNNs alongside with HMM decoding is explored in Section 4.3.
During both experiments, the training process is performed on the standard 3 , 696 sentences uttered byspeakers, while testing is conducted on 192 sentences uttered by 24 speakers of the TIMIT dataset.
A validation set composed of 400 sentences uttered by 50 speakers is used for hyper-parameter tuning.
All the results are from an average of 3 runs (3-folds) to alleviate any variation due to the random initialization.
4.1 Acoustic quaternions
End-to-end and HMM based experiments share the same quaternion input vector extracted from the acoustic signal.
The raw audio is ﬁrst transformed into 40 -dimensional log Mel-ﬁlter-bank coefﬁcients using the pytorch-kaldi 3 toolkit and the Kaldi s5 recipes [30].
Then, the ﬁrst, second, and third order derivatives are extracted.
Consequently, an acoustic quaternion Q ( f, t ) associated with a frequency f and a time-frame t is formed as:
[Equation]
[Equation] Q ( f, t ) represents multiple views of a frequency f at time frame t , consisting of the energy e ( f, t ) in the ﬁlter band at frequency f , its ﬁrst time derivative describing a slope view, its second time derivative describing a concavity view, and the third derivative describing the rate of change of the second derivative.
Quaternions are used to learn the spatial relations that exist between the different views that characterize a same frequency.
Thus, the quaternion input vector length is 160 / 4 = 40 .
4.2 Toward end-to-end phonemes recognition
End-to-end systems are at the heart of modern researches in the speech recognition domain [42].
The task is particularly difﬁcult due to the differences that exists between the raw or pre-processed acoustic signal used as input features, and the word or phonemes expected at the output.
Indeed, both features are not deﬁned at the same time-scale, and an automatic alignment method has to be deﬁned.
This section proposes to evaluate the QCNN compared to traditional CNN in an end-to-end model based on the connectionist temporal classiﬁcation (CTC) method, to see whether the quaternion encoding of the signal, alongside with the quaternion algebra, help to better capture the acoustic signal nature and therefore better generalize.
4.2.1 Connectionist Temporal Classiﬁcation
In the acoustic modeling part of ASR systems, the task of sequence-to-sequence mapping from an input acoustic signal X
=
[ x 1 , ..., x n ] to a sequence of symbols T =
[ t 1 , ..., t m ] is complex due to:
• X and T could be in arbitrary length.
• The alignment between X and T is unknown in most cases.
Specially, T is usually shorter than X in terms of phoneme symbols.
To alleviate these problems, connectionist temporal classiﬁcation (CTC) has been proposed [11].
First, a softmax is applied at each timestep, or frame, providing a probability of emitting each symbol X at that timestep.
This probability results in a symbol sequences representation P ( O | X ) , with O =
[ o 1 , ..., o n ] in the latent space O .
A blank symbol (cid:48) − (cid:48) is introduced as an extra label to allow the classiﬁer to deal with the unknown alignment.
Then, O is transformed to the ﬁnal output sequence with a many-to-one function g ( O ) deﬁned as follows:
[Equation]
Consequently, the output sequence is a summation over the probability of all possible alignments between X and T after applying the function g ( O ) .
Accordingly to [11] the parameters of the models are learned based on the cross entropy loss function:
[Equation]
During the inference, a best path decoding algorithm is performed.
Therefore, the latent sequence with the highest probability is obtained by performing argmax of the softmax output at each timestep.
The ﬁnal sequence is obtained by applying the function g ( . ) to the latent sequence.
4.2.2 Model Architectures
A ﬁrst 2 D convolutional layer is followed by a maxpooling layer along the frequency axis to reduce the internal dimension.
Then, n = 10 2 D convolutional layers are included, together with 3 dense layers of sizes 1024 and 256 respectively for real- and quaternion-valued models.
Indeed, the output of a dense quaternion-valued layer has 256 × 4 = 1024 nodes and is 4 times larger than the number of units.
The ﬁlter size is rectangular (3 , 5) , and a padding is applied to keep the sequence and signal sizes unaltered.
The number of feature maps varies from 32 to 256 for the real-valued models and from 8 to 64 for quaternion-valued models.
Indeed, the number of output feature maps is 4 times larger in the QCNN due to the quaternion convolution, meaning 32 quaternion-valued feature maps (FM) correspond to 128 real-valued ones.
Therefore, for a fair comparison, the number of feature maps is represented in the real-valued space (e.g., a number of real-valued FM of 256 corresponds to 256 / 4 = 64 quaternion-valued neurons).
The PReLU activation function is employed for both models [14].
A dropout of 0 . 2 and a L 2 regularization of 1 e − 5 are used across all the layers, except the input and output ones.
CNNs and QCNNs are trained with the RMSPROP learning rate optimizer and vanilla hyperparameters [18] during 100 epochs.
The learning rate starts at 8 · 10 − 4 , and is decayed by a factor of 0 .
5 every time the results observed on the validation set do not improve.
Quaternion parameters including weights and biases are initialized following the adapted quaternion initialization scheme provided in [26].
Finally, the standard CTC loss function deﬁned in [11] and implemented in [6] is applied.
4.2.3 Results and discussions
End-to-end results of QCNN and CNN are reported in Table 1.
In agreement with our hypothesis, one may notice an important difference in the amount of learning parameters between real and quaternion valued CNNs.
An explanation comes from the quaternion algebra.
A dense layer with 1 , 024 input values and 1 , 024 hidden units contains 1 , 024
≈ 1 M parameters, while the quaternion equivalent needs 256 2 × 4 ≈ 0 .
26 M parameters to deal with the same signal dimension.
Such reduction in the number of parameters have multiple positive impact on the model.
First, a smaller memory footprint for embedded and limited devices.
Second, and as demonstrated in Table 1, a better generalization ability leading to better performances.
Indeed, the best PER observed in realistic conditions (w.r.t to the development PER) is 19 .
5% for the QCNN compared to 20 .
6% for the CNN, giving an absolute improvement of 0 .
5% with QCNN.
Such results are obtained with 32 . 1 M parameters for CNN, and only 8 . 1 M for QCNN, representing a reduction factor of 3 .
96 x of the number of parameters.
[Table]
It is worth noticing that with much fewer learning parameters for a given architecture, the QCNN always performs better than the real-valued one.
Consequently, the quaternion-valued convolutional approach offers an alternative to traditional real-valued end-to-end models, that is more efﬁcient, and more accurate.
However, due to an higher number of computations involved during the Hamilton product and to the lack of proper engineered implementations, the QCNN is one time slower than the CNN to train.
Nonetheless, such behavior can be alleviated with a dedicated implementation in
CUDA of the Hamilton product .
Indeed, this operation is a matrix product and can thus beneﬁt from the parallel computation of GPUs.
In fact, a proper implementation of the Hamilton product will leads to a higher and more efﬁcient usage of GPUs.
4.3 HMM-based phonemes recognition
A conventional ASR pipeline based on a HMM decoding process alongside with recurrent neural networks is also investigated to reach state-of-the-art results on the TIMIT task.
While input features remain the same as for end-to-end experiments, RNNs and QRNNs are trained to predict the HMM states that are then decoded in the standard Kaldi recipes [30].
As hypothetized for the end-to-end solution, QRNN models are expected to better generalize than RNNs due to their speciﬁc algebra.
4.3.1 Model Architectures
RNN and QRNN models are compared on a ﬁxed number of layers M = 4 and by varying the number of neurons N from 256 to 2 , 048 , and 64 to 512 for the RNN and QRNN respectively.
Indeed, as demonstrated on the previous experiments the number of hidden neurons in the quaternion and real spaces do not handle the same amount of real-number values.
Tanh activations are used across all the layers except for the output layer that is based on a softmax function.
Models are optimized with RMSPROP [18] with vanilla hyper-parameters and an initial learning rate of 8 · 10 − 4 .
The learning rate is progressively annealed using an halving factor of 0 .
5 that is applied when no performance improvement on the validation set is observed.
The models are trained during 25 epochs.
A dropout rate of 0 . 2 is applied over all the hidden layers [36] except the output.
The negative log-likelihood loss function is used as an objective function.
As for QCNNs, quaternion parameters are initialized based on [26].
Finally, decoding is based on Kaldi [30] and weighted ﬁnite state transducers (WFST)
[22] that integrate acoustic, lexicon and language model probabilities into a single HMM-based search graph.
4.3.2 Results and discussions
The results of both QRNNs and RNNs alongside with an HMM decoding phase are presented in
Table 2.
A best testing PER of 18 .
5% is reported for QRNN compared to 19 . 0% for RNN, with respect to the best development PER.
Such results are obtained with 3 . 8 M, and 9 .
4 M parameters for the QRNN and RNN respectively, with an equal hidden dimension of 1 , 024 , leading to a reduction of the number of parameters by a factor of 2 .
47 times.
As for previous experiments the QRNN always outperform equivalents architectures in term of PER, with signiﬁcantly less learning parameters.
It is also important to notice that both models tend to overﬁt with larger architectures.
However, such phenomenom is lowered by the small number of free parameters of the QRNN.
Indeed, a QRNN whose hidden dimension is 2 , 048 only has 11 .
2 M parameters compared to 33 .
4 M for an equivalently sized RNN, leading to less degrees of freedom, and therefore less overﬁtting.
the dimension of the hidden layers.
As an example, a QRNN layer of "Hidden dim."
= 512 is equivalent to 128 hidden quaternion neurons.
[Table]
The reported results show that the QRNN is a better framework for ASR systems than real-valued RNN when dealing with conventional multidimensional acoustic features.
Indeed, the QRNN performed better and with less parameters, leading to a more efﬁcient representation of the information.
Conclusion
Summary.
This paper proposes to investigate novel quaternion-valued architectures in two different conditions of speech recognition on the TIMIT phoneme recognition tasks.
The experiments show that quaternion approaches always outperform real-valued equivalents in both benchmarks, with a maximum reduction factor of the number of learning parameters of 3 .
96 times.
It has been shown that the appropriate multidimensional quaternion representation of acoustic features, alongside with the Hamilton product , help QCNN and QRNN to well-learn both internal and external relations that exists within the features, leading to a better generalization capability, and to a more efﬁcient representation of the relevant information through signiﬁcantly less free parameters than traditional real-valued neural networks.
Future Work.
Future investigation will be to develop multi-view features that contribute to decrease ambiguities in representing phonemes in the quaternion space.
In this extend, a recent approach based on a quaternion Fourrier transform to create quaternion-valued signal has to be investigated.