Leveraging Latent Representations of Speech for I ndian Language Identification (2020)

Abstract: Identification of the language spoken from speech utterances is an interesting task because of the diversity associated with different languages and human voices. Indian languages have diverse origins and identifying them from speech utterances would help several language recognition, translation and relationship mining tasks. The current approaches for tackling the problem of languages identification in the Indian context heavily use feature engineering and classical speech processing techniques. This is a bottleneck for language identification systems, as we require to exploit necessary features in speech, required for machine identification, which are learnt by a probabilistic framework, rather than handcrafted feature engineering. In this paper, we tackle the problem of language identification using latent representations learnt from speech using Variational Autoencoders (VAEs) and leverage the representations learnt to train sequence models. Our framework attains an accuracy of 89% in the identification of 8 well known Indian languages (namely Tamil, Telugu, Punjabi, Marathi, Gujarati, Hindi, Kannada and Bengali) from the CMU Indic Speech Database. The presented approach can be applied to several scenarios for speech processing by employing representation learning and leveraging them for sequence models.

## introduction language identification refers to the task of identifying the language being spoken when given a speech utterance. several intelligent agents rely heavily on language identification systems for subsequent speech recognition and processing tasks. this problem is particularly interesting and important in the indian context, given the diverse nature of indian languages. several indian languages suffer regional bias and each language has its own dialect as one travels within each state (region) of the country. in this scenario, language identification would be a challenging task for traditional language identification systems which heavily rely on feature engineering from speech. several of the current indian language identification systems rely on handcrafted features, which end up serving as a bottleneck to such systems. such systems would greatly benefit by learning necessary features in speech utterances using a probabilistic framework and leveraging these representations for training deep sequence models. a few deep neural network based models have also been proposed. the authors in @xcite perform posterior extraction using convolutional neural networks (cnns), and an i-vector based system for subsequent language recognition. the authors in @xcite use an end-toend deep neural network with attention mechanism for indian language identification. we exploit the representation learnt by a vae @xcite trained on speech segments to train several sequence models widely used for natural language processing. these models use distributed wordrepresentations @xcite which model the words in a continuous vector space. we use the latent feature representations learnt by a vae in the stochastic low dimensional latent representation space in a manner similar to how distributed word representations, obtained by pre-training large corpuses in an unsupervised manner, are used in natural language processing to train sequence models. in this paper, we present a framework for indian language identification by pre-training a probabilistic framework for representation learning and leveraging these representation for training sequence models for classification. ## related work language identification from speech has been deeply studied by various research communities. prosodic, phoenetic and phonotactic feature based approaches for identifying language is studied in @xcite and @xcite . many such classical feature engineering based methods require a lot of domain knowledge. with the rise of deep learning and neural networks, automatic feature and representation learning has greatly outperformed all such methods. language identification using deep convolutional recurrent neural networks is studied in @xcite . they use non-overlapping segments of mel spectrograms of speech which are passed through a convolutional neural network and then the features maps are passed through a long short term memory (lstm). the final hidden state of the lstm is used for classification. the cnn captures the spatial features, whereas the lstm captures the temporal features. for long speech utterances, recurrent neural networks, can capture the temporal aspect of speech utterances and this was considered in @xcite . the authors in @xcite give an attention based 1d-cnn for the task of language identification directly from raw audio. this attention greatly enhances the performance of neural network based approaches. indian language identification using deep learning based models have been studied in @xcite , @xcite , @xcite and @xcite . deep neural network based systems take in the speech utterances at each frame, classification performed frame-wise, and this may be considered as a drawback. a deep neural network with attention mechanism was considered in @xcite . this architecture applies attention to specific parts of the input sequence, whilst memorizing important features in long temporal sequences. a 39dimensional mfcc is considered by the authors, each for 5 second chunks of the input sequence, which are passed through a regular dnn to compute hidden layer representations. an attention mechanism is applied over this to memorize the temporal aspect and summarize the features in the whole speech utterance, giving a single context vector and this vector is subsequently passed to a classifier. attention based residual-time delay neural network (res-tdnn) is studied in (man-dava and vuppala, 2019), which further improves over trying to capture the long range temporal dependencies. ## proposed framework our goal is to learn latent representations from speech and use these representations to train sequence models for classification. we use variational autoencoders (vaes) for representation learning on small segments of mel spectrograms of speech utterances (40 mel-scale filter banks). the mel spectrogram is obtained by taking the fourier transform of the signal, followed by mapping the powers of the obtained spectrum onto the mel scale. the mel-frequency scale resembles the resolution of the human auditory system. the segmentation is performed along the time axis. the model is trained in a similar manner adopted in @xcite . after pre-training the vae, the encoder's latent distribution is able to encode mel spectrograms of speech segments into a latent representations space. we use a sequence of such latent representation for each segmented mel spectrogram of speech utterance as input to sequence models. the vae captures important representational features for each segment of the speech utterance and the sequence model captures the temporal aspect of each speech utterance. the unsupervised representation learning parameters are optimized in a different step from when the supervised sequence learning parameters are optimized. ## conclusion in this paper, we have introduced a new framework for indian language identification using vae representation learning and state-of-the-art sequence models to capture the temporal characteristics of speech. the framework performs well on identification of 8 well known languages. the framework also helps improve language identification in the future as sequence models in natural language processing become better capturing long range dependencies and other temporal aspects of sequences. it can be applied in several other speech processing scenarios as well where the task requires representation learning from speech utterances and subsequent classification using a sequence model. we see vaes are powerful probabilistic models which can learn useful representation from speech utterances and these representations can be utilized in several downstream tasks. (a) bi-lstm with self attention (1 layer) (b) bi-lstm with self attention (2 layers) (c) bi-lstm with soft aligned attention (d) rcnn (e) transformer figure 3: t-sne embedding space of representations in the last hidden layer assigned by each sequence model on test data.