A Survey on Paralinguistics in T amil Speech Processing (2021)

Abstract: Speech carries not only the semantic content but also the paralinguistic information which captures the speaking style. Speaker traits and emotional states affect how words are being spoken. The research on paralinguistic information is an emerging field in speech and language processing and it has many potential applications including speech recognition, speaker identification and verification, emotion recognition and accent recognition. Among them, there is a significant interest in emotion recognition from speech. A detailed study of paralinguistic information present in speech signal and an overview of research work related to speech emotion for Tamil Language is presented in this paper.

## introduction the field of paralinguistics deals with how something is being spoken and is limited to non-verbal aspects of speech. it covers the manner of speaking and information characterizing speaker traits and states. paralinguistic content present in the speech signal provides information which is mainly grouped into short-term traits and long-term traits. short-term traits include speaking style, voice quality, and emotional states while long-term traits include biological trait primitives such as height, weight, age and gender, ethnicity, culture, and personality. health state, intoxication, mood, and sleepiness are categorized as medium term traits between short term and permanent traits. @xcite the research on paralinguistics is beneficial in a wide range of speech processing applications. performance of automatic speech recognition (asr) is affected by variability in the speaking style due to speaker traits emotions. analyzing the paralinguistic information in the speech help to compensate for the speaker variability through normalization techniques @xcite . in addition to this, several paralinguistic tasks including speaker recognition, accent recognition, and emotion recognition have been investigated over the past years. interest in paralinguistics has significantly grown in the past years and interspeech conference hosts a computational paralinguistics challenge compare since 2009 with different sets of tasks each year. the most popular paralinguistic task is speech emotion recognition. emotion detection can help in making the human computer interface adapt to user's emotional condition, thus improving the user satisfaction. study of paralinguistics can also be applied in diagnosing and monitoring the disease progression in diseases like neurodegenerative disorders which show speech impairment as one of the early signs. selecting appropriate feature extraction techniques is important for discriminating between classes in paralinguistic tasks. then most common features that capture paralinguistic information are low-level descriptors (lld) @xcite which include mel frequency cepstral coefficients (mfcc), energy, pitch frequency, loudness, zero crossing rate, harmonicity, jitter, shimmer etc. recent research works use deep neural networks (dnns) to learn high-level acoustic features from utterance-level lld or directly from raw speech signal. dnns have shown significant performance in a variety of applications ranging from speech recognition to speaker identification and verification. this paper presents the background of paralinguistics and focuses on the research work related to speech based emotion recognition for tamil language. tamil is a dravidian language natively spoken by south asia's tamil people. tamil is the official language of two sovereign states, singapore and sri lanka, as well as the indian province of tamil nadu and puducherry @xcite @xcite . with a history stretching back to 600 bce, the tamil language is one of the world's longest-surviving classical languages. poetry dominates tamil literature, especially sangam literature, which consists of poems written between 600 bce and 300 ce. the tamil language accounts for more than 55 percent of the epigraphical inscriptions discovered by the archaeological survey of india (approximately 55,000) @xcite . the rest of the paper is organized as follows: section ii describes in detail about paralinguistic information in speech and explores various acoustic features extracted from the speech signal. section iii discusses speech emotion recognition and presents a survey of research work done for tamil language. the paper is concluded in section iv, with discussions in this research area. ## background speech signal carries two types of information: linguistic information which conveys the spoken content and paralinguistic information that covers the speaker attributes. paralinguistic information can be categorized into three types as shown in figure 1 . the following sections discuss about paralinguistic features, its applications and acoustic features that represent them. ## speech emotion recognition for tamil language emotion recognition from speech is one of the major topics in the field of computational paralinguistics. it helps to understand the emotional condition and actual intentions of the speakers which would be beneficial in improving the speech based applications such as automatic speech recognition @xcite @xcite and spoken intent recognition @xcite . it can be used to enhance the human computer interaction and to identify the emotional state of the user in call centers. some of the available tamil speech datasets include mozilla common voice @xcite , open slr -tamil @xcite , microsoft speech corpus (indian languages) -tamil (mic), and tamil speech intent dataset -uom @xcite . however, there is no standard database available for tamil emotional speech. emotional speech corpora can consist of three types of emotional speech: acted emotions which is simulated by actors, spontaneous emotions from real life situations, and elicited emotions stimulated through emotional movies, stories and games. it is desired to have good quality speech recordings of diverse set of emotions collected from a large group of speakers. though spontaneous emotions are preferred, it is difficult to acquire. therefore, most of the available speech corpora contain acted emotional speech. joe (2014) built a tamil emotional speech corpus with speech data consisting of five emotions happy, sad, anger, fear, and neutral. speech recordings were collected from acted emotional speech in tamil audio plays. support vector machine (svm)-based emotion recognition system was used to perform classification with mfccs as input features. this corpus was extended in @xcite and another emotion corpus was developed separately for children using the samples collected from tamil movies. having an emotion corpus consisting of utterances of both the adults and children could help in investigating the influence of age in emotion expression. in @xcite , an acoustic feature-based emotion recognition system is presented. the database used in this work is amritaemo @xcite which consists of speech recordings in tamil and telugu languages with three emotions: anger, happiness, and sadness. k-nearest neighbor (knn) and artificial neural network (ann) were used to classify emotions based on two features namely linear predictive cepstral coefficients (lpccs) and hurst parameter. hurst parameter is the decaying rate of the autocorrelation coefficient function of the speech signal. the experimental results showed that when individual features were considered, for both the languages hurst parameter achieved higher accuracy, precision, and recall when compared to lpccs while the combination of these features resulted in a better performance. @xcite proposed a gaussian mixture model (gmm) -deep belief network (dbn) system for emotion recognition. it models gmm for each emotion independently using mfcc features extracted from the speech signal. the minimum distance between the distribution of features for each utterance with respect to each emotion model is derived as bag of acoustic features and it is fed to dbn as the feature vector. the proposed system was evaluated on berlin emotional speech corpus @xcite as well the emotional database for tamil language. the database used in this work consists of four emotions namely anger, happy, sad, and neutral state for three languages tamil, english, and malayalam. ram and ponnusamy (2018) developed a speech corpus to evaluate the tamil speech emotion recognition of children with autism spectrum disorder. in addition, emotional speech recordings for tamil and telugu language were collected from movies and berlin emotional speech corpus was used as well for training. the selected emotion classes were anger, neutral, happiness, sadness, and fear. mfcc, pitch frequency, and energy were used as the set of features fed to the svm based classifier. an emotion recognition system using weighted features is proposed in @xcite . a speech dataset consisting of audio recordings expressing five emotions namely anger, surprise, disgust, sadness, and happiness was created for three south indian languages tamil, telugu, and malayalam. mean and variance of energy contour, first five formant frequencies, and linear predictive coding (lpc) coefficients were chosen as the speech features and classification was performed using knn, svm, and ann. classification performance using normal features and weighted features were compared and a significant increase in recognition accuracy was observed with weighted features. ann achieved the best accuracy among the considered classifier models. the results indicated that the weight factors are language dependent for the input features. the paper (rajan et al., 2019) presents a multilingual emotional speech corpus tamar-emodb developed for tamil, malayalam, and ravula. the corpus contains short speech utterances in five emotions: anger, anxiety, happiness, sadness, and neutral state. the corpus was built using simulated speech utterances where the subjects were asked to read a sentence, expressing a given emotion. it was evaluated using a dnn-based classifier and the classification was performed using the fusion of mfccs and prosodic features such as short-time energy, zero crossing rate, and standard deviation, skewness, and kurtosis of pitch. sowmya and rajeswari (2020) created a speech dataset for tamil language and built a classifier to predict emotions with mfcc and energy as the input features. the database consists of four emotions anger, sad, neutral, and happiness. knn, svm, and ann were used for emotion classification where svm and ann achieved higher accuracy. ## conclusion the presented study discussed about the paralinguistic information present in speech signal and its relevance in many speech processing applications. the low-level descriptors characterize the underlying paralinguistic content and they can be used as input features to dnn based models. high level speech representations can be extracted directly from raw speech signal as well using dnns. these models have shown significant performance in many paralinguistic tasks. paralinguistic information can also be applied in improving speech recognition systems using normalization techniques. the most popular paralinguistic task, emotion recognition is challenging when applied across different languages and cultures since aspects of emotion expression are language and culture specific. therefore, a proper emotional speech database is required to build effective emotion detection models for tamil language. several research studies have created speech databases for tamil language and evaluated them using svm or dnn based classifiers which reported good performance. however, further improvements are still needed and a standard tamil emotional speech database needs to be built with a diverse set of emotions and a large number of speakers.