Tonic Independent Temporal Music Pattern Recognition in Indian Classical Music
Abstract
A vital aspect of Indian Classical music ( ICM ) is the Raag , which serves as a baseon which improvisations and compositions ( IAC ) are created and presented.
Whileevery Raag represents a unique emotion, it allows a musician to explore and conveyhis interpretation of the lyrical content of a song.
Although many works haveexplored the problem of classiﬁcation of Raag, they have several short comingsowing to the fact that they assume a prior knowledge of the tonic of the audio or isdependent on a preprocessing technique that identiﬁes the tonic.
In this work weintroduce 1) a novel data augmentation technique leveraging an inherent aspect of
ICM that the semantics of IAC are only dependent on the relative position of noteswith respect to the tonic and not the tonic itself 2) Convolutional Neural Networkbased approach to build a robust model that can classify Raag independent of thetonic.
Introduction
ICM is an advanced and complex form of classical music.
Carnatic and Hindustani classical music,which are its two primary branches, have evolved a great deal over a period of more than 500 years.
Acritical component of any Indian Classical Music Concert is the “Manodharma” or the “Spontaneousimprovisation” which is an extremely complex task.
The difﬁculty arises partly due to it beingextemporaneous in nature, but more importantly because the musician has to balance the melodic andthe entertainment aspects equally while being creative at the same time.
A Raag can be deﬁned as a pattern of notes having characteristic embellishments, rhythm andintervals.
The notion of Raag has similarities to the concept of scale in Western Classical Music inthat it deﬁnes the note progressions allowed during ascent and descent of notes.
Every note usedin a given raag is relative to the base note called the Shadaj or the tonic .
Gamaka and Sangathiare features that are unique to a Raag and this is what differentiates a Raag and a scale.
Gamakais a complex version of glissando that enables a musician to express the same progression of notesin multiple ways, due to which two Raags that have a similar set of notes may sound completelydifferent.
A sangathi can be deﬁned as a short progression of notes with or without the using gamaka.
If we assume a Raag to be the depiction of a muscian’s emotions as the theme of a painting, thena Sangathi can be assumed to the brush strokes used to create the Raag, while the painting itself isthe composition or improvisation.
The Gamaka and Sangathis together deﬁne the grammar for a
Raag.
Since Gamaka and Sangathi are the building blocks of a Raag, the key to identify a Raag liesin identifying the characteristic Gamaka and Sangathi of a Raag.
In short, if we imagine Raag to be some kind of distribution, small sequences(or subsequences) aresampled from the same repeatedly and organized in some particular order which would make senseaesthetically and musically and hence results either in a composition or an improvisation.
Sinceevery subsequence is nothing but temporal data, a musician is essentially creating a larger temporalsequence that is comprised of the smaller temporal subsequences.
To add to this we have anotherdimension, the tonic.
Hence we could treat the problem at hand as a kind of a spatio-temporalsequence, where the spatial does not exactly refer to Cartesian co-ordinates, but it refers to an extradimension created due to the tonic.
As mentioned before, every note in a Raag is deﬁned relative to the tonic.
Hence identifying the tonicbecomes very crucial for the task of Raag classiﬁcation.
From an ICM standpoint, the knowledge ofthe Raag is necessary to identify the tonic and vice versa.
Furthermore, the tonic is decided based onthe performers preference.
This hence results in the selection of a non standard note as a tonic, forinstance a note that is halfway between the notes D#4 and E4.
As a result, building a model that iscapable of identifying the Raag, irrespective of the value of the tonic is very challenging.
Also, sinceevery performer has their own way of expressing the same Gamaka, at the same or different speeds,often combining multiple ones to form complex patterns, adds to the complexity of this task.
In thiswork we present a novel approach using CNN to build a robust model that is capable of identifyingthe Raag independent of the tonic.
To the best of our knowledge, we are the ﬁrst to use a CNN basedapproach to solve this problem
Related Work
Raag identity and characteristics is well studied in Indian Music Theory.
There is vast literatureamong the music community as to what features are key in identifying Raags.
In the Carnatic Music
System the Parent Raag scheme (known as Janaka Raag scheme in ICM) deﬁnes set of 72 Raagsfrom which all the other Raags maybe derived.
A detailed analysis on the theoretical aspects of a
Raga is presented in [7].
Another interesting work that describes the characteristics and features of a
Raag which make them similar/dissimilar is presented in [2].[4] describe the recognition of Ragas using pitch class and pitch class dyad distributions.
Althoughthey were able to achieve 75 percent accuracy on an unseen dataset, the model is not fully robust as itassumes prior knowledge of the tonic.
[8] use a non-linear svm based approach where the similaritiesin the audio samples is represented using a combination of two kernels.
[9] use a combination ofhidden markov models, string matching algorithm and automatic note transcription is their model.
The authors assume that the audio is monophonic.
They also make an assumption on the the tonic ofall the audio samples being G. [10] recover characteristic features of the Raga and feed it to a Neural
Network to perform the classiﬁcation.
We note that the features used (arohana and avarohana andset of notes) here, although essential, are not good enough to build a powerful classiﬁer.
They alsoassume the prior knowledge of the tonic of the audio.
Approach3.1 Dataset and Pre processing3.1.1 Dataset
The ﬁrst step in this direction is to prepare a dataset with sufﬁcient number of recordings containingrich musical content to enable the CNN to learn the subtleties and the nuances of the music being fed.
Although there are thousands of recordings available, we observe that most of these recordings are ofsubstandard quality for the purposes of training a NN.
Hence we hand pick recordings that are ofgood quality both in terms of recording standards and music.
The created dataset ’DBICM3’ has atotal length of 2+ hours, featuring 8 artists and the recordings have 10 different tonics.
We create 2sub sets from the dataset, details of which have been outlined below :
• D1 - This contains 9 recordings in 3 Raags and 2 different tonics.
All recordings in thisdataset was performed by the same artist.
It was created as a baseline dataset to compare
our model’s performance with its performance on real world data.• D2 -
This contains 21 recordings in 7 Raags and 8 different tonics.
All the recordings arereal world examples, ie, they have been sampled from Live Recordings of 5 different artists.
Efforts have been made to ensure no two recordings of the same Raag have the same tonic.
3.1.2 Pitch Tracking
As mentioned earlier, the recordings have been hand picked such that they are of good audio
qualityand hence we do not require any additional efforts to improve the quality of the audio.
Since the modelanalyzes the melody component of the audio to identify the Raag, a critical step in preprocessing is to perform pitch tracking of the audio and hence represent the given audio as an array of frequencies.
Weuse Praat [3] (which is an open source software for the analysis of speech and sound) and Parselmouth[6] (which is a Python API for Praat), to perform the pitch tracking of the audio.
3.1.3 Frequency to MIDI conversion
Since Indian Classical Music predominantly characterized by acoustic instruments/ vocals, the audiohence is a continuous waveform.
To be able to effectively analyze a sequence of frequencies, the audiohas to be discretized.
For this we could convert a given frequency into the corresponding Musical
Instrument Digital Interface (MIDI) Note by using the formula, MIDI Note =
+ 12 ∗ log 2 ( f 440 ) .
The issue with this approach is that frequencies in the range of 21 Hz to 4186 Hz is represented by88 discreet levels ( i.e MIDI note 21 to 108 ) which leads to a severe loss of information.
Hencewe deﬁne 10 additional levels (which are technically called cents) between two MIDI notes, thusresulting in a total of 88 ∗ 10 possible levels.
Hence every note is now represented as a tuple (M,C)which can be read off as the MIDI note ’M’ and ’C’ cents above ’M’.
3.1.4 Data Augmentation
Although the process mentioned above helps our classiﬁer in efﬁciently learning the nuances ofmusic, it necessitates that we feed the classiﬁer with data in every possible tonic in order to make theclassiﬁer independent of the tonic.
To address this problem, we propose a novel data augmentationtechnique.
For this we ﬁrst obtain all possible (M,C) tuples and transform them using M*10 + C.
Following this we:
• Once the audio has been converted to a sequence of (M,C) tuples, transform the same usingthe formula : M*10 + C. Hence every Note is represented as a single value and not a tuple.
• The maximum and minimum value in the sequence will be ≤ 109 and ≥
respectively.•
Depending on the maximum value and the minimum value in the sequence, we increment(and decrement)
the values in the sequence n (and m) times by a value delta until the
maximum (and minimum) value becomes equal to 21(109)
• At every step we save the resulting sequence.
This results in the creation of multiple copies
of the audio across different tonics.
• The audio clips from the previous step is then split into smaller clips as mentioned in 3.1.5When we train the model on this augmented dataset, it now has a clear idea as to how a Raag lookslike in different tonics and hence becomes independent of the tonic of the audio.
3.1.5 Sub-sequencing
In this work, we try to emulate the way in which a human listener tries to discern the componentsof the Raag and hence identiﬁes the Raag being presented.
We randomly select a starting point inthe pitch tracked audio and extracting a 500 long array (which corresponds to nearly 5 seconds ofaudio.
We choose 5 seconds as each Sangathi is this long on an average) and use this as a one sample.
The Raag of the recording will be the target for the classiﬁcation.
In a 3 minute long clip (All of thetraining samples are 3-4 minutes long), we repeatedly sample 250 times.
Sampling a 5 second audio250 times in a 4 minutes long clip will create a lot of overlaps between the samples.
We observe thatthis helps the model to better understand the data presented to it.
3.2 Network Architecture
Recently, CNNs have had a lot of success in NLP and speech recognition applications.
[1] presents aconcise explanation of how a CNN can be used for the task of speech recognition.
Authors in [5]have used a novel VDCNN architecture based on deep CNNs to achieve improvements over-state-ofthe-art on many datasets.
Identifying Raags is essentially a sequence classiﬁcation, with invarianceto translation being a critical factor since a gamaka or sangathi can appear in any part of the audio,making it a suitable workload for a CNN.
The ﬁrst layer is an embedding layer with an embeddingvector length of 80.
The embedding layer is succeeded by a 1 dimensional convolution layer (with 30ﬁlters, kernel size of 50, ReLU activation and Dropout), a maxpooling layer (kernel size 3), another 1dimensional convolution layer (with 35 ﬁlters, kernel size of 100, ReLU activation and Dropout).
[Table]
This is then connected to a fully connected layer of size 50 followed by another dense layer with sizeequal to the number of classes (Raags) in the dataset followed by softmax activation.
The network isoptimized on categorical cross entropy loss using adam optimizer.
Model Evaluation
The steps involved in preparing the test data is similar to that of training data, only difference being,it does not require any data augmentation.
We ﬁrst transform the audio to obtain a sequence of (M,C)tuples.
Following which we create multiple sub sequences of the same which acts as test samples.
We test the model on all three datasets 3.1.1.
Note that there is equal representation of all the Raagsin the test set, hence it is a balanced classiﬁcation problem.
We ﬁrst test the model on dataset D1, asit represents a simpliﬁed version of the data that model could expect to see in a real world scenarioand obtain an test accuracy of 85.3 as shown in 3.2.
This shows that the model is able to identify agood portion of the test samples correctly.
We expect to see some error in prediction due to the factthat it might so happen that in the test samples (which are essentially subsequences of the originalaudio 3.1.5) there is only one note present for the entire length of the sample.
This does cause someissue as there is not enough data for the model identify which Raag the sample belongs to.
Next we test our model on dataset D2 which contains 7 Raags with 8 tonics and the recordings havebeen sampled from live recordings of 5 different artists along with accompaniments.
Since this isa real world example, performing the classiﬁcation is extremely difﬁcult due to the prescence ofaccompaniement as a result of which many portions of the recordings will be unclear to the model.
Itis important to note that the model was trained on recordings with 5 different tonics and the test sethad 3 different tonics (totaling to 8 as mentioned above).
Although the possible number of tonics inthe test set is only 3, the model was unaware of the same and had to generalize over a large numberof possibilities.
As summarized in 3.2 we see that the model achieves a test accuracy of 75.7.
Conclusion and Future Work
The data augmentation technique and the approach to solving this have been effective in identifyingthe Raag irrespective of the tonic.
Throught out the work we have only been interested in identifyingthe Raag of the subsequence and not the entire audio as a whole, however, once we have predicted the
Raag for all the sampled subsequences, we can use this array of predictions to infer the Raag contentof the Raag.
We observe that even though the accuracy of the model is around 75.7 on a subsequencelevel, the model is able to correctly identify the Raag of an entire audio with much higher accuracy.
We feel that this approach has tremendous potential and hence we are making efforts to create a largerand comprehensive dataset which will allow us to test the model in various other conditions.