T amil Lyrics Corpus: Analysis and Experiments (2021)

Abstract: In this paper, we present a new Tamil lyrics corpus extracted from Tamil movies captured across a range of 65 years (1954 to 2019). We present a detailed corpus analysis showing the nature of Tamil lyrics with respect to lyricists and the year which it was written. We also present similar- ity score across different lyricists based on their song lyrics. We present experi- mental results based on the SOTA BERT Tamil models to identify the lyricists of a song. Finally, we present future research directions encouraging researchers to pur- sue Tamil NLP research.

## introduction tamil is a classical dravidian language spoken in the states tamil nadu and pondicherry, india. it is also widely spoken in srilanka and is one of the official languages of the country. there are considerable tamil speaking people in countries like singapore, malaysia, canada, and mauritius. historians note that tamil is as old as 2600 years. the earliest identified tamil literature known as the sangam literature dates back to 600 b.c. @xcite . tamilians as an ethnic group appreciate literary works and refer to literary excerpts in their daily lives. for example, thirukkural foot_0 , one of the popular literary works, also called the ulaga pothu marai (translation: universal book of principles) includes short literary phrases emphasizing on principles for a prosperous life. schools teach thirukkural to their students as early as kindergarten. thirukkural excerpts are often found in public transportations and gov-ernment buildings in tamil nadu, india. political leaders, orators and even common people quote thirukkural excerpts in their daily lives. since the inception of the tamil movie industry in the early 1930s, tamil literary works have taken a different form. from the very beginning, tamil movies used the platform to showcase the language's rich literature through movie dialogues and song lyrics. tamil literary works as old as sangam literature have been depicted in early movies, and the recent movies showcase a mix of old and contemporary literary works. thus, tamil movies have become an inseparable part of most tamilians' lives. while independent literary works (poems, hykoos, novels, etc.) exist, tamil movies consistently act as important medium in showcasing rich tamil literary works. the ideologies portrayed in tamil movies closely reflect tamilians' lifestyles. tamil movies and tamil politics go hand in hand @xcite . there are several people in the tamil movie industry with strong political opinions, and it is not uncommon for tamil movie dialogues and tamil songs to reflect their political ideologies. the tamil movie industry releases a movie every four days @xcite . that is, on average there are 90 movies released every year. similar to other indian language movies, tamil movies include several songs. each song in a tamil movie is written by a lyricist coordinating with the movie director and the music director. the movie director usually describes the situation and the emotion where the song is supposed to fit in. depending on the given situation by the movie director, the lyricist writes the song lyrics, and the music director composes the music for the song. though the lyrics of a tamil movie song can stand alone, they are usually an integral part of the movie enhancing an emotion. in this paper, we introduce a new tamil lyrics corpus foot_1 . specifically, we focus only on tamil movie lyrics and not tamil lyrics by independent lyricists. the main contributions of this paper are (a) a corpus including tamil song lyrics, lyricists, and other details; (b) a detailed corpus analysis focusing on tamil lyrics; (c) experimental results to identify the lyricist of a given song lyric; and (d) future research directions about how to utilize the tamil lyrics dataset. ## background songs are an integral part of tamil movies. they represent, enhance, and complement an emotion in the movie. we explain the role of these songs and exemplify with some recent tamil movie songs. one of the most common emotion expressed in tamil songs is romance. this includes falling in love, breakup, marriage, etc. for example, the song nenjukul peidhidum foot_2 from the movie vaaranam aayiram describes a situation when a man meets a woman for the first time. this song emphasizes the emotion of love at first sight. the visualization and the lyrics are dramatic, and do not depict the actual story of the movie. while the flow of the movie is not affected by the song, it plays a vital role in emphasizing a man's emotion towards a woman the first time he sees her. some tamil movie songs are created to complement the scenes in the movie. the movie would not be complete without the song. these songs include sequential scenes from the movie. for instance, the song kadhaipoma foot_3 from the movie oh my kadavulae describes the emotion of taking someone for granted while aptly fitting in the movie scenes. there are several tamil movie songs written with political and social intentions. tamil actors with political ideologies often hint their political intentions in their songs. recently, there are several songs released emphasizing on social equity. the song aalaporan thamizhan 5 from the movie mersal describes the beauty of the language tamil and glorifies tamilians life styles. recently with the advent of youtube, there are several songs that get popular even before the movie release. while releasing songs before releasing a movie has been a practice in the tamil movie industry for a while, youtube has taken these songs sooner to the audience than ever before. until recently, all movie songs were released together in a compact disk, cassette tape or an online streaming site. however, in the past few years tamil movies have started releasing singles from movies. these songs typically include a catchy lyric, music or dance moves. they act as a promotion for the movie and usually have millions of views before the movie release. kutty story 6 from the movie master and bujji 7 from the movie jagamae thanthiram fall under this category. interestingly, the song kutty story has large amount of english songs, which is a phenomena observed in many recent tamil songs. we present youtube links in the footnote to each song exemplified in this section. the english translation of the song lyrics can be viewed by turning on the subtitles in the youtube video. how are tamil song lyrics written? in most cases, the movie director comes up with a situation in the movie.. the music director composes a tune capturing the emotion of this situation. the lyricist writes the lyrics, given the tune. however, there are few music directors who compose tune to a given lyric. ## related work there exists a tamil lyrics dataset (subramanian, 2020) in kaggle for tamil lyrics. the dataset includes the lyrics of tamil songs, alongside song name and movie name. however, we could not track any further experiments or research related to this dataset. it does not include the name of the lyricist or other details. unlike this dataset, our corpus includes the name of the lyricist, the name of the music director, and the year the song was released. we also include the english translit-5 aalaporan thamizhan 6 kutty story 7 bujji erated song lyrics. we believe these information would be useful to researchers. there are some prominent tamil nlp works associated with tamil lyrics. sundara kanchana and ganapathy (2017) work with 1, 000 tamil songs to identify three aspects for a given tamil song lyric. the three aspects are base, mood, and style. the base aspect includes the types character, festival, nature, relationship, romance, occasion, spiritual, patriotic and misc. the emotion aspect includes the types happy, sad, excited, tender, scared and angry. the styles aspect includes traditional, folk, contemporary and mixed. their work is word based, and they use tf-idf as a feature in their classifier to predict these aspects. ramakrishnan a et al. ( 2009 ) work on automatically generating tamil lyrics for melodies. they approach the problem as a sequence labeling problem targeting to guess the syllabic pattern. then they forward this pattern to a sentence generation module, where they use dijkstra's algorithm to choose a meaningful phrase. in their latter work ramakrishnan a and lalitha devi (2010), follow a mapping scheme for matching melody with words. they also use a knowledge-based text generation algorithm on an existing ontology and tamil morphology generator. @xcite design a pleasantness scoring model based not only based on phonetic aspects. @xcite present analysis over a given set of lyrics. in their later work, they @xcite present a visualization tool for lyrics based on the characteristics of the lyrics. @xcite have extensively worked in automatic tamil lyrics generation. in their first work, @xcite they follow a trigram approach using the emotion of a given scene. the use this emotion as a seed word and follow tamil grammatical rules to generate lyrics. in their next work, @xcite they follow a similar approach getting 77% accuracy. following that, they @xcite intend to generate tamil lyrics given a sequence of images and a derived tune. they generate a tune automatically based on carnatic music. and tune; and (ii) context text and images. in their next work @xcite they follow a neuro-linguistic approach for lyric generation intending to add dialect, image, and author specific features. in their recent work @xcite they generate lyrics, combining their two previous works. they make use of visual images, derived tune, and other specification such as the dialect of the speaker. unlike the works described, our work tries to capture the characteristics of the lyric and the lyricist. in an analysis perspective, our work is analogous to @xcite . we go beyond, and present analysis with respect to the year the lyrics were written, and similarity between pairs of lyricists. ## corpus creation we create a corpus of tamil lyrics from www.allnewlyrics.com . the corpus consists of 5,449 tamil songs lyrics and their english transliteration. apart from the lyrics of a song, the corpus includes the name of the lyricist, music director, movie, and the year the song was released. we also present a link to the web page where the data was extracted. the statistics of the corpus are presented in from a linguistics perspective, lyrics are the most interesting. the lyrics of a song reflects the (i) situation in the movie, (ii) writing style of the lyricist, (iii) language style during which the lyrics were released, and (iv) the emotion of the song. on an average each song includes 188 words ranging from minimum 17 words to lyricist music director movie year vaali m.s. vishwanathan kasethan kadavulada 1972 tamil ெமல்ல ேபசுங்கள் பிறர் ேகட்க கூடாது ெசால்லித் தாருங்கள் யாரும் பார்க்க கூடாது [...] english mella pesungal pirar ketkka koodadhu solli thaarungal yaarum paarkka koodadhu [...] lyricist music director movie year vairamuthu a. r. rahman love birds 1996 tamil மலா் கேள மலா் கேள இது என்ன கனவா மைலகேள மைலகேள இது என்ன நிைனவா [...] english malargalae malargalae idhu enna kanavaa malaigalae malaigalae idhu enna ninaivaa aa… [...] lyricist music director movie year na. muthu kumar yuvan shankar raja kaadhal kondein 2003 tamil ெநஞ்ேசாடு கலந்திடு உறவாேல காலங்கள் மறந்திடு அன்ேப நிலேவாடு ெதன்றலும் வரும் ேவைள காயங்கள் மறந்திடு அன்ேப [...] english nenjodu kalanthidu uravaalae kaalangal maranthidu anbae nilavodu thendralum varum velai kaayangal maranthidu anbae [...] lyricist music director movie year gangai amaran ilayaraja paneer pushpangal 1981 tamil ேகாைடக்கால காற்ேற குளிர்த்ெதன்றல் பாடும் பாட்ேட மனம் ேதடும் சுைவேயாடு தினம்ேதாறும் இைசபாடு [...] english kodai kaala kaatrae kulir thendral paadum paattae manam thedum suvaiyodu dhinandhorum isai paadu [...] lyricist music director movie year thamarai darbuka siva enai noki paayum thota 2019 tamil எதுவைர ேபாகலாம் என்று நீ ெசால்லேவண்டும் என்றுதான் விடாமல் ேகட்கிேறன் [...] english ethuvarai pogalaam endru nee sollavendum endruthaan… vidamal ketkiren… [...] ## corpus analysis we present detailed corpus analysis focusing on the tamil lyrics our corpus. all statistics presented in this paper represent the distribu-tion of the data in our corpus and may not reflect the contribution of a lyricist to the tamil movie or music industry foot_4 . figure 1 we treat each word in the corpus as the simplest unit for our analysis. that is, we do not conduct analysis based on lemmas or stems of words. the top 10 lyricists with the highest number of songs and their most frequent 10 words are presented in we noticed that the top 10 most frequent words across the entire corpus (table :3, 1 st row), and the top 10 most frequent words per lyricist were all pronouns or interrogative words. our further investigation revealed that the top 100 words mostly included pronouns, interrogative words, prepositions, and conjunctions which are considered as stop words in english. thus we decided to treat the top 100 most frequent words as stop words. we acknowledge that this process is not perfect, as there are a few non stop words in the top 100 words, and few stop words after the most frequent 100 words. the top 10 most frequent words per lyricist ( in we also present the cosine similarity between the tf-idf representations of the lyrics by the top 5 lyricists. we process the lyrics similar to the processing for calculating jaccard similarity, but do not form a unique set of words for each lyricist. first, we combine all songs by each lyricist into a large text. next, jacc. sim. ## experiments and results we conduct experiments to identify the lyricist of a given lyric. we formulate a binary classification problem by generating a dataset including only the top two lyricists with the most number of songs, vaali and kannadasan. this dataset consists of 1,670 instances from the tamil lyrics corpus, where 873 instances correspond to vaali and 797 instances correspond to kannadasan. we used 80% of this dataset for training and withheld the remaining 20% for testing. we ensured the lyricists distribution is stratified across the train and test sets. ## future research we use the tamil lyrics corpus to identify lyricists formulating a binary classification task. the same corpus could be used for several other tasks. we discuss the previous works relating to tamil lyrics in section 3. in this section, we describe a few possible research that could conducted using the tamil lyrics corpus. ## conclusion in this paper, we introduce a new tamil lyrics dataset. we collect tamil movie lyrics along with the names of the lyricist, the movie, and the music director over a range of 65 years. we also present the english transliteration of the tamil lyrics. we present an elaborate literature review on studies relating to tamil lyrics. we conduct detailed corpus analysis revealing the top 10 frequent words per decade, and top 10 frequent words per lyricist. we also study the similarity of the lyrics between different lyricists using standard similarity approaches. next, we formulate a binary classification task to identify the lyricist, given a song lyric. we fine tuned the state-of-the-art bert models to predict the lyricists. our experimental results show that identifying lyricists is a challenging task, and there is scope for improvement. we go beyond the experiments, and elaborate on the possible future research directions with the tamil lyrics corpus. we intend to add more instances to our corpus to encourage research on tamil lyrics.