Building and Analysis of T amil Lyric Corpus with Semantic Representation (2022)

Abstract: In the new era of modern technology, the cloud has become the library for many things including entertainment, i.e, the availability of lyrics. In order to create awareness about the language and to increase the interest in Tamil film lyrics, a computerized electronic format of Tamil lyrics corpus is necessary for mining the lyric documents. In this paper, the Tamil lyric corpus was collected from various books and lyric websites. Here, we also address the challenges faced while building this corpus. A corpus was created with 15286 documents and stored all the lyric information obtained in the XML format. In this paper, we also explained the Universal Networking Language (UNL) semantic representation that helps to represent the document in a language and domain independent ways. We evaluated this corpus by performing simple statistical analysis for characters, words and a few rhetorical effect analysis. We also evaluated our semantic representation with the existing work and the results are very encouraging.

## introduction tamil (pronounced as tamizh) is one of the ancient and classical indian languages spoken in tamil nadu. at the beginning of the 21st century, about 66 million people were speaking tamil language, one of the oldest languages surviving from 300 bce @xcite . some people in sri lanka, malaysia, singapore, mauritius, fiji islands, canada and south africa are also currently speaking tamil language foot_0 . according to the history of tamil gram mar and lexical changes, the period can be categorized into three, old tamil (300 bce700 ce, sangam period), middle tamil (700 -1600) and modern from 1600 @xcite . very few literature exists from the old tamil containing 2381 poems by 473 poets and about 102 of which remained anonymous @xcite . scholars from the sangam period dealt with emotional and material topics such as war, gov ernance and trade. these literatures are difficult to understand and require literates to explain. in the late 19th century, tamil literature has been simplified so that everyone can understand, which has created interests amongst tamil people. poets like subramaniya bharathi utilized the power of tamil language to create awareness of freedom for both women and british india. in the last century, songs in tamil films have also played a vital role in interacting with people on social and political issues. in the modern form, tamil cinema can be classified as a combination of drama and songs, where the latter is composed with music according to a different genre of the lyrics. in the recent past, the encroachment of foreign languages, tamil film lyrics are mixed with other language words which was a challenging task to build the lyric corpus. mining lyric documents facilitates the users to get the lyric characteristics, such as emotion, genre and a lyricist and also search the lyrics on the web. the linguistic features are however inadequate for identifying lyric characteristics, especially for morphologically rich languages like tamil. for extracting higher level features, we chose semantic representation based on universal networking language (unl), which defines the conceptual structure in the form of a semantic graph foot_1 . to the best of our knowledge, no work has been done previously to build a tamil lyric corpus with semantic graph representation. the rest of the paper is organized as follows. section 2 describes the related work. in section 3, we discuss the linguistic issues faced while building the tamil lyric corpus. section 4 explains about building the tamil lyric corpus and illustrates some statistics about it. the evaluation of the corpus is given in section 5. finally, section 6 concludes the paper with future work. ## related work in this section, we discussed related work carried out for the creation of corpus of tamil and other languages. @xcite created a raw corpus for assamese language. they build the corpus with a total of 1.5 million words from their main categories such as media, learned material and literature. in another work, @xcite explained the procedure and issues of automatic corpus creation for bangla language. @xcite developed the newspaper text corpus for hindi language. they considered 4 online websites of newspapers with a fixed time span of 10 years (20072016) and built a millionword corpus of the hindi newspaper texts. the first corpus for tamil language was created by technological development for indian languages (tdil), the department of electronics, govt. of india. this consists of more than 3 million words from various domains @xcite . later on, various research organiza tions are working for the improvement and annotated of central institute of indian languages (ciil) tamil corpus @xcite . @xcite developed an annotation corpus of discourse connectives and their arguments by using 2,00,000 sentences of tamil en cyclopedia articles. tamil emotional speech corpus was built by @xcite and identified five emotions of happy, sad, anger, fear, and neural using gmm classifier. analysis of lyrics based on the usage of words, concepts cooccuring and rhymes has been done by @xcite . the same author has developed the lyric visualization tool to visualize each lyric characteristic such as emotion, genre, rhyming features, pleasantness and rhetorical style based on statistical modelling @xcite . @xcite built a new tamil lyric corpus with a dataset of 5449 songs and identified the lyricist of the song using the bert model. in our corpus, the number of lyrics is high and also the semantic representation is available for the document. this semantic information is useful to translate the tamil lyric into other natural languages and also to identify the semantic textual similarity @xcite , relation extraction @xcite , search engine (saviya shree et al., 2013) and sentiment analysis @xcite . a language independent semantic representation of unl has been used to convey the word and content based information of the document @xcite . parteek @xcite proposed a unl based mt system for punjabi language by developing punjabi enconverter, punjabi deconverter and a web interface for online enconversion and deconversion process. the author addressed the limitations of the punjabi machine translation. to the best of our knowledge, there is no semantic graph representation for tamil lyric documents. for tamil language, @xcite , have proposed the englishtamil machine translation system using unl. they also developed the sentence formation algorithm to rearrange the tamil words into correct sentences. in another work, unl semantic representation has been developed by @xcite for the tourism domain using rule based approach. this work is not suitable for lyric semantic representation, since the relations and the contextual information in the lyric documents are different. although, several corpora have been developed for tamil language and nontamil languages for different domains, there is not enough corpus size available to perform applications, such as automatic generation of lyrics, lyric similarity, emotion, genre identification or other natural language processing (nlp) related tasks. ## linguistic aspects of traditional and lyric documents from the viewpoint of linguistic criteria, lyric documents have more interesting issues when compared to traditional documents. the processing of lyrics varies from normal text process ing. usually, the document conveys information about a particular topic, whilst lyrics meant to convey emotion and feelings. in the traditional documents, the sentences sometimes follow the standard subject-verb-object (svo) structure where the subject comes first, the verb second, and the object third, whereas in the lyric documents, either the verb will be followed by a con secutive noun or the verb may not present in the sentence at all. in addition, some lyrics may contain compound words, colloquial words and in some cases numeral words. also, the lyric document contains specific properties such as rhyme, meter, freshness and pleasantness to convey emotions and amenable to music. special attention was given to defining and extracting these features. rhyme scheme is used as strings for letters, where each line corresponds to the repetition of same syllables. in tamil lyrics, the internal and external rhyme schemes are presented with three variations, alliteration, rhyme and endrhyme, known in tamil literature as (mōṉai) (first two letters are identical for two words in a rhyme), (etukai) (second two letters are identical for two words in a rhyme) and (iyaipu) (last two letters of the two words in a rhyme are identical) @xcite . the freshness of the word based on various timelines facilitated to identify the characteristics of the lyrics. generation of rhythmic pattern by grouping the strong and weak beats together is measured by using the meter score. moreover, semantic relations between words of the lyric may be similar to those of documents, however, additional context and semantic attributes need to be extracted to convey the lyric characteristics. the need of tackling the above unique features of lyrics require construction of a large corpus and careful categorization is important for computerization. ## building lyric corpus for tamil language in the recent days, most of the researchers focus on the automatic extracting and processing of lyrics @xcite . tamil films consist of several thousand lyrics; however, a huge amount of these lyrics is not yet computerized. hence, it is very important to build a tamil film lyric corpus. in this paper, a tamil lyric corpus has been built by tagging the unannotated corpus and used to determine the lyric characteristics. figure 1 shows the diagram for building a tamil lyric corpus. for building the lyric corpus, we follow four steps, i) data collection, ii) data refinement, iii) data tagging and iv) data validation. ## conclusion and future work in this paper, we described the tamil lyric corpus and the linguistic issues faced during the process. this corpus has been stored in the xml format to add or remove the data easily. we believe that this is the first attempt in creating the tamil lyric corpus. this paper also discussed the semantic representation for extracting the context level information from the documents. we carried out the evaluation by performing statistical and rhetorical analysis of lyric corpus which resulted in promising results. in future, we have planned to increase the corpus size of the lyric documents by crawling in the web and annotating various lyric characteristics of emotion, genre, lyricist inferred based on the semantic features. we have also planned to build the semantic representation using a semisupervised approach. we also planned to extend our work to other morphologically rich languages.