Classification of Buddhist Verses: The Efficacy and Limitations of Transformer-Based Models (2024)

Abstract: This study assesses the ability of machine learning to classify verses from Buddhist texts into two categories: Therigatha and Theragatha, attributed to female and male authors, respectively. It highlights the difficulties in data preprocessing and the use of Transformer-based models on Devanagari script due to limited vocabulary, demonstrating that simple statistical models can be equally effective. The research suggests areas for future exploration, provides the dataset for further study, and acknowledges existing limitations and challenges.

## introduction the term "gāthā" (gatha) denotes a poetic meter primarily employed in legends and folklore, yet it is notably absent from the vedas @xcite . gathas are popular in maharashtra, india, where locals are familiar with the gathas of tukaram @xcite . however, the earliest known reference to gathas appears in the avesta, a zoroastrian scripture compiled during the sasanian empire (224-651 bce) @xcite . the languages in which these ancient gathas were written have since become extinct. consequently, interpreting them is challenging and necessitates reliance on extant languages that exhibit similar, yet distinctly different, structures. this study examines two collections from the buddhist canonical literature: theragathapali and therigathapali, which are, respectively, the linewise utterances attributed to male and female saints. this literature is written in pali, a language believed to be a mixture of prakrit languages, closely related to the vernacular of the common people during the time of siddhartha gautama buddha (circa 600 bce). the authorship of some gathas is debatable. @xcite observes that in pali literature, authorship details are occasionally provided at the beginning or end of the texts. however, not all authors considered it essential to include such information. in examining the authorship of the therigatha, @xcite suggests that the authorship of some verses may be doubtful, indicating that while some verses are traditionally attributed to the female saints themselves, others may have been composed or recited by different individuals, including the possibility of later attribution by compilers. this uncertainty in authorship challenges the straightforward attribution of these texts to the female saints they are associated with. nevertheless, studies demonstrate that the theri gathas differ from the thera gathas. blackstone (2013) argues that the theri gathas focus more on themes of overcoming suffering, societal constraints, and personal liberation. a study by @xcite confirms the uniqueness of topics in therigatha. typically, a gatha is a two-line verse, although variations include verses comprising three or four lines. figure 1 provides a sample two-line gatha in devanagari script. banerjee (2017) suggests that translations of gathas influences the perception of these ancient texts. for example, the gatha from figure 1 is translated by @xcite as "whoever wants to do later what he should have done first, falls away from the easeful state and later burns with remorse", while one of the contributors of this study translates the second line as "he destroys pleasure producing points and regrets later". the abundance of transformer-based models @xcite and their proficiency across various domains @xcite @xcite , particularly in classification tasks @xcite @xcite @xcite , inspired us to conduct a study on their performance in classifying verses from low-resource pali texts. while we acknowledge the debates around the authorship of some therigatha verses, we deliberately avoid this discussion in our study due to the lack of definitive evidence regarding authorship. consequently, we treat therigatha verses as authored by female authors and theragatha verses as authored by male authors. the goal of this study is to determine whether transformer-based models can outperform traditional machine learning models in the binary classification of the verses. we hypothesize that transformer-based models, even when pretrained on languages other than pali, can still identify patterns specific to each class. additionally, we aim to assess the performance difference of these models when using devanagari script versus roman script. through this investigation, we aim to highlight the challenges associated with this task and suggest directions for future research. ## related work research on poetry classification in the pali language using machine learning is scarce, however, insights can be drawn from related areas, including poetry classification in other languages, text classification in low-resource settings, and computational analysis of pali texts. one of the earliest studies in poetry classification is by @xcite , who use logistic regression to examine stylistic and content features that distinguish professional poets from amateurs. the authors extract features related to diction, sound devices, affect, and imagery to identify elements contributing to poetic sophistication. similarly, pal and patel (2020) classify hindi poems using machine learning, providing insights into poetry classification in an indo-aryan language closely related to pali. the authors employ classical models, such as naïve bayes, random forest, and svm, achieving a maximum accuracy of 64% with naïve bayes, highlighting the challenges of poetry classification due to the morphological richness and varied sentence structures. in the context of text classification for lowresource languages, recent research suggests that cross-lingual models, such as xlm @xcite , may sometimes offer performance gains compared to classic machine learning models like svm or naïve bayes. for instance, @xcite introduce a model called agglutifit, fine-tuned from a cross-lingual pre-trained model (xlm-r), which significantly outperforms strong baselines in terms of accuracy. additionally, @xcite benchmark multilabel topic classification in the kyrgyz language, evaluating several baseline models, including classical approaches and neural models like xlm-roberta. their findings indicate that the multilingual model xlm-roberta outperforms classical models in terms of f1 score. however, transformer-based models do not always surpass traditional machine learning models for low-resource languages. for example, lalthangmawii and singh (2023) found that the svm model achieved the highest accuracy (75%) on a sentiment classification task for the mizo language, performing similarly to the xlm-roberta model using a transfer learning approach. another method for handling low-resource languages is leveraging machine translation. recent work by @xcite provides valuable insights into sentiment classification for low-resource indian languages using machine-translated datasets. the results highlight the potential of datasets translated with tools like google translate and indicate that models such as lstm can effectively preserve sentiment by accounting for sequential patterns. focusing specifically on pali texts, zigmond (2021) conduct a computational analysis of the pali canon. the author uses computational text mining to examine various volumes of the canon, extracting linguistic and thematic insights. by employing techniques such as k-means clustering and principal component analysis, they reveal differences between older texts (vinaya and suttas) and later ones (abhidhamma). the research also underscores the complexity of pali language processing, including multiple word declensions, elisions, and compound formations. ## dataset the dataset utilized in this study comprises the thera and theri gatha texts from the khuddakanikaya volume of the sutta-pitaka, which is the third part of the buddhist canonical literature, tipitaka 1 . each gatha is categorized into chapters based on the number of verses attributed to each author: single verses are compiled in the chapter named ekaka-nipaat, meaning 'collection of ones', while chapters such as dukanipaat-'collection of twos'-contain texts with two verses from a single author, and so forth. the theragatha consists of 1,288 verses spread across 21 chapters, whereas the therigatha contains 524 verses distributed over 16 chapters, with all verses sequentially labeled within their respective compendiums. to study the potential impact of script on the training of the classifier, both the devanagari and roman versions were used. the manual preprocessing involved several steps: 1. punctuation handling: we agreed on approaches to interpret punctuation marks, considering variations in their usage across different scripts. lenges in separating these combined words, we decided to work with the combined forms as they appear in the text. after the manual preprocessing of the text, we encountered discrepancies in the counts of distinct words when tokenizing the verses by spaces. assuming a one-to-one correspondence between tokens in the devanagari and roman scripts, a dictionary-based test was applied to identify these discrepancies. the test revealed several transliteration nuances. for instance, some symbols such as and in devanagari are represented by two utf-8 code points, which leads to confusion with symbols and , respectively. another challenge was caused by complex compounding rules; for example, space-based tokenization ambiguously mapped the symbol to either 'muni' or 'munin', depending on the neighboring tokens (a one-tomany case). similarly, both symbols and map to 'ti' in the romanized script (a many-to-one case). these cases demonstrate that space-based tokenization may not adequately capture the nuances of these complex verses. for this study, we decided to exclude three nuanced verses from the theri gathas and sixteen nuanced verses from the thera gathas where we were unable to easily resolve the inconsistencies. this resulted in 1793 verses in our dataset foot_0 . ## experiments and results the overall task can be defined as a binary classification problem with two categories: 'm' for theragathas and 'f' for therigathas. the dataset, divided by script type into devanagari and roman subsets, was split into training (75%) and test (25%) sets. considering the dataset's imbalance, we report key metrics such as roc-auc, matthews correlation coefficient (mcc), as well as precision, recall, f1scores, and average precision (ap) for both classes. we deliberately avoided sampling to address the imbalance due to the dataset's small size. however, by providing a comprehensive set of metrics, we aim to give a detailed comparison of the models' performance across different aspects. first, we applied traditional machine learning models: multinomial naïve bayes, logistic regression, random forest classifier (rfc), support vector classifier (svc), gradient boost classifier (gbc), and k-nearest neighbors classifier (knn), on the roman script to classify gathas. space tokenization and a tf-idf matrix were used for all models except for the multinomial naïve bayes, which served as a baseline model using simple count vectorization. the multinomial naïve bayes assumes conditional independence of tokens and positional independence of features. naïve bayes can be optimal under certain circumstances, such as when the conditional independence assumption holds @xcite . to assess whether transformer-based models could improve specific aspects of classification, such as precision and recall, we experimented with fine-tuning the following models: xlm (lample and conneau, 2019), xlm pre-trained additionally on our training corpus, t5-base @xcite , and electra-small @xcite . figure 2 presents the classification results for the roman script. similar experiments with the devanagari script revealed that while transformer-based models underperformed relative to their counterparts in roman script, the performance disparities among traditional models were minimal, as depicted in figure 3 . additionally, our trials with a byte-level t5 @xcite model yielded substantially lower performance (auc 0.58 for devanagari), which we attribute to its inability to effectively handle scriptspecific complexities, leading to its exclusion from our study. when investigating why transformer-based models exhibit inferior performance compared to classic machine learning algorithms, we analyzed the number of tokens generated by tokenizers for both devanagari and roman scripts in the test subsets. our attempt to employ the shap framework (lundberg and lee, 2017) on the best-performing models to explain their discrimination decisions did not reveal any specific features that contribute significantly to either of the classes. ## discussion and further research the study highlights persistent challenges in using original, non-romanized scripts with modern transformer-based models for classification tasks, primarily due to inadequate token coverage in the models' vocabularies. previous studies, such as the one by @xcite , showed that the compatibility of tokenizations is crucial in multilingual language models, discussing the importance of vocabulary size. more recently, @xcite confirmed that the choice of tokenizer significantly impacts a model's downstream performance. they suggest that tokenizers not tailored to handle a variety of scripts can lead to inefficient tokenization, directly affecting model performance, and that larger vocabulary sizes are required for multilingual tokenizers compared to those designed for english only. although romanized versions of the scripts enabled the use of a broader range of models, these models still did not surpass the performance of traditional machine learning algorithms. this outcome suggests that the employed models failed to identify any class-specific patterns within the dataset, likely because these models lacked sufficiently relevant data during their pretraining stages. notably, additional pre-training of the xlm model improved the auc on the classification task, and a paired bootstrap test with 10 5 iterations confirmed the statistical significance of this improvement (@xmath1.05). extended research is necessary for the authorship attribution task. our next step is to identify therigathas that are consistently misclassified by the majority of models and perform a detailed analysis of these cases. this includes annotating and analyzing specific gathas whose authorship is disputed by scholars. statistical sampling to identify whether the differences between the theri and thera gathas are significant may help reveal if there are substantial distinctions between the two classes of gathas from a machine learning perspective. additionally, compiling an extensive pali corpus to pre-train a transformer model would enable us to experiment with its discriminatory abilities and its capability to generate novel gathas. ## limitations first, our dataset is small and imbalanced, with only slightly over 10% of words shared between the thera and theri gathas. this low overlap might explain why classical machine learning algorithms were able to effectively discriminate between the classes, primarily by relying on words unique to specific classes. the second limitation pertains to the existing transformer models, which often lack the comprehensive vocabulary necessary for thorough evaluation.