An Annotated Dataset and Automatic Approaches for Discourse Mode Identification in Low-resource B engali Language (2022)

Abstract: The modes of discourse aid in comprehending the convention and purpose of various forms of languages used during communication. In this study, we introduce a discourse mode annotated corpus for the low-resource Bangla (also referred to as Bengali) language. The corpus consists of sentence-level annotation of three different discourse modes, narrative, descriptive, and informative of the text excerpted from a number of Bangla novels. We analyze the annotated corpus to expose various linguistic aspects of discourse modes, such as class distributions and average sentence lengths. To automatically determine the mode of discourse, we apply CML (classical machine learning) classifiers with n-gram based statistical features and a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) based language model. We observe that fine-tuned BERT-based approach yields more promising results than n-gram based CML classifiers. Our created discourse mode annotated dataset, the first of its kind in Bangla, and the evaluation, provide baselines for the automatic discourse mode identification in Bangla and can assist various downstream natural language processing tasks.

## introduction discourse is the notion of conversation that is expressed through language. based on @xcite , discourse indicates the relationship between states, events, or beliefs manifested within one or multiple sentences in a given mode of communication. understanding discourse structures and identifying relationships between various modes can help downstream natural language processing tasks including text summarization @xcite , question answering @xcite , anaphora resolution @xcite , and machine translation @xcite . the modes of discourse, also referred to as rhetorical modes, represent the variety, conventions, and purposes of the dominant types of language used in communication (both oral and written). the discourse modes have high importance while writing composition because they attribute to several factors that would affect the quality and coherence of a text. the combination and interaction of various discourse modes make a text organized and unified @xcite ). to give an example, the writer may start an expressing an event through narration, then provide details regarding using descriptive modes and establish ideas with argument. discourse modes have also importance in rhetorical research as they are closely related to rhetoric @xcite that provides guidelines for effectively expressing content. researchers categorized modes of discourse into various categories @xcite @xcite . based on @xcite , discourse modes can be classified into four categories, narration, description, exposition, and argument. narration mode primarily focuses on governing the progression of the story by presenting and connecting events; exposition mode instructs or explains; the argument aims to provide a convincing or persuasive statement; description tries to provide detailed mentions of characters, objects, and scenery, in a figurative language. @xcite categorized the mode of discourse into five categories, narration, exposition, description, argument and emotion expressing sentences in narrative essays, while @xcite annotated discourse mode of short story into argumentative, narrative, descriptive, dialogic and informative categories. although a piece of text can be labeled as a specific mode of discourse, it is not uncommon to have text snippets with multiple modes of discourse @xcite where one of them possesses the dominant role. although discourse structure and mode have a significant role in various downstream natural language processing tasks, research in this area is largely unexplored in bengla. although bengali is the 7th most spoken language in the world foot_0 , nlp resources are scarcely available except few areas such as sentiment analysis @xcite or inappropriate textual content detection @xcite . regarding discourse analysis, only a limited number of works performed research (chatterjee and @xcite @xcite @xcite . however, to the best of our knowledge, no study related to automatic discourse mode identification has been carried out yet. thus, in this study, we introduce an annotated dataset and present a set of techniques for the automatic identification of discourse modes. following the rough guidelines provided by @xcite and @xcite for discourse mode annotation, we manually categorize a dataset of 3310 sentences from bengali novels into various discourse modes. the sentences are annotated in three modes of discourse, narrative, descriptive and informative. for automatic identification of the discourse mode, we extract word n-gram based features from the text and then employ several classical machine learning (cml) classifiers such as logistic regression (lr), support vector machine (svm), random forest (rf). in addition, the transformer-based multilingual bert language model is leveraged and fine-tuned for discourse mode determination. we observe that the multilingual bert model yields better performance than the cml classifiers, although the difference is not substantial compared to lr or svm. ## data annotation and collection the data collection process starts with identifying a set of novels from bengali literature. we select six 20th-century bengali novels েগালেমেল েলাক, পেথর পাঁ চািল, আরণয্ক,পটাশগেড়র জঙ্গ,নি ত নরেক, িহমু ) written by three famous bengali novelists, 'shirshendu mukhopadhyay', 'bibhutibhushan bandyopadhyay', and 'humayun ahmed'. unlike english, the electronic versions (i.e., ebooks) of bengali books are hardly available as ebooks are not popular among bengali readers. moreover, we notice that most of the ebooks available in pdf format were created by scanning images of the print versions; therefore, they are not suitable for text extraction. we find a website that provides a set of bengali fiction in epub format. from there, we manually download the abovestated six bengali novels and extract the text for annotation. three native bengali speakers with university-level education perform the annotation. annotating the mode of discourse in a piece of text (i.e., sentence) is often challenging since a sentence may have multiple modes, or the distinction is often not obvious. thus annotators are provided a set of online resources and guidelines from a number of publications. the discourse modes are selected based on the existing works of @xcite and @xcite al. (2020). song et al. (2017) categorized modes of discourse into five categories, narration, exposition, description, argument and emotion in narrative essays, while dhanwal et al. (2020) annotated discourse modes into argumentative, narrative, descriptive, dialogic and informative categories. as our annotated content (i.e., excerpted sentences of bengali novels) are more similar to the content (i.e., short stories) of dhanwal et al. ( 2020 ), our annotated discourse modes are more similar to their annotation. however, we notice that the presence of the argumentative mode in a fictional novel is rare as instead of establishing any opinion, a novel tells a story in chronological order. besides, it is observed that the dialogic category itself does not comprise any new mode. instead, it echoes the narrative or descriptive or other modes from a third-person point of view; thus, we do not include it as a separate mode. ## results and discussion the results reveal that all the four cml classifiers, lr, svm, rf, and et, yield high performance for the narrative class prediction; they achieve f1 scores between 0.84-0.89 and an accuracy of around 97%. for the descriptive class prediction, lr and svm perform better than the rf and et; they obtain f1 scores over 0.60 compared to 0.4 scores of decision tree-based classifiers. however, we observe that for informative class prediction all the classifiers perform poorly. we observe that the performances of cml classifiers are affected by the class distribution of the dataset. since the narrative class contains close to 70% of the instances in the dataset, the classifiers are biased towards it ( the transformer-based multilingual language model yield slightly better performance than the cml classifiers. for the dominant narrative class, it attains an f1 score of 0.912. for other classes, it obtains similar f1 scores of the lr and svm, around 0.67 and 0.05, respectively. it is noticed that all the classifiers perform poorly for the minor informative class prediction. the results suggest that the transformerbased multilingual bert model can be effective for discourse mode classification in bengali text. although we do not notice signif-icant improvement compared to cml classifiers in this study, it could be attributed to limited labeled data. with more labeled data incorporated, the improvement could be higher ( transformer-based models have shown state-of-the-art performances for various nlp tasks across languages). low resource language such as bengali suffers from data annotation issues, as there are not enough resources to create a large labeled dataset. thus, incorporating a pre-trained model can help address the scarcity of annotated data in the bengali language to some extent. ## summary and future work in this study, we introduce a corpus consisting of sentences level annotation of various modes of discourse. the corpus consists of excerpted text from bengali novels annotated with three different discourse modes: narrative, descriptive and informative. we provide details of the annotation procedure, such as annotation guidelines and annotator agreements, and investigate the characteristics of various discourse modes. finally, we employ cml and deep learning-based classification approaches for automatic discourse mode identification. we observe that transformer-based fine-tuned language models yield the best performance. our future work will expand the size of the corpus and demonstrate the usefulness of discourse mode annotated data for downstream tasks such as automated essay scoring and sentiment analysis in the low-resource bengali language.