An Aid to A ssamese Language Processing by Constructing an Offline A ssamese Handwritten Dataset (2024)

Abstract: Recent years have seen a growing interest in analyzing Indian handwritten documents. In pattern recognition, particularly handwritten document recognition, the availability of standard databases is essential for assessing algorithm efficacy and facilitating result comparisons among research groups. However, there is a notable scarcity of standardized databases for handwritten texts in Indian languages. This paper presents a comprehensive methodology for the development of a novel, unconstrained dataset named OAHTD (Offline Assamese Handwritten Text Dataset) for the Assamese language, derived from offline handwritten documents. The dataset, which represents a significant contribution to the field of Optical Character Recognition (OCR) for handwritten Assamese, is the first of its kind in this domain. The corpus comprises 410 document images, each containing a diverse array of linguistic elements including words, numerals, individual characters, and various symbols. These documents were collected from a demographically diverse cohort of 300 contributors, spanning an age range of 10 to 76 years and representing varied educational backgrounds and genders. This meticulously curated collection aims to provide a robust foundation for the development and evaluation of OCR algorithms specifically tailored to the Assamese script, addressing a critical gap in the existing literature and resources for this language.

## introduction the recognition of language and characters has gained importance as a crucial field of research across the globe. the study of optical character recognition (ocr) systems is essential to enable automated visual perception and reading. ocr involves a document image analysis process where machine-printed or handwritten documents are subjected to a learning process to convert them into machine-readable and editable format @xcite . the study of ocr revolves around the different modes of acquisition and modes of writing of the document/texts. based on the mode of writing, ocr is categorized into handwritten and printed, while based on the mode of acquisition, it is categorized in online and offline. thus, ocr recognizes both handwritten and printed text that is obtained online as well as offline mode @xcite . in the online recognition mode, the temporal information (position of the pen, trajectory, etc.) is considered. it requires direct interaction with the user as its handwriting is considered. offline recognition refers to optically capturing the handwritten text as an image and further analyzing and processing it for recognition. it does not require direct interaction with the user whose handwriting is taken under consideration (figure 1 ). the initial stage in ocr is scanning the documents if they are in a physical form. preprocessing such as cropping, noise removal, skew removal, binarization, etc., are performed as required. the next important step is the segmentation process, which incorporates segmenting or extracting or isolating the text lines in the document images (handwritten or printed). segmentation of handwritten documents is a challenging task compared to the printed. the next phase in segmentation is the extraction or isolation of the constituent words and characters from the extracted text lines. the final stage incorporates a classification model that is able to classify and recognize the final characters obtained after segmentation. a general pipeline of this process is shown in figure 2 . there are many challenges associated with the segmentation or recognition of characters in handwritten ocr. some of them are: among the various challenges witnessed in literature for the segmentation or recognition of languages and characters @xcite , the availability of the dataset in the desired form and language is a challenging one. literature has witnessed a surge in ocr technologies for various languages such as arabic, chinese, roman, etc., but ocr techniques for regional languages such as assamese are comparatively less. the structure of various languages across the globe is different from one another. thus techniques developed for the segmentation and recognition of one language are different from the other. due to this, ocr techniques for different languages will also be different. the main motivation of the present work delves significantly into the development of an offline dataset for the assamese language, as there is no offline public database of assamese script. assamese is one of the most dominant languages of north-east india. so creating a standardised database will greatly help the people of the region for improved accessibility. ## related work this section discusses the related work for different handwritten datasets of indic languages. the dataset selection carries significant weight due to the inherent diversity in language patterns, resulting in variations in learning outcomes. this variability is especially evident in tasks such as optical character recognition (ocr), where different datasets have been extensively examined and experimented with. for instance, languages like assamese and hindi incorporate diacritical marks (matras), which affect character recognition differently than languages like telugu and english, which lack such components. consequently, researchers have explored a range of datasets to understand these variations and optimize ocr systems for various linguistic contexts. good datasets are essential for understanding and interpreting handwriting in any language. the field of handwritten document analysis for indian languages has witnessed significant advancements in recent years, largely facilitated by the development of diverse and comprehensive datasets. these datasets have played a crucial role in enabling researchers to develop, evaluate, and benchmark various algorithms and methodologies for optical character recognition (ocr) and related tasks. a chronological examination of the literature reveals a progression in both the scope and sophistication of these datasets. in 2008, (bhattacharya and chaudhuri, 2008) made a notable contribution with the isi-hdnd dataset, which comprised 22,556 devanagari and 23,392 bangla numerals, addressing the need for numeral recognition in two major indian scripts. the following year, @xcite introduced the chars 74k dataset, a substantial collection of 74,000 english and kannada characters, which proved invaluable for cross-lingual character recognition studies. @xcite focused on online handwriting recognition with their dataset of 8,235 assamese characters, while @xcite contributed significantly to kannada script research with the khtd dataset, encompassing 4,298 text lines and 26,115 words. this period also saw @xcite introduce the cma-terdb1.1, a collection of 100 pages of bengali text lines, further enriching the resources for bengali script analysis. @xcite made a substantial contribution with the dhcd dataset, featuring 92,000 devanagari characters, which has since become a benchmark for devanagari character recognition tasks. @xcite expanded the cmater database with two new additions: cmaterdb2.2.3, containing 15,528 devanagari words, and cmaterdb2.1.3, comprising 18,931 bengali words, both of which have proven instrumental in word-level recognition studies. a significant leap towards multi-script analysis came with (obaidullah et al., 2018)'s phdindic_11 dataset, which included 1,458 pages of text lines from 11 different scripts, including bangla, oriya, roman, and urdu. this dataset has been particularly valuable for developing script-independent recognition techniques and cross-script studies. more recent contributions have focused on addressing gaps in specific language resources. (dutta and muppalaneni, 2021) developed a dataset of 512 assamese handwritten digits, addressing the scarcity of resources for this less-studied language. similarly, @xcite contributed to gujarati script research with their dataset of 8,000 handwritten digits. these datasets have significantly contributed to the advancement of handwritten document analysis for various indian languages and scripts. the progression from character-level to word-level and eventually to multi-script datasets reflects the evolving challenges and ambitions in the field of handwritten document analysis for indian languages. it is apparent from ## dataset description due to the lack of an existing assamese dataset suitable for our specific task, we have undertaken the initiative to create our own custom dataset. the novel dataset developed for this research has been designated as offline assamese handwritten text dataset (oahtd). this nomenclature reflects the nature of the data collected, emphasizing its focus on offline handwritten text in the assamese language. the process of data collec-tion and compilation for the oahtd followed a structured workflow, which is visually represented in figure 5 . this workflow diagram delineates the systematic approach employed in the dataset's creation, encompassing the various stages from initial data gathering to final dataset compilation.