Pipeline Enabling Zero-shot Classification for B angla Handwritten Grapheme (2023)

Abstract: This research investigates Zero-Shot Learning (ZSL), and proposes CycleGAN-based image synthesis and accurate label mapping to build a strong association between labels and graphemes. The objective is to enhance model accuracy in detecting unseen classes by employing advanced font image categorization and a CycleGAN-based generator. The resulting representations of abstract character structures demonstrate a significant improvement in recognition, accommodating both seen and unseen classes. This investigation addresses the complex issue of Optical Character Recognition (OCR) in the specific context of the Bangla language. Bangla script is renowned for its intricate nature, consisting of a total of 49 letters, which include 11 vowels, 38 consonants, and 18 diacritics. The combination of letters in this complex arrangement provides the opportunity to create almost 13,000 unique variations of graphemes, which exceeds the number of graphemic units found in the English language. Our investigation presents a new strategy for ZSL in the context of Bangla OCR. This approach combines generative models with careful labeling techniques to enhance the progress of Bangla OCR, specifically focusing on grapheme categorization. Our goal is to make a substantial impact on the digitalization of educational resources in the Indian subcontinent.

## introduction ocr, a significant technological innovation, has revolutionized the processing and examination of textual content in the contemporary digital age. ocr technology, specifically designed for the purpose of identifying and converting printed or handwritten text into text that can be processed by machines, has facilitated the retrieval, searchability, and manipulation of vast quantities of information across various languages, including bangla. bangla/bengali has a rich and complicated writing system that makes it hard for ocr systems to read because of its complex ligatures, unique letters, and complicated calligraphy. ocr for bangla characters aims to bridge the disparity between physical documents and digital databases by offering solutions for activities such as document digitization, language translation, and text analysis. this study is performed on a global ai competition, bengali.ai handwritten grapheme classification hosted by kaggle and bengali.ai where our study topped the final leaderboard. the main objective of this research was not exclusively to categorize handwritten characters into predetermined classes, but rather to construct a model with the ability to identify and classify classes that were not explic- in a recent publication by @xcite , a comprehensive analysis of thirteen papers on ocr for the bangla language was published. the authors reported that the bidirectional long short-term memory (blstm) model, as proposed by @xcite , demonstrated higher accuracy among the investigated approaches. ## dataset in the realm of modern bangla literature, a distinct collection of graphemes is frequently utilized, with their recognition being established by transcriptions derived from the google bangla asr dataset as the primary point of reference @xcite . the dataset utilized for this objective is extensive, comprising 127,565 spoken utterances that were transcribed, resulting in a cumulative count of 609,510 words and 2,111,256 graphemes. the dataset consists of 1,295 frequently used bangla graphemes based on specific criteria, including occurrence in words and frequency. these graphemes comprise three main components: vowel diacritics, consonant diacritics, and grapheme roots. vowel diacritics, represented by 11 classes, are typically found at the end of unicode strings, with a null diacritic for cases of absent vowels. consonant diacritics, forming diverse combinations, resulted in 8 classes. the remaining grapheme roots, including vowels, consonants, and conjuncts, are limited to 168 classes based on their prevalence in everyday language. the painstaking compilation of metadata obtained from several sources has been a great resource for conducting comprehensive investigations into the relationship between handwriting and various categories of metadata. it is noteworthy to mention that the metadata pertaining to the training set has been made publicly accessible; however, access to the metadata of the test set can be acquired through a formal request to the authority @xcite . the dataset has been made available to the public domain as a fundamental element of the bengali.ai handwritten grapheme classification kaggle com-petition foot_0 . in this dataset, a meticulous distribution was implemented, whereby 200,840 samples were assigned to the training set, 98,661 samples were allocated to the public test set, and 112,381 samples were selected for the private test set. significantly, a rigorous standard was implemented to guarantee the absence of any duplication in contributions within these sets. it is worth mentioning that the graphemes that occur less frequently were predominantly allocated to the private test set, and none of them were incorporated into the training subset. during the duration of the competition, players strive to improve their performance by analyzing the results obtained from the public test set. on the other hand, the outcomes obtained from the private test set are kept undisclosed for every submission and are solely disclosed once the competition is concluded. significantly, a deliberate decision was made to include 88.4% of the out-ofdictionary (ood) graphemes in the private test set. this strategic choice was intended to discourage the development of models that rely entirely on public standings from overfitting. the aforementioned strategy functioned as a source of motivation for the participants to devise techniques that are capable of categorizing out-of-distribution graphemes by autonomously identifying the desired variables. ## method our model classifies against 14784 (168 × 11 × 8) class which are all the possible combinations. therefore, we needed to know the combinations of labels made up of grapheme. prediction of the relationship between the combination of labels and grapheme are done from the label of the given train data. the generation of synthetic data and the conversion of the prediction results into three components are based on this correspondence. data splitting process was conducted randomly. as a result, an unintended grapheme root class was generated during the evaluation stage, making it impractical to conduct a thorough examination. data splitting method presented a significant difficulty due to the considerable computational resources and time investment it demanded. consequently, the local cross-validation (cv) procedures were implemented utilizing the data in its present condition, notwithstanding the aforementioned constraints. split counts for train and validation for seen and unseen classes are presented in figure 3 . ## experiments out of distribution detection model utilizes the au-toaugment policy for preparing the svhn dataset. the model employed in the study is based on the efficientnet-b7 (tan and le, 2019) architecture, which has been pretrained on the imagenet dataset. the optimization process leverages the adamw optimizer with default parameters. the management of the learning rate scheduling is handled by the warmupandlineardecay module. the output layer is composed of layernorm-fc with dimensions 2560 to 1295, and it utilizes the bcewith-logitsloss function. the model undergoes training for a total of 200 epochs, with a batch size of 32. the dataset is divided in a 1:0 ratio, indicating the adoption of a single-fold methodology. seen class model employed in this study is based on the efficientnet-b7 architecture, which has been pretrained on the imagenet dataset. the optimization process utilizes the adamw optimizer with the default configuration. the management of the learning rate scheduling is handled by the warmu-pandlineardecay module. the output layer is comprised of a layer normalization followed by a fully connected layer with dimensions ranging from 2560 to 14784. this is then followed by the application of a softmax cross-entropy activation function. the model is trained for a total of 200 epochs using a batch size of 32. the dataset is divided randomly into a 9:1 ratio, which follows a single-fold methodology for both training and evaluation purposes. the optimization procedure in cyclegan training utilizes the adam optimizer, using a learning rate of 0.0002 and beta values of (0.5, 0.999). the implementation of learning rate scheduling involves the utilization of the lineardecay method. the training process consists of 40 epochs, where each epoch involves a batch size of 32. the training is performed on a machine configuration consisting of 4 tesla v100 gpus, and the entire training process takes approximately 2.5 days to complete. the key hyperparameters of the model consist of lambda_consistency, which is set to 10, and lambda_cls, which ranges from 1.0 to 5.0. these hyperparameters play a crucial role in determining the performance of the model. the training process plays a crucial role in attaining uniformity and efficacy in tasks related to image translation. the proposed cyclegan training module is presented in figure 1 . the leaderboard scores and submissions are assessed using a hierarchical macro-averaged recall (hmar). for each component (grapheme root, vowel diacritic, or consonant diacritic), a standard macroaveraged recall (mar) is first calculated. the grapheme root receives double the weight in the final result, which is calculated as the weighted average of those three scores. ## results and discussion in the domain of handwritten character recognition, zsl has been a prominent research focus, particularly in the context of chinese and japanese character recognition. many studies have explored the sub-categorization and identification of characters through the manipulation of constituent components. for chinese characters @xcite , which pose a considerable challenge due to their complex grapheme structure, approaches involving the classification of approximately 500 components using recurrent neural networks (rnn) series have been adopted. in contrast, for bangla characters, an attempt was made to categorize them into three component-based groups, simplifying the multi-class classification process compared to rnn-based methods. however, it became evident that this approach led to significant overfitting in zero-shot recognition, as evidenced by both private validation experiments and competition outcomes. the issue of overfitting in multi-class classification arises from the model's reliance on the entire character for predicting each class, necessitating intricate engineering to dissect the relevant features effectively. motivated by the need to address these challenges, an unconventional approach was pursued, where each character was treated as an individual class, even when data for certain classes were scarce. a fundamental assumption underlying this approach was that if there is a software running on a computer that can recognize a certain character, then that computer is capable of handling the character code and can output it as an image. this assumption was deemed valid within the context of the competition. several attempts foot_2 were made to construct models that output characters, and the most successful method among these is presented in this paper. the subsequent sections outline the proposed method and discuss the results obtained. the findings suggest that the model trained on font images acquired statistically informed, finegrained character components to efficiently discriminate among font images. furthermore, the h to f image conversion model appeared to perform the desired transformation, emphasizing recognizable components. this transformation operated on local features of handwriting, with the range of local features being statistically inferred and generalized from the font image-trained model through back-propagation. this generalization facilitated the recognition of zero-shot classes, rendering it feasible. it is worth noting that this technique's strengths extend beyond zero-shot class generalization, encompassing its universal applicability to languages and its straightforward implementation. using a dataset composed of 1168 seen classes and 127 unseen classes, the out of distribution detection model results in 0.9967 local cv area under the receiver operating characteristic (auroc). the cv score achieved for the seen class model is 0.9985, while the leaderboard score stands at 0.9874. to determine the threshold for this model, a local cv was created weighted to replicate the predicted ratio of seen/unseen classes in the leaderboard. the model's threshold was adopted when this local cv was maximized. at the initial stage, the loss calculation of the discriminator incorporated a supervised loss obtained from a font classifier that had been pretrained. the configuration that yielded the maximum performance, as indicated by a lambda_cls value of 4.0, produced the subsequent cv scores: a local cv score of 0.8804 for previously unobserved classes, and a cv score of 0.9377 for previously observed classes. it is worth noting that the calculation of the mar involved the exclusion of non-existent classes, hence preventing the assignment of recall values of either 0.0 or 1.0 to these classes. following that, the hyperparameter modification was conducted, and two further models were trained using different font data, without assessing their local cross-validation performance. these models exhibited a noteworthy achievement on the leaderboard (lb) when compared to the original model, indicating the potential for improved ability to generalize to unfamiliar classes in private data. in contrast to early hypotheses, the development of synthetic data that closely resembles real samples did not yield the anticipated enhancement in consistency and discriminator losses. furthermore, this unforeseen inclination did not yield a higher level of generalization towards classes that were not before encountered. during the process of finetuning hyperparameters, an observation was made that the model's overall performance exhibited improvement in terms of generalization. however, this improvement was accompanied by a deterioration in the visual quality of the generated images. the aforementioned observations suggest the potential existence of complex interconnections among hyperparameters, model architecture, and the ability to generalize to novel classes. consequently, it is imperative to do additional research in order to explore this matter in greater depth. hmar scores on the final lb for the top 5 performing architectures are mentioned in ## conclusion this work introduces an innovative architecture for zero-shot learning (zsl) in the context of optical character recognition (ocr), specifically for the complex bangla script. our objective was not solely to assign characters to seen classes, but also to enhance our model's ability to identify and classify classes that were unseen. by utilizing the cyclegan for image synthesis and implementing accurate label mapping techniques, a robust correlation between labels and graphemes has been developed. by actively engaging in the bengali.ai handwritten grapheme classification competition, we achieved the highest rank on the scoreboard, effectively demonstrating the exceptional capabilities of our novel model. the performance of our system in detecting out-of-distribution instances was exceptional. it achieved an area under the receiver operating characteristic (auroc) score of 0.9967 for a dataset incorporating 1168 seen classes and 127 unseen classes. additionally, the system demon-strated a cross-validation (cv) score of 0.9985 for the classes it had encountered during training, and a leaderboard score of 0.9874. and finally, it achieved a hierarchical macro-averaged recall (hmar) score of 0.9762 and topped the leaderboard amongst other contestants. through our investigation, we have uncovered the intricate relationships among hyperparameters, model architecture, and the ability to generalize to unfamiliar classes. this study represents a noteworthy achievement in the field of character recognition. it is posited that our proposed methodology possesses the capacity to fundamentally transform the field of bangla ocr development, hence expediting the process of digitizing educational materials in the indian subcontinent. furthermore, it is suggested that the prospective uses of this strategy may expand beyond the realm of character recognition. the comprehensive examination of these complex interconnections is important in order to fully realize the potential of our pioneering approach.