Qalam: A Multimodal LLM for A rabic Optical Character and Handwriting Recognition (2024)

Abstract: Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR) pose unique challenges due to the cursive and context-sensitive nature of the Arabic script. This study introduces ***Qalam***, a novel foundation model designed for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder architecture. Our model significantly outperforms existing methods, achieving a Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We train ***Qalam*** on a diverse dataset, including over 4.5 million images from Arabic manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably, ***Qalam*** demonstrates exceptional handling of Arabic diacritics, a critical feature in Arabic scripts. Furthermore, it shows a remarkable ability to process high-resolution inputs, addressing a common limitation in current OCR systems. These advancements underscore ***Qalam***’s potential as a leading solution for Arabic script recognition, offering a significant leap in accuracy and efficiency.

## introduction optical character recognition (ocr) technology has revolutionized the way we interact with written and printed materials. it enables the conversion of various documents, including scanned paper documents, pdf files, or images captured by a digital camera, into editable and searchable data. the ability of ocr to digitize text has found applications in numerous domains, ranging from banking and healthcare to education and historical research, among others @xcite . in this work, our focus is on handling arabic ocr and hwr. arabic ocr and hwr pose substantial challenges due to several distinctive features of the arabic script. the arabic writing system is cursive and context-dependent, a characteristic that complicates the design of robust ocr models. further intricacies such as diacritical marks, loops, overlaps, ligatures, dots, and multi-context joining contribute to the complexity of the task. in addition to these structural challenges, the vast diversity of arabic fonts and individual handwriting styles further complicate character recognition and segmentation tasks. finally, the lack of comprehensive and diverse annotated datasets increases the difficulty of arabic ocr and hwr tasks. figure 2 visually represents these challenges. ocr systems can be classified based on the type of input data they process: handwritten or printed. in this paper, for clarity, we refer to printed text recognition specifically as ocr, although both types broadly fall under the ocr umbrella. our objective in this work is to tackle these complex tasks, making several key contributions that collectively underscore the novelty of our research. concretely, our contributions can be summarized as follows: the rest of this paper is structured as follows: section 2 discusses the related work in hwr and ocr. section 3 introduces midad , our arabic hwr and ocr benchmark. in section 4, we present our methodology. section 5 describes the experiments conducted in this work. section 6 introduces qalam, the proposed model for arabic hwr and ocr, and evaluates qalam. we conclude in section 8 and spell out many limitations related to our work in section 9. ## related works traditional hidden markov model (hmm) @xcite @xcite approaches in sequence modeling have been largely surpassed by deep learning techniques that do not require explicit segmentation. connectionist temporal classification (ctc) models @xcite and encoder-decoder models @xcite with attention mechanisms @xcite represent two primary deep learning categories. recent advances include transformer models @xcite and pre-trained models @xcite . ## methods in our pursuit to address the challenges of both arabic hwr and ocr, we employ a vision encoder-decoder (ved) framework that ingeniously brings together transformer-based models in a novel manner. this framework leverages the power of transformer-based vision models as encoders, adeptly processing image data, and pairs it with the linguistic sophistication of transformerbased language models as decoders. the result is a synergistic pairing that skillfully transcodes visual information into meaningful textual output, thus overcoming the intricate complexities of arabic ocr. furthermore, we extensively analyze various encoder and decoder combinations, investigating their implications for model performance. in ved design, the encoder ingests image data, while the decoder manages ground-truth caption inputs via teacher forcing @xcite during training. we ensure autoregressive training for the next token prediction using causal self-attention in the model. this mechanism restricts a token's attention only to its predecessors, maintaining the sequential nature of the input text. ## experiments we present our experimental settings, including the selection process for the encoder and decoder from various options. we also discuss the challenges encountered during their selection. ## building qalam to further optimize the performance of our vision encoder-decoder framework, we introduce additional pre-training strategies for the encoder and the decoder, including the swinv2 and roberta models. this is undertaken to capitalize on their specific architectural strengths in handling large image sizes and next-token prediction tasks. encoder upgrades. to enhance the model's performance, we further pre-train the swin transformer v2 encoder. we choose this model due to its proficiency in handling high-resolution inputs and capturing rich spatial information. the training strategy involved augmenting the input image size and utilizing a robust dataset comprising 4.5m images extracted from arabic manuscripts and books. the masked language modeling approach is employed during training, promoting the encoder's adaptability to real-world ocr challenges and the complexities inherent in arabic script. decoder upgrades. on the decoder side, we further pre-train the the roberta model using the masked language modeling approach @xcite . we select roberta as the decoder for qalam for its superior performance in nexttoken prediction tasks. a substantial and diverse training dataset provided the model with various language patterns, including arabic wikipedia and arac4 @xcite . the model was also modified to handle longer sequences during training, improving its comprehension and generation of complex sentence structures. we used a sentence-piece tokenizer that also can understand diacritics, which is crucial for processing the arabic language, thereby enhancing the decoder's effectiveness in arabic ocr tasks. figure 7 shows our final architecture. synthetic data. this data comprises 60k imagetext pairs and features more than 28 diverse fonts, comprehensively representing potential inputs. qalam is subsequently fine-tuned using this synthetic data in a supervised manner. figure 10 shows a few examples of this dataset. we generate this dataset using text files from hindawi arabic books @xcite . we subsequently augment it into different fonts and organize it into pdf files of size 760x640. the model was then fine-tuned on this dataset to teach it about the arabic script. ## discussion the exemplary performance of qalam across arabic and persian ocr and hwr tasks ( the stark disparity between qalam and trocr, despite trocr's strength, underscores the limitation of models pretrained on english data when applied to different scripts, emphasizing the need for script-specific training. the results also highlight the limitations of task-specific models like tesseract, which excels in line-based recognition but underperforms in character or word-based tasks. finally, the exceptional performance of qalam can be attributed to the synergy between swinv2 encoder and roberta decoder, effectively tackling ocr and hwr complexities. examples of the demo can be found in the appendix a.6. ## conclusion we introduced qalam, a foundation model for arabic ocr and hwr. qalam establishes a new standard in arabic ocr and hwr tasks. the robustness of its architecture, comprising the swinv2 encoder and roberta decoder, outperforms previous state-of-the-art systems. our study demonstrates that arabic script's unique challenges can be effectively addressed by leveraging the strengths of transformer-based models. the performance of qalam on the midad benchmark validates the scalability and flexibility of our approach, suggesting its potential application to ocr and hwr tasks in other complex scripts. moving forward, qalam offers a compelling basis for further innovation in arabic ocr and hwr systems, contributing to the advancement of this critical area of research.