Multilingual Multimodal Text Detection in I ndo- A ryan Languages (2023)

Abstract: Multi-language text detection and recognition in complex visual scenes is an essential yet challenging task. Traditional pipelines relying on optical character recognition (OCR) often fail to generalize across different languages, fonts, orientations and imaging conditions. This work proposes a novel approach using the YOLOv5 object detection model architecture for multilanguage text detection in images and videos. We curate and annotate a new dataset of over 4,000 scene text images across 4 Indian languages and use specialized data augmentation techniques to improve model robustness. Transfer learning from a base YOLOv5 model pretrained on COCO is combined with tailored optimization strategies for multi-language text detection. Our approach achieves state-of-theart performance, with over 90% accuracy on multi-language text detection across all four languages in our test set. We demonstrate the effectiveness of fine-tuning YOLOv5 for generalized multi-language text extraction across diverse fonts, scales, orientations, and visual contexts. Our approach’s high accuracy and generalizability could enable numerous applications involving multilingual text processing from imagery and video.

## introduction language identification is essential in various fields such as speech recognition @xcite , nlp, and multilingual content processing @xcite . language detection from images is a challenging task in the field of computer vision @xcite and nlp. the task is more difficult because the text may appear in different fonts, sizes, and orientations and may be accompanied by graphics or noise. deep learning models, particularly convolutional neural networks (cnns) @xcite , have shown significant progress in this area, with the state-of-the-art performance achieved by models like yolov5. different approaches for detecting multiple languages have been proposed, including rule-based, statistical, and machine-learning approaches. with the increasing volume of multilingual content available on the internet, the need for accurate and efficient language identification algorithms has become more critical. approaches such as deep learning-based (n..p and s.s., 2019) approaches have shown significant progress in language recognition tasks. machine learning-based approaches, like svm @xcite , nn and deep learning models, have been quite popular due to their ability to learn intricate features from the data and make accurate predictions. for example, the model proposed by @xcite used a cnn to extract features from text data and achieved high accuracy in language identification. similarly, @xcite used a deep learning-based approach to identify multiple languages in tweets with high accuracy. multi-language text detection and recognition in images and videos is an increasingly essential capability enabling numerous applications such as machine translation, image/video captioning, and multilingual content analysis @xcite . however, robustly identifying text regions and recognizing multiple languages within complex visual scenes remains challenging @xcite . this work proposes a novel approach for multilanguage text detection and recognition in images and videos using the state-of-the-art yolov5 object detection model @xcite . yolov5 has demonstrated tremendous success in detecting natural objects, but its application to multilanguage text in complex visual scenes remains relatively unexplored. we make the following key contributions: our approach provides a robust and generalizable framework for multi-language scene text processing in diverse visual media. the high accuracy could enable numerous applications, including automated translation, captioning, and content analysis for images and videos containing text in multiple languages. ## literature survey this literature review discusses the existing methods and techniques used for language recognition, emphasising the recent advancements in deep learning-based approaches. ## methodology this research focuses on classifying four native indian languages, including english, assamese, malayalam, and telugu, using the yolov5 model.yolov5 is an advanced object detection model that employs a single neural network to perform simultaneous predictions of bounding boxes and class probabilities for objects present in input images or videos. @xcite . the model architecture consists of a backbone network based on cspnet (cross stage partial network) @xcite , which improves feature extraction efficiency by reducing redundant computation, followed by a series of detection heads that predict the location and class of objects. the detection heads use anchor boxes, predefined boxes of different sizes, and aspect ratios that help the model accurately localize objects of varying sizes. the yolov5 algorithm performs object detection at high speed, making it well-suited for real-time applications. in this study, a pre-trained yolov5 model has been utilized to detect the text regions in input images or videos. a language recognition model was subsequently applied to classify each detected text region into one of the five targeted languages. to ensure high accuracy, the pre-trained yolov5 model has been fine-tuned on a dataset of images containing text in the targeted languages that were curated manually. for the language recognition model, a cnn-based approach has been used that has been proven to perform really good classification tasks that are quite similar to the tasks in the proposed model @xcite . data augmentation techniques, namely random rotation and horizontal flipping, have also been implemented to enhance the model's performance further @xcite . promising results have been obtained by our proposed methodology, which can have significant practical implications in various applications, such as language identification in multi-lingual environments and automatic captioning in videos containing multiple languages. for language recognition, the suggested design as shown in figure 2 consists of a number of essential elements. the dataset is first set up, then it is preprocessed and annotated, which is then the input to the model. the yolo model, which has four essential components-the backbone, neck, head, and detection-is then used for image analysis. the neck refines and integrates the basic features that the backbone extracted from the input images. the detection component recognises the languages present while the brain analyses the improved information and generates pertinent predictions. overall, this architecture makes it possible to accurately identify languages through systematic language recognition by efficiently processing and analysing images. ## experimental analysis the experimental analysis entails input data preprocessing, model training, and evaluation. the effectiveness of the language recognition system is quantitatively assessed using a variety of measures, including accuracy, precision, recall, and f1-score. additionally, in order to verify the improvements made by the suggested methodology, comparison studies with baselines and existing approaches are carried out. ## conclusion and future work in conclusion, our work proposes detecting multiple languages in images and videos using yolov5. a new dataset of over 4,000 annotated real-world images in four indian languages was created. through transfer learning and custom optimization, yolov5 achieves over 90% accuracy in detecting and identifying languages, outperforming prior methods. unlike previous ocr-reliant approaches, it flexibly extracts text irrespective of style or quality. the high performance demonstrates yolov5's capabilities for generalized multilingual text extraction, enabling impactful applications. several promising research directions exist. the model can be extended to more languages, including similar scripts, to improve disambiguation. video-specific techniques like temporal modelling can boost performance. transfer learning can enable low-resource language recognition. realworld deployment on multilingual image/video captioning and ocr is worth exploring. this work provides a robust framework for multilingual text extraction that can enable future advances.