OJ 4 OCRMT : A Large Multilingual Dataset for OCR - MT Evaluation (2025)

Abstract: We introduce OJ4OCRMT, an Optical Character Recognition (OCR) dataset for Machine Translation (MT). The dataset supports research on automatic extraction, recognition, and translation of text from document images. The Official Journal of the European Union (OJEU), is the official gazette for the EU. Tens of thousands of pages of legislative acts and regulatory notices are published annually, and parallel translations are available in each of the official languages. Due to its large size, high degree of multilinguality, and carefully produced human translations, the OJEU is a singular resource for language processing research. We have assembled a large collection of parallel pages from the OJEU and have created a dataset to support translation of document images. In this work we introduce the dataset, describe the design decisions which we undertook, and report baseline performance figures for the translation task. It is our hope that this dataset will significantly add to the comparatively few resources presently available for evaluating OCR-MT systems.

## introduction relatively few datasets exist for studying the translation of document images. the manual labor associated with obtaining suitable digital images and producing high-quality transcriptions of the source image and translations in the target language(s) is an impediment. we survey some of the available datasets in the official journal of the european union (ojeu) is available in digital form in the official languages of the eu and it contains content going back decades. the ojeu is in the public domain, and its quantity of data, high quality translations, and large number of supported languages covering three writing systems, make it an attractive source for developing a open source dataset to support translation of document images. the ojeu and related eu publications have previously been used as corpora in the development and evaluation of language technologies. for example, koehn produced parallel texts from transcripts of the european @xcite . similarly, steinberger and colleagues at the jrc have released parallel texts such as jrc-acquis @xcite and dgt-acquis @xcite , and they even foresaw the use of these collections for supporting ocr research @xcite ). 1 our present focus is in the development of corpora to support the evaluation of document image translation, which can be accomplished through pipelines of ocr and mt systems, or through use of newly available vision language models such as claude @xcite or pali gemma @xcite . in section 2 we survey datasets for this task. section 3 describes the creation of oj4ocrmt, including the design choices we undertook and key characteristics of the dataset. in section 4 we describe our experimental setup. finally, we present and discuss our baseline results in section 5. ## related work datasets for ocr-mt can be classified by the type of images used (see with the text from 60 languages in the flores dataset. 2 the advantage of rendering approaches is that any quantity of images can be synthesized. for example, @xcite rendered a 4 million pair image-translation training set based on wmt bitext. a notable disadvantage is that great effort is required if we want to match the variety of image layouts found in the real world. a second approach to ocr-mt datasets is based on collecting natural images from the wild. for example, @xcite take existing chinese street sign ocr datasets like @xcite and translate the text portion into english using professional human translators. in a similar vein, @xcite collected 770 natural photographs consisting of english in-scene text from the web and hired professional translators for translation into german, spanish, russian, and chinese. the dataset of @xcite significantly increased the scale (10 million collected images) but relied on a machine translation api to generate the translated text; it covers 14 languages and 8 scripts. a third approach focuses on collecting natural documents from the web. the distinction with the second approach is not clear-cut, but the focus here is on collecting machine-printed documents that are text-rich and sentence-like, as opposed to photographs like street signs where in-scene text may consist of short phrases. for example, @xcite collected and annotated 70k images for bounding boxes from the web, covering 35 languages and including both scanned and machineprinted documents like newspapers, books, journals, and web pages. a subset of ∼15k images covering 13 languages were transcribed: note this is a multilingual collection where images contain different languages and scripts; it must be trans-2 they also include some natural pdfs from the universal declaration of human rights database. lated to create an ocr-mt dataset. most relevant to our work are ditrans @xcite and dota @xcite , which like our work, focus on natural pdfs that are textrich documents containing a diversity of layouts. ditrans consists of political reports, newspapers, and advertisements; dota consists of scientific papers from arxiv. the test sets of both of these have been professionally translated from english to chinese. additionally, they have provided french and german translations performed by mt. our dataset is different in that we have a larger set of languages (23 in total) with translations professionally produced by the data provider and aligned in a multi-way parallel fashion. in general, these kinds of document pdfs, when converted to images, are challenging from the ocr perspective due to diverse layouts and reading orders; they are also challenging from the mt perspective due to the richer vocabulary and syntax. last but not least, there is work on translation of handwritten text, c.f. @xcite . this is a substantially different problem than scanned or born-digital machine-print documents. ## dataset an ideal dataset for ocr-mt evaluation consists of three components: (a) document images; (b) ground truth transcripts in the source languages; and, (c) human-produced translations in the target languages. we downloaded pdf files for each ojeu document in the available languages and extracted images and text for each individual page. files were obtained by crawling the eur-lex online portal 3 , the official repository for the ojeu. we decided to focus on content from recent years because previous datasets such as dgt-acquis have released translation memories that include some ojeu content, and machine translation systems are often trained using these data, which are available in the popular opus portal @xcite . as a general rule, ojeu pages in different languages contain equivalent content for a given published page. in other words, the ith page of document d in language l 1 , matches the content of the ith page of document d in language l 2 . 4 we thus have page-wise alignments, and not sentence-wise alignments which are more commonly used for machine translation. we elected to work directly with page-level alignments and not perform automated alignment of text fragments. this avoids the considerable expense required in manual determination of reading order and sentence selection and alignment. the ground truth text for each page was obtained using pdftotext. the extracted text contains blank lines and many broken up lines or text fragments. the order of the extracted content can vary by language, but usually only slightly. we used multiple newline characters as hard breaks between sections of text (i.e., paragraphs, list items), but conjoined other text fragments and then ran automated sentence boundary detection using the multilingual sentence splitter, ersatz @xcite . lossless images in png format were produced in three resolutions: 72, 150, and 300 dpi. this corresponds to the historical default resolution for web images, an intermediate value for experimentation, and the current standard high-quality resolution. due to the fact that a number of the articles were not available in the irish language, we made the decision to exclude it from the set of languages in the dataset. every page in the dev and test sets is available in the other 23 eu languages. 5 a sample image and its extracted and reconstituted text are shown in figure 1 . ## experimental results to demonstrate the utility of oj4ocrmt and to document the performance attainable by contemporary ocr-mt systems we report several experimental results. we studied: (a) translation into english (section 5.1); (b) ocr-mt performance using images of differing quality (section 5.2); and, (c) multilingual translation between any source/target language pair (section 5.3). ## conclusions we have created a new dataset, oj4ocrmt, which adds to the set of resources available for assessing the performance of document image translation systems. the dataset is large, supports 23 languages and 3 writing systems, and contains interesting visual layouts of natural documents in the government domain. we reported benchmark experiments on this translation task using two cascaded systems and one vlm-based end-to-end system. some of our findings include: (a) the vlm system (claude) generally outperformed the cascaded systems; (b) tesseract generally outperformed easyocr; (c) the ocr models performed poorly on 72dpi images; and, (d) the presence of tables or figures in images led to poorer translation quality. the dataset can be obtained from https:// huggingface.co/hltcoe .