M 3 T : A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation (2024)

Abstract: Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assuming flawless extraction of text from documents along with their precise reading order. These systems also tend to disregard additional visual cues such as the document layout, deeming it irrelevant. However, real-world documents often possess intricate text layouts that defy these assumptions. Extracting information from Optical Character Recognition (OCR) or heuristic rules can result in errors, and the layout (e.g., paragraphs, headers) may convey relationships between distant sections of text. This complexity is particularly evident in widely used PDF documents, which represent information visually. This paper addresses this gap by introducing M3T a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications.

## introduction traditional machine translation (mt) systems primarily focus on textual content at the sentence level, ignoring both global context and visual layout structure of a document. given that long-range contextual dependencies are crucial for generating high quality translations @xcite @xcite , these systems fail short of achieving human translation quality when considering entire documents @xcite . several auhors have sought to address these gaps by including additional context, but these focused solely on text content (i.e. preceeding sentences) @xcite ; visual cues represent an important yet overlooked set of features which can provide contextual clues. for instance, real world documents typically include different typeface in denoting section headings to different parts of the document which indicates to human readers that parts of a document should be treated as a logically distinct. text blocks can also indicate that sections of text are contextually related and multi-columnar text is easily understood by human readers. layout elements can also indicate that certain text should not be translated (e.g. address or name fields) or that tabular data is present. disregarding layout and visual information can lead to catastrophic translation errors, as demonstrated in in recent years, there has been significant focus on vision-language (vl) architectures, which has led to notable advancements in multi-modal reasoning @xcite @xcite @xcite . specifically, recent research has dedicated considerable attention to understanding structured documents, particularly in the domain of intelligent document processing and information extraction (e.g. visual question answering) tasks @xcite @xcite . in terms of translation, earlier work has explored correcting single-word translation using visual features @xcite @xcite . with recent advances in multi-modal foundation models (fms) @xcite @xcite , that combine visual encoders with large language models (llms), development of multi-modal models to handle visually and textually challenging tasks like translation is within reach. the ability to benchmark model performance on a challenging document understanding task that takes into account long range contextual clues is of growing importance. publicly available datasets for benchmarking document-level mt mainly consist of plain text documents with sentence or paragraph level alignment. the widely used jrc-acquis consists of 8,000 documents in 20 official eu languages aligned at sentence and paragraph level @xcite . similarly, the web fiction benchmark @xcite consists of manually aligned sentence pairs that capture book and chapter-level contexts. these benchmarks address the text domain, assuming perfect segmentation of document structure (e.g. document and paragraph boundaries). in this work, we present a multi-modal benchmark dataset for evaluating mt models on translating visually rich pdf documents. our dataset is unique in that it focuses on machine translation at the document level and test models on both their ability to translate and their ability to use visual features as contextual clues. our contributions are: (1) we introduce m3t , a new challenging benchmark testset for evaluating end-to-end multi-modal mt of structured documents (figure 2 ). to further aid in research in document understanding, we have also annotated layout and reading order of the extracted text; (2) we conduct initial experiments using a multi-modal foundation model, llava-v1.5 @xcite , and find that multi-modal features improve translation quality, though there is significant room for improvement; (3) finally, we release synthetically generated parallel data to aid future model development by the community. 1 2 dataset description m3t focuses on pdf documents which are a commonly utilized format that pose several challenges for modern language models. even in digitally generated pdfs, certain artifacts, such as white characters strategically placed for spacing adjustments or the duplication of characters in the absence of bold-face options, can complicate straightforward text extraction. ## multi-modal document translation multi-modal document translation is under explored because of a lack of annotated benchmark datasets. given the creation of a novel benchmark dataset in this work, we conducted experiments testing how recent multi-modal fms perform on multi-modal document translations. to motivate future research, we experiment with a recent llava-v1.5 model @xcite , a large multi-modal model that combines a clip vision encoder vit-l @xcite and vicuna-13b @xcite for general-purpose visual and language understanding. we conduct experiments adding various visual and layout features to the model. for visually rich documents, evaluating ocr text translation directly is difficult since an alignment between extracted source segments with clean source segments needs to be found. we leave the development of such an alignment model to future work. instead, our experiments focus on whether the visual features are utilized by a multi-modal fm to improve translations by (1) recovering from ocr errors or (2) understanding the context of the document. we focused our experiments on @xmath0. to test these hypothesizes, we controlled for the effects of incorrect layout prediction by conducting experiments using the gold annotated layout elements (e.g., annotated bounding boxes). for hypothesis (1), we experimented with different granularity of visual features (at the text block level or at the document level). we used tesseract for ocr in our experiments to extract text blocks and passed a single text block at a time to the llava model. we also compared with using translations gold source text translation to understand the upper bound on translation quality. finally to test hypothesis (2), we experimented with translations using the gold text to see if less granular visual features enable a multi-modal fm to understand contextual clues better. as a baseline, we tested random images from the ms coco dataset @xcite into the llava model which allows us to measure improvements from including relevant visual features. we conducted ablations using images of identified text blocks or images of the entire document. lastly for evaluation, since the reading order of the text was annotated, we used document level comet scores foot_2 @xcite with a context size of two (i.e., two previous segments as context). results we found that including relevant visual features improved upon random visual features from the ms coco dataset. moreover, features focused on the relevant text blocks improved translation quality (as measured by doc-comet score) on both eur-lex and doclaynet subsets of data. when looking at document level visual features, we found that these universally improved translation quality beyond the baseline with random visual features. while there were improvements, visual features did not help translation appreciably. this presents an opportunity for future researchers to find methods to leverage contextual clues in the visual domain more fully. one issue was that the vicuna-13b model suffered from hallucinations especially when the inputs are short, single word, or numerical segments which is the case for many of the tabular data from both eur-lex and doclaynet documents. in the case of @xmath1, we found that the model misinterpreted the prompt and did not produce translations. ## conclusions in conclusion, we release m3t , a new benchmark for assessing multi-modal machine translation of visually-rich documents. we conducted experiments using a multi-modal fm built ontop of vicuna-13b and found that while these models attain impressive results on visual questionanswering and captioning tasks, multi-modal document translation is still an area for future research.