You’ve translated it, now what? (2022)

Abstract: Humans use document formatting to discover document and section titles, and important phrases. But when machines process a paper–especially documents OCRed from images–these cues are often invisible to downstream processes: words in footnotes or body text are treated as just as important as words in titles. It would be better for indexing and summarization tools to be guided by implicit document structure. In an ODNI-sponsored project, ARLIS looked at discovering formatting in OCRed text as a way to infer document structure. Most OCR engines output results as hOCR (an XML format), giving bounding boxes around characters. In theory, this also provides style information such as bolding and italicization, but in practice, this capability is limited. For example, the Tesseract OCR tool provides bounding boxes, but does not attempt to detect bold text (relevant to author emphasis and specialized fields in e.g. print dictionaries), and its discrimination of italicization is poor. Our project inferred font size from hOCR bounding boxes, and using that and other cues (e.g. the fact that titles tend to be short) determined which text constituted section titles; from this, a document outline can be created. We also experimented with algorithms for detecting bold text. Our best algorithm has a much improved recall and precision, although the exact numbers are font-dependent. The next step is to incorporate inferred structure into the output of machine translation. One way is to embed XML tags for inferred structure into the text extracted from the imaged document, and to either pass the strings enclosed by XML tags to the MT engine individually, or pass the tags through the MT engine without modification. This structural information can guide downstream bulk processing tasks such as summarization and search, and also enables building tables of contents for human users examining individual documents.

## introduction as you decided whether to read this paper, you probably read the title first, then maybe the abstract. you might have looked at the pictures to see if they whetted your interest, and looked at the title of this section before starting to read the section itself. you may also have noticed the bolding in the paragraph above, and thought that perhaps those were clues to what the paper is about-which of course would be right. in short, you will undoubtedly made use of the paper's formatting, both to decide whether reading a paper is worth your time, and if so, to better understand what the paper is about. now suppose you have a pdf of a paper written in a language you don't know, or an image of a document that has to be ocred, or a paper document. you extract the text and pass it through a machine translation system, and out comes translated text-and if you're lucky, the translation is both fluent and accurate. but where is the formatting that you found so helpful? ## the problem documents are often written using overt or covert markup, where the markup more or less explicitly defines the document structure. overt structural markup is used in formats like the following: other document formats, like microsoft word, use covert markup, i.e. markup that the user assigns but which then becomes more or less invisible, so that the document appears to the eye to be formatted without any markup. whether markup is overt or covert, the final document contains only a visual display which readers have come to understand: paragraphs are separated by spaces and possibly indented; qand-a lists are lists of paragraphs starting with a 'q' or 'a'; lines of poetry use a ragged right margin; section titles are short lines, often in a larger font, and possibly preceded by a section number, and text which is bolded is represented by glyphs whose strokes are thicker. in general terms, the task we are attempting is to reverse engineer the visual display of a document into an overt markup, by inferring a document's covert structure-the same thing a proficient human reader does when they read a document. we illustrate this problem with a few examples, before turning to some (in-progress) solutions. figure 1 shows a small snippet from a pdf document. 1 two terms are italicized, indicating in this particular document that the terms are being defined. but the italicization shows up neither in the plain text extracted by the tesseract ocr program (figure 2 ), nor in tesseract's more verbose hocr output(figure 3 ). @xcite figure 1 : excerpt from pdf, showing italicization and information relating to meaning. we call the information relevant to a morpheme's pronunciation its phonological representation, and we call the information relevant to its meaning its semantic representation. bold text is similarly not tagged as such in tesseract's output. @xcite the implication of this is that the fact that the original author has highlighted something as important, cannot be recovered; structural information has been lost. even more problematic is the situation where a textual document represents a highly structured database, and one wishes to reconstruct the structure of the data from the formatting. a common example of this is dictionaries, where lexical information is contained in fields delineated by the formatting. figure 4 shows an example of this, here a polish-english dictionary. @xcite notice that some of the fields within this entry are indicated by switching between bold and nonbold, such as the headword and its definition, or between the sub-entry 'z zapartym tchem' and its definition "with bated breath"; other fields are indicated by switching between an italic font and a non-italic font, such as 'bez tchu' and "out of breath." at a higher level in the document, we also wish to infer things like chapters and sections with their titles, itemized lists, sidebars, footnotes, tables and their captions, and a host of other things that make a document structure apparent to the human reader. the red text at the top highlights one of four 'judgments' in the document; a list of these judgments appears in the blue side bar box to the right. a short paragraph of plain black text introduces a bulleted list, with each list item beginning with a bolded word, and using a dark blue font. more plain black text follows the list. a header and footer appear at the top and bottom of the page. not readily visible in this image are several footnote numbers, indicated by a raised digit in a smaller font. this document is about 750 pages. if one wanted to know the committee's judgments, a search for simply the word 'judgment' would pull up both the committee's judgments and document image analysis (dia) tasks include document image classification and layout detection in images; the technology can be used to convert images of documents into structured data. early work on dia involved rule-based approaches, but deep learning (neural net) approaches are now more common. shen et al. 2021 describes layout parser, a trainable deep learning toolkit and model repository for dia (see also their section on 'related work'). they use image processing and ocr to find rectangular boxes of text (and pictures) in document images being processed and label the function of each box; it also provides pre-trained structural models for certain types of documents. but since a given document can have a very different structure from the documents in layout parser's existing models, it is desireable to fine-tune a model that is similar to the target documents, or in the worst case to train a new model from scratch. in either case, a set of hand-labeled documents is required for the tuning or training. clausner, pletschacher, and antonacopoulos 2011 describes the university of salford's aletheia, a dia system including tools for annotating document structure with the assistance of various semi-automated mechanisms. the university also maintains a dataset of annotated documents at https://www.primaresearch.org/dataset/ . in the next section, we describe on-going work in the evaluation and use of dia tools (including the layout parser), while section 5 discusses experiments in improving the detection of bold text. ## general format detection we experimented with the layout parser on various documents, including the nscai report (a partial page of which was shown in figure 5 ). our goal was to recognize significant text boxes, join boxes that were broken over page or column boundaries, and if possible label the boxes as to type (body text, footers and headers, section titles, etc.). we selected a pre-trained model that worked reasonably well with our document's format, but even this best model showed some spurious or overlapping boxes in the output. however, we were able to eliminate most of these spurious boxes without losing valid boxes by ignoring boxes to which the program assigned a low confidence; this in turn eliminated most overlapping boxes. in addition, boxes frequently clipped off pieces of characters at the edges, but we addressed this by adding a small padding to increase the size of each box. the model classified regions as (body) text, titles, lists, tables or figures; it was however confused by sidebar elements (like the one shown in the upper right-hand side of figure 5 ). we experimented with the detection of titles by layout parser's publaynet dataset and its corresponding model (faster rcnn r 50 fpn 3x) foot_3 by running this over a set of test samples with 15 images. the aim of this experiment was to estimate the quality of the model performance, including a raw count of three categories: figure 6 shows some examples from the test set. notice that the raw number of incorrect bold text segments is relatively small compared to the number of bold text segments that are correctly classified. however the number of incorrectly classified non-bold texts is quite high, and another method should be used to lower this number. this model could perhaps be combined with other models or methods to decrease the number of incorrect non-bold text tags (ensemble models), but we did not have time to test this. ## the special problem of bold font detection as mentioned, tesseract does not currently attempt to distinguish between bold and non-bold (or italic) text. since bolding is often used to emphasize important words, as well as distinguishing between fields in semi-structured data (like dictionary entries), we experimented with figure 6 : yellow boxes are marks for titles assigned by the layout parser (lp). picture a shows two non-bold text segments that lp correctly tagged as titles. picture b shows bold text that lp correctly boxed as titles (although it missed some bold text continued on to the next line). picture c shows short bold texts at the beginning of the paragraphs that lp did not tag as titles, and red but non-bold text that is incorrectly tagged as a title. methods for detecting bolded words, using for test data a tzeltal-spanish dictionary (cruz, gerdel, and slocum 1999) and a cubeo-spanish dictionary (morse and m. b. maxwell 1999), as well as the previously mentioned nscai document. the initial attempt used unsupervised clustering based on features such as bounding box dimensions, pixel counts and scaling by letter shape, but this unsupervised approach did not work well. the best performing method used the opencv python library, an open source library for image processing ( https://github.com/opencv/opencv-python ). we first converted the color image to gray scale, then 'thresholded' it to convert the gray scale into a black-and-white image (where gray turned into white), and finally used cv2's dilation() function to partially erode areas with black pixels. this had the effect of removing most of the pixels for non-bold characters, while leaving enough pixels in bold characters to enable approximate ocr of the bolded words. the result of applying this to the page shown in figure 5 is illustrated in figure 7 ; running this image through ocr gives the text shown in figure 8 . clearly the ocr output from this degraded image is not good enough for downstream use by itself, however by using the bounding boxes in the hocr output (not shown here), it was possible to pick out the spans in the original hocr output that contained bold text in the original image. the cv2 dilation() function has several adjustable parameters. the best settings are doubtless dependent on a number of factors (including the particular font and its size). for our experiments, we set these by hand, but for best results the parameter settings should be adjusted for particular documents or document classes based on a sampling of the outputs. nscai judgments regarding al-enabled and autonornous weapon systems listing tign preaperticnvality . be at figure 8: excerpt of plain text ocr output from figure 7 6 passing formatting through machine translation the output of dia has several potential uses: if search (1) or summarization (2) is done in the original document's language, and results are passed through machine translation (mt), then the inferred formatting tagging (likely in xml or json) can be stripped before passing the outputs of these processes through mt. however, if search or summarization is instead done on the mt output, or if the mt output is to be formatted according to the input (3) foot_4 , then the formatting needs to be passed transparently through the mt process. how this would be done depends on the particular mt engine; one method would be to train the mt system to ignore such text. much the same issue arises if named entity tagging is done in the source language, and must be transferred into the target language (cf. hermjakob, knight, and daumé iii 2008 for some observations on this problem, albeit in the context of statistical rather than neural net mt). another method would be to only send blocks of text that are entirely inside a discovered box through translation, and to assemble the translated document using the boxes and their translated contents. ## future work clearly much remains to be done before document image analysis can be considered plug-andplay. too much currently requires manual control: parameter setting (e.g. for the detection of bolding or the choice of confidence-based cutoffs in the boxing of document chunks), choosing pre-trained models, or deciding whether to use an existing pre-trained model as-is or to tune it. we did not attempt to detect italicization (or underlining, which did not appear in the documents we experimented with). font size detection is also relevant, e.g. for detecting section headers (which are often in a larger font) or captions or footnotes (often in a smaller font). font size can probably be inferred from bounding boxes on ocred text, provided one pays attention to capitalization, and ascenders and descenders. finally, some documents may be one-of-a-kind. figure 9 shows a page from a hand-written arabic book on islamic rulings. @xcite the page header includes a 'ruling' title on the left, a page number in the middle, and the 'book' (chapter) title on the right. below the header is the section title, in red. a horizontal rule about a third of the way down the page separates the original text above from the commentary below. a red font has been used in the commentary to refer to words in bold characters above, as indicated by the orange lines i have drawn in. the words in the commentary circled in green are citations to sources from which the commentator drew, like 'muraqi al-falah: page 699'. the words circled in brown are someone's explanation of the words they appear immediately under. whether dia is relevant for such unusual documents probably depends on the size and usefulness of the document in question, and in this case on whether ocr is even possible on this hand-written arabic document. in sum, while tools like the layout parser require substantial automated training material, for long documents (such as the nscai document) or for many documents of a single type (e.g. medical reports or other standardized documents), automating the document image analysis (dia) process would pay a return on investment by making it easier for users to make sense of machine translation output, enabling parsing of structured data such as dictionaries, and informing search by highlighting "hits" that occur in regions of higher importance. 10 proceedings of the 15th biennial conference of the association for machine translation in the americas orlando, usa, september 12-16, 2022. volume 2: users and providers track and government track