A Probabilistic Generative Model for Typographical Analysis of Early Modern Printing (2020)

Abstract: We propose a deep and interpretable probabilistic generative model to analyze glyph shapes in printed Early Modern documents. We focus on clustering extracted glyph images into underlying templates in the presence of multiple confounding sources of variance. Our approach introduces a neural editor model that first generates well-understood printing phenomena like spatial perturbations from template parameters via interpertable latent variables, and then modifies the result by generating a non-interpretable latent vector responsible for inking variations, jitter, noise from the archiving process, and other unforeseen phenomena associated with Early Modern printing. Critically, by introducing an inference network whose input is restricted to the visual residual between the observation and the interpretably-modified template, we are able to control and isolate what the vector-valued latent variable captures. We show that our approach outperforms rigid interpretable clustering baselines (c.f. Ocular) and overly-flexible deep generative models (VAE) alike on the task of completely unsupervised discovery of typefaces in mixed-fonts documents.

## introduction scholars interested in understanding details related to production and provenance of historical documents rely on methods of analysis ranging from the study of orthographic differences and stylometrics, to visual analysis of layout, font, and printed characters. recently developed tools like ocular @xcite for ocr of historical documents have helped automate and scale some textual analysis methods for tasks like compositor attribution @xcite and digitization of historical documents @xcite . however, researchers often find the need to go beyond figure 1 : we desire a generative model that can be biased to cluster according to typeface characteristics (e.g. the length of the middle arm) rather than other more visually salient sources of variation like inking. as depicted in figure 1 , we are interested in identifying clusters of subtly distinctive glyph shapes as these correspond to distinct metal stamps in the type-cases used by printers. however, other sources of variation (inking, for example, as depicted in figure 1 ) are likely to dominate conventional clustering methods. for example, powerful models like the variational autoencoder (vae) (kingma and welling, 2014) capture the more visually salient variance in inking rather than typeface, while more rigid models (e.g. the emission model of ocular @xcite ), fail to fit the data. the goal of our approach is to account for these confounding sources of variance, while isolating the variables pertinent to clustering. hence, we propose a generative clustering model that introduces a neural editing process to add expressivity, but includes interpretable latent variables that model well-understood variance in the printing process: bi-axial translation, shear, and rotation of canonical type shapes. in order to make our model controllable and prevent deep latent variables from explaining all variance in the data, we introduce a restricted inference network. by only allowing the inference network to observe the visual residual of the observation after interpretable modifications have been applied, we bias the posterior approximation on the neural editor (and thus the model itself) to capture residual sources of variance in the editor -for example, inking levels, ink bleeds, and imaging noise. this approach is related to recently introduced neural editor models for text generation @xcite . in experiments, we compare our model with rigid interpretable models (ocular) and powerful generative models (vae) at the task of unsupervised clustering subtly distinct typeface in scanned images early modern documents sourced from early english books online (eebo). ## model our model reasons about the printed appearances of a symbol (say majuscule f) in a document via a mixture model whose k components correspond to different metal stamps used by a printer for the document. during various stages of printing, random transformations result in varying printed manifestations of a metal cast on the paper. figure 2 depicts our model. we denote an observed image of the extracted character by x. we denote choice of typeface by latent variable c (the mixture component) with prior π. we represent the shape of the k-th stamp by template t k , a square matrix of parameters. we denote the interpretable latent variables corresponding to spatial adjustment of the metal stamp by λ, and the editor latent variable responsible for residual sources of variation by z. as illustrated in fig. 2 , after a cluster component @xmath0, the corresponding template t k undergoes a transformation to yield tk . this transformation occurs in two stages: first, the interpretable spatial adjustment variables (λ) produce an adjusted template ( §2.1), @xmath1, λ), and then the neural latent variable transforms the adjusted template ( §2.2), @xmath2, z). the marginal probability under our model is ## experiments we train our models on printed occurrences of 10 different uppercase character classes that scholars have found useful for bibliographic analysis @xcite because of their distinctiveness. as a preprocessing step, we ran ocular @xcite on the grayscale scanned images of historical books in eebo dataset and extracted the estimated image segments for the letters of interest. ## conclusion beyond applications to typeface clustering, the general approach we take might apply more broadly to other clustering problems, and the model we developed might be incorporated into ocr models for historical text.