Scalable Font Reconstruction with Dual Latent Manifolds (2021)

Abstract: We propose a deep generative model that performs typography analysis and font reconstruction by learning disentangled manifolds of both font style and character shape. Our approach enables us to massively scale up the number of character types we can effectively model compared to previous methods. Specifically, we infer separate latent variables representing character and font via a pair of inference networks which take as input sets of glyphs that either all share a character type, or belong to the same font. This design allows our model to generalize to characters that were not observed during training time, an important task in light of the relative sparsity of most fonts. We also put forward a new loss, adapted from prior work that measures likelihood using an adaptive distribution in a projected space, resulting in more natural images without requiring a discriminator. We evaluate on the task of font reconstruction over various datasets representing character types of many languages, and compare favorably to modern style transfer systems according to both automatic and manually-evaluated metrics.

## introduction the majority of written natural language comes to us in the form of glyphs, visual representations of characters generally rendered in a font with a contextually appropriate style. in order to be legible a glyph must be recognizable as its corresponding character from its underlying shape, but it must also be stylistically consistent with the other glyphs in that font. while this labor intensive process is typically performed manually by human artists, the number of character types that a font may be expected to support is extremely large, with unicode 13.0.0 including as many as 143,859 character types (unicode). as a result, graphic designers often create glyphs only for a subset of these characters, which tends to be determined by their own cultural context. this can create an accessibility gap for users seeking to create or read digital content in languages with less widespread orthographies, due to the relative lack of available options. figure 1 shows that within the google fonts library @xcite there is a long tail of fonts with a large proportion of missing glyphs. font reconstruction is a task that attempts to solve this problem. the goal is for a model, given a small set of example glyphs from an incomplete font, to generate glyphs for the remaining characters in a consistent style. prior work has approached this in various ways, albeit with some limitations. while some approaches use a variational framework @xcite , generally their models only treat the font style as a latent variable, and not the character shape. such methods can therefore only handle a small, fixed, and an a priori known set of character types. other work such as that by by @xcite ; @xcite use discriminative models which dynamically compute both embeddings, allowing them to generalize to unseen characters. however these networks typically require a pre-specified number of observations as input, and their lack of a probabilistic prior can lead to learning a brittle manifold on datasets with a large number of infrequently observed characters. by contrast, our method learns two smooth manifolds over character shape and font style in order to better share parameters across structurally similar characters, letting it scale to a larger set and more effectively generalize to characters never seen during training. our model treats font reconstruction as a matrix factorization problem, where we view our corpus as a matrix with rows corresponding to character type, and columns corresponding to fonts. each row and column is assigned a latent variable that determines its structure or style respectively. a decoder network consisting of transposed convolutional layers parameterizes the model's distribution on each cell in that matrix, i.e. an image of a glyph, conditioned on the corresponding row and column embeddings. this approach can be thought of as a generalization of @xcite , who used a similar factorization framework, but with only one manifold over font style. in addition to model structure, the loss function is also important in font reconstruction as pixel independent losses like l 2 tend to produce blurry output, reflecting an averaged expectation instead of something realistic. some have used generative adversarial networks (gans) to mitigate this @xcite , but these can suffer from missing modes and collapse issues. we instead introduce a novel adaptive loss to font reconstruction that operates on a wavelet image representation, while still permitting a well formed likelihood. specifically, in this paper we make the following contributions: (1) propose the "dual manifold" model which treats both style and structure representations as latent variables (2) propose a new adaptive loss function for synthesizing glyphs, and demonstrate its improvements over more common losses (3) put forward two datasets that emphasize few-shot reconstruction, along with a preprocessing technique to remove near-duplicate fonts resulting in more challenging train/test splits. we evaluate on the task of few-shot font reconstruction, reporting the structural similarity (ssim) -a popular metric for image synthesis better correlated with human judgement than l 2 @xcite -between reconstructions and a gold reference. these experiments are further split into known characters, which the model observed in at least one font at train time, and unknown characters, which can be thought of as a few-shot task. in addition we also perform human evaluation using amazon mechanical turk. our approach outperforms various baselines including nearest neighbor, the single manifold approach we build on @xcite , and the previously mentioned discriminative model @xcite . ## related work a variety of style transfer work has focused specifically on font style, and therefore, font reconstruction. some approaches have sought to model the style aspect as a transformation on an underlying topological or stroke-based representation which must be learned for each character @xcite @xcite . however this requires characters to have consistent topologies across fonts. other work has learned a font skeleton manifold using gaussian process latent variable models @xcite . one of the more philosophically similar approaches to ours is the bilinear factorization model of @xcite which also learns vector representations for each font and character type, albeit in a non-probabilistic and linear manner. some more recent research has treated font reconstruction as a discriminative task, using modern neural architectures and techniques from the style transfer literature @xcite @xcite ## dual manifold model figure 2 depicts our model's generative process. for a corpus consisting of j fonts, each defined over up to i character types, we characterize each particular glyph image as a combination of properties relating to the style of that particular font and to the shape of that character. our model effectively factorizes the data by assigning a vector representation to every row and column which correspond to char-acter and font respectively. therefore, our approach works by leveraging the fact that all glyphs of the same character type (i.e. an entire row in our data) share the same underlying structural shape, and all glyphs within the same font (i.e. an entire column) share the same stylistic properties. by forming separate representations over each of these two axes of variation, we can reconstruct missing glyphs in our data by separately inferring the relevant row and column variables, and then pushing new combinations of those inferred variables through our generative process. this can be thought of as a form of matrix completion, where unobserved entries correspond to characters not supported by particular fonts. given a corpus x consisting of i characters across j fonts, we assign to each observed glyph x ij a pair of latent variables which model the properties of that glyph's character type and font style. specifically we define these as y @xmath0, which we draw from a standard gaussian prior n (0, i k ), with y modeling the shape of the character (e.g. a q @xmath1, and z modeling the properties of the font (e.g. times new roman or roboto light italic). given a particular y i and z j , we combine them via a neural decoder to obtain a distribution p(x ij |y i , z j ; θ) which scores the corresponding glyph image x ij . this yields the following likelihood function: both y and z are unobserved, and we must therefore infer both to train our model and pro-duce reconstructions at test time. note that by contrast, @xcite represents characters as fixed parameters, and must only perform inference over font representations. we use a pair of encoder networks to perform amortized inference, as depicted in figure 3 . ## adaptive wavelet loss our decoder architecture outputs a grid of values, but an important decision is what distribution (and therefore loss) these should finally parameterize to score actual pixels. traditional approaches using variational autoencoders have modeled each pixel as an independent normal distribution, which results in the model minimizing the l 2 loss between its output and gold. this however leads to oversmoothed images, as it treats adjacent pixels as independent despite their strong correlations (bell and sejnowski, 1997), and fails to account for the heavy-tailed distribution of oriented edges in natural images @xcite . as a result l 2 penalizes the model for generating images that are realistic but slightly transposed or otherwise not perfectly aligned with gold, which encourages models to produce fuzzy edges in order to be closer on average. gans are often employed to force sharper output @xcite , but following recent work we instead use a projected loss for a similar effect. at a high level, our approach will first project images to a feature space, and let the model's output parameterize a distribution on this projection. if that projection is invertible and volume-preserving, this is equivalent to directly parameterizing a distribution on pixels, but allows for more expressivity @xcite @xcite . ideally, such a loss requires a distribution expressive enough to capture the variable frequency characteristics of natural images, and a representation of the image that explicitly reasons about spatially-localized edges. a good example of this technique is that of @xcite , which modeled images by placing a cauchy distribution on a 2d discrete cosine transform (dct) representation of glyphs. though this is an improvement over the default choice of placing a normal distribution on individual pixels as it both decorrelates pixels and is tolerant of outliers, this approach is limited in its expressiveness and its ability to model spatially localized edges: cauchy distributions are excessively heavy-tailed and so have difficulty modeling inliers, and since dct is a global representation it does not allow the model to reason about where image gradient content is located. ## learning and inference we now describe our approach to training this model. this process mirrors that of previous variational work, although since we are learning a dual manifold, our model will require two separate inference networks. the projected loss we add (section 4) will not fundamentally affect the learning process, but does change how the reconstruction term is computed. as our model is generative, we wish to maximize the log likelihood of the training data with respect to the model parameters, which requires summing out the unobserved variables y and z. however, this integral is intractable and does not permit a closed form solution. we therefore resort to optimizing a variational approximation, a strategy which has seen success in similar settings @xcite . rather than directly optimize the likelihood (which we cannot compute the gradient of), we maximize a lower bound on it known as the evidence lower bound (elbo). we compute the elbo via a function q(y, z|x) = q(y |x) * q(z|y, x) which approximates the posterior p(z, y |x) of the distribution defined by our decoder network. figure 3 : overview of the inference procedure. first the character encoder infers a representation of structure over each row, and then the font encoder infers a representation of style conditioned on a (perhaps partially observed) column and the character embeddings. we define q(y |x) and q(z|y, x) via a pair of encoder networks which operate over one row or column of the matrix respectively. an encoder passes each glyph in that row or column through a series of convolutional layers, and then max pools the output features across all glyphs, ensuring it can handle a variable number of observations (see figure 3 ). note that the method of pooling (e.g. min, max, avg), as well as the order in which to infer y and z are important choice points that allow for different inductive biases. the pooled feature representation is then passed through an mlp which outputs parameters µ and σ to define a gaussian posterior over y i or z j . given these, we compute approximate gradients on the elbo via the reparameterization trick described by @xcite . ## experiments we evaluate on the task of font reconstruction, in which given a small random subset of glyphs from a held out font, models must reconstruct the remaining ones. we separately report performance on known (i.e. observed at least once during training) and unknown character types. during training, we mask out a randomly chosen 20% of character types to serve as unknowns. at test time, models observe examples of previously masked characters to infer their representations for reconstruction. this can be thought of as a few-shot task, where models must generate glyphs for character types they did not observe at train time based on limited test-time examples. ## conclusion in this work introduced a generative model for typography capable of reconstructing characters in a novel font, of a novel shape, or both, and demonstrated its improvements over previous approaches on two datasets containing large numbers of characters. we analyzed the results qualitatively, and inspected learned manifolds for smoothness. in future work, this methodology has potential value not just to fonts, but to any domain which can also be factored over independent axes of variation, such as handwriting by different authors. one could also incorporate this model into more complex downstream tasks such as ocr. that being said, these domains also feature complex interactions between physically adjacent glyphs (our model treats different characters within a font as conditionally independent), so some further innovation would likely still be required. there are also extensions to the model itself that might be worth exploring in future work, for instance operating on a stroke-based representation in order to perform reconstruction in the original ttf space instead of raw pixel space as we do here. this would also likely assist with smoothness of edges and reduce the incidence of "corroded"output glyphs.