Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition (2023)

Abstract: Handwritten mathematical expression recognition (HMER) is a multidisciplinary task that generates LaTeX sequences from images. Existing approaches, employing tree decoders within attention-based encoder-decoder architectures, aim to capture the hierarchical tree structure, but are limited by CFGs and pre-generated triplet data, hindering expandability and neglecting visual ambiguity challenges. This article investigates the distinctive language characteristics of LaTeX mathematical expressions, revealing two key observations: 1) the presence of explicit structural symbols, and 2) the treatment of symbols, particularly letters, as minimal units with context-dependent semantics, representing variables or constants. Rooted in these properties, we propose that language models have the potential to synchronously and complementarily provide both structural and semantic information, making them suitable for correction of HMER. To validate our proposition, we propose an architecture called Recognize and Language Fusion Network (RLFN), which integrates recognition and language features to output corrected sequences while jointly optimizing with a string decoder recognition model. Experiments show that RLFN outperforms existing state-of-the-art methods on the CROHME 2014/2016/2019 datasets.

## introduction handwritten mathematical expression recognition (hmer), a demanding subsection of optical character recognition (ocr), constitutes an interdisciplinary crossroad of computer vision, pattern recognition, and natural language processing (nlp). the unfolding of deep learning advancements has notably enhanced the effectiveness of hmer, ushering its adoption in diverse arenas, including intelligent education. nonetheless, the precision of these technologies is continuously challenged by 1 https://github.com/zui-c/rlfn inherent ambiguities in handwritten characters and the complexity of mathematical formulas. these hurdles underscore the pivotal role that nlp could play in enhancing the robustness of current visual models grappling with these issues. the encoder-decoder architecture is the prevalent method for hmer, which recasts the problem as an image-to-sequence translation task, converting a handwritten formula image into a latex markup sequence. in contrast to traditional ocr tasks, the two-dimensional structure of handwritten formulas necessitates an approach that doesn't rely on direct segmentation. since @xcite introduces a decoder using rnn with attention, subsequent work has concentrated on enhancing the accuracy of the visual attention @xcite @xcite . currently, various tree decoders and methods of syntactic analysis, such as @xcite and @xcite , are employed to focus on analyzing the expression structure and the relations of symbols. while structure-focused methods have undeniably enriched recognition model capabilities, they have also precipitated two notable challenges: 1) they rely on complex context-free grammars (cfgs), necessitating the pre-transformation of the latex markup sequence into specific tuple representations, which limits their extensibility. 2) the issue of visual ambiguity is left behind. tree decoders pay less attention to context when predicting triples, often unable to distinguish differences such as '2' and 'z'. @xcite try to employ a language model (lm) for post-correction, but @xcite underline the inherent risk of wholly depending on a lm for the correction of low-redundancy information, such as numbers, which is particularly susceptible to biases introduced by probabilistic skewing. however, as a formal language designed for mathematical structures, latex mathematical expressions possess unique language characteristics. we believe that advance structural analysis separately following normal nlp methods may not be a prerequisite to catch complicated structures of latex mathematical expressions. two key characteristics of latex mathematical expressions are: 1) they have explicit structural symbols. 2) minimal units are symbols, and they are directly assigned specific semantics. based on this, we propose that lms can proffer both structural and semantic information, making them suitable for correction of hmer. an in-depth theoretical and statistical exploration of this perspective is articulated in section 3. specifically, regarding the current limitations of structure-focused methods, we believe that: 1) the structural information can be represented by structural symbols' semantic with the context provided by mathematical notations. this circumvents the necessity for complex cfgs to generate triplet data. we substantiate our propositions through experiments on hmer correction. specifically, we deploy a math lm to rectify an hmer model reliant on unstructured-based methods, demonstrating the suitability of lm in addressing current limitations. additionally, we argue against the sole reliance on lms in a post-correction method. by leveraging information from the recognition model, we can constrict the correction space. finally, we propose our architecture called recognition and language fusion network (rlfn), which integrates recognition and language feature to output correct sequences and optimizes jointly with the recognition model. experiments show that rlfn outperforms existing state-of-theart methods and achieves expression recognition rates (exprate)s of 57.00/54.23/54.13% on the crohme 2014/2016/2019 datasets. encoder-decoder based methods are led by @xcite and @xcite . based on cnn encoder and rnn decoder, @xcite design a coarse-to-fine process, while @xcite design coverage attention to avoid over-parsing and under-parsing. @xcite use densenet @xcite as encoder and introduce a decoder with multi-scale attention. @xcite integrate left-to-right attention to simulate the progressive nature of human perception. @xcite use multi-modal attention aim to fully utilize both online and offline information. @xcite replace rnn-based decoder with a bidirectionally trained transformer, leading to enhance global coverage and parallelization capabilities. @xcite apply mutual learning to enhance bidirectional learning and design a multiscale coverage attention for longer expressions. several works focus on the tree structure of math expressions. @xcite regarded the expression as a tree represented by triples that include parent, children, and relation; then designed a tree decoder to predict each triple. based on this work, @xcite expanded prediction of symbols into attribute prediction and position prediction, then purposed a transformer-based decoder to predict triples. @xcite utilized grammar constrained attention to transform the whole image into a parse tree. @xcite added thinking attention to tree decoder, assisted by pixel-level auxiliary loss to improve recognition of complex expressions. @xcite designed a structural string representation, attempting to utilize both language model and tree structure. these structured representations are specifically designed, limiting their extensibility. ## method in an endeavor to empirically corroborate our hypotheses, we propose a novel architecture, the recognition and language fusion network (rlfn), engineered specifically to address the dual challenge of visual ambiguity and the complex structural issues that are inherent to string decoder recognition models. the rlfn is built with a string decoder recognition module, a language module that extracts language information, and a fusion module to refine the recognition output by utilizing the language information. ## conclusion in this research, we have examined the unique language characteristics intrinsic to latex mathematical expressions, with a keen focus on the minimal semantic unit and explicit structural symbols. our investigation underscores that these characteristics give hmer systems the potential to obtain both se-mantic and structural information through language models. we subsequently propose an innovative architecture that harmoniously integrates recognition and language features to yield corrected sequences. this framework eliminates the requirement to construct complex cfgs for resolving structural issues, and serves to ameliorate the challenge of visual ambiguities. this integrative approach offers fresh insights and promising theoretical groundwork for the development of hmer and related mathematical endeavors.