Multimodal Attention Network for Handwritten Mathematical Expression Recognition In this paper, we propose a novel multimodal attention network (MAN), which is based on encoderdecoder framework, for handwritten mathematical expression recognition (HMER). Here, multimodal means two specific modalities: online and offline, where online modality employs dynamic trajectories as input and offline modality employs static images as input. More specifically, the proposed method first feeds dynamic trajectories and static images into online and offline channels of the multimodal encoder respectively. The output of the encoder is then transferred to the multimodal decoder to generate a LaTeX sequence as the mathematical expression recognition result. To make full use of the complementary information that comes from the two modalities, we propose a reattention mechanism as an enhanced version of the multimodal attention mechanism which can further improve the recognition performance. Evaluated on a benchmark published by CROHME competition, the proposed approach achieves an expression recognition accuracy of 54.05% on CROHME 2014 and 50.56% on CROHME 2016 which substantially outperforms the stateofthearts using the single online or offline modality.

Residual BiRNN Based Seq2Seq Model with Transition Probability Matrix for Online Handwritten Mathematical Expression Recognition In this paper, we present a Seq2Seq model for online handwritten mathematical expression recognition (OHMER), which consists of two major parts: a residual bidirectional RNN (BiRNN) based encoder that takes handwritten traces as the input and a transition probability matrix introduced decoder that generates LaTeX notations. We employ residual connection in the BiRNN layers to improve feature extraction. Markovian transition probability matrix is introduced in decoder and longterm information can be used in each decoding step through joint probability. Furthermore, we analyze the impact of the novel encoder and transition probability matrix through several specific instances. Experimental results on the CROHME 2014 and CROHME 2016 competition tasks show that our model outperforms the previous stateoftheart single model by only using the official training dataset.