Fast Distributional Smoothing for Regularization in CTC Applied to Text Recognition Many recent text recognition studies achieved successful performance by applying a sequentiallabel prediction framework such as connectionist temporal classification. Meanwhile, regularization is known to be essential to avoid overfitting when training deep neural networks. Regularization techniques that allow for semisupervised learning have a greater impact than those that do not. Among widely researched singlelabel regularization techniques, virtual adversarial training (VAT) performs successfully by smoothing posterior distributions around training data points. However, VAT is almost solely applied to singlelabel prediction tasks, not to sequentiallabel prediction tasks. This is because the number of possible candidates in the label sequence exponentially increases with the sequence length, making it impractical to calculate posterior distributions and the divergence between them. Investigating this problem, we have found that there is an easily computable upper bound for divergence. Here, we propose fast distributional smoothing (FDS) as a method for drastically reducing computational costs by minimizing this upper bound. FDS allows regularization at practical computational costs in both supervised and semisupervised learning. An experiment under simple settings confirmed that upperbound minimization decreases divergence. Experiments also show that FDS improves scene text recognition performance and enhances stateoftheart regularization performance. Furthermore, experiments show that FDS enables efficient semisupervised learning in sequentiallabel prediction tasks and that it outperforms a conventional semisupervised method.

Analysis of Unsupervised Training Approaches for LSTMBased OCR In the context of historical documents, where labeled training data is especially expensive to acquire, the prospect of using unlabeled training data to improve and speed up the overall training is desirable. The most common way to use unlabeled data is unsupervised pretraining, which has been successfully applied to various CNN and RNN architectures in different domains. There is however not sufficient work for its application in the field of OCR. In this paper we investigate multiple architectures and how unlabeled data could be applied to them. We show that in combination with Connectionist Temporal Classification (CTC), a reconstruction objective has no apparent synergistic effect, with both objectives learning different representations. We therefore investigate the use of an LSTMbased Seq2Seq OCR architecture which shows promise regarding unsupervised pretraining.