Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition (2025)

Abstract: Handwritten text recognition aims to convert visual input into machine-readable text, and it remains challenging due to the evolving and context-dependent nature of handwriting. Character sets change over time, and character frequency distributions shift across historical periods or regions, often causing models trained on broad, heterogeneous corpora to underperform on specific subsets. To tackle this, we propose a novel loss function that incorporates the Wasserstein distance between the character frequency distribution of the predicted text and a target distribution empirically derived from training data. By penalizing divergence from expected distributions, our approach enhances both accuracy and robustness under temporal and contextual intra-dataset shifts. Furthermore, we demonstrate that character distribution alignment can also improve existing models at inference time without requiring retraining by integrating it as a scoring function in a guided decoding scheme. Experimental results across multiple datasets and architectures confirm the effectiveness of our method in boosting generalization and performance. We open source our code at https://github.com/pkaliosis/fada.

## introduction handwritten text recognition (htr) enables the automatic conversion of handwritten input into machine-readable text, supporting critical applications such as historical manuscript digitization, document analysis, and archival research. although often overlooked, systematic shifts in character frequency distributions, caused by linguistic evolution, orthographic reforms, and contextual usage, are an important consideration when building robust htr systems across diverse time periods or regions @xcite . in the transition from old english to modern english, for example, grammatical † work done during internship at archimedes research center before joining stony brook university. htr models face challenges when transcribing historical manuscripts, where orthographic conventions and character usage shift considerably over time @xcite . a common mitigation strategy involves training on diverse corpora to increase generalization, but this does not necessarily yield optimal performance @xcite . in initial experiments, we observed that models trained on heterogeneous data tend to underperform in specific subsets where character frequency distributions deviate, leading to systematic biases and suboptimal transcription accuracy, a phenomenon that to the best of knowledge has not been thoroughly explored in prior work. to address these challenges, we propose fada (frequency-aware distribution alignment), a novel training framework that aligns a model's predicted character-level frequency distributions with empirically observed ones. when such distributions are available, fada enhances robustness to temporal and contextual variation by incorporating a distance-based loss function that penalizes discrepancies between predicted and empirical character frequencies during training. this alignment loss allows the model to learn general task-relevant representations while also reflecting the expected subset-specific statistical patterns. in addition, we introduce a guided decoding algorithm that integrates the frequency-aware objective into the beam search process at inference time. by treating empirical character frequency distributions as soft constraints, our method refines predictions to better match expected patterns, improving transcription accuracy without the need for retraining. fada is the first framework to explicitly address characterlevel frequency distribution shifts in text recognition. in contrast to prior work, which largely overlooks intra-dataset variation, fada introduces both a trainable alignment mechanism and an inferencetime correction strategy, providing a comprehensive solution for mitigating distributional shifts. our main contributions are: (i) we introduce fada, a novel training framework that explicitly addresses character-level frequency distribution shifts in recognition tasks. @xcite we propose a guided decoding strategy that incorporates empirical frequency priors into beam search, enhancing alignment between predicted and expected character frequency distributions at inference time. (iii) we demonstrate the effectiveness and generality of fada through extensive experiments on htr, showing consistent performance gains across multiple datasets, languages, and model architectures. in scene text recognition (str), @xcite proposed shape-driven attention to guide models toward higher-quality character features using geometric priors. in vision-language models (vlms), distributional alignment has also been explored @xcite . for example, @xcite minimized distributional gaps between positive and negative captions to improve image-text alignment by promoting stronger visual grounding. in speech-related tasks, alignment techniques have been applied to adapt to domain-specific variation. @xcite investigated distribution alignment for multi-genre speaker recognition, showing that within-between distribution alignment (wbda) @xcite improved robustness but did not fully eliminate genre-specific variability. moreover, @xcite aligned the character-level audio representations between a source and a target domain in an unsupervised manner via maximum mean discrepancy. standard decoding strategies like greedy or beam search often lack mechanisms to enforce external constraints, which are crucial in tasks requiring adherence to linguistic structure or statistical properties. to address this, guided decoding methods steer generation based on predefined constraints @xcite , typically grouped as semantic, structural, or lexical @xcite . semantic constraints enforce stylistic or topical alignment @xcite @xcite , or integrate task-specific priors @xcite @xcite . structural approaches promote syntactic coherence @xcite , while lexical constraints guide the inclusion or exclusion of specific tokens @xcite @xcite . despite this progress, existing approaches do not explicitly address the impact of character-level frequency distribution shifts, which are particularly prominent in recognition tasks involving historical or domain-specific data. most alignment methods focus on feature-level or semantic alignment, overlooking statistical priors at the character level that can guide model predictions. fada fills this gap by introducing a frequency-aware training and decoding framework that directly incorporates characterlevel distributional knowledge into both learning and inference. ## the proposed fada method our proposed method, fada, introduces a novel training framework that aligns the predicted character-level frequency distribution of the model with an empirical target distribution at each training step, thereby reducing discrepancies caused by temporal and contextual shifts. for example, if the model predicts a character frequency of 9.3% for the letter "a", while the empirical distribution indicates an expected 11.1%, fada encourages the model to adjust its predictions to reduce this gap. to achieve this, we introduce an auxiliary loss term that penalizes divergence between the predicted and empirical character frequency distributions. this alignment loss is jointly optimized with the task-specific objective (e.g., ctc or cross-entropy loss), enabling the model to preserve recognition performance while adapting to subset-specific distributional properties. as a result, the model becomes more robust to intradataset shifts and better aligned with the statistical patterns present in the data. ## experiments we evaluate the performance of fada primarily on htr, using two model architectures across three ## discussion importance of inference-time alignment advantages over temporal/regional tagging (tr-tt baseline) contextual-conditioning strategies, such as prepending special tokens to indicate conditioning attributes (e.g., language or time) are widely used in recognition tasks @xcite . while effective, these methods control the model in an implicit way that is hard to interpret or adjust. in contrast, by explicitly aligning predicted character distributions with empirical ones, computed from the training data, fada makes the alignment process interpretable and transparent. also, it gives control to users over the output, who can modify the target distribution, encouraging (or discouraging) certain characters. ## conclusion we present fada (frequency-aware distribution alignment), a framework mitigating character relative frequency distribution shifts in text and speech recognition, consisting of two components: (i) a training-time alignment loss minimizing discrepancies between predicted and empirical character distributions; (ii) an inference-time guided decoding frequency-based algorithm that dynamically adjusts predictions. our experiments on htr and asr demonstrate consistent performance improvements across datasets, languages, writing systems, and model architectures. directions of future work include extending fada by incorporating bi-grams and tri-grams, enabling alignment at the sequence level rather than for individual characters (unigrams). also, we plan to explore case-sensitive frequency distributions, and to improve recognition of named entities and specialized terminology. finally, we aim to develop adaptive weighting strategies for λ, allowing a dynamic adjustment instead of relying on a fixed hyperparameter.