CCIM : Cross-modal Cross-lingual Interactive Image Translation (2023)

Abstract: Text image machine translation (TIMT) which translates source language text images into target language texts has attracted intensive attention in recent years. Although the end-to-end TIMT model directly generates target translation from encoded text image features with an efficient architecture, it lacks the recognized source language information resulting in a decrease in translation performance. In this paper, we propose a novel Cross-modal Cross-lingual Interactive Model (CCIM) to incorporate source language information by synchronously generating source language and target language results through an interactive attention mechanism between two language decoders. Extensive experimental results have shown the interactive decoder significantly outperforms end-to-end TIMT models and has faster decoding speed with smaller model size than cascade models.

## introduction text image machine translation (timt) aims at translating text in images from the source language into the target language, which has been widely used in various applications such as photo translation, scene text translation, digital document translation, and so on. existing research on timt is mainly divided into two categories of methods: cascade method and end-to-end method. cascade method @xcite @xcite @xcite takes a text image recognition (tir) model for source language text recognition @xcite @xcite @xcite and then translates them into target language texts with a machine translation (mt) model @xcite @xcite @xcite @xcite . to explicitly recognize the source language embedded in text images, the cascade model combines tir models and mt models for the timt task. however, two individual models in the cascade frame have double parameters and the decoding speed is slow. meanwhile, errors in the tir model are further propagated in the mt model leading to performance decrease. the end-to-end method directly translates the source language text image into target language through a unified encoder-decoder architecture, which is more parameter-efficient than cascade models with faster decoding speed @xcite @xcite @xcite . however, the performance of end-to-end models is limited because the translation process lacks explicit source language guidance from recognition texts. an intuitive solution is to incorporate the recognition history into the translation decoder to offer more efficient guidance. recently, multisource interaction has been studied to incorporate effective information into target model @xcite @xcite @xcite @xcite . although multi-source interaction is vital to enhance the encoding capacity of timt model through attending recognition information explicitly, it has not been explored yet. to address the above issues, we propose a novel cross-modal cross-lingual interactive model (ccim) for timt, which effectively incorporates source language recognition information into the timt decoder through interactive attention. the interactive decoder has two decoding modules, one for source language and the other one for target language generation. a cross-lingual interactive attention mechanism is introduced to bridge the two language decoders. when generating translation results, the target language decoder not only receives the hidden states from the encoder and previous decoded translation history but also attends to the ## conclusion this paper proposes a novel interactive decoder based end-to-end timt model, which explicitly incorporates recognized hidden states into the translation process. through the interactive attention mechanism, recognition and translation results are generated synchronously and mutually enhanced. by making full use of the source language recognition information, ccim outperforms existing endto-end and multi-task based timt models on both synthetic and real-world evaluation sets. furthermore, with the end-to-end architecture, ccim has fewer parameters and faster decoding speed than cascade models. ablation study of different interactive attention types shows hierarchical interactive attention has stronger interactive ability across recognition and translation tasks. in the future, we will explore more interactive methods for end-toend text image machine translation.