  Image captioning model is a cross-modality knowledge discovery task, which
targets at automatically describing an image with an informative and coherent
sentence. To generate the captions, the previous encoder-decoder frameworks
directly forward the visual vectors to the recurrent language model, forcing
the recurrent units to generate a sentence based on the visual features.
Although these sentences are generally readable, they still suffer from the
lack of details and highlights, due to the fact that the substantial gap
between the image and text modalities is not sufficiently addressed. In this
work, we explicitly build a Modality Transition Module (MTM) to transfer visual
features into semantic representations before forwarding them to the language
model. During the training phase, the modality transition network is optimised
by the proposed modality loss, which compares the generated preliminary textual
encodings with the target sentence vectors from a pre-trained text
auto-encoder. In this way, the visual vectors are transited into the textual
subspace for more contextual and precise language generation. The novel MTM can
be incorporated into most of the existing methods. Extensive experiments have
been conducted on the MS-COCO dataset demonstrating the effectiveness of the
proposed framework, improving the performance by 3.4% comparing to the
state-of-the-arts.
