  Medical image captioning automatically generates a medical description to
describe the content of a given medical image. A traditional medical image
captioning model creates a medical description only based on a single medical
image input. Hence, an abstract medical description or concept is hard to be
generated based on the traditional approach. Such a method limits the
effectiveness of medical image captioning. Multi-modal medical image captioning
is one of the approaches utilized to address this problem. In multi-modal
medical image captioning, textual input, e.g., expert-defined keywords, is
considered as one of the main drivers of medical description generation. Thus,
encoding the textual input and the medical image effectively are both important
for the task of multi-modal medical image captioning. In this work, a new
end-to-end deep multi-modal medical image captioning model is proposed.
Contextualized keyword representations, textual feature reinforcement, and
masked self-attention are used to develop the proposed approach. Based on the
evaluation of the existing multi-modal medical image captioning dataset,
experimental results show that the proposed model is effective with the
increase of +53.2% in BLEU-avg and +18.6% in CIDEr, compared with the
state-of-the-art method.
