  The area of automatic image caption evaluation is still undergoing intensive
research to address the needs of generating captions which can meet adequacy
and fluency requirements. Based on our past attempts at developing highly
sophisticated learning-based metrics, we have discovered that a simple cosine
similarity measure using the Mean of Word Embeddings(MOWE) of captions can
actually achieve a surprisingly high performance on unsupervised caption
evaluation. This inspires our proposed work on an effective metric WEmbSim,
which beats complex measures such as SPICE, CIDEr and WMD at system-level
correlation with human judgments. Moreover, it also achieves the best accuracy
at matching human consensus scores for caption pairs, against commonly used
unsupervised methods. Therefore, we believe that WEmbSim sets a new baseline
for any complex metric to be justified.
