Mutual Information Divergence: A Uniﬁed Metric for Multimodal Generative Models
Abstract
Text-to-image generation and image captioning are recently emerged as a new experimental paradigm to assess machine intelligence.
They predict continuous quantity accompanied by their sampling techniques in the generation, making evaluation complicated and intractable to get marginal distributions.
Based on a recent trend that multimodal generative evaluations exploit a vison-and-language pre-trained model, we propose the negative Gaussian cross-mutual information using the CLIP features as a uniﬁed metric, coined by Mutual Information Divergence (MID).
To validate, we extensively compare it with competing metrics using carefully-generated or human-annotated judgments in text-to-image generation and image captioning tasks.
The proposed MID signiﬁcantly outperforms the competitive methods by having consistency across benchmarks, sample parsimony, and robustness toward the exploited CLIP model.
We look forward to seeing the underrepresented implications of the Gaussian cross-mutual information in multimodal representation learning and future works based on this novel proposition.
The code is available at https://github.com/naver-ai/mid.metric .
Introduction
A multimodal generative model, including text-to-image generation [1] and image captioning [2] models, is an emerging research topic showing interpretative multimodal understanding, text or image retrieval, machine creativity, etc .
The gist of learning multimodal generative models is to understand how to connect one modality to the other and generate the corresponding representations following the desired data distribution.
However, measuring the distance or divergence between the model and data distributions is generally intractable due to the ﬁnite data and generation cost.
Therefore, the proposed metrics attempt to approximate it with polynomial-sized data and generated samples [3].
For text-to-image generation, the widely-used metrics are Inception Score (IS) [4] and Fréchet
Inception Distance (FID) [5].
These metrics are originally proposed for non-conditional generative models, which are repurposed to measure the distance between the data and model conditional distributions using a validation split.
This idea was supported by the effectiveness of deep features as a perceptual metric [6], although these metrics use the Inception V3 [7].
Not surprisingly, there are attempts to develop more robust metrics using multimodal pre-trained models, such as object
C o rr e l a t i o n
[Table]
Reference Usage
[Table]
Reference
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
r e q u e n c y
[Figure]
S p e n t
T i m e
[Figure]
N o r m a l
i z e d
S c o r e - - -
[Figure]
ViT-B/32 ViT-B/16 ViT-L
[Figure]
CLIP-R-
[Figure]
Caption - - -
RN101 ViT-B/32 ViT-B/16 ViT-L/ - - - RN101 ViT-B/32 ViT-B/16 ViT-L/
[Figure]
VQ-Diffusion
DM-GAN
[Figure]
detectors [8], image captioning models [9], and vision-and-language pre-trained models [10, 11].
Signiﬁcantly, the image captioning method transforms text-to-image measurement into image-to-text measurement, providing a different viewpoint with cyclic consistency.
For image-to-text generation, or image captioning, the COCO Caption Evaluation toolkit seems to be the standard to measure the divergence from ground-truth captions having BLEU [12], METEOR [13], ROUGE [14], CIDEr [15], and SPICE [16]. Similar to the IS and FID of text-to-image generation metrics, these metrics are merely n-gram-based statistical methods neglecting conditional images. The breakthrough in improving the correlation with human judgment comes from the utilization of the pre-trained vision-and-language models, e.g. , TIGEr [17], ViLBERTScore-F [18], and RefCLIPS [19].
We speculate that these two directional metrics are getting closer to measuring the generative divergence of text-image alignment.
This paper proposes a uniﬁed metric for multimodal generative models.
In probability theory and information theory, mutual information (MI) measures how much one random variable tells us about the other.
In a multimodal generation, the MI of two modalities quantitatively measures how much the generated is well-aligned with the condition.
From this motive, we propose to use the Gaussian mutual information where the probability distributions are deﬁned by the means and covariances of visual and textual features and borrow the idea of cross-mutual information [20] to measure the MI divergence from the real data distribution, which is the expectation of point-wise mutual information with respect to evaluating samples.
Surprisingly, the proposed method outperforms previous works with signiﬁcant margins on the assorted benchmarks of text-to-image generation and image captioning, including standard human judgment correlation benchmarks.
Section 2 introduces the previous works on text-to-image generation and image captioning metrics, and the work on cross-mutual information.
Section 3 describes the proposed method deﬁning the continuous mutual information using multivariate Gaussian distributions and the negative crossmutual information, which is the Mutual Information Divergence (MID) that we term.
Sectionconsists of two parts, evaluation on text-to-image generation and image captioning evaluation, including related discussions.
Section 6 concludes the work with remarks.
We summarize our contributions as follows:
• To the best of our knowledge, we ﬁrstly propose the negative cross-mutual information under the Gaussian assumption as a uniﬁed metric for multimodal generative models.
• We provide three theoretical analyses on the proposed method MID, out-of-distribution detecting by the squared Mahalanobis distances, bias and variance decomposition, and its relation to the Kullback-Leibler divergence.
• We achieve the state-of-the-art in text-to-image generation and image captioning benchmarks including the generated and human Likert-scale judgment correlations, visual reasoning accuracy, Flickr8K-Expert, Flickr8K-CF, Pascal-50S, and FOIL hallucination detection.
2.1 Metrics for assessing text-to-image generation
Traditional metrics.
One of the widely used metrics is the Fréchet Inception Distance (FID) [5] that measures the distributional difference between synthetic (fake) and real-world images (real).
Training and validation distributions are independent, making a model that fails to match the conditional distributions if it does not reﬂect the given textual information.
Although it measures ﬁdelity along with Inception Score [4], it cannot directly measure the alignment of text and image.
The alternative metrics [21–23] are proposed to evaluate the ﬁdelity and diversity.
Text-to-image metrics.
Dedicated to assessing text-to-image generation, the R-Precision exploits the Deep Attentional Multimodal Similarity Model (DAMSM) [24] to calculate the top-1 retrieval accuracy from one hundred text candidates for the generated image as a query.
Besides, the CLIP R-Precision [11] exploits the CLIP [10] showing a better retrieval performance and human judgment correlation.
However, falsenegative candidates (accidentally correlated) or strong negative candidates (totally unrelated) may interfere with the accurate assessment [25].
To evaluate the quality of individual objects, the SOA [8] attempts to measure the object detection accuracy using the YOLOv3 [26] based on the object classes that appeared in the text but cannot consider other factors.
Caption generation [9] is another approach using the vision-and-language pre-trained model.
The motivation is the cyclic consistency that the generated caption from the generated image should match with the text for image generation.
However, the model bias including object hallucination [27] and the accumulated errors from metrics are drawbacks.
Diagnostic datasets.
Park et al.
[11] provide the curated splits of the CUB [28, 29] and Flowers [30] to assess unseen color and shape compositions in the narrow domains.
DALLEval [31] proposed a diagnostic dataset PaintSkills to evaluate visual reasoning skills to assess models based on this dataset.
Since the dataset is generated from a 3D simulator using limited conﬁgurations, the data distributions deviate from other real-world datasets.
2.2 Metrics for assessing image captioning
Reference-only metrics.
Borrowing from machine translation literature, image captioning models are evaluated using reference-based textual metrics where ﬁve references are usually used.
BLEU-4 [12], ROUGE-L [14], and METEOR [13] are n-gram precision or recall-based metrics, CIDEr [15] uses tf-idf weighting and stemming, while SPICE [16] uses semantic parsing and scene graph analysis.
Notably, BERT-S++ [32] considers inter-reference variance using the ﬁne-tuned BERTScore [33] for image captioning.
Reference-with-image metrics.
The recently proposed metrics are considering the images used for generating captions.
TIGEr [17] uses a pre-trained SCAN [34] while ViLBERTScoreF [18] uses a pre-trained ViLBERT [35] exploiting the vision-and-language alignments from largescale data and multiple tasks.
Similarly, CLIP-S and RefCLIP-S [19] use the pre-trained CLIP, a more powerful vision-and-language model, outperforming the previous methods.
Implications.
CLIP R-Precision for text-to-image generation and RefCLIP-S for image captioning share the same motivation exploiting the same vision-and-language pre-trained model.
Here, we remark on the unifying metrics in two different tasks ( e.g. , CLIP-S), text-to-image generation and image captioning, and propose a new uniﬁed metric for the multimodal generative models based on the continuous mutual information considering the covariances of two modality groups. 2.3 Metrics for assessing other tasks
One of the other applications worth mentioning is text-based motion generation tasks [36].
Since traditional metrics, e.g. , average position error or average variance error, merely rely on numerical differences instead of high-level semantics, they proposed to use a motion-text alignment pre-trained model (mCLIP), enabling us to measure CLIP-S and our proposed MID.
It validates our speculation that the uniﬁed metric can effectively apply beyond vision and language.
2.4 Cross-mutual information in machine translation
Bugliarello et al.
[20] proposed the cross-mutual information (XMI) as a metric of machine translation exploiting a probabilistic view in neural machine translation models.
XMI is an analogue of mutual information for cross-entropy deﬁned as XMI ( S !
T )
=
H q LM ( T )  H q MT ( T | S ) .
H q LM ( T ) denotes the cross-entropy of the target sentence T under a language model q LM and H q MT ( T | S ) is the cross- conditional entropy under a cross-lingual model q MT .
In practice, they exploit two model distributions q LM ( t ) and q MT ( t | s ) to approximate the XMI as follows:
[Equation]
where N denotes the number of held-out evaluating samples.
Since language models consider a ﬁnite size of vocabulary, the cross-entropy can be efﬁciently approximated using the target sentences; however, it is limited to readily applying to other generative models.
Method 3.1 Continuous mutual information
We introduce a uniﬁed metric for conditional generative models not depending on the modalities of condition and generation .
To measure the alignment of condition and generation, we ﬁrst consider the continuous mutual information of the condition x and generation y as follows:
[Equation]
where the probability and joint probability distributions are multivariate Gaussian, which is the maximum entropy distribution for the given mean µ and covariance ⌃ [37].
The ﬁrst two moments are used for practical reason [5].
The multivariate Gaussian distribution is deﬁned as:
[Equation]
where D is the dimension of x .
The mutual information with the Gaussian distributions is reduced to:
[Equation]
where ⌃ x and ⌃ y 2 R D ⇥ D are the covariances of the condition and generation and ⌃ z 2 R 2 D ⇥ 2 D is the covariance matrix of the concatenation z of x and y representing the joint distribution.
The proof can be found in Appendix A.1.
Note that we use log det( ⌃ ) =
P i log  i for numerical stability, where  i is the eigenvalue of ⌃ .
For high-dimensional features, we consider two encoders f x and f y to get ˜ x and ˜ y , respectively, maximizing the mutual information of the encoded representations, I ( ˜ X ; ˜ Y ) .
Speciﬁcally, the image and text encoders of the CLIP [10] are used since these are pre-trained on 400M image-text pairs using the InfoNCE loss, maximizing a lower bound on mutual information [38].
Without loss of generality, we use x , y instead of ˜ x , ˜ y to denote the feature vectors to calculate the moments.
3.2 Point-wise mutual information for pair-wise evaluation
Based on the previous continuous mutual information, we derive the point-wise mutual information (PMI) for pair-wise evaluation.
Please see Appendix A.1 for the detail.
The PMI is deﬁned as:
[Equation]
where D 2 M denotes the squared Mahalanobis distance (SMD), D 2 M ( x ) =
( x  µ x ) | ⌃  1 x ( x  µ x ) where µ x and ⌃ x are the mean and covariance of x , and similarly for D 2 M ( y ) and D 2 M ( z ) .
z denotes [ x ; y ] .
The MI is from the normalization of the Gaussians and the SMDs are from the exponential of the Gaussians.
The second term measures the difference between the distances, D 2 M ( x ) + D 2 M ( y ) and D 2 M ( z ) , assessing the deviation from the MI.
Notice that the expectation of the second term with respect to the sample distribution is zero (see Appendix A.1). [Table]
3.3 Mutual Information Divergence: the expectation of PMI w.r.t evaluating samples
We propose to use the expectation of PMI with respect to the evaluating sample (ˆ x , ˆ y ) , measuring the divergence from the ground-truth or reference samples ( X , Y ) .
The metric is deﬁned as follows:
[Equation]
where D and (ˆ x , ˆ y ) denote the set of evaluating samples and a pair of evaluating sample, respectively, ˆ z denotes [ˆ x ; ˆ y ] .
Notice that the expectation of D 2 M (ˆ x ) can be decomposed to the bias and variance terms as follows (Appendix A.2 for the proof):
[Equation]
considering the mean and covariance deviations from the reference, N ( µ x , ⌃ x ) .
E ˆ y
[Equation]
[Equation]
⇤ is D when ˆ y =
y as a generative condition since the two moments are equal to the counterparts.
By the way, we can show that E (ˆ x , ˆ y )
⇠ D PMI (ˆ x ; ˆ y ) is related to the Kullback–Leibler divergence as follows (The proof can be found in Appendix A.3):
[Equation]
For simplicity, we denote our proposed method E (ˆ x , ˆ y )
⇠ D PMI (ˆ x ; ˆ y ) as Mutual Information Divergence (MID) , comparable of the FID.
In practice, we use Equation 5 using the double-precision
CLIP features.
For point-wise evaluation, we use the PMI (ˆ x , ˆ y ) without the expectation.
Experiment 4.1 Evaluation on text-to-image generation
Implementation details.
Without an explicit mention, we use the CLIP (ViT-L/14) to extract image and text embedding vectors.
Note that it is crucial to use double-precision for numerical stability.
Generated Likert-scale judgments.
To carefully assess the text-image alignment, we consider the four-scale alignment using the real and fake images from the COCO dataset [2].
We regard the real images as a four-point set, the fake images generated by the ground-truth captions as a three-point set, the fake images generated by the foiled captions 2 [40] as a two-point set, and the randomly sampled (misaligned) fake images as one-point set.
We assume that the fake images generated by the foiled captions should be inferior compared with the fake images generated by the ground-truth captions because the model cannot exploit the critical information to generate key objects.
The current state-of-the-art LAFITE [41] pre-trained on the COCO 3 is used for our text-to-image generation https:
[Table]
l
[Table] [Table]
Expert [Table]
[Figure]
M u t u a l I n f o r m a t
i o n[-0 10 20 30 40 
-]
Text-Image Shuffling Ratio 0.
CUBCOCOLN-COCOMM-CelebA-HQ C L I P - S 0.
Text-Image Shuffling Ratio 0.
[Figure]
[Figure] [Figure] F r e q u e n c y
[Figure]
We randomly sample 30K captions from the FOIL dataset [40] to build 120K judgments. We report the Kendall’s ⌧ coefﬁcient [42] to measure the rank correlation, a variant of ⌧ c or ⌧ b accounting for ties.
In Table 1, our method consistently outperforms competing methods.
InfoNCE denotes the negative InfoNCE loss [38] calculating the softmax function over the 30K captions.
Remind that the InfoNCE maximizes a lower bound on mutual information [38].
Since this is estimated using a smaller batch size when optimizing, it shows a limited capability as a metric for the text-image alignment.
Please refer to Table 8 and Figure 8 in Appendix for the VQ-Diffusion [43] benchmark.
Figure 2 (top left) shows the histogram of the frequencies of PMI showing the four categories.
Fake samples have generally lower values than Real ’s. Foiled Fake has lower values than Real and
Fake having a long tail.
We observe that foiled caption broadly impacts the text-image alignment as we expected.
The negative PMI is often observed for the fake images, which are deviated from the distribution of real images.
Whereas, the histogram of CLIP-S 4 (bottom right) and InfoNCE (bottom left) show Fake samples where its scores are higher than Real ’s, while the overlapping areas of Fake and Foiled Fake (62.2% and 61.6% for CLIP-S and InfoNCE, respectively) are greater than PMI’s (55.2%), making it difﬁcult to differentiate the degree of text-image alignment.
For the caption generation method [9], we exploit the current state-of-the-art image captioning model of the
OFA-Large [39] pre-trained on a huge mixture of publicly available datasets to generate captions for [Table] CLIP ViT [Table]
CLIP ViT
[Table]
L/ [Table] the generated images. Then, CLIP-S [19] is used to assess the quality of image captioning.
Notice that CLIP-S outperforms traditional image captioning metrics as shown in Section 4.2.
Visual reasoning accuracy using the foiled caption trick.
Inspired by the foiled caption trick, we extend to four visual reasoning skills for object, count, color, and spatial relationship.
For each category, we deﬁne a set of tokens, and we foiled those tokens in the caption by randomly swapping to the other token.
We build three sets of images, the real images, the fake images, and the foiled fake images.
We measure the accuracy that getting one point for the foiled fake images having the lowest score, or zero for the other cases.
Table 3 shows that our method achieved the best performance across all categories.
The runner-up was InfoNCE, while SOA was ineffective to differentiate among the two fake images and the real image.
Although DALL-Eval [31] proposed to use a detector and its dedicated heads for the count, color, and spatial relationship tasks, this was limited to the 3D-generated images with their near-perfect detection ability.
Remind that the accuracy of random guessing is 33.3%, where MID requires a powerful feature extractor of the CLIP ViT-L/14 to get meaningful performances on the count, color, and spatial relationship tasks.
For the detail, please refer to the text in Appendix C.
Human Likert-scale judgment.
We collect 10K one-to-four Likert-scale human judgments for 2K fake images using the LAFITE and VQ-Diffusion from the Amazon Mechanical Turk (AMT) for the ﬁne-grained comparison with the competing metrics.
For each image, we collect ﬁve annotations from unique workers taking its median for a reliable correlation measurement.
Table 2 shows the consistent results with our generated judgment correlation benchmark.
Since this benchmark aims for the ﬁne-grained judgment among only fake images, overall scores are relatively lower than the generated benchmark.
All correlation results are signiﬁcant having the p-value < 0.001.
For the details of the collection procedure and data statistics, please refer to Appendix D, where Figureshows the visualization of some examples comparing with human judgment scores.
We also report the comp-t2i benchmark [11] results for the compositional evaluation of the CUB and Flower datasets in aspects of color and shape in Appendix F.
4.1.1 Discussions
Consistent metric across datasets.
A distinct property of our method is the consistency across datasets.
As shown in Figure 1, the cosine similarity-based method, CLIP-S suffers the inconsistent results.
For example, the CUB [28, 29] and MM-CelebA-HQ [44] have narrow domains, birds and human faces, respectively, which is prone to get a similar score for all samples in the datasets by cosine similarity.
To validate our hypothesis, we vary the ratio of text-image shufﬂing using the real datasets where the counterparts in the selected pairs are deliberately shufﬂed, depicting misalignment.
For CLIP-S, we observe the inconsistency depending on datasets, while MI shows a consistent tendency.
Notice that the expectation of PMI is reduced to MI for the real images (Appendix A.1).
Inspecting possible over-ﬁtting with the CLIP features.
Figure 3 shows the normalized scores of metrics across feature extractors.
To get the normalized score, we subtract the score of RN101 and divide by the standard deviation of four scores across feature extractors reminiscent of z-score.
We expect the footprint of metrics should be consistent across different generative models if the model is not over-ﬁtted to the metrics.
LAFITE used both encoders of CLIP ViT-B/32 and VQ-Diffusion used the text encoder of CLIP ViT-B/32, while DM-GAN did none of them [45].
InfoNCE and
CLIP-R-Precision are related to the contrastive training losses (LAFITE, and DM-GAN for the
DAMSM loss [24]), which may lead to drastic change of the normalized scoring signature across the feature extractors.
While the proposed MID was relatively stable across the generative models. [Table] [Table]
Assignment 200 400 600 800 1K
LAFITE N o r m a l i z e d S c o r e - - -
RN101 ViT-B/32 ViT-B/16 ViT-L/
MID
CLIP-S
CLIP-R-Precision
SOA
InfoNCE
Caption
VQ-Diffusion - - -
RN101 ViT-B/32 ViT-B/16 ViT-L/
DM-GAN - - -
RN101 ViT-B/32 ViT-B/16 ViT-L/
[Table]
[Table] [Table] Comparison with the state-of-the-art models using MID. Table 9 in Appendix E exhibits the MID performance of the recent text-to-image generative models along with the other major metrics.
4.2 Evaluation on image captioning
Implementation details.
For a fair comparison with the current state-of-the-art evaluation metric (RefCLIP-S), we use the same pre-trained CLIP (ViT-B/32) used in the prior work to extract image and caption embedding vectors.
We use the images and the corresponding reference captions to build the covariance matrices ⌃ x , ⌃ y and the joint covariance matrix ⌃ z .
For the numerical stability of the inverse of covariance matrix, we replace ⌃  1 x with ˜ ⌃  1 x =
( ⌃ x + ✏ I )  1 , which handles the near-zero eigenvalues of covariance.
We found that ✏ of 5e-4 generally works across all benchmark evaluations, except for the FOIL benchmark where we used ✏ of 1e-15, which was slightly better.
Note that we use an identical prompt “A photo depicts” for all caption embeddings as employed in RefCLIP-S [19].
Flickr8K-Expert and Flickr8k-CF.
We measure the correlation of the proposed method with the Likert-scale judgments, which indicate the relative correctness of given captions.
Fliker8KExpert [48] provides 17K human expert judgments for 5,664 images with a four-scale where the higher is better.
Following prior works, we ﬂatten all human judgments to a list of 16,992 (5,664 ⇥ 3) samples, and we exclude 158 pairs where their captions appear in the reference set.
Flickr8K-CF [48] has 145K binary judgments from the CrowdFlower for 48K image-caption pairs.
Each pair receives at least three judgments, and we take the proportion of positive as a corresponding score.
Kendall’s ⌧ coefﬁcient [42] measures the rank correlation, and ⌧ c and ⌧ b are used for Fliker8K-Expert and
Flickr8K-CF, respectively.
Although ⌧ c is more suitable when the underlying scales differ in two variables, we follow the previous works for a fair comparison.
Tables 4 and 5 show the evaluation results.
For both cases, our MID signiﬁcantly improves the correlation of human judgments with [Table] [Table] [Table]
[Table] 54.9 and 37.3, respectively. Notably, MID improves further than RefCLIP-S, which uses the same vision-and-language pre-trained CLIP.
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
n
[Table]
[Figure]
[-Text-Image Shuffling Ratio 0 0.
 CUBCOCOLN-COCOMM-CelebA-HQ C L I P - S 0. Text-Image Shuffling Ratio 0.-][Table]
[Figure]
[Table]
[Figure]
[Table]
Shuffling Ratio
[Table]
[Figure]
[Table]
[Figure]
[-CLIP-S (Hessel et al., 2021) 0. RealFakeFoiled Fake Misaligned Fake Misaligned Real N u m b e r o f S a m p l e s 0 1000 2000 
-][Table]
[Figure]
Flickr8K reference parsimony.
Compared with the other methods, our method does not directly rely on the corresponding references, but through the mean and covariance.
Therefore, we could exploit the sample statistics with a limited number of references.
Figureshows the Kendall’s ⌧ correlation utilizing a subset of the available references.
Interestingly, even though 30-40% of images are available, it retains the majority of performance.
The dashed lines indicate the correlations of RefCLIP-S, which exploits all references.
Notice that our method can be positioned between the reference-with-image and reference-free metrics.
Because a sufﬁcient amount of samples are required to assess image captioning models, the sample statistics can be reliable from the sufﬁcient samples.
Pascal-50S. For a different evaluation setting of accuracy, the Pascal-50S [15] offers 4K pair-wise preference judgments between two captions, evenly splitting four categories, two human correct captions (HC), both human written, but one is incorrect (HI), one is from human, the other is by a model (HM), and both are generated by machine (MM).
For each pair, there are 48 human judgments and the majority of votes decides which caption is preferred, where ties are broken randomly.
As in the previous work [19], we randomly sample 5 references among 48 candidates and average over ﬁve evaluations.
Table 6 shows the consistent results outperforming competitive methods.
Except for HI, we achieve the state-of-the-art while the margin of HI is 0.1 having a near-perfect score of 99.7.
[Figure] e l [Figure] o n 0.
Reference Usage Ratio 0.
Flickr8K-ExpertFlickr8K-CF
A cc u r a c y 0.
Reference Usage Ratio 0.
PMI (4-ref) PMI (1-ref)
RefCLIP-S (1-ref)
M u t u a l I n f o r m a t
i o n
Text-Image Shuffling Ratio 0 0.
CUBCOCOLN-COCOMM-CelebA-HQ
C L I P - S 0.
Text-Image Shuffling Ratio 0.
CUBCOCOLN-COCOMM-CelebA-HQ
Object hallucination sensitivity.
Rohrbach et al.
[27] argue that image captioning models prone to generate the objects not presented in the image due to learned bias.
To assess this aspect, the FOIL-COCO [40] builds the carefully modiﬁed captions from the COCO captions [2] by swapping a single noun-phrase, e.g. , substituting “cat” for “dog”.
To measure the accuracy whether assigning a higher score to the ground-truth caption over the FOIL caption, we evaluate 32K test images with exclusive four reference captions of the COCO dataset.
Table 7 shows the competitive scores of our method.
Since RefCLIP-S [19] directly accesses the reference captions of the evaluating image, they can exploit the original words (before foiling) in the references, which is roughly 87% in four references and 67% in a randomly selected reference.
The RefCLIP-S is deﬁned as follows:
[Equation]
where H is harmonic mean, x and y denotes image and caption embeddings, and R is a set of the references.
Figure 5 supports this.
Our method is robust via covariance estimation in both 1-ref (green line) and 4-ref (blue line), while RefCLIP-S is reducing to its reference-free version of CLIP-S degrading performance.
We interpolate the scores of RefCLIP-S and CLIP-S as a reference (yellow line).
If we similarly utilize the term of max(max r 2 R cos( r , y ) , 0) as in Equation 8, MID gets 92.
and 93.7, outperforming RefCLIP-S (Appendix G for the detail, Figure 11 for the visualization).  General discussion
Reliance on the pre-trained CLIP models. Do the CLIP-based models take advantage of overﬁtting to the metric comparing other models? In Section 4.1.1, we partly answer that. Notice that multiple versions of the pre-trained CLIP vary the architectural type of visual backbone networks.
We found that the normalized scores are generally consistent across generative models and the backbones, especially for our metric (Figure 3).
However, it is worth noting that we encourage to use of the feature extractor ( e.g. , CLIP) trained by the loss maximizing mutual information of two modalities ( ref.
the last paragraph of Section 3.1).
Whether two independently-trained encoders perform well is under-explored; however, since we deﬁne the joint distribution in the proposed metric using the multivariate Gaussian distribution of concatenated features, we expect it reasonably works.
Relations to the FID and CLIP-score.
The Fréchet inception distance (FID) assesses the perceptual quality of fake images from a generative model [5].
It is a symmetric distance where two differences between two means and two variances are independently considered and summed.
While our method is an asymmetric divergence (Equation 7), the distances of means and variances are re-scaled to have unit variance as in the Mahalanobis distance (Equation 6), which is the source of domain robustness (Figure 1 Right).
Note that the CLIP-S is the cosine similarity using CLIP features.
The CLIP features are trained to minimize the InfoNCE loss by increasing mutual information approximately; however, our experiment shows that their approximation signiﬁcantly degrades the evaluating performance.
Measurement on diversity.
Our method does not entirely ignore the diversity by considering the covariance matrices from real and fake samples, similarly to the FID.
However, a few unique fake samples ( e.g. , mode collapse) may have small errors in the estimated statistics.
Ethical considerations.
Multiple reports show that the CLIP and the generative models based on this pre-trained CLIP have problematic social biases toward racial or ethnic groups [31, 49].
Unless appropriate measures toward these biases in the CLIP, our metric is potentially subject to the risk of fairness, accountability, and transparency.
Conclusion
We, to our best knowledge, ﬁrstly argue that the negative cross-mutual information with multivariate
Gaussian distributions can be used as a uniﬁed metric for multimodal generative models.
We provide the theoretical analyses of the proposed metric by out-of-distribution detecting by the squared
Mahalanobis distances, bias and variance decomposition, and the relation to the Kullback-Leibler divergence, along with the empirical experiments.
We achieve the state-of-the-art performances on text-to-image generation and image captioning benchmarks, the generated and human Likert-scale judgment correlations, visual reasoning accuracy, Flickr8K-Expert, Flickr8K-CF, Pascal-50S, and FOIL hallucination detection.
We look forward to seeing future works on the Gaussian cross-mutual information in multimodal representation learning based on this novel proposition. Acknowledgments
We sincerely thank Dongyoon Han for reviewing our manuscript and providing helpful comments.
Also, we give thanks to Jung-Woo Ha for early discussions and suggestions throughout the project.
And, we thank anonymous reviewers for their constructive feedback to solidify the manuscript.
The
NAVER Smart Machine Learning (NSML) platform [50] has been used in the experiments.