Ensemble of MRR and NDCG models for Visual Dialog (2021)

Abstract: Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., ‘yeah’ and ‘yes’). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as ‘I don’t know.’ Crafting a model that excels on both MRR and NDCG metrics is challenging. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg .

## related work visual conversation evaluation: early attempts to marry conversation with vision used street scene images, and binary questions @xcite . while binary answers are easy to verify, such an approach is limiting for an ai agent. on the other hand, analysis of generative metrics for dialog often show no correlation with human judgements @xcite . intuitively, metrics like bleu-scores rely on corresponding words with the ground-truth answer and often miss synonyms or the subjective nature. more importantly, generative metrics are geared toward textual assessment rather than visual reasoning, which results in models mainly relying on textual cues @xcite . malinowski and fritz (2014) suggest wu-palmer similarity metric that calculates similarity based on the depth of two words based on the wordnet taxonomy @xcite . a different approach suggested in the vqa dataset focus only on brief, mostly 1-word answers @xcite . in this setup, the task turns into popular answers classification, alleviating many text-generation challenges. notably, vqa requires 3 out of 10 annotators to agree on the answer, which is robust to inter-person variation. still, accuracy ignores the reasoning process. @xcite propose gqa, which extends the accuracy metric and uses a scene graph for both question generation and evaluation. following, @xcite propose the visdial dataset for the visual dialog task, which formulates multiple image-language interactions via a dialog. concurrently, de vries et al. @xcite propose guess-what, a goal-driven dialog dataset for object identification. different from vqa and goal-driven dialogs, the visdial answers are detailed and more human-like. for instance, in fig. 1 , the answer is "can't tell...cloth", while a vqa answer would be "cloth". therefore, metrics that require exact matching are no longer suitable. instead, each question is accompanied with 100 candidate answers. consequently, the metric has been shifted from accuracy to retrieval-based metrics, e.g., mrr and ndcg. prior works focus on optimizing a single metric @xcite @xcite . differently, murahari et al. (2020) attempt to optimize both metrics with a joint loss. still, a dedicated single metric model is superior. instead, we propose principals to ensemble two dedicated models, one for ndcg and one for mrr. our approach allows most of the mrr and ndcg to be preserved simultaneously. 2020 ) propose a lightweight transformer that handles the interplay between many modalities. the three methods mentioned above finetune using the dense annotation (i.e., human assessment of all the candidates), resulting in a substantial improvement on the ndcg metric. importantly, murahari et al. find that finetuning a model for ndcg hurts mrr performance. this work demonstrates that re-ranking mrr model (e.g., fga) and ndcg model (e.g., ls) with simple principles keeps most mrr and ndcg performance. ## two-step rank ensemble the mrr metric depends on a single humanderived answer. hence, given that this answer is ranked highly, the remaining candidates can be ranked according to the ndcg model. in the following, we describe two steps: (i) the mrr step responsible for preserving the human-derived rank high, and (ii) the ndcg step responsible for ranking the remaining candidates based on the ndcg model. ## results we show our results on the visdial v1.0 dataset, where 123,287 images are used for training, 2,000 images for validation, and 8,000 images for testing @xcite . each image is associated with ten questions, and each question has 100 corresponding answer candidates. we use two mrr models (i.e., n @xmath0, fga (schwartz et al., 2019b) and an ensemble of ls (murahari et al., 2020) with fga. we use ls(ce) as the ndcg model. we set ρ @xmath1, ρ @xmath2, ρ n @xmath3, ρ n @xmath4, and @xmath5. we tune these parameters using the validation set. comparison to state-of-the-art: in tab. 1 we compare our method to naïve ensembles and previous baselines. we first ensemble the ls's output with the fga's output. by combining them, we achieve the new mrr state-of-the-art (71.24% vs. 69.37%). next, we build a naïve ensemble of the mrr model and the ndcg model. we do so by adding the mrr ensemble scores (denoted by s m ) and ls(ce) scores (denoted by s n ), as follows: α • s m + (1 -α)s n , where α ∈ r calibrates the trade-off between mrr and ndcg performance. we show in fig. 2 an analysis of different α values on the validation set. in tab. 1, we report results for α = 0.8. our two-step method outperforms the mrr (70.41% vs. 68.78%) and ndcg (72.16% vs. 69.22%) metrics, despite lacking the output scores and only requiring rankings. we also compare our approach to previous baselines. most methods use the sparse annotations, i.e., the human-derived answer, while mreal-bdai, vd-bert, and ls(ce) finetune using the dense annotations. finetuning with the dense annotations tremendously boosts the ndcg performance but loses mrr performance. the mrr performance decline can be attributed to ndcg being biased toward uncertain answers. we also note that ls leverages large-scale image-text corpora. ls(ce+nsp) optimizes both the dense and sparse annotations but still suffers from a performance drop compared to metric-dedicated ls models, i.e., mrr (63.92% vs. 67.50%) and ndcg (68.08% vs. 74.47%). unlike the method mentioned above, our method re-rank the candidates based on two distinct models, with two distinct steps, to keep the human-derived answer high. in doing so, we achieve a good mrr performance (70.41% vs. 71.24%), yet notably with limited ndcg drop (72.25% vs. 75.35%). this property comes in handy in the recent visual dialog challenge, where the winners were picked based on both the ndcg and mrr evaluation metrics. our method performs well on both metrics simultaneously and won the challenge. ## conclusions we describe a non-parametric method to ensemble the candidate ranks of two strong mrr and ndcg models into a single ranking that excels on both ndcg and mrr. intuitively, we use the mrrmodel for non-ambiguous questions with certain answers. the dense-annotations cue is more applicable in ambiguous questions than the sparse annotations. thus, in the case of low certainty, our method relies almost entirely on the ndcg model. we hope the proposed principles can guide the community towards a parametric model that can employ answers' semantics to measure certainty. zhe gan, yu cheng, ahmed ei kholy, linjie li, jingjing liu, and jianfeng gao. 2019. multi-step reasoning via recurrent dual attention for visual dialog. acl. donald geman, stuart geman, neil hallonquist, and laurent younes. 2015. visual turing test for computer vision systems. nas. dalu guo, chang xu, and dacheng tao. 2019. imagequestion-answer synergistic network for visual dialog. in cvpr. ronghang hu, jacob andreas, marcus rohrbach, trevor darrell, and kate saenko. 2017. learning to reason: end-to-end module networks for visual question answering. iccv. drew a hudson and christopher d manning. 2019. gqa: a new dataset for real-world visual reasoning and compositional question answering. in cvpr. unnat jain, svetlana lazebnik, and alexander schwing. 2019. two can play this game: visual dialog with discriminative question generation and answering. cvpr. xiaoze jiang, jing yu, zengchang qin, yingying zhuang, xingxing zhang, yue hu, and qi wu. 2020. dualvd: an adaptive dual encoding model for deep visual understanding in visual dialogue. aaai. gi-cheon kang, jaeseo lim, and byoung-tak zhang. 2019. dual attention networks for visual reference resolution in visual dialog. acl. satwik kottur, josé mf moura, devi parikh, dhruv batra, and marcus rohrbach. 2018. visual coreference resolution in visual dialog using neural module networks. in eccv. chia-wei liu, ryan lowe, iulian v serban, michael noseworthy, laurent charlin, and joelle pineau. 2016. how not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. arxiv preprint arxiv:1603.08023. mateusz malinowski and mario fritz. 2014. a multiworld approach to question answering about realworld scenes based on uncertain input. in nips. george a miller. 1995. wordnet: a lexical database for english. communications of the acm. vishvak murahari, dhruv batra, devi parikh, and abhishek das. 2020. large-scale pretraining for visual dialog: a simple state-of-the-art baseline. eccv. van-quang nguyen, masanori suganuma, and takayuki okatani. 2020. efficient attention mechanism for handling all the interactions between many inputs with application to visual dialog. eccv. yulei niu, hanwang zhang, manli zhang, jianhong zhang, zhiwu lu, and ji-rong wen. 2019. recursive visual attention in visual dialog. in cvpr. jiaxin qi, yulei niu, jianqiang huang, and hanwang zhang. 2020. two causal principles for improving visual dialog. cvpr. idan schwartz, alexander g schwing, and tamir hazan. 2019a. a simple baseline for audio-visual scene-aware dialog. in cvpr. idan schwartz, seunghak yu, tamir hazan, and alexander g schwing. 2019b. factor graph attention. in cvpr. paul hongsuck seo, andreas lehrmann, bohyung han, and leonid sigal. 2017. visual reference resolution using attention memory for visual dialog. in nips. iulian vlad serban, alessandro sordoni, ryan lowe, laurent charlin, joelle pineau, aaron c courville, and yoshua bengio. 2017. a hierarchical latent variable encoder-decoder model for generating dialogues. in aaai. piyush sharma, nan ding, sebastian goodman, and radu soricut. 2018. conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. in acl.