Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations (2020)

Abstract: A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings. To do this, it is critical to ensure that our evaluation protocols are correct, and benchmarks are reliable. In this work, we set forth to design a set of experiments to understand an important but often ignored problem in visually grounded language generation: given that humans have different utilities and visual attention, how will the sample variance in multi-reference datasets affect the models’ performance? Empirically, we study several multi-reference datasets and corresponding vision-and-language tasks. We show that it is of paramount importance to report variance in experiments; that human-generated references could vary drastically in different datasets/tasks, revealing the nature of each task; that metric-wise, CIDEr has shown systematically larger variances than others. Our evaluations on reference-per-instance shed light on the design of reliable datasets in the future.

## introduction natural language generation (nlg) is a challenging problem in natural language processing (nlp)-the complex nature of nlg tasks arise particularly in the output space. in contrast to text classification or regression problems with finite output space, generation could be seen as a combinatorial optimization problem, where we often have exponentially many options |v | ℓ (here |v | is the size of the vocabulary and ℓ is the sentence length). with the advances of both computer vision and nlp techniques in deep learning, there have been growing interests in visually grounded nlg tasks, such as image captioning @xcite @xcite , video captioning @xcite 1. this group of folks comprising runners and bikers, some wearing identifying numbers, look like they are getting ready for a marathon. 2. a runner in yellow has a convoy of motorcycles following behind him on a highway as bystanders watch. 3. marathon runners are running down a street with motorcyclists nearby. 4. a runner in the middle of a race running along side the road. 5. a man in a yellow shirt is running in a race. in this paper, instead of crunching numbers and modifying model architectural designs to achieve new "state-of-the-art" results on leaderboards, we focus on re-assessing the current practices in visually grounded language generation research, including problems, datasets, evaluations, and tasks, from the sample variance angle. given the differences in annotators' utility function and human visual attention models, how could the sample variance in captions teach us building robust and reliable visually grounded language generation agents? more specifically, we empirically investigate the variance among the multiple parallel references in different datasets, and its effect on the training performance and evaluation result of corresponding tasks. we further study the number of references per visual instance, and how it affects the training and testing performance. a simple search in acl anthology and cvf open access site shows that 58 out of 60 papers on vision-based text generation do not report variance in experimental results, while they often claim that their methods outperform previous state-of-the-art. our evaluation suggests that the variance cannot be ignored and must be reported, and that cider @xcite has shown higher variance than other metrics. fi-nally, introducing more training visual instances in the image and video captioning task on ms coco and vatex results in better performance on automatic metrics, while the visual storytelling task in vist favors more references in the training set. for future dataset collection, we recommend the inclusion of more references when each reference is distinctive and complicated. ## research questions and settings to understand sample variance, we conduct a series of experiments on multiple visually grounded nlg datasets, aiming to answer the following questions: we focus on multi-reference visually grounded nlg tasks where each visual instance is paired with multiple parallel text references. below we describe the datasets we investigate into, the models used for training, and the metrics for evaluation. datasets seven commonly used datasets in ## reference variance within datasets in this section, we examine the sample variance among text references within seven visually grounded nlg datasets. to quantify the sample variance, we define a consensus score c among n parallel references @xmath0: ## effect of sample variance on evaluation performance for visually grounded nlg tasks, models are trained on preset training samples and evaluated on preset testing samples, and then results are reported on leaderboards. but would training or evaluating with different samples affect their performance? how reliable are those numbers? in this section, we study to what extent the sample variance during either training or testing affects the evaluation results. for simplicity, the number of parallel references per visual instance used for training or testing is denoted by rpi. ## conclusion we study the sample variance in visually-grounded language generation, in terms of reference sample variance within datasets, effects of training or testing sample variance on metric scores, and the trade-off between the visual instance number and the parallel reference number per visual. along with some intriguing findings, we urge researchers to report sample variance in addition to the metric scores when comparing models' performance. we also recommend that when collecting a new dataset, the test set should include more parallel references for fair evaluation, while for the training set, when the text generations are expected to be distinctive and complicated, more parallel references should be collected otherwise a larger variety of visual appearances is more favorable.