Learning to Rank Visual Stories From Human Ranking Data (2022)

Abstract: Visual storytelling (VIST) is a typical vision and language task that has seen extensive development in the natural language generation research domain. However, it remains unclear whether conventional automatic evaluation metrics for text generation are applicable on VIST. In this paper, we present the VHED (VIST Human Evaluation Data) dataset, which first re-purposes human evaluation results for automatic evaluation; hence we develop Vrank (VIST Ranker), a novel reference-free VIST metric for story evaluation. We first show that the results from commonly adopted automatic metrics for text generation have little correlation with those obtained from human evaluation, which motivates us to directly utilize human evaluation results to learn the automatic evaluation model. In the experiments, we evaluate the generated texts to predict story ranks using our model as well as other reference-based and reference-free metrics. Results show that Vrank prediction is significantly more aligned to human evaluation than other metrics with almost 30% higher accuracy when ranking story pairs. Moreover, we demonstrate that only Vrank shows human-like behavior in its strong ability to find better stories when the quality gap between two stories is high. Finally, we show the superiority of Vrank by its generalizability to pure textual stories, and conclude that this reuse of human evaluation results puts Vrank in a strong position for continued future advances.

## introduction in visual storytelling (vist) @xcite , a generation model tells a short story to describe the given five images. automatic generation of visual stories is challenging because it has the complexity of cross-modal understanding with the diversity and sophistication of creative writing @xcite . extensive efforts in model developments have decreased the distance between machinegenerated and human-written stories, but research on vist evaluation remains stagnant. automatic metrics and human evaluation are widely used to examine natural language generation. traditional n-gram-based or referencebased autometrics such as bleu @xcite , @xcite , and me-teor @xcite are common for vist evaluation. however, preliminary findings have shown that these metrics have many drawbacks and hence are incompatible with vist @xcite . in particular, they assume that human-written stories are always better than machine-generated stories, limiting the advance of models yet not conforming to our observation on human judgment. rethinking this postulation in evaluation, we believe the dependence on references should be minimized and human evaluation results should be fully utilized instead, because human judgements contain more meaningful signals. recent hybrid and referencefree metrics such as bleurt @xcite and union @xcite have not yet been implemented or studied in vist. nevertheless, bleurt utilizes few human results in finetuning, and union still regards human references as gold labels, which results in poor correlation to human judgment. on the other hand, human evaluations are relatively reliable for performance reports, and recent studies often include them to provide more convincing experimental results @xcite . however, human evaluations are expensive, time-consuming, and difficult to reproduce. therefore, results should be recycled to benefit future evaluations. accordingly, we re-collected the human evaluation results from multiple published papers and organized the data into story pairs @xcite as the vhed (vist human evaluation data) dataset. we then re-purposed vhed to create a better metric vrank for vist to rank visual stories. vrank is a reference-free simcse @xcite based metric trained on vhed to learn to rank visual stories. we believe a storytelling metric should be independent of the references because stories are highly diverse by nature @xcite , and it is reasonable for them to be dissimilar to the references @xcite as shown in fig. 1 , the story generated by model 1 is assigned a higher bleu score because larger portions of text overlap with the reference. however, human rankers recognize description in isolation and object detection error in model 1, and instead rank model 2 better. we conduct experiments to show that vrank is superior to existing metrics, many of which lack properties essential to evaluating stories in a human-like fashion. therefore, we utilize vhed to understand and analyze human judgment in evaluating visual stories, and to provide additional metric assessments to reveal the shortcomings of existing metrics. the metric assessment experiments are conducted as the story-pair ranking task in which two stories are ranked based on their story quality. we observe three characteristics and design corresponding assessments to demonstrate vrank's merits. first, larger rank differences in story quality are easier for people to differentiate. we measure the performance of metrics in story pairs with large gaps versus small gaps to determine whether all metrics have this property. our assessment indi-cates this property is exclusively hold by vrank. second, human-written stories are not always better than machine-written stories. indeed, 38% of machine-generated stories are better than the references, which suggests that the afore-mentioned assumption may need to be revisited @xcite . we examine the ability of metrics to rank such human-machine pairs, which vrank performs relatively well. finally, most generated stories still contain many errors, which serve as signals for human rankers @xcite . hence we evaluate the ability of metrics to detect errors and show that vrank is a better indicator of errors. also, we show that vrank is able to generalize to other datasets without bias to vhed. in conclusion, vrank excels in the above assessments and able to follow human behaviors in ranking. moreover, vrank can rank machine and human stories decently and is better at detecting story errors. specifically, we make three major contributions: visual storytelling (vist) visual storytelling was introduced by @xcite as the task of generating a coherent story given five images. they provided a dataset, sequential images narrative dataset (sind), containing images and references in which references are human-written short stories describing images. for every image prompt (one sequence of photos), there are 2 to 5 references. vist requires deeper understanding of the photo events to prevent descriptions in isolation (i.e., image captions). researchers have proposed various methods for this task. knowledge graphs are often integrated in models to encourage diversity of terms and plots in the stories @xcite @xcite . some studies use reinforcement learning to reward models that generate stories that contain fewer errors and are more topically-focused @xcite . however, existing evaluation methods are unable to capture the true quality of the generated stories. thus we examine automatic metrics to devise a better way for machines to evaluate stories. ## vrank we propose vrank, a reference-free automatic metric that inputs story pairs to predict human preferences between the two stories. we utilize sim-cse @xcite to leverage better sentence representations. simcse uses contrastive learning with dropout as augmentation, then trained on natural langauge inference datasets to obtain better sentence embeddings from bert @xcite . first, we pre-trained the simcse model using sind reference stories with the masked language model objective. then, we input two stories with a [sep] token in between through the pre-trained model. we use the acquired sentence embeddings and feed it through a regression layer to predict a ranking gap. we used mean squared error to calculate the loss between the predicted ranking gap and true ranking gap. after obtaining the ranking gap, we predict which story is better according to the sign of the predicted ranking gap. although vrank is a simple model fine-tuned solely on human judgment, it still outperforms current existing metrics in our assessments. this suggests further potential for use with vhed; more studies can be conducted to replace vrank with stronger neural network models. during model training, since the number of positives and negatives were not balanced in the original dataset, we augmented the data to create a symmetric dataset of vhed to minimize dataset bias. 3 the ranking gap in the resulting dataset was close to normally distributed. we hypothesize that utilizing this feature makes it possible to extract more information, making it easier for the model to learn human judgment. however, due to the small amount of data available, high variance is likely @xcite to occur during inference. hence, we used all data from vhed, including human @xmath0. ## metric assessment in this section, we describe a series of assessments conducted on existing metrics on vhed, in which the assessment methods are based on the analyses in vhed. the objective is to examine whether vrank is superior to other metrics based on our analysis of vhed. ## dataset generalization in addition to vist, we expect vrank to reasonably evaluate the quality of text as well. to determine whether vrank generalizes to textual stories, we selected mans dataset @xcite , an imagefree storytelling dataset in which the stories are derived from the rocstories corpus @xcite . mans includes 200 story prompts, where each prompt includes five model-generated stories and a reference. however, it does not contain human story rankings. thus, for each story prompt, we asked five workers from amazon me-chanical turk to rank the five stories to obtain ranking scores. following the vhed construction procedure, the ranked stories were converted into story pairs, making for 1,112 story pairs for which 3 workers agreed on the ranking, 605 story pairs for which 4 workers agreed, and 132 story pairs for which 5 workers agreed. likewise, we evaluate story pairs with @xmath1. the results of vrank and the baseline automatic metrics when ranking mans are shown in ## conclusion and discussion we present vhed and vrank, the first dataset of human evaluation results and evaluation metric for vist. we show that vrank performs significantly better in three assessment tasks and generalizes to other datasets. also, recent automatic metrics are ill-suited to evaluating visual stories, especially human-level written stories. we welcome researchers to share their human evaluation results to the community to broaden the data domain to obtain more knowledge about human judgment and improve the performance of vrank. as the gap between machines and humans continues to decrease, stronger metrics will be needed to evaluate machine and human stories. improving vrank performance to replace reference-based metrics is our future goal. 9 appendix application in this section, we introduce an application for vrank and other reference-free metrics. our assessment indicates that vrank's predictions strongly agree with human judgment. we quantify the distance between humans and machines by pairing up reference and generated stories and calculating the ratio of generated stories that outmatch the references. unlike human evaluation, which can be conducted only on a portion of the testing data, this method allows researchers to evaluate the proposed model over the entire testing dataset. after applying vrank to assess five recent vist models, we present the results in figure 4 : the models are gradually approaching human-level writing, outlining an exciting development of nlg in vist. error type examples and correlation in without reference absent algorithm here, we show the results of automatic metric accuracy in story-pair ranking without the proposed reference absent algorithm. as expected, the accuracies for h&m pairs are the same. since all references are regarded as ground truth for reference-based automatic metrics, the accuracy is shown as the percentage of the human-written stories that are better