  In many research areas, for example motion and gesture generation, objective
measures alone do not provide an accurate impression of key stimulus traits
such as perceived quality or appropriateness. The gold standard is instead to
evaluate these aspects through user studies, especially subjective evaluations
of video stimuli. Common evaluation paradigms either present individual stimuli
to be scored on Likert-type scales, or ask users to compare and rate videos in
a pairwise fashion. However, the time and resources required for such
evaluations scale poorly as the number of conditions to be compared increases.
Building on standards used for evaluating the quality of multimedia codecs,
this paper instead introduces a framework for granular rating of multiple
comparable videos in parallel. This methodology essentially analyses all
condition pairs at once. Our contributions are 1) a proposed framework, called
HEMVIP, for parallel and granular evaluation of multiple video stimuli and 2) a
validation study confirming that results obtained using the tool are in close
agreement with results of prior studies using conventional multiple pairwise
comparisons.
