  Automatic video summarization is still an unsolved problem due to several
challenges. The currently available datasets either have very short videos or
have few long videos of only a particular type. We introduce a new benchmarking
video dataset called VISIOCITY (VIdeo SummarIzatiOn based on Continuity, Intent
and DiversiTY) which comprises of longer videos across six different categories
with dense concept annotations capable of supporting different flavors of video
summarization and other vision problems. For long videos, human reference
summaries necessary for supervised video summarization techniques are difficult
to obtain. We explore strategies to automatically generate multiple reference
summaries from indirect ground truth present in VISIOCITY. We show that these
summaries are at par with human summaries. We also present a study of different
desired characteristics of a good summary and demonstrate how it is normal to
have two good summaries with different characteristics. Thus we argue that
evaluating a summary against one or more human summaries and using a single
measure has its shortcomings. We propose an evaluation framework for better
quantitative assessment of summary quality which is closer to human judgment.
Lastly, we present insights into how a model can be enhanced to yield better
summaries. Sepcifically, when multiple diverse ground truth summaries can
exist, learning from them individually and using a combination of loss
functions measuring different characteristics is better than learning from a
single combined (oracle) ground truth summary using a single loss function. We
demonstrate the effectiveness of doing so as compared to some of the
representative state of the art techniques tested on VISIOCITY. We release
VISIOCITY as a benchmarking dataset and invite researchers to test the
effectiveness of their video summarization algorithms on VISIOCITY.
