  Datasets in the computer vision academic research community are primarily
static. Once a dataset is accepted as a benchmark for a computer vision task,
researchers working on this task will not alter it in order to make their
results reproducible. At the same time, when exploring new tasks and new
applications, datasets tend to be an ever changing entity. A practitioner may
combine existing public datasets, filter images or objects in them, change
annotations or add new ones to fit a task at hand, visualize sample images, or
perhaps output statistics in the form of text or plots. In fact, datasets
change as practitioners experiment with data as much as with algorithms, trying
to make the most out of machine learning models. Given that ML and deep
learning call for large volumes of data to produce satisfactory results, it is
no surprise that the resulting data and software management associated to
dealing with live datasets can be quite complex. As far as we know, there is no
flexible, publicly available instrument to facilitate manipulating image data
and their annotations throughout a ML pipeline. In this work, we present
Shuffler, an open source tool that makes it easy to manage large computer
vision datasets. It stores annotations in a relational, human-readable
database. Shuffler defines over 40 data handling operations with annotations
that are commonly useful in supervised learning applied to computer vision and
supports some of the most well-known computer vision datasets. Finally, it is
easily extensible, making the addition of new operations and datasets a task
that is fast and easy to accomplish.
