  Forecasting competitions are the equivalent of laboratory experimentation
widely used in physical and life sciences. They provide useful, objective
information to improve the theory and practice of forecasting, advancing the
field, expanding its usage and enhancing its value to decision and
policymakers. We describe ten design attributes to be considered when
organizing forecasting competitions, taking into account trade-offs between
optimal choices and practical concerns like costs, as well as the time and
effort required to participate in them. Consequently, we map all major past
competitions in respect to their design attributes, identifying similarities
and differences between them, as well as design gaps, and making suggestions
about the principles to be included in future competitions, putting a
particular emphasis on learning as much as possible from their implementation
in order to help improve forecasting accuracy and uncertainty. We discuss that
the task of forecasting often presents a multitude of challenges that can be
difficult to be captured in a single forecasting contest. To assess the quality
of a forecaster, we, therefore, propose that organizers of future competitions
consider a multi-contest approach. We suggest the idea of a forecasting
pentathlon, where different challenges of varying characteristics take place.
