  During the COVID-19 pandemic, a significant effort has gone into developing
ML-driven epidemic forecasting techniques. However, benchmarks do not exist to
claim if a new AI/ML technique is better than the existing ones. The
"covid-forecast-hub" is a collection of more than 30 teams, including us, that
submit their forecasts weekly to the CDC. It is not possible to declare whether
one method is better than the other using those forecasts because each team's
submission may correspond to different techniques over the period and involve
human interventions as the teams are continuously changing/tuning their
approach. Such forecasts may be considered "human-expert" forecasts and do not
qualify as AI/ML approaches, although they can be used as an indicator of human
expert performance. We are interested in supporting AI/ML research in epidemic
forecasting which can lead to scalable forecasting without human intervention.
Which modeling technique, learning strategy, and data pre-processing technique
work well for epidemic forecasting is still an open problem. To help advance
the state-of-the-art AI/ML applied to epidemiology, a benchmark with a
collection of performance points is needed and the current "state-of-the-art"
techniques need to be identified. We propose EpiBench a platform consisting of
community-driven benchmarks for AI/ML applied to epidemic forecasting to
standardize the challenge with a uniform evaluation protocol. In this paper, we
introduce a prototype of EpiBench which is currently running and accepting
submissions for the task of forecasting COVID-19 cases and deaths in the US
states and We demonstrate that we can utilize the prototype to develop an
ensemble relying on fully automated epidemic forecasts (no human intervention)
that reaches human-expert level ensemble currently being used by the CDC.
