  Perhaps surprisingly sewerage infrastructure is one of the most costly
infrastructures in modern society. Sewer pipes are manually inspected to
determine whether the pipes are defective. However, this process is limited by
the number of qualified inspectors and the time it takes to inspect a pipe.
Automatization of this process is therefore of high interest. So far, the
success of computer vision approaches for sewer defect classification has been
limited when compared to the success in other fields mainly due to the lack of
public datasets. To this end, in this work we present a large novel and
publicly available multi-label classification dataset for image-based sewer
defect classification called Sewer-ML.
  The Sewer-ML dataset consists of 1.3 million images annotated by professional
sewer inspectors from three different utility companies across nine years.
Together with the dataset, we also present a benchmark algorithm and a novel
metric for assessing performance. The benchmark algorithm is a result of
evaluating 12 state-of-the-art algorithms, six from the sewer defect
classification domain and six from the multi-label classification domain, and
combining the best performing algorithms. The novel metric is a
class-importance weighted F2 score, $\text{F}2_{\text{CIW}}$, reflecting the
economic impact of each class, used together with the normal pipe F1 score,
$\text{F}1_{\text{Normal}}$. The benchmark algorithm achieves an
$\text{F}2_{\text{CIW}}$ score of 55.11% and $\text{F}1_{\text{Normal}}$ score
of 90.94%, leaving ample room for improvement on the Sewer-ML dataset. The
code, models, and dataset are available at the project page
https://vap.aau.dk/sewer-ml/
