  Landmark correspondences are a widely used type of gold standard in image
registration. However, the manual placement of corresponding points is subject
to high inter-user variability in the chosen annotated locations and in the
interpretation of visual ambiguities. In this paper, we introduce a principled
strategy for the construction of a gold standard in deformable registration.
Our framework: (i) iteratively suggests the most informative location to
annotate next, taking into account its redundancy with previous annotations;
(ii) extends traditional pointwise annotations by accounting for the spatial
uncertainty of each annotation, which can either be directly specified by the
user, or aggregated from pointwise annotations from multiple experts; and (iii)
naturally provides a new strategy for the evaluation of deformable registration
algorithms. Our approach is validated on four different registration tasks. The
experimental results show the efficacy of suggesting annotations according to
their informativeness, and an improved capacity to assess the quality of the
outputs of registration algorithms. In addition, our approach yields, from
sparse annotations only, a dense visualization of the errors made by a
registration method. The source code of our approach supporting both 2D and 3D
data is publicly available at
https://github.com/LoicPeter/evaluation-deformable-registration.
