  The research field concerned with the digital restoration of degraded written
heritage lacks a quantitative metric for evaluating its results, which prevents
the comparison of relevant methods on large datasets. Thus, we introduce a
novel dataset of Subjective Assessments of Legibility in Ancient Manuscript
Images (SALAMI) to serve as a ground truth for the development of quantitative
evaluation metrics in the field of digital text restoration. This dataset
consists of 250 images of 50 manuscript regions with corresponding spatial maps
of mean legibility and uncertainty, which are based on a study conducted with
20 experts of philology and paleography. As this study is the first of its
kind, the validity and reliability of its design and the results obtained are
motivated statistically: we report a high intra- and inter-rater agreement and
show that the bulk of variation in the scores is introduced by the images
regions observed and not by controlled or uncontrolled properties of
participants and test environments, thus concluding that the legibility scores
measured are valid attributes of the underlying images.
