  How can we evaluate the performance of a disambiguation method implemented on
big bibliographic data? This study suggests that the open researcher profile
system, ORCID, can be used as an authority source to label name instances at
scale. This study demonstrates the potential by evaluating the disambiguation
performances of Author-ity2009 (which algorithmically disambiguates author
names in MEDLINE) using 3 million name instances that are automatically labeled
through linkage to 5 million ORCID researcher profiles. Results show that
although ORCID-linked labeled data do not effectively represent the population
of name instances in Author-ity2009, they do effectively capture the 'high
precision over high recall' performances of Author-ity2009. In addition,
ORCID-linked labeled data can provide nuanced details about the
Author-ity2009's performance when name instances are evaluated within and
across ethnicity categories. As ORCID continues to be expanded to include more
researchers, labeled data via ORCID-linkage can be improved in representing the
population of a whole disambiguated data and updated on a regular basis. This
can benefit author name disambiguation researchers and practitioners who need
large-scale labeled data but lack resources for manual labeling or access to
other authority sources for linkage-based labeling. The ORCID-linked labeled
data for Author-tiy2009 are publicly available for validation and reuse.
