  For sensitive text data to be shared among NLP researchers and practitioners,
shared documents need to comply with data protection and privacy laws. There is
hence a growing interest in automated approaches for text anonymization.
However, measuring such methods' performance is challenging: missing a single
identifying attribute can reveal an individual's identity. In this paper, we
draw attention to this problem and argue that researchers and practitioners
developing automated text anonymization systems should carefully assess whether
their evaluation methods truly reflect the system's ability to protect
individuals from being re-identified. We then propose TILD, a set of evaluation
criteria that comprises an anonymization method's technical performance, the
information loss resulting from its anonymization, and the human ability to
de-anonymize redacted documents. These criteria may facilitate progress towards
a standardized way for measuring anonymization performance.
