The Devil’s in the Details: the Detailedness of Classes Influences Personal Information Detection and Labeling (2025)

Abstract: In this paper, we experiment with the effect of different levels of detailedness or granularity—understood as i) the number of classes, and ii) the classes’ semantic depth in the sense of hypernym and hyponym relations — of the annotation of Personally Identifiable Information (PII) on automatic detection and labeling of such information. We fine-tune a Swedish BERT model on a corpus of Swedish learner essays annotated with a total of six PII tagsets at varying levels of granularity. We also investigate whether the presence of grammatical and lexical correction annotation in the tokens and class prevalence have an effect on predictions. We observe that the fewer total categories there are, the better the overall results are, but having a more diverse annotation facilitates fewer misclassifications for tokens containing correction annotation. We also note that the classes’ internal diversity has an effect on labeling. We conclude from the results that while labeling based on the detailed annotation is difficult because of the number of classes, it is likely that models trained on such annotation rely more on the semantic content captured by contextual word embeddings rather than just the form of the tokens, making them more robust against nonstandard language.

## introduction personal information is ubiquitous in many text genres, posing a unique challenge for those seeking to create and share corpora. while access to collections of texts is highly desirable from the perspective of researchers in fields such as linguistics, natural language processing (nlp), or digital humanities, the potential presence of clues indicating the identity of the writer or other natural persons makes them fall under the general data protection regulation (gdpr, official journal of the european union, 2016). the gdpr itself suggests potential solutions to the problem: de-identification methods such as anonymization -the "[c]omplete and irreversible removal [...] of any information that, directly or indirectly, may lead to a subject's data being identified" -or pseudonymization, the "[p]rocess of replacing direct identifiers with pseudonyms or coded values," for which there must exist a mapping between the original data and the pseudonyms, which is securely stored separately from the pseudonymized texts @xcite . both of these privacy-preserving procedures presuppose a stage where the personally identifiable information (pii) found in the data is detected. while this can be done manually, it is time-consuming. while automatic approaches for both anonymization and pseudonymization have been proposed @xcite , @xcite show that there appears to be very little uniformity in how researchers and corpus creators choose to classify piis. the taxonomies range in terms of granularity or detailedness, understood as the number of classes that piis are divided into and their semantic depth in terms of hypernym and hyponym relations (as in wordnet @xcite ). for example, @xcite utilize only one label, person, to refer to elements such as names, surnames, nicknames, usernames, etc., which can be differentiated in other corpora (e.g. @xcite @xcite ). very little work has been done on determining what level of granularity of pii annotation is the most suitable for subsequent removal or replacement of personal information. it is worth noting that while the term detection often includes labeling in other research on ## prior research data for research or training language models needs to be free from personal information to protect those who generate it, and the work on automatic de-identification methods, especially for texts belonging to domains other than medical or legal, has gained much traction in the recent years @xcite . much research has gone into testing what kinds of models perform best for pii detection or labeling. @xcite evaluate 9 different model architectures and embedding combinations on the pii-annotated corpus of german emails, code alltag, reaching the best performance with a transformer-based architecture and embeddings, optionally combined with noncontextual word embeddings. @xcite successfully utilize a combination of a generic named entity recognition (ner) model with a gazetteer to detect and classify piis in english (the tab corpus and a set of annotated wikipedia biographies) and employ privacy risk estimation methods to determine whether a span should be anonymized or not. @xcite frame the closely related task of protected health information (phi) detection as a named entity recognition and classification (nerc) task and obtain good results on it using two bert-type language models on swedish medical data from the stockholm epr phi corpus. szawerna et al. (2024b) also use models of this kind to detect piis in the swell corpus, a collection of learner essays in swedish. notably, they forego the labeling step, differentiating only between pii and non-pii tokens. it is worth noting that all of the previously mentioned pii or phi detection or labeling studies utilized different data, and only the texts used by papadopoulou et al. (2022, 2023 ) -representing a vastly different domain and a more general tagset than the texts we work with -are openly available with the original piis in place. additionally, all of the papers employed different categories for the labeling task. as szawerna et al. (2024a) point out, differences between pii taxonomies employed in the de-identification of corpora can be quite considerable, not only in terms of class granularity but also class overlap. this may be motivated by the specific characteristics of the de-identified domains or the end goal: taxonomies used for pseudonymization seem to feature more classes than those intended for anonymization, likely because the class of the pii is later used to generate a suitable pseudonym. this leads to the results not being fully comparable. the tab corpus features fewer, semantically more general classes (grouping together many different concepts into one category); it also lexical or grammatical correction annotation foot_0 . this makes it unsuitable for addressing our research questions without a considerable amount of time going into manual reannotation. however, it remains unclear how and to what extent the types of classes used in personal information detection affect the detection step itself. in szawerna et al. (2024a) we consider a more detailed taxonomy more favorable, but we do not test that. we do, however, point out that what is per-sonal is context-dependent and may vary between domains, so the choice of the labels can also depend on the domain. to the best of our knowledge, the only study that investigated whether a more diverse class division facilitates better pii detection is the one by @xcite . in this case, the authors adapted the tab corpus by automatically translating it into spanish and projecting the pii categories back into the text. they later re-annotated the corpus with refined, less ambiguous classes, leading to an increase in the number of classes. notably, they also discard the misc class, which is used to annotate very semantically diverse elements. they note an increase in performance on the dataset annotated using the refined tagset, which could be due to the new tagset being easier for their models to train on, but also due to manual re-annotation being more reliable than projection, and some information not being as revealing after translation. ## results the mean detection results and the standard deviation over the runs are presented in both tables show that the iob-type annotation appears to be more difficult to predict. this is likely due to relatively few piis spanning more than one token, leading to the classifiers having more issues determining those boundaries; in our case both the iob component and the class label have to match for a token to be counted as correctly classified. yet another aspect worth men-tioning is that in many cases (firstname male, month word, etc.) the original swell annotation is intended to describe only one token, whereas other classes (e.g. school) are likely to consist of more than one token (see when it comes to pii detection ( in the labeling task, the basic type of annotation excels in all evaluation metrics. in the case of basic annotation, detection and labeling are the same. if we consider the types of annotation where these two tasks are different, then the runs with the best recall, f1, and f2 for detection are the ones finetuned on the general type of annotation. in the case of labeling, these runs would be the best on all of the evaluation metrics. this partly contradicts the findings of @xcite , who report that more detailed classes facilitate better pii detection and labeling. however, the detailed classes in their experiment were refined based on what they found to be ambiguous in the original tagset. in our case, we utilized an existing hierarchical tagset. the difference in granularity between general and specific classes is also much larger, as @xcite split the original classes into at most 2 new classes, while in our data one general class can correspond to as many as 12 specific classes. we can only examine the interplay between the identification of piis and the tokens that were labeled for grammatical or lexical errors in different tagsets ( figure 2 and figure 3 show the per-class accuracy (disregarding the i and b distinction). points indicating the number of instances of the respective class in the data are overlaid atop the accuracy bar charts. figure 2 shows these statistics for the general and general iob tagsets. for some of the classes, prevalence in the data correlates with accuracy -nearly 1200 tokens belong to the geographic class, which has high accuracy, while institution, with around 200 tokens, shows worse results and the extremely infrequent transportation class practically never gets correctly predicted. however, there are classes that diverge from this trend: despite having almost as many instances as geographic, other has noticeably lower accuracy, implying that they are difficult to predict. less frequent classes like personal name, date, and age achieve high accuracy scores despite not being as numerous as some other classes, indicating that they are easier to predict. a similar phenomenon can be observed in figure 3 , which represents the specific and specific iob tagsets. classes like city, fam, and country, have high frequency and high accuracy. many of the infrequent classes practically never get correctly predicted, and classes with intermediate frequency, like firstname male or other nr seq have mediocre accuracy. once again there are also frequent classes with low accuracy (sensitive) and less frequent classes with high accuracy (e.g. age digits, month words) -which, once again, suggests that some classes can be easy or difficult to predict regardless of their frequency. these results suggest that while having many examples helps the models to learn to predict a given class, some classes are much easier or much more difficult to predict than others. the performance of some classes is high because of their high frequency in the dataset, whereas some other classes are easy to predict despite not being all that frequent. it therefore appears that it is not only class frequency, but also class semantics that influence the accuracy of predictions, with frequent classes and classes with little internal variation in meaning performing better. this might also explain why sierro et al. ( 2024 ) observed an improvement with a larger number of classes, as their increase in the number of classes happened once they split (and subsequently narrowed down the semantics of) vague classes and disregarded the misc class, their equivalent of our sensitive or other. this confirms that identifying semantically distinct classes for annotation is crucial for the success of the annotation scheme and its application in classification tasks. such labeling requires a good understanding and knowledge of the domain. while the results show what kind of annotation facilitates the best detection or labeling, the results of the experiments do not allow us to identify the overall best type of pii annotation, as this depends on the subsequent steps. for example, if the final corpus should contain more specific labels for anonymized spans, then it may be worth to split the process into detection followed by labeling, as detection outperforms labeling at this level of tagset detail; there are some results from other tasks which may suggest that such a separation could be beneficial, e.g. @xcite . another related observation is that pii entities tend not to appear directly adjacent to other pii elements belonging to the same class, which suggests that such boundaries (i.e. iob-type annotation) need not be included, but it may vary for different labels and domains. ## conclusions we have compared the performance of kb-bertbased classifiers on detecting and classifying personally identifiable information distinguished by a different number of classes and the semantic depth or specificity of these classes. we have found that for pii detection, basic, non-iob annotation yields the best results. when it comes to labeling, more specific classes do not ensure better results, possibly due to some of those classes being underrepresented, since frequency does appear to play some role in how well various classes are detected. an iob-style annotation also results in a decrease in performance versus not differentiating between beginnings and insides of spans. we have also found that models fine-tuned on more basic annotation tend to misclassify words that are misspelled, misplaced, or syntactically incorrect more often than models fine-tuned with more specific classes. we have also observed that it is not only class imbalance and a low frequency of a number of the classes, but also the classes' semantics that influence the accuracy of the predictions. semantically less coherent or less constrained classes make it much more difficult for the models to make correct predictions, pointing to the need for well-defined classes. this emphasizes the role of understanding the domain for which the annotation scheme is designed and raises an important issue concerning the cross-domain transfer of annotation schemes as different classes will have different frequencies and semantics across these classes. while the choice of pii taxonomy is likely to depend on the needs of the specific case, the results suggest that using over-detailed classes for automatic pii detection and labeling may not lead to optimal performance, at least not without a large dataset for the model to learn from. the same applies to the differentiating between the beginning and the inside of a pii span in iob-type annotation, which does not lead to better performance, and therefore should only be included if required in the specific case. in these experiments we have shown what kinds of annotation facilitate pii detection and labeling, the final choice also depends on the subsequent task, such as generating pseudonyms or removing pii spans. as long as the classes are required by the subsequent steps in a pipeline (e.g. pseudonym generation) or desired in the final version of the text (e.g. as placeholders in the anonymized text), there is a need for a more detailed annotation than the basic one utilized in our experiment. this also signals a need for investigating whether the label-ing step can be separated from the detection step, and how the performance of such a setup compares to classifying the piis in a single step. the overt class imbalance (including the lack of any piis of certain kinds of specific labeling in the data) highlights the need for well-curated training datasets that feature a sufficient number of piis of each kind, either by collecting more data or adjusting the annotation; alternatively, one could also opt to combine machine learning and rule-based detection methods (many of the absent specific classes, such as account nr, could be more easily identified using e.g. regular expressions).