I ndonesian Speech Content De-Identification in Low Resource Transcripts (2025)

Abstract: Advancements in technology and the increased use of digital data threaten individual privacy, especially in speech containing Personally Identifiable Information (PII). Therefore, systems that can remove or process privacy-sensitive data in speech are needed, particularly for low-resource transcripts. These transcripts are minimally annotated or labeled automatically, which is less precise than human annotation. However, using them can simplify the development of de-identification systems in any language. In this study, we develop and evaluate an efficient speech de-identification system. We create an Indonesian speech dataset containing sensitive private information and design a system with three main components: speech recognition, information extraction, and masking. To enhance performance in low-resource settings, we incorporate transcription data in training, use data augmentation, and apply weakly supervised learning. Our results show that our techniques significantly improve privacy detection performance, with approximately 29% increase in F1 score, 20% in precision, and 30% in recall with minimally labeled data.

## introduction a considerable amount of private data is readily accessible online @xcite , often utilized for machine learning research leveraging publicly available information. while privacy concerns for text data have received attention @xcite , strategies to protect speech data remain underdeveloped. this imbalance highlights the critical need to implement robust privacy safeguards for all modality. speech privacy comprises two main categories: speaker identity and content privacy, with the latter, including sensitive spoken utterances like personally identifiable information (pii), being relatively underexplored @xcite . this con-tent may include spoken utterances that contain sensitive information, such as personally identifiable information (pii). exposure to pii risks severe consequences, such as losing control over personal information @xcite . to protect the privacy of speech content, a method called speech content de-identification can be employed. this technique focuses on identifying private information and either removing it or substituting it with uniform noise. on the surface, de-identified data might seem unusable, but @xcite have demonstrated that such data can still be useful for training a privacy-preserving speech recognition models without a significant drop in performance. figure 1 : overview of the speech content @xcite multiple studies on speech content deidentification have used text transcripts as intermediaries @xcite @xcite . these systems typically consist of a speech recognition module, an information extraction module, and a masking module, as shown in figure 1 . although prior research achieved positive results, their resource-intensive implementations are challenging to apply to lowresource languages, which often lack advanced privacy protection tools. consequently, sensitive data in these languages are at greater risk of privacy breaches. to ensure privacy, de-identification systems must operate effectively in the target language despite limited resources. the objective of this research is to develop a speech de-identification system that overcomes the challenges related to transcription and limited resources. to this end, we incorporate transcription data in the training, utilize data augmentation techniques, and apply weakly supervised learning. our work contributes to improving the system efficiency, particularly when working with lowresource transcripts. ## related work speech content de-identification involves the systematic removal of any pii from recorded speech, positioning it as a new entity recognition task @xcite . for example, in a recorded speech that reads, "john came from indonesia," a speech content de-identification system would process the data to redact any private information like "john" and "indonesia." this ensures the anonymization of sensitive information within the speech data. the main challenges in developing a speech deidentification system for low-resource languages like indonesian include, but are not limited to, transcribing speech in these languages, processing the transcribed text, and effectively handling the unique characteristics of text in low-resource languages. @xcite explains that the system performance is mostly dependent on the transcription result from the speech processing component. this is inline with @xcite as it states that the transcription text is a different domain than a normal text, which needs a transfer learning to improve the performance. numerous applications (amazon; microsoft) and research efforts have focused on speech deidentification systems in english @xcite @xcite and other languages such as french @xcite . however, none of these studies address low-resource languages like indonesian language, which suffer from a lack of annotated datasets and pre-trained models. this presents a significant problem, as such systems are highly language-dependent and may perform poorly when applied to languages that are either underrepresented in training data or fall outside the system's distribution. there are ways to improve the system performance with multiple low-resource handling methods. dai and adel ( 2020 ) proves that using simple augmentation on a low-resource data could improve the performance on named entity recognition (ner). this is because the variation sentence that it creates from augmentation can be learned as a new sentence by the language model. other than that, methods like weakly-supervision training can improve the robustness with low quality data that usually can be seen on low-resource language data. a method like @xcite to make model learn on so-called "predicted" false-negative data can be useful to increase the performance. our research aims to combine, adapt, and improve multiple method to develop a speech content de-identification system and data creation pipeline tailored for low-resource languages, with the goal of enabling similar systems for languages with resource levels similar to indonesian, ensuring privacy in speech-related contexts. ## proposed method to develop and improve the speech content deidentification system and methods, we need to create a dataset for the training and validation and a data processing pipeline that includes 3 main component, speech processing component, information extraction component, and masking component. after establishing the system, we imple-mented three optimization methods tailored to the data domain: training on audio transcription text, dataset augmentation, and weakly supervised learning. ## experimental setup for the experiment, we split the current dataset into 80:20 for training and testing. to create variations of the systems, we created tags as follows. to simulate a low-resource environment (distantly annotated data), we removed percentages of labels on the dataset based on the "missing label" variable. the variable varies from 0 to 0.8 with 0.2 steps, where 0 means the label is complete and 0.8 means 80% of the label is randomly missing. to make sure there were no random variables in the experiment, we performed the evaluation five times and averaged the results. to evaluate the speech content de-identification system, we utilized multiple evaluation metrics as follows: ## results this section summarizes the experimental results, including the asr evaluation, text component evaluation, reliability evaluation, error analysis, variant performance analysis, and overall system performance. ## conclusion we successfully developed a de-identification system for indonesian speech comprised of speech recognition, information extraction, and masking components. using a dataset without missing labels, the system achieved a recall of 69.2%, precision of 79.4%, and an f1 score of 73.9%. when tested on a dataset with 60% labeled data, the performance showed a recall of 36.2%, precision of 63.0%, and an f1 score of 45.6%. however, on a dataset with only 20% labeled data, the system's performance dropped significantly, achieving a recall of 0.00%, precision of 0.20%, and an f1 score of 0.00%. the system's performance decreased with the percentage of labeled data, showing that the system gained its knowledge from the given data. the addition of various techniques into the baseline model resulted in improved performance. specifically, the combination of domain-specific transcription data, dataset augmentation, and weakly supervised learning methods yielded a significant performance boost. the de-identification system incorporating all techniques achieved a recall of 75.2%, precision of 75.6%, and an f1 score of 75.3% on perfect data; a recall of 69.4%, precision of 77.0%, and an f1 score of 72.9% on 60% labeled data; and a recall of 47.8%, precision of 70.0%, and an f1 score of 56.7% on 20% labeled data. these results indicate a significant improve-