Automatic Named Entity Obfuscation in Speech (2023)

Abstract: Sharing data containing personal information often requires its anonymization, even when consent for sharing was obtained from the data originator. While approaches exist for automated anonymization of text, the area is not as thoroughly explored in speech. This work focuses on identifying, replacing and inserting replacement named entities synthesized using voice cloning into original audio thereby retaining prosodic information while reducing the likelihood of deanonymization. The approach employs a novel named entity recognition (NER) system built directly on speech by training HuBERT (Hsu et al, 2021) using the English speech NER dataset (Yadav et al, 2020). Name substitutes are found using a masked language model and are synthesized using text to speech voice cloning (Eren and team, 2021), upon which the substitute named entities are re-inserted into the original text. The approach is prototyped on a sample of the LibriSpeech corpus (Panyatov et al, 2015) with each step evaluated individually.

## introduction privacy concerns, particularly where an individual could be identified, preclude sharing and therefore automatic exploitation of many data sources. anonymization, the removal of identifying information, has been automated for text @xcite , including large scale applications such as in clinical @xcite or legal settings @xcite , with off-the-shelf systems having reported performance of 90+% @xcite . to minimize the risk of re-identification, obfuscation -replacing identifying information with a different substitute of the same type -has been explored as an alternative to replacing identifying information with a generic marker @xcite . the main focus in speech has been on voice anonymization, which may not be a problem with speaker consent, with the removal of identifying information receiving less attention. to our knowledge, this is the first prototype to perform named entity obfuscation directly, in the original speaker's voice. aside from voice cloning, it explores a named entity recognition approach based directly on audio signal and uses language model masking to find appropriate substitutions. recent advances in speech models, particularly the inclusion of language models within the speech model itself (e.g. hubert @xcite ) gives models greater insight into expected contexts. previous work on named entity recognition (ner) in speech frequently employs a two step approach, transcribing speech first, followed by the application of existing named entity techniques @xcite . however, this process has the potential to compound errors as errors in transcription will increase the probability of error in ner. we suggest that the addition of language models into the speech model gives these sufficient power to perform ner directly, and therefore that transcribing (automatic speech recognition, asr) and ner can be separated, and used to provide a confidence measure in their performance. divided, the two do not propagate errors in the same way; in fact, treating asr and ner separately allows one to fix (some of the) errors of the other. the proposed second (final) asr pass merely produces a confidence value in the result to decide whether a manual check should be performed. the success of few shot learning, where a limited number of examples is used to generalize a pre-trained deep learning model to a new situation, for text-to-speech -and specifically voice cloning @xcite -enables an alternative, equivalent but different, entity to be inserted in the audio signal in place of the original while preserving the prosody information throughout. while large databases of potential replacement entities can be used to select a substitution, these may not preserve necessary properties (such as gender). al-ternatively, word embeddings have been used to suggest close (in the multi-dimensional space) alternatives @xcite , however these can suffer from the same drawback. we propose using a more contextualized alternative to word embeddings, a masked language model @xcite , where the model is trained by hiding (masking) words and predictions of the original word are made based on their context. this work makes the following contributions: (1) a complete obfuscation pipeline for names in speech foot_0 , (2) a named entity recognizer built directly on speech without requiring text transcription first, (3) alternative (obfuscated) entity replacement selection via masking language model, and (4) confidence annotated system output, allowing for manual correction and / or selection of shareable instances. section 2 contains the methodology with results in section 3. section 4 presents the conclusions and future work. ## methodology the steps of the overall pipeline, which takes in an audio file and produces an obfuscated audio file along with a confidence value, can be found in figure 1 . the approach comprises of three main parts: 1) identification of named entities (nes) in the audio, 2) finding an equivalent alternative for the original nes, and 3) reconstructing the original audio to incorporate the replacement nes. the reconstructed audio can further be used to obtain a confidence value. ## conclusion the prototype described shows the steps of an obfuscation pipeline for speech, which results in substituted person named entities uttered in the original speakers voice and replaced in the original audio signal. the prototype makes use of a named entity recognizer built directly on top of audio input, and employs masked language models to generate the substituted entity. it offers an end-to-end automatic solution enabling the sharing of speech with identifying information removed. the resulting obfuscated speech remains in the original speaker's voice, allowing for the application of traditional speaker anonymization approaches to mask the speaker's identity. the original prosody can be protected by applying a transformation such as waveform change, offering a significant advantage over a technique which generates a complete obfuscated transcription (instead of splicing an obfuscated entity into original speech).