C opy NE : Better Contextual ASR by Copying Named Entities (2024)

Abstract: End-to-end automatic speech recognition (ASR) systems have made significant progress in general scenarios. However, it remains challenging to transcribe contextual named entities (NEs) in the contextual ASR scenario. Previous approaches have attempted to address this by utilizing the NE dictionary. These approaches treat entities as individual tokens and generate them token-by-token, which may result in incomplete transcriptions of entities. In this paper, we treat entities as indivisible wholes and introduce the idea of copying into ASR. We design a systematic mechanism called CopyNE, which can copy entities from the NE dictionary. By copying all tokens of an entity at once, we can reduce errors during entity transcription, ensuring the completeness of the entity. Experiments demonstrate that CopyNE consistently improves the accuracy of transcribing entities compared to previous approaches. Even when based on the strong Whisper, CopyNE still achieves notable improvements.

## introduction end-to-end automatic speech recognition (asr) systems have achieved impressive performance in general scenarios @xcite @xcite . however, in the contextual asr scenario where speech often contains numerous contextual entities, it remains a challenge for asr systems to get accurate transcriptions @xcite . for instance, when utilizing personal voice assistants like siri or alexa, it is common to encounter contextual entities such as personal names, place names, and organization names. asr models trained solely on speech-text data often struggle to transcribe these personalized entities due to their infrequent occurrence in the training set @xcite . since contextual entities always cover a wealth of semantic information. it is important to improve the accuracy of transcribing entities for downstream natural language processing tasks such as information retrieval and spoken language understanding @xcite . recently, researchers have started leveraging the information of textual modality as additional contextual knowledge to help contextual asr. the most typical approach, premised on the assumption that entities are already known before, use a contextual named entity (ne) dictionary as contextual knowledge @xcite @xcite @xcite . two representative approaches are "contextual listen, attend and spell" (clas) @xcite and contextual bias attention (cba) @xcite . clas employs the knowledge of the dictionary to aid the prediction of each token. they use dictionary representation as extra inputs for token prediction in the decoder. the decoder attends to each entity, and the dictionary representation is an aggregated representation of all entities, weighted by the attention scores. cba extends clas and uses an extra training loss. the loss explicitly makes use of the attention scores and force the model attend to a proper entity in the dictionary if the token to be predicted is related with the entity. previous methods have achieved considerable improvements, especially in transcribing entities. however, they all treat entities as individual tokens. these models utilize contextual knowledge to aid in predicting independent tokens without considering the role of these tokens in constituting a complete entity. in other words, a multi-token entity is broken into isolated tokens during decoding. we argue that this is problematic. for instance, model may erroneously generate the subsequent tokens of an entity, despite correctly producing the preceding tokens. as shown in figure 1 , when transcribing the speech "他来自安徽铜陵" (he comes from anhui tongling), an incorrect output of "他来自安徽铜 铃" (he comes from anhui copper bell) is obtained. despite the model's awareness of the location entity "铜陵" in the ne dictionary and its accurate prediction of the first token "铜", it mistakenly transcribes "陵" (ling) as "铃" (bell) during the tokenlevel prediction process. this occurs because the model predicts tokens independently, neglecting the integrity of the token span as a complete entity. furthermore, "陵" and "铃" share the same pronunciation "líng", with "铃" being a more frequently occurring token in the training data. consequently, the model tends to generate the wrong token "铃". in this paper, we propose a new approach for contextual asr called copyne. unlike previous approaches, we view entities as indivisible wholes. to the best of our knowledge, we are the first to introduce the idea of copying into asr. we design a systematic and effective mechanism to copy entities from a dictionary. specifically, copyne uses a copy loss that guides the model to copy the correct entity from the dictionary. during inference, our copyne has the flexibility to either predict a token from the token vocabulary or copy an entity from the ne dictionary at each decoding step. by copying multiple tokens simultaneously, we can alleviate errors within the entity, thus ensuring the token span as a complete entity. experiments on chinese aishell @xcite , st-cmds 1 , and english eng @xcite datasets show that our copyne achieves significant improvements across all scenarios, particularly in the contextual asr scenario. compared to previous methods using dictionary, copyne achieves relative reductions in cer of 13.5% and 1 https://www.openslr.org/38/ 20.8% on aishell and st-cmds in the contextual scenario. notably, copyne shows more remarkable improvements when it comes to transcribing entities, with relative reductions of 55.4% and 53.9% in the ne-cer metric on aishell and stcmds. moreover, when based on whisper @xcite and evaluated in its domain of expertise, eng dataset, copyne still achieves an impressive 6.4% and 16.8% relative reductions in wer and ne-wer. we will release our codes, configurations, and models at https://github. com/zslin177/copyne . ## the ctc-transformer model in this work, we build our proposed approach on the end-to-end ctc-transformer model, since it is the most widely used and achieves competitive performance in the asr field @xcite @xcite @xcite . however, it is worth noting that our idea can be applied to other asr approaches as well. the ctc-transformer is built upon the seq-toseq transformer @xcite , with a connectionist temporal classification (ctc) layer added after the audio encoder. as shown in figure 2 , it takes a sequence of acoustic frames @xmath0, ..., x t ) as input and generates the corresponding transcription text @xmath1, ..., y u ) as output. the model consists of two main components: an encoder and a decoder. first, the encoder encodes the acoustic frames x into hidden states @xmath2, ..., h t ). then, the decoder predicts the target sequence y in an auto-regressive manner. at each decoding step u, the decoder predicts the next target token y u+1 based on the encoder's output h and the previously predicted tokens @xmath3, ..., y u ). this process is expressed as follows: here, d @xmath4, and p (y u+1 |@xmath5. @xmath6, where v is the token vocabulary, and |v| is the size of the vocabulary. x audio encoder h decoder du h ctc layer @xmath7: 安1 ān 徽2 huī 铜3 tóng 陵4 líng . . . dot-product attention ne encoder z @xmath8. . . y: ltrans e2 e0 e1 e0 . . . σ: lcopy figure 3: our copyne model. the loss of transformer, l trans (y), comes from minimizing the negative log probability of y. as commonly used in previous works, the ctc loss is also applied here. ctc aligns each acoustic frame with a token from left to right. for a given target sequence y, there may be multiple valid alignments. the ctc loss is derived from maximizing the sum of these valid alignments, and has been proved to be able to enhance the representational capacity of the audio encoder @xcite . finally, the overall loss is the a weighted sum of the l trans (y) and l ctc (y), as follows: in inference, the model selects the most probable transcription using beam search as follows: here, there are many ways to use scores in decoding, such as combining ctc scores and transformer scores as in training, or using ctc-prefix beam search followed by re-scoring with transformer to select the optimal result. to compare with most previous works, we use the simplest decoding strategy, as shown in equation 6. ## our copyne model this section describes our proposed copyne model. the basic idea is that the model incorporates a contextual ne dictionary as external knowledge and can choose to directly copy nes from the dictionary. we design a systematic framework to implement the idea. during training, a copy loss is designed to encourage the model to copy corresponding entities from the dictionary. during inference, at each generation step, the model can either predict a single token from token vocabulary or directly copy a entity from the given dictionary. ## conclusion in this paper, we consider entities as indivisible elements and introduce a copy mechanism into asr for the first time to assist in transcribing entities. we devise a systematic copy framework that can copy all the tokens of an entity from the ne dictionary at once, preserving the token span as a complete entity. our approach demonstrates substantial improvements on both english and chinese datasets. in summary, copyne represents a significant advancement in contextual asr, providing a promising direction in this field.