D iffusion NER : Boundary Diffusion for Named Entity Recognition (2023)

Abstract: In this paper, we propose DiffusionNER, which formulates the named entity recognition task as a boundary-denoising diffusion process and thus generates named entities from noisy spans. During training, DiffusionNER gradually adds noises to the golden entity boundaries by a fixed forward diffusion process and learns a reverse diffusion process to recover the entity boundaries. In inference, DiffusionNER first randomly samples some noisy spans from a standard Gaussian distribution and then generates the named entities by denoising them with the learned reverse diffusion process. The proposed boundary-denoising diffusion process allows progressive refinement and dynamic sampling of entities, empowering DiffusionNER with efficient and flexible entity generation capability. Experiments on multiple flat and nested NER datasets demonstrate that DiffusionNER achieves comparable or even better performance than previous state-of-the-art models.

## introduction named entity recognition (ner) is a basic task of information extraction @xcite , which aims to locate entity mentions and label specific entity types such as person, location, and organization. it is fundamental to many structured information extraction tasks, such as relation extraction @xcite and event extraction @xcite . most traditional methods @xcite formulate the ner task into a sequence labeling task by assigning a single label to each token. to accommodate the nested structure between entities, some methods @xcite achieved his master's degree from the university of sydney ñamed entities noisy spans forward process reverse process denoise ( 8, 9 | gpe ) ( 16, 18 | org ) ( 1, 9 | per ) premier of the western canadian province of british columbia { { ( 4, 5 | loc ) ( 3, 9 | gpe ) figure 1: boundary diffusion in named entity recognition. the fixed forward diffusion process adds gaussian noise to the entity boundaries at each timestep, and the noisy boundaries recover their original state by denoising with the learnable reverse diffusion process. for inference, the reverse diffusion process generates entity boundaries and performs entity typing based on the noisy spans sampled from the gaussian distribution. inspired by this, we present diffusionner, a new generative framework for named entity recognition, which formulates ner as a denoising diffusion process @xcite on entity boundaries and generates entities from noisy spans. as shown in figure 1 , during training, we add gaussian noise to the entity boundaries step by step in the forward diffusion process, and the noisy spans are progressively denoised by a reverse diffusion process to recover the original entity boundaries. the forward process is fixed and determined by the variance schedule of the gaussian markov chains, while the reverse process requires learning a denoising network that progressively refines the entity boundaries. for inference, we first sample noisy spans from a prior gaussian distribution and then generate entity boundaries using the learned reverse diffusion process. empowered by the diffusion model, diffusion-ner presents three advantages. first, the iterative denoising process of the diffusion model gives diffusionner the ability to progressively refine the entity boundaries, thus improve performance. second, independent of the predefined number of noisy spans in the training stage, dif-fusionner can sample a different number of noisy spans to decode entities during evaluation. such dynamic entity sampling makes more sense in real scenarios where the number of entities is arbitrary. third, different from the autoregressive manner in generation-based methods, diffusion-ner can generate all entities in parallel within several denoising timesteps. in addition, the shared encoder across timesteps can further speed up inference. we will further analyze these advantages of diffusionner in § 6.2. in summary, our main contributions are as follows: ## preliminary in diffusion models, both the forward and reverse processes can be considered a markov chain with progressive gaussian transitions. formally, given a data distribution x 0 ∼ q (x 0 ) and a predefined variance schedule {β 1 , . . . , β t }, the forward process q gradually adds gaussian noise with variance β @xmath0, 1) at timestep t to produce latent variables x 1 , x 2 , . . . , x t as follows: an important property of the forward process is that we can sample the noisy latents at an arbitrary timestep conditioned on the data x 0 . with the notation α t := 1β t and ᾱt := t @xmath1, we have: as ᾱt approximates 0, x t follows the standard gaussian distribution: p (x t ) ≈ n (x t ; 0, i). unlike the fixed forward process, the reverse process p θ (x 0:t ) is defined as a markov chain with learnable gaussian transitions starting at a prior p (x t ) = n (x t ; 0, i): ## method in this section, we first present the formulation of diffusion model for ner (i.e., the boundary denoising diffusion process) in § 4.1. then, we detail the architecture of the denoising network for boundary reverse process in § 4.2. finally, we describe the inference procedure of diffusionner in § 4.3. ## conclusion in this paper, we present diffusionner, a novel generative approach for ner that converts the task into a boundary denoising diffusion process. our evaluations on six nested and flat ner datasets show that diffusionner achieves comparable or better performance compared to previous stateof-the-art models. additionally, our additional analyses reveal the advantages of diffusionner in terms of inference speed, progressive boundary refinement, and dynamic entity sampling. overall, this study is a pioneering effort of diffusion models for extractive tasks on discrete text sequences, and we hope it may serve as a catalyst for more research about the potential of diffusion models in natural language understanding tasks.