D iffusion SL : Sequence Labeling via Tag Diffusion Process (2023)

Abstract: Sequence Labeling (SL) is long-standing in Natural Language Processing (NLP). Traditionally, discriminative models have been widely used to capture the conditional distribution of sequence tags, rather than generative models. In this paper, we present DiffusionSL, a framework that utilizes a conditional discrete diffusion model for generating discrete tag data, resulting in a Tag Diffusion Process. We treat the natural language sequence as the conditional signal and the sequence tags as the generation target, iteratively refining the noisy tags to obtain clean ones. To address the discreteness issue, we propose the Bit-Tag Converter (BTConverter) to model the target in continuous data space. Furthermore, we introduce the Bit Diffusion Transformer (BitDiT) to model the process of noise elimination. Leveraging the powerful iterative refinement capability of the diffusion model, DiffusionSL achieves superior performance against previous state-of-the-art (SOTA) baselines and outperforms gpt-3.5-turbo significantly across multiple benchmark datasets and various tasks.

## introduction sequence labeling (sl) is a basic paradigm in the natural language processing (nlp) field, which assigns a predefined label to every meaningful unit (e.g., word or character) in a given sequence @xcite @xcite . many nlp tasks belong to this category, such as named entity recognition (ner) @xcite @xcite @xcite , chinese word segmentation (cws) @xcite @xcite , part-of-speech (pos) tagging @xcite . most current methods tackled sl in a discriminative manner @xcite , which usually employed a language model @xcite to encode the sentence and added a token-level classifier to capture the conditional tag distribution @xcite @xcite @xcite . recently, generative pre-trained language models have demonstrated their versatility in various nlp tasks @xcite . in consequence, researchers attempt to formulate sl as a sequence generation problem @xcite @xcite . this paradigm offers the advantage of flexible generation formats @xcite , allowing models to be trained to produce output in any desired format using teacher forcing. in these generative approaches, the input sentence is encoded first, and the label sequence is then decoded in an autoregressive manner, as depicted in figure 1(a) . nevertheless, the intrinsic token-by-token generation style of the autoregressive model results in inefficient inference, and the difference in conditional signal during training and inference leads to the exposure bias problem @xcite . to this end, this paper resorts to the diffusion probabilistic model (dpm) @xcite @xcite @xcite , renowned as one of the most powerful generative models in the field of artificial intelligence generated content (aigc) @xcite , especially in the realm of image synthesis @xcite @xcite . dpm gradually introduces noises into clean data through a predefined forward process and subsequently employs a denoising network in a reverse process to recover the original data from pure gaussian noise. the generation process operates in parallel. compared with an autoregressive decoder, encoder eu rejects german call to boycott british lamb it could naturally solve the exposure bias problem. furthermore, the diffusion model can generate the targets progressively to refine the contents step-bystep, thereby facilitating the correction of some mistakes during the generation process. in comparison, the conventional autoregressive generative model generates the targets at once. the aforementioned advantages contribute to the success of the diffusion model in generating data across various modalities @xcite @xcite . intuitively, we expect that dpm could also present superior performance on sl tasks. therefore, this paper proposes a novel framework called diffusionsl. the input sentence serves as the conditional signal, while the corresponding task-related tag sequence is the generation target. we call the entire diffusion process as the tag diffusion process. during the training phase, a fixed forward process is utilized to sequentially perturb the tags and a reverse diffusion process is learned to eliminate noise based on the noisy tags and the encoded sentence. in the inference phase, we sample noisy tags from a standard gaussian distribution and employ the well-trained denoising model to refine these tags into clean ones in an incremental manner (shown in figure 1(b) ). however, directly applying a dpm on sl task is challenging. classical dpm models the image data and gaussian noise in a continuous space. instead, tags in sl are discrete. a few research ef-forts have addressed this issue (e.g., text symbols @xcite , segmentation maps @xcite ). following them, this paper introduces a lightweight and flexible module named bit-tag converter (btconverter) to represent the tags as binary bits, which can be viewed as real numbers in continuous space. additionally, we propose the bit diffusion transformer (bitdit) to model the noise-removal reverse process, which enjoys faster convergence and stronger performance. currently, @xcite also introduces a diffusion model into ner, an important sl task. however, their approach only models the diffusion process on entity span boundaries, restricting itself to ner and impeding the potential for extension to other sl tasks. in summary, our contributions are as follows: ## preliminary the diffusion model is characterized as a latent generative model. it encompasses a forward process that involves the gradual introduction of noise to clean data x 0 . additionally, it includes a reverse process that incrementally eliminates the noise, thereby generating data following the original distribution. the forward process q(x t |x t-1 ) can be mathematically expressed as: here, n represents the gaussian distribution and β t represents the noise introduced at timestep t. the magnitude of the noise is governed by a noise schedule β = {β t } t @xmath0. generally, the noise intensity usually follows either a linear @xcite or cosine (nichol and dhariwal, 2021) function. in this study, we adopt the linear noise schedule. by employing the notations α @xmath1, we can deduce an equation to directly incorporate noise into x 0 , yielding the noisy data at any given timestep: additionally, the reverse process q(x t-1 |x t ) allows us to reconstruct samples from pure gaussian noise, which is not tractable directly. we can use a neural network p θ to estimate it: the conditional distribution of x t-1 given x t in reverse process could be obtained analytically as follows when x 0 is available: therefore, we train the µ θ to predict the μt . due to x @xmath2, we could get µ θ by predicting it directly or predicting ϵ, x 0 indirectly. the variance σ θ could be learned or fixed. we keep it fixed in this work. the training objective is the variational lower bound of the negative loglikelihoodlog p θ (x 0 ), which can be simplified to predict the clean data x 0 or injected noise ϵ. to expedite inference, it is possible to skip certain timesteps in the reverse process. by taking in the more noisy samples at timestep t i , we can generate the less noisy samples at timestep t i-1 : ## methodology the details of diffusionsl are illustrated in figure 2 . we will provide a comprehensive explanation of it in this section. firstly, we formally describe the overall procedure of the tag diffusion process employed by diffusionsl. secondly, we present a detailed description of two components we proposed to help modeling the tag diffusion process: btconverter and bitdit. lastly, we summarize the training and inference algorithms of diffusionsl. ## case study we visually show the denoising process details by one chinese ner case. in figure 4 , we input the sentence 中国中央致中国致公党十一大的贺词 (the cpc central committee's congratulatory message to the 11th national congress of the chinese zhigong party) as condition signal, then we generate the corresponding tags in ten decoding steps by iterative refinement. we also show the label accuracy of tags in the bottom for every step, which increases with the number of reverse steps. finally, we get the exactly correct ner tags of the input sentence at the final step. then, we can decode the desired entities (中共中央 (cpc central committee's), nt), (中国致公党十一大 (the 11th national congress of the chinese zhigong party), nt) by the semantics of these tags. to the best of our knowledge, we are the first to introduce this iterative denoising process and shift the traditional classification paradigm to the non-autoregressive generation paradigm for sequence labeling. ## conclusion in this paper, we cast the sequence labeling (sl) task as a conditional generation problem. furthermore, we introduce a novel framework diffusionsl to model sl using a non-autoregressive stepwise generation approach. meanwhile, we propose bt-converter to handle the discreteness problem and bitdit to model the progressive noise-removal process. experimental results indicate that our model performs better on ner, cws, and pos tagging compared to previous strong baselines, thus revealing the superiority of diffusionsl. our work is pioneer research to cast natural language understanding task as a non-autoregressive corresponding generation task. we hope dissusionsl could be used in more tasks that could be formulated as sequence labeling and leave this as the future work, e.g. semantic role labeling, text chunking.