When Generative Adversarial Networks Meet Sequence Labeling Challenges (2024)

Abstract: The current framework for sequence labeling encompasses a feature extractor and a sequence tagger. This study introduces a unified framework named SLGAN, which harnesses the capabilities of Generative Adversarial Networks to address the challenges associated with Sequence Labeling tasks. SLGAN not only mitigates the limitation of GANs in backpropagating loss to discrete data but also exhibits strong adaptability to various sequence labeling tasks. Unlike traditional GANs, the discriminator within SLGAN does not discriminate whether data originates from the discriminator or the generator; instead, it focuses on predicting the correctness of each tag within the tag sequence. We conducted evaluations on six different tasks spanning four languages, including Chinese, Japanese, and Korean Word Segmentation, Chinese and English Named Entity Recognition, and Chinese Part-of-Speech Tagging. Our experimental results illustrate that SLGAN represents a versatile and highly effective solution, consistently achieving state-of-the-art or competitive performance results, irrespective of the specific task or language under consideration.

## introduction generative adversarial networks (gans) have achieved remarkable success in the realm of computer vision. nonetheless, their applicability in the field of natural language processing (nlp) has been limited due to the discrete nature of natural language text. this discretization challenge arises from the inability to directly back-propagate gradients from the discriminator to the discrete data, thereby impeding the integration of gans into nlp tasks. while alternative approaches, such as reinforcement learning (rl) @xcite methodologies or smooth approximation policies @xcite , have been proposed to circumvent this limitation, the predominant utilization of gans in nlp has been confined to text generation tasks, including summarization, poetry generation, dialog systems, and machine translation. notably, there are scarce applications in sequence labeling tasks, primarily due to the inherent distribution disparities between natural language text and sequence labels. furthermore, existing approaches for sequence labeling predominantly employ a pre-trained model and sequence tagger to model data distributions and subsequently produce sequences of tags. the efficacy of these methods is significantly constrained by the quality of the training data and the inherent limitations of the network's learning capabilities. it is well-recognized that the fundamental principle of generative adversarial networks (gans) revolves around harnessing adversarial dynamics between the discriminator and the generator, with the ultimate aim of enhancing the generator's data generation performance through the critical feedback loop provided by the discriminator. leveraging this intrinsic strength of gans, we introduce a gan-based framework and introduce a specialized discriminator in a bystander role. the discriminator's role is to guide the existing sequence labeling model, with the overarching objective of elevating the quality of the training process. we present a unified framework for sequence labeling, denoted as slgan, which integrates generative adversarial network (gan) principles. sl-gan enhances the training process of the sequence labeling model by incorporating a purposefully designed discriminator. the schematic representation of the slgan framework is shown in figure 1 , which comprises two pivotal constituents: the generator and the discriminator. we have constructed a generator, which relies on a pre-trained bert (kenton and toutanova, 2019) model and a conventional sequence tagger crf @xcite , to produce the sequence of tags. consequently, the loss function for the generator comprises two main components. one component stems from the intrinsic supervised information of the generator, while the other component is derived from the loss feedback originating from the discriminator. in contrast to the discriminator in the standard gan, which is primarily responsible for distinguishing between generated and real data distributions, the discriminator in our slgan framework is explicitly tailored to evaluate the correctness of each tag within the tag sequence. this evaluation is carried out by observing both the textual content and the accompanying labels simultaneously. the discriminator in our framework simultaneously processes two distinct data streams: (i) a composite data stream consisting of textual information combined with labels generated by the generator. (ii) another composite data stream comprises textual information and ground truth labels. the fusion of text and labels is facilitated through an attention layer. the loss incurred from the first data stream is back-propagated into the system to continuously refine the generator's output, striving for high-quality label generation while concurrently intensifying the complexity of the discrimination task. in summary, slgan leverages the supervised guidance provided by the auxiliary discriminator to train an improved generator for sequence labeling tasks. ## related work generative adversarial networks (gans) have achieved remarkable success in the field of computer vision @xcite @xcite and image generation @xcite . through the process of adversarial training, where the generator engages in a competitive learning paradigm with the discriminator, the generator generates data that closely approximates the distribution of real-world data. nevertheless, to the best of our knowledge, the effective application of gans in the domain of natural language processing (nlp) has been limited due to the inherently discrete nature of textual data. to tackle this challenge, the work proposed by @xcite and long short-term memory (lstm) networks in their adversarial training approach, aimed at generating realistic textual content. notably, they replaced the traditional gan objective with a feature distribution matching criterion during the generator's training process. on one hand, researchers introduced a method to handle the non-differentiability of the multinomial distribution by replacing the softmax function with the gumbel-softmax distribution, which offers a continuous approximation of the multinomial distribution (kusner and hernández-lobato, 2016). on the other hand, researchers suggested furnishing the discriminator with the intermediate vector generated by the generator, as opposed to the entire sequence output @xcite . it is noteworthy that gans primarily center their applications around text generation tasks, while their participation in the domain of sequence labeling tasks remains limited. typically, within the sequence labeling framework, an initial feature extractor is employed, such as lstm or cnn @xcite architectures. subsequently, a sequence tagger, notably conditional random field (crf) networks @xcite , is utilized to assign tags to each character. with the advancement of pre-trained language models like bert (kenton and toutanova, 2019) and elmo @xcite , a gradual shift has occurred, replacing lstm and cnn with these potent models. electra @xcite was proposed in an economically efficient way, by replacing the masked language modeling (mlm) task in bert with a task focused on detecting replaced tokens. electra engaged in training a discriminator to predict whether each token is being substituted with a sample generated by the model. slgan draws inspiration from the elec-tra. the distinguishing factor between slgan and electra lies in their objectives. slgan's core emphasis is on optimizing performance in sequence labeling tasks, whereas electra aims to establish a robust pre-trained model. after the pre-training phase, the authors discarded the gen-erator, only retaining the discriminator component, which represents the electra model. consequently, electra is a discriminator. differing from electra, slgan, in line with the established paradigm of gans and their derivative models, pursues the goal of enhancing generator quality. ## proposed method the architecture of the slgan framework is illustrated in figure 1 . it is important to note that two distinct bert models are employed within the generator and discriminator components. in both instances, the role of the bert model remains consistent, primarily focused on obtaining representations of the input sentences. the input sentences undergo independent processing within the respective bert models, yielding two distinct streams of input representations. the initial input stream is dedicated to the task of generating tag sequences within the generator, while the second stream is specifically utilized for participating in attention operations. given that the input sequences and the predicted labels reside within disparate data distributions, an attention layer is deployed to integrate these distinct data distributions. specifically, the logits generated by the generator and those derived from the ground truth labels are individually merged with the original input representations via the attention layer. the logits from the generator and those from the conversion of ground truth labels are individually integrated with the original input representations through the use of the attention layer. it is noteworthy that the discriminator remains agnostic to the data's source, be it the generator or the original input data. the primary mission of the discriminator is to evaluate the accuracy of each tag within the tag sequences within both data streams. ## experiments the evaluation encompasses a diverse set of tasks, spanning in-domain chinese word segmentation (cws), cross-domain cws, korean word segmentation (kws), japanese word segmentation (jws), chinese named entity recognition (ner), english ner, and chinese part-of-speech (pos) tagging. these tasks encompass four distinct languages and are evaluated across fourteen datasets. to illustrate the specifics of our experimental setup, for the cws and chinese ner tasks, we employ bertbase-chinese as the feature extractor. for jws and kws tasks, bert_multilingual is employed, serving as the feature extraction layer. for the english ner experiments, the bert-base-cased is utilized. it is noteworthy that a consistent "bmes" tagging schema is applied uniformly across all experiments to ensure a standardized evaluation framework. ## conclusion slgan demonstrates its effectiveness in addressing sequence labeling tasks by leveraging gan architecture. it offers a universal and efficient solution for tasks such as ws, ner, and pos tagging. slgan's performance is evaluated across six tasks spanning four languages, consistently achieving sota or near sota results in all experiments. notably, slgan excels in handling oov words, showcasing its sota performance in terms of r_oov. furthermore, slgan extends its applicability to cross-domain cws tasks. moreover, slgan's impressive performance in ner and pos tagging tasks reiterates its versatility across different labeling tasks, transcending language barriers.