Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing (2023)

Abstract: Sequence-to-Sequence (S2S) models have achieved remarkable success on various text generation tasks. However, learning complex structures with S2S models remains challenging as external neural modules and additional lexicons are often supplemented to predict non-textual outputs. We present a systematic study of S2S modeling using contained decoding on four core tasks: part-of-speech tagging, named entity recognition, constituency, and dependency parsing, to develop efficient exploitation methods costing zero extra parameters. In particular, 3 lexically diverse linearization schemas and corresponding constrained decoding methods are designed and evaluated. Experiments show that although more lexicalized schemas yield longer output sequences that require heavier training, their sequences being closer to natural language makes them easier to learn. Moreover, S2S models using our constrained decoding outperform other S2S approaches using external resources. Our best models perform better than or comparably to the state-of-the-art for all 4 tasks, lighting a promise for S2S models to generate non-sequential structures.

## introduction sequence-to-sequence (s2s) models pretrained for language modeling (plm) and denoising objectives have been successful on a wide range of nlp tasks where both inputs and outputs are sequences @xcite @xcite . however, for non-sequential outputs like trees and graphs, a procedure called linearization is often required to flatten them into ordinary sequences @xcite @xcite @xcite , where labels in non-sequential structures are mapped heuristically as individual tokens in sequences, and numerical properties like indices are either predicted using an external decoder such as pointer networks @xcite or cast to additional tokens in the vocabulary. while these methods are found to be effective, we hypothesize that s2s models can learn complex structures without adapting such patches. to challenge the limit of s2s modeling, bart @xcite is finetuned on four tasks without extra decoders: part-of-speech tagging (pos), named entity recognition (ner), constituency parsing (con), and dependency parsing (dep). three novel linearization schemas are introduced for each task: label sequence (ls), label with text (lt), and prompt (pt). ls to pt feature an increasing number of lexicons and a decreasing number of labels, which are not in the vocabulary (section 3). every schema is equipped with a constrained decoding algorithm searching over valid sequences (section 4). our experiments on three popular datasets depict that s2s models can learn these linguistic structures without external resources such as index tokens or pointer networks. our best models perform on par with or better than the other stateof-the-art models for all four tasks (section 5). finally, a detailed analysis is provided to compare the distinctive natures of our proposed schemas (section 6). 1 2 related work s2s @xcite architectures have been effective on many sequential modeling tasks. conventionally, s2s is implemented as an encoder and decoder pair, where the encoder learns input representations used to generate the output sequence via the decoder. since the input sequence can be very long, attention mechanisms @xcite focusing on particular positions are often augmented to the basic architecture. with transfer-learning, s2s models pretrained on large unlabeled corpora have risen to a diversity of new approaches that convert language problems into a text-to-text format @xcite @xcite @xcite . among them, tasks most related to our work are linguistic structure predictions using s2s, pos, ner, dep, and con. pos has been commonly tackled as a sequence tagging task, where the input and output sequences have equal lengths. s2s, on the other hand, does not enjoy such constraints as the output sequence can be arbitrarily long. therefore, s2s is not as popular as sequence tagging for pos. prevailing neural architectures for pos are often built on top of a neural sequence tagger with rich embeddings @xcite and conditional random fields @xcite . ner has been cast to a neural sequence tagging task using the iob notation @xcite over the years, which benefits most from contextual word embeddings @xcite . early s2s-based works cast ner to a text-to-iob transduction problem @xcite @xcite , which is included as a baseline schema in section 3.2. @xcite augment pointer networks to generate numerical entity spans, which we refrain to use because the focus of this work is purely on the s2s itself. most recently, @xcite propose the first template prompting to query all possible spans against a s2s language model, which is highly simplified into a one-pass generation in our pt schema. instead of directly prompting for the entity type, @xcite propose to generate its concepts first then its type later. their two-step generation is tailored for few-shot learning, orthogonal to our approach. moreover, our prompt approach does not rely on non-textual tokens as they do. con is a more established task for s2s models since the bracketed constituency tree is naturally a linearized sequence. top-down tree linearizations based on brackets @xcite or shift-reduce actions @xcite rely on a strong encoder over the sentence while bottom-up ones @xcite can utilize rich features from readily built partial parses. recently, the in-order traversal has proved superior to bottom-up and top-down in both transition @xcite and s2s @xcite ) constituency parsing. most recently, a pointer networks augmented approach @xcite is ranked top among s2s approaches. since we are interested in the potential of s2s models without patches, a naive bottom-up baseline and its novel upgrades are studied in section 3.3. dep has been underexplored as s2s due to the linearization complexity. the first s2s work maps a sentence to a sequence of source sentence words interleaved with the arc-standard, reduce-actions in its parse @xcite , which is adopted as our lt baseline in section 3.4. @xcite introduce a stack-based multi-layer attention mechanism to leverage structural linguistics information from the decoding stack in arc-standard parsing. arc-standard is also used in our ls baseline, however, we use no such extra layers. apart from transition parsing, @xcite directly predict the relative head position instead of the transition. this schema is later extended to multilingual and multitasking by @xcite . their encoder and decoder use different vocabularies, while in our pt setting, we re-use the vocabulary in the s2s language model. s2s appears to be more prevailing for semantic parsing due to two reasons. first, synchronous context-free grammar bridges the gap between natural text and meaning representation for s2s. it has been employed to obtain silver annotations @xcite , and to generate canonical natural language paraphrases that are easier to learn for s2s @xcite . this trend of insights viewing semantic parsing as prompt-guided generation @xcite and paraphrasing @xcite has also inspired our design of pt. second, the flexible input/output format of s2s facilitates joint learning of semantic parsing and generation. latent variable sharing @xcite and unified pretraining @xcite are two representative joint modeling approaches, which could be augmented with our idea of pt schema as a potentially more effective linearization. our finding that core nlp tasks can be solved using lt overlaps with the translation between augmented natural languages @xcite . however, we take one step further to study the impacts of textual tokens in schema design choices. our constrained decoding is similar to existing work @xcite @xcite . we craft constrained decoding algorithms for our proposed schemas and provide a systematic ablation study in section 6.1. ## schemas this section presents our output schemas for pos, ner, con, and dep in ## decoding strategies to ensure well-formed output sequences that match the schemas (section 3), a set of constrained decoding strategies is designed per task except for con, which is already tackled as s2s modeling without constrained decoding @xcite . formally, given an input x and any partial @xmath0, a constrained decoding algorithm defines a function nexty returning the set of all possible values for y i that can immediately follow @xmath1. for brevity, subwords handling is separately explained in section 5. ## experiments for all tasks, bart-large @xcite is finetuned as our underlying s2s model. we also tried t5 @xcite , although its performance was less satisfactory. every model is trained three times using different random seeds and their average scores and standard deviations on the test sets are reported. our models are experimented on the ontonotes 5 @xcite each token is independently tokenized using the subword tokenizer of bart and merged into an input sequence. the boundary information for each token is recorded to ensure full tokens are generated in lt and pt without broken pieces. to fit in the positional embeddings of bart, sentences longer than 1,024 subwords are discarded, which include 1 sentence from the penn treebank 3 training set, and 24 sentences from the ontonotes 5 training set. development sets and test sets are not affected. ## conclusion we aim to unleash the true potential of s2s models for sequence tagging and structure parsing. to this end, we develop s2s methods that rival state-of-the-art approaches more complicated than ours, without substantial task-specific architecture modifications. our experiments with three novel prompting schemas on four core nlp tasks demonstrated the effectiveness of natural language in s2s outputs. our systematic analysis revealed the pros and cons of s2s models, appealing for more exploration of structure prediction with s2s. our proposed s2s approach reduces the need for many heavily engineered task-specific architectures. it can be readily extended to multi-task and few-shot learning. we have a vision of s2s playing an integral role in more language understanding and generation systems. the limitation of our approach is its relatively slow decoding speed due to serial generation. this issue can be mitigated with non-autoregressive generation and model compression techniques in the future.