Transforming Sequence Tagging Into A S eq2 S eq Task (2022)

Abstract: Pretrained, large, generative language models (LMs) have had great success in a wide range of sequence tagging and structured prediction tasks. Casting a sequence tagging task as a Seq2Seq one requires deciding the formats of the input and output sequences. However, we lack a principled understanding of the trade-offs associated with these formats (such as the effect on model accuracy, sequence length, multilingual generalization, hallucination). In this paper, we rigorously study different formats one could use for casting input text sentences and their output labels into the input and target (i.e., output) of a Seq2Seq model. Along the way, we introduce a new format, which we show to to be both simpler and more effective. Additionally the new format demonstrates significant gains in the multilingual settings – both zero-shot transfer learning and joint training. Lastly, we find that the new format is more robust and almost completely devoid of hallucination – an issue we find common in existing formats. With well over a 1000 experiments studying 14 different formats, over 7 diverse public benchmarks – including 3 multilingual datasets spanning 7 languages – we believe our findings provide a strong empirical basis in understanding how we should tackle sequence tagging tasks.

## introduction the advent of powerful generative language models (lms) such as t5 @xcite and mt5 @xcite have unlocked new opportunities. recent works have shown great success applying these models to a wide range of sequence tagging and structured prediction tasks @xcite @xcite . the key to leveraging these models is to cast the sequence tagging task as a suitable seq2seq task -by transforming the original input * corresponding author text and output labels into the input and target (i.e., output) of a seq2seq model. slot-filling example from @xcite input: add kent james to the disney soundtrack target: add [ kent james | artist ] to the [ disney | playlist ] soundtrack srl example from paolini et al. (2021) (@xmath0: the luxury auto maker last year [ sold ] 1,214 cars in the u.s. target: [ the luxury auto maker |subject ] [ last year | temporal ] sold [ 1,214 cars | object ] [ in the u.s. | location ] while the list of nlp problems tackled in this manner are diverse -named entity recognition, co-reference resolution, semantic role labeling (srl), slot-filling, entity and relation extraction to name a few -the seq2seq transformation used across different works is surprisingly similar (with slight variants). as seen from the examples in while exciting results have been achieved using this format, there has been no principled study understanding this format choice. towards that end, we make the following contributions in this paper: taggedspans format (as well as other similar formats) are highly susceptible to hallucinations @xcite , i.e., the proclivity of large lms adding, modifying, or deleting input words while generating the output sequence. while the taggedspans had hallucinated tokens in over 50% of examples in some settings, the proposed sentinel-based format is virtually hallucination-free (< 0.15% hallucinations). with well over a 1000 experiments involving 14 different formats, 7 datasets (3 multilingual), we believe this to be the most comprehensive study on this topic to-date. ## related work several recent papers have shown the strength of the large seq2seq lms for sequence tagging -in the context of named entity recognition @xcite @xcite , slot-labeling for semantic parsing @xcite @xcite , and semantic role labeling @xcite . @xcite proposed a general seq2seq framework for ner tasks, both for disjoint spans and nested overlapping spans. @xcite showed that seq2seq models can significantly outperform the encoder-only models for slot-filling tasks. @xcite converted a wide range of sequence tagging tasks (e.g., answer span extraction for question-answering) into seq2seq problems. all these methods, however, apply fairly similar text-to-text formats in that: they (1) prepend the input with a task prefix and leave rest of it mostly unchanged and (2) use the taggedspans format (or slight variants) for the target (output) sequence i.e., copy over input tokens and interlace with the tags for the associated spans. in this paper, we systematically evaluate this approach, and demonstrate its relative strengths and weaknesses across a large collection of benchmarks. recent papers investigated the impact of carefully hand-crafted input prompts for large language models and showed different prompt templates can significantly change the prediction accuracy, espe-cially in zero-shot or few-shot settings @xcite @xcite . for example, @xcite has demonstrated that different prompts describing classification labels can have large impact on the zero-shot classification accuracy. similar nuances are likely at play for output sequence formats as well, but there hasn't been enough systematic study to understand the impact of different output format choices. we present empirical evaluations of different input/output formats for sequence tagging tasks, and how they impact sequence tagging accuracy in english-only, multilingual, and zero-shot multilingual settings. ## the seq2seq transformation applying pretrained generative models on these sequence labeling / structured predictions problems requires creating the input and target strings from the original input sentences and labels. the examples in ## datasets to gain a comprehensive understanding of how these different formats perform, we ran experiments on 7 different public datasets -4 monolingual and 3 multilingual spanning 7 languages from different language groups. the monolingual datasets include: snips @xcite , mit movie trivia foot_2 , mit movie corpus, and mit restaurant corpus. the multilingual datasets include: matis (upadhyay et al., 2018) -a multilingual generalization of the atis dataset @xcite , mtop @xcite , and mtod @xcite . these datasets span different domains -such as cinema, dining, travel, dialog . . . -and contain different kinds of texts -including queries, reviews, and conversations. for instance, matis, snips, mtop, and mtod contain conversational queries or dialog-like texts, and involve slot labelling for semantic parsing. the mit movie trivia, mit movies, and mit restaurants datasets, on the other hand, are widely used for named entity recognition (ner). as seen from the statistics listed in ## empirical setting we studied different input/output formats using the mt5 pretrained models @xcite , which have been used to obtain state-of-the-art results on multiple public benchmarks. in particular, we used the base-sized model for the majority of our experiments as it enabled us to maximize our experimentation given our compute budget, while still providing for a very powerful initialization. however, results and trends largely stay the same for other model sizes as we discussed in sec 9. in all cases, we trained these models using the t5x foot_3 code for 5000 steps -selecting the checkpoint with the highest validation set sequence accuracy. as recommended in the t5 paper we used the default learning rate of 0.001 for all our experiments, and used a batch size of 128. we appended eos tokens at the ends of all input and target formats for consistency. lastly, we used an input sequence length of 128 and target sequence length of 256, while also using packing. the impact of training sequence length on accuracy and speed is discussed further in sec 9. to evaluate the different seq2seq formats, we primarily relied on two metrics: ## performance on english-benchmarks the first question we looked to answer was: how well do these seq2seq approaches perform compared with the previous encoder-only approaches?. to do so, as a baseline we used the previous benchmark setting mbert @xcite model -which is an encoder-only model having a similar size as the t5 base model's encoder. we evaluated the models on the simpler english-only setting to start with. the results on all 7 english benchmarks can be found in the next question we looked to answer was: how do the different basic formats described in sec 3 compare against each other? in particular, we wanted to understand whether not repeating the input tokens hurts performance. as seen from the third to sixth columns in these results also show that while the tag only approach is quite competitive, using either the sentinels or input tokens leads to better results. additionally, we also find that the input + tag is actually marginally better than the more commonly used tagged spans approach (though this gap is too narrow to be statistically significant). ## multilingual capabilities a major benefit of large lms is their generalization capabilities in the multilingual setting. in this section, we evaluated the different formats in the more challenging multilingual setting. specifically we ran experiments on 3 benchmarks (spanning 7 different languages) in two different settings: these gains are all the more noteworthy when compared to the performance of the tag only (no sentinels) and input+tag (input token instead of as seen in a few additional observations: (a) as on english ## performance of extractive methods as discussed in sec 3.5, certain applications may prefer extractive formats. to verify if our findings extend to these settings we also evaluated these extractive formats on the same datasets. tables 8 (zeroshot) and 9 (joint multilingual) summarize the results. a few key observations: ## robustness & efficiency beyond performance, efficiency and robustness of a model are equally important factors in practice. we focus on measuring these in this section. ## conclusion in this paper, we rigorously studied different input and output sequence formats for casting sequence tagging tasks as seq2seq problems. using experiments across 7 public benchmarks, we found that the formats of the input and output sequences can significantly impact performance. to remedy the shortcomings of the existing formats, we also introduced a new format that uses sentinel tokens. along with its variants, this new format proved to not only be simpler and more computationally efficient but also more accurate. the sentinel-based formats significantly outperform all other formats when it comes to multilingual generalization, with 30+% relative improvement in accuracy. while current formats are plagued by hallucination in a large percentage of examples, the new format rectifies this issue and is virtually hallucination free.