On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART (2022)

Abstract: Word ordering is a constrained language generation task taking unordered words as input. Existing work uses linear models and neural networks for the task, yet pre-trained language models have not been studied in word ordering, let alone why they help. We use BART as an instance and show its effectiveness in the task. To explain why BART helps word ordering, we extend analysis with probing and empirically identify that syntactic dependency knowledge in BART is a reliable explanation. We also report performance gains with BART in the related partial tree linearization task, which readily extends our analysis.

## introduction the task of word ordering @xcite @xcite , also known as linearization @xcite , aims to assign a valid permutation to a bag of words for a coherent sentence. while early work uses word ordering to improve the grammaticality of machine-generated sentences @xcite , the task subsequently manifests itself in applications such as discourse generation @xcite , machine translation @xcite , and image captioning @xcite . it plays a central role in linguistic realization @xcite of pipeline text generation systems. advances in word ordering are also relevant to retrieval augmented generation @xcite , with outputs additionally conditioned on retrieved entries, which can constitute a bag of words. word ordering can be viewed as constrained language generation with all inflected output words provided, which makes it more amenable for error analysis ( §3.4). the task can be extended to tree linearization @xcite or partial tree linearization @xcite with syntactic features as figure 1 : illustration of our approach for word ordering. bag-of-words inputs are first turned into pseudosequences with arbitrarily word permutations, and then fed to a sequence-to-sequence transformer for coherent outputs. intuitively, the rich knowledge in plms can readily help word ordering. however, the unordered bag-of-words inputs may seem incompatible to plms with sequential inputs. the role of plms in word ordering thus remains an interesting research question. we fill the research gap by empirically investigating bart @xcite , a pre-trained sequence-to-sequence transformer @xcite , as an instance of plms for word ordering. specifically, we assign an arbitrary permutation for the input bag of words to obtain a pseudo-sequence, and use sequence-to-sequence transformers to generate ordered outputs, as illustrated in figure 1 . as bart uses subword inputs @xcite instead of words @xcite @xcite , we implement constrained beam search using prefix trees ( §3.1; figure 2 ). results show that bart substantially improves word ordering compared to our transformer baseline, which already outperforms previous best ( §3.3). with transformer models (including bart), we further investigate consequences of two major modeling decisions, which remain unexamined in the literature. first, while all previous studies assume output sequences constrained within permutations of input words, @xcite eliminate such constraints. we find the latter leads to a consistent performance drop, which is further attributed to missing words in outputs, a phenomenon related to the coverage issue in machine translation @xcite . second, @xcite use conditional probability p(y|x) instead of the unconditional one p(y) of @xcite to score candidate output permutations y of input bag of words x. we provide the first fair comparison for the two approaches, and find that with small decoding beam, conditional models substantially outperform unconditional ones. interestingly, such an advantage fails to persist as we increase the beam size. our transformer word orderers may be sensitive to arbitrary word permutations in the input pseudosequence ( §3.6). recent studies @xcite show that transformers are relatively insensitive to word permutations in sequential inputs. they are more sensitive to local orders than global orders of input subwords on the glue benchmark @xcite . in contrast, we find that transformer (including bart) word orderers are relatively insensitive to both word and subword permutations in inputs. such result can be relevant to modeling unordered inputs with plms @xcite . we finally aim to explain why bart helps word ordering ( §4). analysis with probing @xcite provides speculated explanations for the utility of plms with the possession of numerous types of knowledge. however, for a reliable explanation, we need to identify the specific type of knowledge relevant to the task. in addition, the amount of the knowledge should be nontrivial in the plm. with a procedure based on feature importance @xcite and probing @xcite , we empirically identify that knowledge about syntactic dependency structure reliably explains why bart helps word ordering. our analysis can be readily extended to partial tree linearization @xcite , for which we also report performance gains with our models ( §5). ## related work word ordering modeling early work uses syntactic models @xcite and language models @xcite to rank candidate permutations of input words. @xcite and @xcite discuss their relative importance. syntactic models rank candidates with the probability of the jointly predicted parse tree. they can be linear models @xcite or neural networks @xcite with hand-crafted features. language models use the probability of the output sentence for ranking. early work uses statistical ngram models @xcite , which are later replaced by recurrent neural networks @xcite . most related to our work, @xcite and @xcite formulate word ordering as conditional generation. @xcite uses an lstm decoder with attention @xcite and an encoder degenerating to an embedding layer. @xcite stack self-attention @xcite layers as the encoder and use a decoder from pointer network @xcite . both encode bag-of-words inputs with permutation invariant word encoders. in contrast, we turn bagof-words inputs into subword pseudo-sequences and feed them to standard sequence-to-sequence models. instead of investigating features, prediction targets, and model architectures as in previous work, we focus on the utility of bart in the task. word ordering decoding early work relies on time-constrained best-first-search @xcite . as it lacks an asymptotic upper bound for time complexity @xcite , later work with syntactic models @xcite , language models @xcite , and conditional generation models @xcite adopt beam search for decoding. all previous work assumes an output space constrained to permutations of input words except for @xcite , who assume the output to be any sequences permitted by the vocabulary. however, the effect of such unconstrained output space is unexamined. we compare the difference between beam search with constrained and unconstrained output spaces. tasks related to word ordering word ordering was first proposed by @xcite as a surrogate for grammaticality test, and later formulated by @xcite as a standard task. a closely related task is commongen @xcite , which aims to generate a coherent sentence subjective to commonsense constraints given a set of lemmatized concept words. in contrast, word ordering is a constrained language modeling task given inflected output words. tree linearization @xcite is a related task with full dependency trees as inputs. dropping subsets of dependency arcs and part-of-speech tags results in partial tree linearization @xcite . further removing functional words and word inflections results in surface realization @xcite . different from commongen and surface realization, the provided output bag-of-words limit reliance on domain knowledge and reduce ambiguity in output, making word ordering a concentrated case for testing generic linguistic capacity @xcite ) of text generation models. in addition, word ordering requires no labeling in contrast to all these tasks. plms and non-sequential inputs plms with the transformer @xcite decoder are amenable for sequence generation @xcite . they have been used for sequence generation tasks with non-sequential inputs, such as amr-to-text @xcite , @xcite , and common-gen @xcite . typically, non-sequential inputs are turned into sequential ones before being fed to plms. additionally aiming to understand why bart helps word ordering, we adopt a similar approach and refrain from task-specific engineering, which allows the same sequence-to-sequence model for multiset and tree inputs, limiting extra confounding factors in our analysis. analysis with probing previous work on probing @xcite has identified various types of knowledge in plms, such as syntax @xcite , semantics @xcite , and commonsense @xcite . they are speculated to explain the utility of plms in target tasks. we make such explanations reliable for bart in word ordering by establishing the relevance of specific types of knowledge to the task, in addition to probing their existence in bart. ## word ordering with bart we describe our formulation of word ordering and how to adopt the sequence-to-sequence bart for the task ( §3.1), report results on the standard ptb benchmark ( §3.2 and §3.3), and analyze effects of different modeling decisions ( §3.4-3.6). a c e 2 q 0 i m f i c b f x r j 1 o 9 x 5 9 / y g v g o e j i m j + f c m 3 t 6 g p g z b v z 3 2 y m s r k 6 t b x q 3 s 1 v b o 7 t 7 5 f 2 d r y 0 t r w i t s c 5 v o 8 c a c i z o 0 z d d a t t w f e c b p 6 1 g d j p 5 r t f v m k n x y c y x 9 s m 8 e c x k b b s r + d 0 i m 6 g k 0 n s p z b r x r r h v n w d a j t 6 c v o p h k k p r k 3 9 1 + 5 i k e r w g c k x 1 x 3 n j 4 6 d y g u y 4 n z a 6 i a y x j i m 8 o b 1 l b y 6 o 9 t m 8 9 b s d w q w p q q n s e w b l 6 u + n f e d a t 6 l a t m y h 9 a k x i f 9 5 n c s e v 3 7 k r j w y k s j s u j h w z c t k g k b 9 p i g x f g i j j o r z r i g m s c l e 2 j 5 k t g r v 8 c v l 5 p g 8 6 l 1 u a 3 e 1 s v 1 6 1 g y u 4 r h o 4 a w 8 u i q 6 3 e i d m k d g c z 7 h f d 6 c s f p i v d s f s 9 g c m 9 8 5 h d 9 w p n 8 a 2 m i s 2 @xmath0 figure 2 : prefix tree at initialization for constraints {"she", "li_ kes", "li_ stening", "music"}. each path from the root to a leaf corresponds to a subword sequence. except for the root, each node corresponds to a subword and tracks its count (superscript) in the constraints. a pointer (bold outline) denoting the previously generated subword y t-1 points to the root at initialization. at decoding step t, only subwords of child nodes with nonzero count are valid as the next token y t . once a subword is selected for y t , we move the pointer to the corresponding node and decrement its count by 1. the pointer is reset to the root after reaching leaves. decoding for each candidate finishes when all counts are zero. ## understanding why bart helps we empirically establish the intuition that knowledge in bart helps target tasks. though numerous types of knowledge have been identified in plms by probing @xcite , they may not necessarily improve the target task @xcite . thus a reliable explanation should identify the relevant type of knowledge for word ordering. candidate types of knowledge can be selected using a procedure akin to feature importance ( §4.1): we feed different types of knowledge as additional features and select the one bringing the most salient gain in word ordering. their relevance should be further verified by a strong correlation ( §4.2) between the probing performance and word ordering performance, as models can utilize unexpected shortcut rules @xcite instead of distilling the intended knowledge provided in the features. 7 such a correlation is estimated using models with different amount of the knowledge. we finally probe for the nontrivial existence of the knowledge in bart ( §4.2). ## extension to partial tree linearization our analysis in §4 can be readily extended to partial tree linearization @xcite , a generalized word ordering task provided with additional syntactic input features. unlike settings in §4, additional features can be arbitrary subsets of part-of-speech tags and labeled dependency arcs. the task can be helpful for applications such as machine translation @xcite . following zhang (2013), we use the same dependency trees in §4.1 and report bleu on the ptb development set with different proportions of syntactic features. previous studies @xcite use linear models with hand-crafted features. each base noun phrase (bnp; noun phrases without decedent noun phrases) is regarded as a single word for computation efficiency. we adopt the same sequence-to-sequence models rand and bart in §3.2 and report results with and without special treatment for bnps. similar to §4.1, partial trees are turned into penman sequences and fed to sequence-to-sequence transformers. we use the same model for different proportions of input features. for each tree in the training set, we sample 0%, 50% and 100% of part-of-speech tags and labeled dependency arcs, respectively, resulting in 9 training instances with different input features for the same ordered output sequence. to keep inputs consistent, we put brackets around words that have no additional features (see brac in figure 5 ). results are shown in ## conclusion we investigated the role of plms in word ordering using bart as an instance. non-sequential inputs are turned into sequences and fed to sequenceto-sequence transformers and bart for coherent outputs. we achieve the best-reported results on word ordering and partial tree linearization with bart. with transformers and bart, we provide a systematic study on the effects of output space constraints, conditional modeling, and permutation sensitivity of inputs for word ordering. our findings can shed light on related pre-trained models such as t5 @xcite in related tasks such as commongen. our analysis with feature importance and structural probing empirically identifies that knowledge about syntactic dependency structure reliably explains the utility of bart in word ordering. such a procedure is general and can be readily used to explain why a given plm helps a target task.