Hierarchical Phrase-Based Sequence-to-Sequence Learning (2022)

Abstract: This paper describes a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative parser based on a bracketing transduction grammar whose derivation tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one. We use the same seq2seq model to translate at all phrase scales, which results in two inference modes: one mode in which the parser is discarded and only the seq2seq component is used at the sequence-level, and another in which the parser is combined with the seq2seq model. Decoding in the latter mode is done with the cube-pruned CKY algorithm, which is more involved but can make use of new translation rules during inference. We formalize our model as a source-conditioned synchronous grammar and develop an efficient variational inference algorithm for training. When applied on top of both randomly initialized and pretrained seq2seq models, we find that it performs well compared to baselines on small scale machine translation benchmarks.

## introduction despite the improvements in performance and data-efficiency enabled by recent advances in pretraining, standard neural sequence-to-sequence (seq2seq) models can still fail to model the hierarchical structure of sentences and be brittle with respect to to novel syntactic structures @xcite @xcite . it has moreover not been clear how to incorporate explicit constraints such as translation rules (e.g., for translating idioms) into these blackbox models without changing the underlying architecture. @xcite code available at: https://github.com/berlino/btg-seq2seq . figure 1 : an overview of our approach. during training a variational synchronous parser hierarchically segments and aligns source-target phrases (top), which provides training pairs for a seq2seq model (bottom). the seq2seq loss provides learning signal to the synchronous parser which subsequently provides signal to (and is regularized by) a monolingual source parser. after training, there are two inference models: one mode in which both parsers are discarded, which results in a regular seq2seq model that has been regularized by latent hierarchical phrase pairs; another mode in which the seq2seq model can be combined with the source parser to derive a neural synchronous grammar. inference in this mode is done via translation-as-parsing with cube-pruned cky. the transducer consists of two components learned end-to-end: a discriminative parser based on a bracketing transduction grammar (btg; @xcite whose derivation tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one. importantly, a single seq2seq model is trained to translate phrases at all scales, including at the sentence-level, which results in two inference modes. in the first mode, we discard the parser and simply decode with the resulting seq2seq model at the sequence level, which maintains fast inference but still makes use of a seq2seq model that has been exposed to (and regularized by) latent hierarchical phrase alignments. this approach can be seen a form of learned data augmentation @xcite , wherein a model is guided away from wrong generalizations by additionally being trained on crafted data that coarsely express the desired inductive biases. in the second mode, we view the combined parser and seq2seq model as a sourceconditioned neural synchronous grammar to derive a set of synchronous grammar rules along with their scores (a "neural phrase table"), and decode with a modified version of the classic cube-pruned cky algorithm @xcite . while more involved, this decoding scheme can incorporate explicit constraints and new translation rules during inference. we formulate our approach as a latent variable model and use variational learning to efficiently maximize a lower bound on the log marginal likelihood. this results in an intuitive training scheme wherein the seq2seq component is trained on phrase tables derived from hierarchical alignments sampled from a variational posterior synchronous parser, as shown in figure 1 . when applied on top of both randomly initialized and pretrained seq2seq models across various small-scale machine translation benchmarks, we find that both modes improve upon baseline seq2seq models. ## approach notation. we use x, y to denote the source/target strings, x i:k , y a:c to denote the source/target spans, and x i to denote the token with index i. ## experimental setup we apply btg-seq2seq to a spectrum sequence-tosequence learning tasks, starting from diagnostic toy datasets and then gradually moving towards real machine translation tasks. additional details (dataset statistics, model sizes, etc.) are given in appendix e. baselines. we use the standard transformerbased seq2seq models @xcite as the backbone module for all experiments. we consider three baselines: (1) a seq2seq model trained in a standard way, (2) a seq2seq model pretrained with phrase pairs extracted with an off-the-shelf tool, fast-align @xcite ), foot_4 and (3) a trivial baseline in which the model is pretrained on phrase pairs generated by randomly splitting a source-target sentence pair into phrase pairs. the latter two approaches study the importance of using learned alignments. btg-1 seq2seq and btg-n seq2seq. we focus on evaluating the two decoding modes of our model: cky decoding where phrase-level translations are structurally combined to form a sentence-level prediction (btg-n seq2seq), and standard "flat" seq2seq decoding where sentencelevel translations are generated directly by forcing @xmath0. in this way we test the effectiveness of hierarchical alignments both as a structured translation model and as a regularizer on standard seq2seq learning. btg-n seq2seq is sometimes referred simply as btg-seq2seq in cases where cky decoding is activated by default (e.g., constrained inference). ## conclusion we presented a neural transducer that maintains the flexibility and seq2seq models but still incorporates hierarchical phrases both as a source of inductive bias during training and as explicit constraints during inference. we formalized our model as a synchronous grammar and developed an efficient variational inference algorithm for training. our model performs well compared to baselines on small mt benchmarks both as a regularized seq2seq model and as a synchronous grammar.