The de-facto standard decoding method for semantic parsing in recent years has been to autoregressively decode the abstract syntax tree of the target program using a top-down depthfirst traversal . In this work , we propose an alternative approach : a Semi-autoregressive Bottom-up Parser ( SMBOP ) that constructs at decoding step t the top-K sub-trees of height ≤ t. Our parser enjoys several benefits compared to top-down autoregressive parsing . From an efficiency perspective , bottom-up parsing allows to decode all sub-trees of a certain height in parallel , leading to logarithmic runtime complexity rather than linear . From a modeling perspective , a bottom-up parser learns representations for meaningful semantic sub-programs at each step , rather than for semantically-vacuous partial trees . We apply SMBOP on SPIDER , a challenging zero-shot semantic parsing benchmark , and show that SMBOP leads to a 2.2x speed-up in decoding time and a ∼5x speed-up in training time , compared to a semantic parser that uses autoregressive decoding . SMBOP obtains 71.1 denotation accuracy on SPIDER , establishing a new state-of-the-art , and 69.5 exact match , comparable to the 69.6 exact match of the autoregressive RAT-SQL+GRAPPA . Semantic parsing , the task of mapping natural language utterances into programs ( Zelle and Mooney , 1996 ; Zettlemoyer and Collins , 2005 ; Clarke et al . ; Liang et al . , 2011 ) , has converged in recent years on a standard encoder-decoder architecture . Recently , meaningful advances emerged on the encoder side , including developments in Transformer-based architectures ( Wang et al . , 2020a ) and new pretraining techniques ( Yin et al . , 2020 ; Herzig et al . , 2020 ; Yu et al . , 2020 ; Deng et al . , 2020 ; Shi et al . , 2021 ) . Conversely , the decoder has remained roughly constant for years , where the abstract syntax tree of the target program is autoregressively decoded in a top-down manner ( Yin and Neubig , 2017 ; Krishnamurthy et al . , 2017 ; Rabinovich et al . , 2017 ) . Bottom-up decoding in semantic parsing has received little attention ( Cheng et al . , 2019 ; Odena et al . , 2020 ) . In this work , we propose a bottom-up semantic parser , and demonstrate that equipped with recent developments in Transformer-based ( Vaswani et al . , 2017 ) architectures , it offers several advantages . From an efficiency perspective , bottom-up parsing can naturally be done semiautoregressively : at each decoding step t , the parser generates in parallel the top-K program sub-trees of depth ≤ t ( akin to beam search ) . This leads to runtime complexity that is logarithmic in the tree size , rather than linear , contributing to the rocketing interest in efficient and greener artificial intelligence technologies ( Schwartz et al . , 2020 ) . From a modeling perspective , neural bottom-up parsing provides learned representations for meaningful ( and executable ) sub-programs , which are sub-trees computed during the search procedure , in contrast to top-down parsing , where hidden states represent partial trees without clear semantics . Figure 1 illustrates a single decoding step of our parser . Given a beam Z t with K = 4 trees of height t ( blue vectors ) , we use cross-attention to contextualize the trees with information from the input question ( orange ) . Then , we score the frontier , that is , the set of all trees of height t + 1 that can be constructed using a grammar from the current beam , and the top-K trees are kept ( purple ) . Last , a representation for each of the new K trees is generated and placed in the new beam Z t+1 . After T decoding steps , the parser returns the highest-scoring tree in Z T that corresponds to a full program . Because we have gold trees at training time , the entire model is trained jointly using maximum likelihood . We evaluate our model , SMBOP1 ( SeMiautoregressive Bottom-up semantic Parser ) , on SPI- What are the names of actors over 60 ? Prune frontier In this work we present the first semiautoregressive bottom-up semantic parser that enjoys logarithmic theoretical runtime , and show that it leads to a 2.2x speed-up in decoding and ∼5x faster taining , while maintaining state-of-the-art performance . Our work shows that bottom-up parsing , where the model learns representations for semantically meaningful sub-trees is a promising research direction , that can contribute in the future to setups such as contextual semantic parsing , where sub-trees often repeat , and can enjoy the benefits of execution at training time . Future work can also leverage work on learning tree representations ( Shiv and Quirk , 2019 ) to further improve parser performance .

Challenge: Although bottom-up approaches for semantic parsing can theoretically achieve efficiency and produce meaningful sub-trees, they have not been studied well.
Approach: They propose a transformer-based semi-autoregressive bottom-up parser for semantic parsing and couple it with a state-of-the-art encoder.
Outcome: The proposed bottom-up parser can substantially improve over existing models in performance and also in decoding speed.