Paper: Semantic Parsing with Candidate Expressions for Knowledge Base Question Answering (ICLR-2024)

Reviewer comment (Novelty):
In the era of LLMs, the improvement seems marginal comparing with using large LMs and with more training data.

Author response:
>In the era of LLMs, the improvement seems marginal comparing with using large LMs and with more training data.

As your comment, LLMs can achieve high performance with many training examples.
In addition, LLMs such as GPT3 enables in-context learning, where our method cannot be directly applied.
However, we've recently found interesting semantic parsers that generate tokens (not production rules) but contrain tokens with respect to SCFGs [2,3].
Following [3], our method can be combined with Earley’s algorithm [12], where our intermediate representations can replace dotted production rules.
In our revision, we'll describe [2,3] and the idea of applying our method to [2,3].

Reviewer comment (Experiments):
Comparing with using production rules etc, have you tried unstructured decoding method? Maybe this should be a good baseline.

Author response:
>Comparing with using production rules etc, have you tried unstructured decoding method? Maybe this should be a good baseline.

BART KoPL [1] can be a good baseline that uses unstructured decoding.
BART KoPL predicts logical forms written in KoPL, which is linearized in postfix representations.
An example output sequence of BART KoPL is
```

FindAll <func> FilterStr <arg> official website <arg> http://www.thesiege.com/ <func> FilterConcept <arg> visual artwork <func> QueryAttrQualifier <arg> publication date <arg> 1999-01-21 <arg> place of publication

```
which are represented in our model as
```
(query-attr-qualifier (filter-concept (filter-str all-entities "official website" "http://www.thesiege.com/")
                                      "visual artwork")
                      "publication date" "1999-01-21" "place of publication")
```

Unlike ours, BART KoPL splits symbols (e.g. FindAll or FilterStr) into tokens (e.g. Find, All, Filter, Str), then the model has slightly longer output sequences.

We also analyzed the speed of ours and BART KoPL in the reply to the reviewer pMus.

Reviewer comment (Experiments):
Clearly with more data, the constrained decoding doesn't help much. Have you explored data augmentation?

Author response:
>Clearly with more data, the constrained decoding doesn't help much.

We think that constraints on the syntax don't much enhance performance with a large training set,
since the limited number of syntactic patterns are repeated in the training set.
As your comment, experiment results also show that candidate expressions are less effective with a large training set.
However, we still think that candidate expressions are useful, especially when inferring KB components (e.g. entities or relations) that are unseen during training.
Similarly with GENRE [4], candidate expressions would guide semantic parsers when a KB is updated after training (e.g. new entities are added).

>Have you explored data augmentation?

KQAPRO [1] was semi-automatically constructed in two steps [9]; (1) automatically generating pairs of canonical utterances and logical forms by using SCFG, and (2) paraphrasing the canonical utterances by crowdsourcing.
In addition, zero-shot semantic parsing [10] showed decent performance without manually annotated training and validation sets.
Another interesting work generates syntactic pairs of canonical utterances and logical forms during weakly-supervised learning [11].

The core of data synthesis method is using SCFGs, which consist of pairs of production rules that generate canonical utterances and logical forms.
If we additionally add templates for canonical utterances to all node classes, our grammar can be considered as a SCFG, where a complete intermediate representation can generate a pairs of a canonical utterance and a logical form from two kinds of templates.

We think that grammar-based data synthesis is a good research area for semantic parsing.

Reviewer comment (Novelty):
The main contribution, as the author points out, is “To the best of our knowledge, our work is the first to use production rules as actions for semantic parsers based on pre-trained seq2seq models.” First, I’m not sure what is the underlying challenge of extending traditional grammar-based seq2seqs to their pretrained counterparts. That is, I'm not sure about the technical contribution of the paper.

Author response:
**Additional Explanation**

Please let us explain more about our paper with respect to your comments.

>The main contribution, as the author points out, is “To the best of our knowledge, our work is the first to use production rules as actions for semantic parsers based on pre-trained seq2seq models.”.
>First, I’m not sure what is the underlying challenge of extending traditional grammar-based seq2seqs to their pretrained counterparts? That is, I'm not sure about the technical contribution of the paper.
>Second, there are already existing works in this direction (as cited in the related work). For example, RAT-SQL (based on BERT) extends TRANX which expands production rules incrementally during decoding.

We regret using the arrogant expression "To the best of our knowledge ..." to describe the difference with previous neural seq2seq semantic parsers.
At the time of writing, we hadn't seen semantic parsers that take production rules as actions and that are based on pretrained seq2seq models, such as BART or T5.
Since we consider BERT, which are used by [8], as a pretrained auto-encoding model, which has no pretrained decoder, rather than a pretrained seq2seq model, we thought our model has difference with the previous semantic parsers.
However, as you commented, applying [5,6,7] to BART or T5 is not technically difficult.
In addition, we've recently found other previous work [2,3] that take tokens (not production rules) as actions but constrain the tokens by using production rules.
We'll remove "To the best of our knowledge ..." and add descriptions of [2,3].

**TODO**

We're going to update our paper during the discussion period.

- The sentence beginning with "To the best of our knowledge ..." will be removed.
- Constrained decoding methods [2,3] for pretrained seq2seq will be added to "Related work".
  - We will also briefly describe how our method can be combined with [2,3].

**Gratitude**

We again greatly thank you for your reviews and feedback.
If you have any more comments or questions, please feel free.

**Reference**

[1] KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base  
[2] From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding  
[3] Constrained Language Models Yield Few-Shot Semantic Parsers  
[5] A Syntactic Neural Model for General-Purpose Code Generation  
[6] TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation  
[7] Neural Semantic Parsing with Type Constraints for Semi-Structured Tables  
[8] RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Reviewer comment (Novelty):
There are already existing works in this direction (as cited in the related work). For example, RAT-SQL (based on BERT) extends TRANX which expands production rules incrementally during decoding.

Author response:
**Additional Explanation**

Please let us explain more about our paper with respect to your comments.

>The main contribution, as the author points out, is “To the best of our knowledge, our work is the first to use production rules as actions for semantic parsers based on pre-trained seq2seq models.”.
>First, I’m not sure what is the underlying challenge of extending traditional grammar-based seq2seqs to their pretrained counterparts? That is, I'm not sure about the technical contribution of the paper.
>Second, there are already existing works in this direction (as cited in the related work). For example, RAT-SQL (based on BERT) extends TRANX which expands production rules incrementally during decoding.

We regret using the arrogant expression "To the best of our knowledge ..." to describe the difference with previous neural seq2seq semantic parsers.
At the time of writing, we hadn't seen semantic parsers that take production rules as actions and that are based on pretrained seq2seq models, such as BART or T5.
Since we consider BERT, which are used by [8], as a pretrained auto-encoding model, which has no pretrained decoder, rather than a pretrained seq2seq model, we thought our model has difference with the previous semantic parsers.
However, as you commented, applying [5,6,7] to BART or T5 is not technically difficult.
In addition, we've recently found other previous work [2,3] that take tokens (not production rules) as actions but constrain the tokens by using production rules.
We'll remove "To the best of our knowledge ..." and add descriptions of [2,3].

**TODO**

We're going to update our paper during the discussion period.

- The sentence beginning with "To the best of our knowledge ..." will be removed.
- Constrained decoding methods [2,3] for pretrained seq2seq will be added to "Related work".
  - We will also briefly describe how our method can be combined with [2,3].

**Gratitude**

We again greatly thank you for your reviews and feedback.
If you have any more comments or questions, please feel free.

**Reference**

[1] KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base  
[2] From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing via Synchronous Semantic Decoding  
[3] Constrained Language Models Yield Few-Shot Semantic Parsers  
[5] A Syntactic Neural Model for General-Purpose Code Generation  
[6] TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation  
[7] Neural Semantic Parsing with Type Constraints for Semi-Structured Tables  
[8] RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Reviewer comment (Evaluation):
What is the generation speed of the grammar-based parser compared with the baseline BART? I’m wondering that with more constraints, the parser might suffer from much slower inference speed.

Author response:
We first reply to your questions.

>What is generation speed of the grammar-based parser compared with the baseline BART?

Does "the baseline BART" mean BART KoPL [1]?
We could run BART KoPL on the validation set by using the publicly released code [1].
In the following table, we compare the average time to convert tokens of utterances to tokens of logical forms (or actions), where the batch size is 64.

| Model                             | Time (ms) |
|---|---|
| BART KoPL                         |       4.5 |
| Ours with $\Psi^\textrm{NONE}$    |       3.8 |
| Ours with $\Psi^{\textrm{TYPE}-}$ |      10.0 |
| Ours with $\Psi^\textrm{TYPE}$    |      10.0 |
| Ours with $\Psi^\textrm{HYBR}$    |      10.2 |

Our decoding with $\Psi^\textrm{NONE}$ is slightly faster than BART KoPL, although both don't use constraints during decoding.
It's because the average length of action sequences of our model is shorter than the average length of output token sequences of BART KoPL; the average length is 28.8 for ours and 35.1 for BART KoPL.

An example output sequence of BART KoPL is
```

FindAll <func> FilterStr <arg> official website <arg> http://www.thesiege.com/ <func> FilterConcept <arg> visual artwork <func> QueryAttrQualifier <arg> publication date <arg> 1999-01-21 <arg> place of publication

```
which is identical with the following logical form.
```
(query-attr-qualifier (filter-concept (filter-str all-entities "official website" "http://www.thesiege.com/")
                                      "visual artwork")
                      "publication date" "1999-01-21" "place of publication")
```

Since BART KoPL splits symbols (e.g. FindAll or FilterStr) into tokens (e.g. Find, All, Filter, Str), the model has slightly longer output sequences.

>I’m wondering that with more constraints, the parser might suffer from much slower inference speed

We think that additional contraints by types or candidate expressions don't much slow down the speed.
Type constraints depend on valid actions $\Psi^\textrm{TYPE}(r({\boldsymbol{a}^\star}))$ for the current intermediate representation $r({\boldsymbol{a}^\star})$, and $\Psi^\textrm{TYPE}(r({\boldsymbol{a}^\star}))$ can be efficiently cached for the type of the leftmost non-terminal $\nu(r({\boldsymbol{a}^\star}))$.
Constraints by candidate expressions depend on the time complexity of trie data structures.
Retrieving valid actions $\Psi^\textrm{CAND}(r({\boldsymbol{a}^\star}))$ from a trie only takes $O(\text{length of prefix})$ time.

Reviewer comment (Experiments):
In this paper, the performance of the model is recorded only in one KB-QA dataset. It is not known how good is this method among all KB-QA tasks.

Author response:
We haven't performed experiments on another dataset, although we're considering other datasets, whose tasks could be KBQA, Text-to-SQL or others.
As you and the reviewer 4fJL commented, using only one dataset is a critical disadvantage of our paper.
We acknowledge that experimenting on more than one dataset could enhance the validity of our method.

However, at least in KQAPRO [1], we tried to compare our method with previous work as fairly as possible.
We and previous work use the BART-base model, and we adapt hyperparameters from BART KoPL [1],
Nevertheless, we admit that it cannot be an excuse for using only one dataset.

Reviewer comment (Experiments):
[Not critical] This paper compares their decoding method with Picard's paper to show the proposed method has a good latency. Why are the two models in different tasks (KBQA vs Spider), different model architectures (BART vs T5), and different hardwares comparable?

Author response:
We wanted to compare our method with constrained decoding for __pretrained seq2seq model__, such as BART or T5.
At the time of writing, PICARD was only method we know for the scenario (however, we've recently found other previous work [2,3]).
We think that methods based on production rules have better efficiency, so we wanted to emphasize it.
As your comment, the comparison would be fair if we could implement our method on both Spider and KQA Pro.

With respect to hardware, CPUs and GPUs may affect latency, so we specified the exact spec of our own hardware, where our model could run __solely__ in the machine.
However, when we measured latency on other environments (e.g. a cluster with V100), the latency was not much different.