Paper: A Branching Decoder for Set Generation (ICLR-2024)

Reviewer comment (Novelty):
It seems reasonable to expect that a sufficiently powerful One2Seq model may ultimately perform better than a more specialized set generation model.

Author response:
Thanks for this inspiring question.

Our parallel decoding **does not lose the benefit of autoregressive reasoning**. Indeed, recall that our decoder in Equation (2) is still an autoregressive decoder, so the generation of each individual sequence in our model still follows an autoregressive manner and benefits its reasoning capability. Therefore, in theory, our model will be equally effective if used as a Seq2Seq model (see our response to W1).

What our model **intentionally ignores** is the dependence among the generation of different sequences in the set; multiple sequences are independently generated in parallel. For set generation tasks, such decoupling is considered beneficial; it is exactly the motivation of our work. The benefit of decoupling is demonstrated by our experimental results in Table 2 where even the Base version of our model outperforms the Large version of One2Seq in most settings, both in accuracy and in throughput. Therefore, we believe that our One2Branch is a promising generalization of One2Seq.

Reviewer comment (Evaluation):
Limited analysis of how performance varies across different set sizes and domains.

Author response:
As to domains, please note that **we have already used four datasets from different domains** in the experiments: science (KP20k), news (KPTimes), forum (StackEx), and open-domain (MSQA), and we drew consistent conclusions.

As to set size, we empirically observed that the performance of our approach generally increases when the size of the gold-standard set increases, possibly because large sets are often associated with relatively easy samples. **This performance result with a varying set size is in ready availability. Following the reviewer's suggestion, we will easily add it to the appendix of the camera-ready version.**

Reviewer comment (Experiments):
The factorization of decoder embedding seems unnecessary given modern hardware.

Author response:
We did not perform embedding/matrix factorization in our decoder. Maybe we did not fully understand the comment. We would appreciate if the reviewer could clarify the question.

A possibly related fact is that considering the slow speed of long sequence generation even on modern hardware, our proposed branching decoder generates text in parallel which accelerates inference by several times without additional GPU requirements, which we believe is beneficial to user experience.

Reviewer comment (Writing):
I am not familiar with the multi-decoder technique, can you elaborate on what "a decoder with shared parameters" means or provide a reference? In the text, you mention that "each sequence is input to a decoder independently" — does that mean there are multiple decoders with shared parameters, or is it the exact same decoder that the inputs are fed into, although independently?

Author response:
We clarify that it is the exact same decoder that the inputs are independently fed into.

Reviewer comment (Experiments):
What loss is the model trained on? Is it only optimized on the ZPLR loss? If so, what happens if only positive labels are considered? How does your method compare with simply selecting the highest scoring sequences instead of selecting sequences that have positive scores on average? During decoding, the $k_{min}$ highest-scored tokens are guaranteed to be selected at each iteration anyway. Does your proposed method work without the ZPLR loss?

Author response:
Our model is only optimized on the ZPLR loss. It cannot be trained without negative labels. Please note the **difference between negative label (i.e., token) and negative sequence**. Both positive and negative tokens can be straightforwardly derived from positive sequences; they complement to each other and they collectively form the entire vocabulary. They are both needed for training. However, our model can be trained without negative sequences. Indeed, as mentioned at the end of Section 3.3, we firstly train our model only using positive sequences (i.e., without using negative sequences, but only using positive and negative tokens derived from positive sequences).  

The difference is that it would be challenging to determine **how many** highest scoring sequences should be selected. Our model design allows to **dynamically** choose this number (by comparing each score with zero), which represents a distinguishing advantage of our approach. **This advantage is sourced from the ZPLR loss**, which therefore is a key component of our approach.

Reviewer comment (Experiments):
What value of $k_{min}$ is chosen for the computation of throughput / GPU memory usage? How do they vary with different values of $k_{min}$?

Author response:
As mentioned in Section 4.3, we tuned $k^\text{min} \in [1,15]$ on the dev set and finally fixed $k^\text{min}=8$.

**A smaller value of $k^\text{min}$ would allow a smaller number of paths to be explored, thus increasing throughput.** For example, based on T5-Base on the KP20k dataset, the throughput of our One2Branch model is 7.6 at $k^\text{min}=1$, being 1.1 times as fast as the throughput of 6.8 at $k^\text{min}=8$. The throughput further decreases to 5.6 at $k^\text{min}=15$, but still, it is much faster than the baseline One2Seq. **The influence of varying $k^\text{min}$ on GPU memory usage is negligible.** We will add these results to Appendix A.2 where we have reported the influence of $k^\text{min}$ on F1 scores.

Reviewer comment (Experiments):
The sota comparisons are focused on pretrained models and sequential decoders. Diverse decoding and generation methods (see some relevant papers below) would also potentially be good for set generation tasks, and it would be interesting to know how branching decoders perform against them.

Author response:
Many thanks for providing references about diverse decoding and generation methods. While we would argue that diversity is not necessarily an optimization objective to be pursued in set generation tasks, following the reviewer's idea, we conducted an experiment to compare our One2Branch with Diversity Beam Search (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17329), a representive diverse decoding and generation method mentioned in the review. Due to time limitations, we only completed our experiments on the multi-span question answering problem. As shown in the following tables, our One2Branch outperforms Diversity Beam Search in almost all the settings. More experiments will be added to the camera-ready version.

|  w/ doc (extraction task) | EM  | PM | EM | PM |
|  ----  | ----  | ----  | ----  | ----  |
|  | ***T5-Base*** | | ***T5-Large*** | | | 
| One2Seq (Diversity Beam Search)   | 69.1 | **83.1** | 65.3 | 81.2 |
| One2Branch    | **70.2**  |  82.9 |  **73.3** |  **85.5** |

|  w/o doc (generation task) | EM  | PM | EM | PM |
|  ----  | ----  | ----  | ----  | ----  |
|  | ***T5-Base*** | | ***T5-Large*** | | | 
| One2Seq (Diversity Beam Search)   | 14.1 | 34.1 | 15.1 | 35.5 |
| One2Branch      |  **20.9**  | **37.2**   | **23.3** |**39.7**  |

Reviewer comment (Experiments):
The authors solely focus on the keyphrase generation problem, where the keyphrases are 1-3 words long which is very small compared to natural generation tasks. Natural language generation is inherently a set generation problem; for example, there could be multiple paraphrases for a sentence (paraphrase generation), the same questions can be asked differently (question generation), and summaries can be written differently (summarization). It would be very interesting to see how branch decoders perform on some of these tasks (see some relevant papers below).

Author response:
We would like to clarify that in Appendix A.5, we also presented experiments on the **multi-span question answering** problem, where the length of an answer varies from one to **dozens of words**. Inspired by the reviewer's comment, we examined the performance of our approach on long-answer questions, that is, questions with an average of **more than 10 words per answer**. As shown in the following tables, our One2Branch considerably outperforms the baseline One2Seq in almost all the settings.

|  w/ doc (extraction task), &#124;ans&#124; > 10 | EM  | PM | EM | PM |
|  ----  | ----  | ----  | ----  | ----  |
|  | ***T5-Base*** | | ***T5-Large*** | | | 
| One2Seq  | 29.2 | 61.8 | 22.6 | 60.4  |
| One2Branch  | **30.1** | **64.4** | **31.2** | **63.1**  |

|  w/o doc (generation task), &#124;ans&#124; > 10| EM  | PM | EM | PM |
|  ----  | ----  | ----  | ----  | ----  |
|  | ***T5-Base*** | | ***T5-Large*** | | | 
| One2Seq  | 4.8 | **19.6** | 3.0 | 19.8 |
| One2Branch | **8.3** | 19.2 | **7.5** | **20.5** |

Excitedly, we appreciate the reviewer's suggestion about other potential applications of our approach. We did not foresee that many opportunities! We will definitely add them to our future work.

Reviewer comment (Theory):
In section 3.4, why do we need the minimum number of explored paths (k^min) given we are using the threshold-based decoding algorithm?

Author response:
It is used to strengthen the robustness of our decoder. As mentioned in Section 3.4, for each generated sequence we require its **average** token score to be positive, but some token scores can be negative. This is a practical relaxation of our threshold-based decoding. To enable it, we introduce $k^\text{min}$ to keep top-$k^\text{min}$ tokens at each time-step, even if their scores are negative, because they have a chance to utilamtely lead to a sequence having a positive average token score.