Paper: Branch-GAN: Improving Text Generation with (not so) Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Both generator and discriminator need to be tuned. This results in expensive computation and memory consumption, especially for large LLMs.

Author response:
**Regarding the main weakness of having to tune both a generator and discriminator.**
It is true that the current bare-bones implementation of Branch-GAN always tunes both models. However, we do not agree that this diminishes the academic contribution, and is thus not a weakness of the paper in itself. Rather we find that introducing these concepts in a straight-forward manner to be a strength.

That being said, the increased per-sample computational cost pales in comparison to that of the original pre-training. Consider that we utilize less than 0.01% of the original training data. Due to the efficient encoding method, training with a depth of 16 means we perform 16 extra forward passes per sample. We train two models (Gen + Disc) for 4 epochs. giving us: 0.01% x 16 x 2 x 4 = 1.28% 
In a very over-simplified manner, this means that all our training is roughly 1.28% of a **single** epoch of the pre-training. 

For our biggest model this training can be comfortably executed on a single DGX server (8 GPUs) under 48 hours. We fully agree that this deserves further explanation in the paper, and we have created the additional Appendix section D.1.

Reviewer comment (Experiments):
For different generation tasks (such as Wiki, Fiction generation), was the proposed method tuned separately? In other words, did the current version support multi-task tuning?

Author response:
**Regarding Question-1:**
No, the Branch-GAN model was not tuned separately for the different tasks. There is just the continued training on the 0.01% of the Pile dataset. Paragraph 2 in Section 4.1 has been updated to more explicitly explain this.

Reviewer comment (Experiments):
For sequence S with length N, K unique starting points is randomly sampled from [1, N-1] as branches. As each generated branch consists of d (e.g., 16) tokens, if the sampled starting point is N-1, is the corresponding generated branch easy to be recognized than that corresponding to other points? Or should we sample the starting point from the range [1, N - d]?

Author response:
**Regarding Question-2:**
A very astute question! Indeed the discriminator could easily learn to distinguish tokens that surpass N. In our experiments any token that exceeds N is automatically masked, and yields no loss for either the generator and discriminator. This is mentioned in the second paragraph of Section 3.3


Only sampling starting points in the range [1, N-d], as you propose, would ensure that compute is not wasted on branches exceeding N. If further work can demonstrate that it does not introduce any detrimental bias in the training it would indeed be preferable. We avoided this since it means that the last d tokens would have less probability of being generated than with our masking schema.

Reviewer comment (Reproducibility):
For each branch, if the termination token was generated, was the branch still generated until its length = d?

Author response:
**Regarding Question-3:**
Admittedly, we have not considered this possibility. It would make sense that the loss is masked after the termination token. Something to be incorporated in future work, thank you!

Reviewer comment (Experiments):
How about the training complexity w.r.t. branch number and depth of generation?

Author response:
**Regarding Question-4:**
We are not sure exactly what you’re asking for, as training complexity is somewhat nebulous. Could you elaborate upon your question?

The depth and number of branches explored in Section 4.6 affects training complexity in different ways. The number of branches mainly affects the memory load for each sample. The depth increases both the memory, but also the sequential time it takes to compute a single sample.

Reviewer comment (Reproducibility):
For robustness analysis, how to repeat the random selected characters? It is better to give an example. Is the noise sensitive to the position of selected characters?

Author response:
**Regarding Question-5:**
This is a fair request. We will create an example of the noise generation process shortly and notify you when we have upload an updated version.

We have now added Appendix E.4 where we explain the noise generation in more detail and show some examples

Reviewer comment (Presentation):
Confusing Illustrations: While many of the important elements have been shown in Fig1, I would really appreciate if the authors can considering showing some technical details of the discriminator in Fig 1 too. With the current illustration it's really difficult to know enough details about how the dense adversarial rewards are constructed, which are actually the most novel part of the proposed method. (The efficient pretext encoding reusing trick is good, but it's not what I think that differentiates BranchGAN from other language GANs)

Author response:
**Regarding Confusing Illustrations:**
Your intuition about the benefits of the dense reward is fully aligned with ours. We did admittedly struggle to properly illustrate all we wanted within the page limit. Considering that both the dense reward and the pretext coding needs to be introduced, would you be satisfied with a more extensive illustration in the Appendix that covers both the generator and discriminator? If so, then we will do our best to have one done prior to the rebuttal deadline.

Reviewer comment (Evaluation):
Lacking Mode Collapse Detection and/or Diversity Measure [Addressed during Rebuttal]: According to some previous criticism against language GANs from Language GANs Falling Short [https://arxiv.org/abs/1811.02549 ], most previous language GANs that have shown effectiveness in producing more coherent output sequences generally can be fragile against the comparison of simply annealing the temperature of language models. In other words, it could be the case that BranchGAN, too, sacrifices diversity/mode coverage in exchange for precision/quality. A proper study on this is needed to respond to such challenges. **This is dangerously needing attention** because the remarkably increased PPL of the BranchGAN-generated samples by Pythia 6.9b and remarkably increased generation quality reminds me of some previous language GANs that also over-claim themselves to be able to generate high-quality samples, eventually turned out to be rather trading-off between quality and diversity and had zero improvement when considering both aspects.

Author response:
**Regarding Lack of Diversity Measure:**
We are grateful for this important observation. Evaluation results in this setting seems indeed crucial given the prior work and criticism. To rectify this oversight Appendix E.4 now contains a Diversity-Quality experiment with a temperature sweep for selected models according to the proposed experiment.

That being said, we struggle with the intuition of the evaluation process of the proposed paper. Measuring quality via the BLEU overlap between generated texts and a given test set seems infeasible in open-ended text generation. Indeed, as one can tell from results in Appendix E.4 there is a clear contradiction in the perceived quality given the BLEU metrics and our human evaluation.

Faced with these contradicting results, we carefully tried to find any bugs in our code, and instead decided to using the accompanying code of the “Language GANs Falling Short” paper: https://github.com/pclucas14/GansFallingShort/blob/master/real_data_experiments/eval_bleu.py

However, the contradicting results still remained, and upon further investigation we found others who questioned this evaluation process or the metrics being used. In 
https://arxiv.org/pdf/1806.04936.pdf they point out some limitations of automatic evaluation scores for diversity and quality. Additionally, https://arxiv.org/pdf/1905.09922.pdf mentions a model that generates very poor quality samples yet still received close to a perfect BLEU-5 score.

We acknowledge the merit of evaluating text generation models over a sweep of temperatures. Especially if prior work has found this to be a weak point of GAN’s. But, we recognize that the automatic evaluation in these experiments are somewhat controversial. Thus, we hope that you find our results, which include human evaluation, to be a worthy contribution to this ongoing discourse.

Reviewer comment (Experiments):
**Lack of comparison**: The paper does not compare Branch-GAN with other language GANs or cooperative language GANs that use pre-trained Transformers.

Author response:
**Regarding lack of GAN comparisons:**
To the best of our knowledge, no previous GAN has shown improvements on long open-ended text generation, compared to that of SOTA LLMs’. Indeed, as our human study shows, even 7B LLMs struggle to generate convincing sequences of 96 tokens.
Considering that no GAN is seemingly used for these scenarios, we opted on only evaluating LLMs (not to mention how relatively compute expensive it would be to apply other GANs to this sequence length.)

Reviewer comment (Experiments):
**Limited evaluation**: My main concern is that the paper only evaluates text generation on two domains (Wikipedia and Fictional Stories) and does not test the generalization ability of Branch-GAN on other domains or tasks, such as summarization, dialogue, or question answering.

Author response:
**Regarding your main concern of limited evaluation:**
We fully share your curiosity of further evaluation on other tasks. The current evaluation is mainly due to space limitations, and that previous conference papers have seemingly focused on a single task when introducing new algorithmic concepts.
Therefore, we decided on the very general task of open-ended text-generation, and to evaluate it as rigorously as possible. Hoping that this would give us enough space to properly introduce our ideas.

However, we can at least partially accommodate your concern of only performing text generation on two datasets. Appendix E.3 now contains a table that evaluates the relevant models on 5 additional datasets using the two automatic metrics we established in the paper. This means that in total we now evaluate on the following types of datasets:

Wikipedia, Stories, CC News, Dialogsum, Legal Summarization, Medical transcripts, OpenWeb

Reviewer comment (Experiments):
How well does Branch-GAN generalize to other domains or tasks, such as summarization, dialogue, or question answering? What are the challenges or limitations of applying Branch-GAN to these scenarios?

Author response:
**Regarding the question of generalisation to other tasks:**
There is no inherent limitation to applying Branch-GAN to any particular domain or task. As now depicted in Appendix E.3, the same Branch-GAN models seemingly perform fairly well across a wide range of datasets.

That being said, for tasks such as summarization or translation, it would admittedly be beneficial to only generate branch sequences from the summarized/translated text tokens. This means that one would change the algorithm to apply loss and generations only to the target translation tokens.

Reviewer comment (Experiments):
Though the human evaluation on text completion is good, I'm really quite curious to see how the Branch-GAN performs on tasks that are more divergent from the pre-training setup, such as prompting tasks.

Author response:
**Regarding evaluation on text completion only.**
We fully share your curiosity of evaluation on other tasks. Considering the many new concepts that had to be condensed into the main paper, it seemed natural to focus on a single task evaluation. Future work should definitely explore other areas such as summarization, translation, instruction tuning and so on.

Reviewer comment (Theory):
I also find the loss function in Eq. (1) to be poorly motivated. Why choose this variant, over say REINFORCE/PPO with a baseline/advantage function? (I would also at least move more of the description of the loss in A.1 to the main text.)

Author response:
**Regarding the motivation of the loss function.**
Due to spatial reasons we are currently struggling to squeeze more motivation regarding the loss function into the main paper. PPO and REINFORCE were evaluated in early stages of development, but lead to a very unstable training dynamic. The motivation for having two value heads is mainly intuitive, as explained in Appendix A.1, but since switching to this loss we encountered zero problems with collapse of either the generator or discriminator. Appendix A.2 has been created to share these insights in the paper.

Reviewer comment (Presentation):
I'm not clear about Table 4 in the appendix --- which model is being evaluated? I would have thought Branch-GAN, but don't understand why it would be compared "vs. Branch-GAN 1B*".

Author response:
**Regarding table-4 in appendix.**
Table 4 contains the aggregated data from all collected annotations. We have updated the title and description to clarify this further.

Reviewer comment (Evaluation):
I'm also curious about win-rate _between_ models (it's noted in Sec. 5 that GPT-4 is consistently preferred over Branch-GAN, but I could only find comparisons to original texts in the paper).

Author response:
**Regarding win rate between models.**
This table was not included in the main paper due to spatial reasons, but we see no reason to not include it. Clearly an oversight on our part, thank you. It now exists in appendix E.1