Paper: Parameter-Efficient Detoxification with Contrastive Decoding (ICLR-2024)

Reviewer comment (Novelty):
The authors claim that they "are the first to apply parameter-efficient learning to controllable text generation for detoxification". However, there has been other work (such as "Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models" by Wang et al) that use PEFT.

Author response:
* Previous work has applied parameter-efficient learning to detoxification

Thank you for pointing this out. Indeed Wang et al. (2022) applied parameter-efficient tuning to detoxification. We have toned down our claim in the revision. We do note a key difference though that their work assumes access to the Perspective API during training, while our work and the baselines we compare with do not, making the results not comparable to each other.

Reviewer comment (Evaluation):
One of the disadvantages of the proposed method is the cost as it needs both the generator as well as the detoxifier to do inference. The authors do not address this fact when they compare to other methods.

Author response:
* By using both the generator and the detoxifier during inference, the cost is higher as compared to other methods

We respectfully disagree. In fact, cost is our approach’s strength rather than a weakness. Here we will compare our GPU memory usage and inference latency with the two top-performing baselines, namely GeDi or DExperts:
(1) Our GPU memory usage is only around a third of the baselines because our detoxifier is trained with a parameter-efficient method and shares exactly the same backbone parameters as the generator (i.e., the LLM). This is different from the baselines which need to train an expert and an anti-expert LM of opposing attributes with full finetuning.

(2) Our inference latency is only two-thirds of the baselines because our detoxifier only needs to rely on the toxic part of the training data. Hence instead of running the LLM three times for each prompt during inference (one for the LLM itself, one for the expert LM, and one for the anti-expert LM), our method only runs twice because there is no anti-expert involved.

We sincerely hope that this detailed explanation alleviates the cost concern that may have negatively and greatly impacted our score.

Reviewer comment (Experiments):
The effect on fluency, measured by perplexity, has been evaluated only on the RealToxicityPrompt dataset. A more diverse set will show broader impact.

Author response:
* Evaluate the framework on more benchmark datasets and downstream tasks

We selected the current evaluation benchmark because it makes it easier to compare with previous work that also focused on the same dataset. That said, we do agree that more benchmarks and downstream tasks can shed more light on the effectiveness of our approach.

Reviewer comment (Experiments):
The effect of the proposed approach on downstream tasks is not studied and it is not clear how the performance is affected.

Author response:
* Evaluate the framework on more benchmark datasets and downstream tasks

We selected the current evaluation benchmark because it makes it easier to compare with previous work that also focused on the same dataset. That said, we do agree that more benchmarks and downstream tasks can shed more light on the effectiveness of our approach.

Reviewer comment (Presentation):
In Table 1, what are the perplexity for the original model (alpha = 0)?

Author response:
* In Table 1, what is the perplexity for the original model (alpha = 0)?

That would be column 1 in Tables 6 and 7, where we present results with the vanilla generator (i.e., without the detoxifier).

Reviewer comment (Evaluation):
I would appreciate more qualitative examples of detoxification in the paper.

Author response:
* Qualitative examples of detoxification

We thank the reviewer for the suggestion. We did not include the qualitative examples due to ethical concerns as the outputs without detoxification could seem very offensive to the readers. In our code release, we plan to provide some sample toxicity-inducing prompts and include instructions for researchers and practitioners to compare the difference in outputs.

Reviewer comment (Reproducibility):
I do not see mention of code release.

Author response:
* No mention of a code release

We thank the reviewer for bringing this up. We promised code release on Page 4 Footnote 1, and we have initiated the application process to open-source this project through our institution. We have migrated this footnote to page 1 in the revision to make it more salient.

Reviewer comment (Evaluation):
Is perplexity the best measure of fluency? I would expect to see some human evaluations of generated text to confirm.

Author response:
* Effectiveness of perplexity as a measure of fluency

We agree that perplexity alone may favor models that output generic, dull continuations that lead to low perplexity. That is the motivation that our work and previous baselines add the diversity metrics as a sanity check.

Reviewer comment (Writing):
I am not sure if most readers are familiar with the "Distinct-2/3" metrics of diversity, I would appreciate a brief explanation of this metric in the paper.

Author response:
* A detailed explanation of Distinct-N metrics

We thank the reviewer for the suggestion. We did not elaborate on the Distinct-N metrics due to space constraints. We have added the details in the revision.

Reviewer comment (Experiments):
While I respect the author's choice of sticking to a small set of reasonably chosen design decisions, I would have liked to trade some of the comprehensiveness on the model-size experiments for a broader look at some other hyperparameters, such as the method for creating the *detox* model (there are both more effective efficient fine-tuning methods like LoRA, and cheaper, more straightforward non-fine-tuning methods like plain-old prompting).

Author response:
We thank the reviewer for considering our work to have the potential to have a significant impact by introducing a straightforward and simple approach in a well-presented paper.

* Explore other parameter-efficient tuning methods such as LoRa

We went with Prompt Tuning as it is a more straightforward parameter-efficient approach that requires no change in model architecture (other than the few additional parameters from the soft token embeddings). That said, we agree that LoRa would be a nice baseline to compare with as part of the ablation studies. As the reviewer mentioned, due to space constraints, there is a trade-off on what studies to focus on — we figured that the experiments on model sizes would be very helpful for the audience to make design choices when applying our work.

Reviewer comment (Evaluation):
The authors should show some qualitative examples to further back up table 2.

Author response:
We thank the reviewer for the suggestion. We did not include the qualitative examples due to ethical concerns as the outputs without detoxification could seem very offensive to the readers. In our code release, we plan to provide some sample toxicity-inducing prompts and include instructions for researchers and practitioners to compare the difference in outputs.

Reviewer comment (Experiments):
Only one benchmark dataset is used.

Author response:
We selected the current evaluation benchmark because it makes it easier to compare with previous work that also focused on the same dataset. That said, we do agree that more benchmarks and downstream tasks can shed more light on the effectiveness of our approach.