Paper: Controlled Text Generation via Language Model Arithmetic (ICLR-2024)

Reviewer comment (Writing):
It is not quite clear how or when such a method can be applied. Two example applications are given.

Author response:
**Q: Where can the full feature set of model arithmetic be best utilized to improve over prior work?**

We use the full feature set of model arithmetic in several places to significantly outperform previous work. First, in Section 5.1, while the union operator by itself suffices to outperform existing work, the ability of model arithmetic to combine the union operator with a classifier allows us to widen the gap even further. The new sentiment control task presented in Appendix F displays this utility even further, as the combination widens the gap by a much bigger margin, especially as evaluated by GPT-4. This shows that the feature set of model arithmetic can be used to improve results, even on the simple tasks that can be expressed using prior work. Secondly, Section 5.2 discusses the application of conversational agents, with fine grained control over specific attributes. The examples discussed there cannot be expressed using prior CTG works because of the added features of model arithmetic. Since we show that model arithmetic indeed has fine grained control over the attributes at almost no cost in perplexity, it essentially solves this specific task and its improvement over prior work lies in the fact that we can express this.

Reviewer comment (Experiments):
In the first application, toxicity, both the linear and union operators are used but only in conjunction with weights that seem to be quite hand-tuned (0.9, 0.96, 0.99). It would be useful to also see a simpler framing of this problem with model arithmetic, e.g., simply subtracting the M_toxic instead of the union. This raises the question of whether toxicity requires the full complexity of model arithmetic, as it seems like a fairly simple application that may only require a subtraction, in which case the need for the more complex proposed model arithmetic might not be supported here.

Author response:
**Q: How did you select the weights for model arithmetic in Table 5?**

We selected these weights by gradually increasing the biasing strength until we found a degradation in the perplexity of the results. In an initial version of model arithmetic, we denoted these formulas using the equivalent notation $5M - 4M_\text{toxic}$, $25M - 24M_\text{toxic}$ and $100M - 99M_\text{toxic}$. We thus increased the first term from 5 to 25 to 100 because we noticed that increasing the strength remained fluent.

**Q: Does toxicity reduction require the full complexity behind model arithmetic?**  

While toxicity reduction does not require the full depth in which we introduced model arithmetic, it can make use of all the parts we introduced. In particular it allows us to showcase that our newly derived operator, union, empirically outperforms prior work and further that model arithmetic allows for combining different methods, leading to even better results.  Note that subtraction expresses the baselines, PreAdd, which we clearly outperform.

Reviewer comment (Experiments):
Do the authors have other examples of how model arithmetic might intuitively be used?

Author response:
**Q: Do you have other examples of how model arithmetic might intuitively be used?**

Model arithmetic can intuitively be used in several areas. Firstly, as shown in the example in Table 3, model arithmetic can be used to avoid detection of AI generated text, by biasing away from the classifier. This can in turn be used to train better detectors, since we can use this ability to generate adversarial examples to the current detector. 

Furthermore, ensembling models or prompts to achieve better performance can be achieved by using linear operators and the union/intersection operators. Classifier-Free Guidance (Sanchez et al, 2023), a method we subsume, for example shows that performance of language models can be improved by biasing away from models that do not get any context (e.g. by omitting the input question).

Finally, by using the fact that one can simply optimize the constants over a certain corpus, model arithmetic can characterize texts. This makes it possible to identify whether a newspaper is politically more left- or right-leaning, or to replicate personal styles by analyzing personal messages.

Reviewer comment (Evaluation):
On the experimentation side again, it'd be great to have a section on using human eval to evaluate a small subset of the generations to see if humans agree that it indeed is reducing toxicity or whether its an artifact or oddity of the perspective api evaluation.

Author response:
**Q: Can you include human evaluation for the experiments in Table 5?**

We included an evaluation by GPT-4 for both the toxicity reduction (see Table 6, Section 5.1) and sentiment control (see Table 10 and 12, Appendix F) tasks as a surrogate for human evaluation. We find that GPT-4 confirms our results and prefers our method over prior work. For further details, please see Q1 and Q2 of our main reply.

Reviewer comment (Experiments):
On the experimentation side, it'd be nice to see how well these methods work for smaller models (e.g. GPT2, GPT2-large)

Author response:
**Q: Can you evaluate how well model arithmetic works for smaller models (e.g., GPT2, GPT2-large)?**

We extended the toxicity results with results for the GPT-2 model family in Appendix  I.1. Interestingly, we find that Fudge is more powerful for the smallest model GPT-2. This is caused by the smaller capacity of GPT-2 (with only 124M parameters, compared to the 355M parameters of the classifier) which makes it harder for a prompt-based method to generate non-toxic samples. For the larger models in the model family, our union operator performs better than prior work. Furthermore, we can leverage the flexibility of model arithmetic by applying a combination of a classifier and the union operator. This formula doubles the toxicity reduction with respect to previous work for the entire GPT-2 model family, thereby validating the results that model arithmetic is more effective at toxicity reduction.

Reviewer comment (Experiments):
How straight forward and performant would it be to swap in different models for each part here and capture the output distribution, if the vocabulary was shared?

Author response:
**Q: How straight forward and performant would it be to swap in different models for each part here and capture the output distribution, if the vocabulary was shared?**

Swapping different models is very easy, but the exact performance would depend on the use case.  

A formula can use a mixture of several models under the mentioned condition that the vocabulary is shared. The current implementation is built on top of the popular transformers Python package from HuggingFace (Wolf et al., 2020) allowing us to use all models from its library and making the operation of swapping models as simple as changing a string from one name to another.

How performant the resulting formulas are depends on the underlying task and the models themselves. For example, we see that the same formula works very well for all tested models in Section 5.1 and Appendix F. In Appendix I.1, when applying the formulas on the smaller and less performant model family, the performance of the formulas is a bit different, with the Fudge method now being almost as good as our union operator.

Reviewer comment (Novelty):
The core concept closely resembles "PREADD" (Pei et al., 2023), which also relies on composing multiple LLMs to control desired attributes and using prompts to construct different LLMs with desired attributes.

Author response:
**Q: How does model arithmetic differ from PreAdd?**

Model Arithmetic improves upon PreAdd (Pei et al., 2023) in several significant ways: First, PreAdd only allows the use of 2 different LLMs with a negative coefficient between them. This restriction is a consequence of the limited theoretical framework which model arithmetic does provide, giving us the correct normalization and allowing us to compose more than 2 LLMs with positive and negative coefficients. Second, our method is able to use classifiers and the union operator. These expand the possible applications of model arithmetic by allowing attributes that cannot be expressed in natural language and provides a stronger biasing method. Specifically, the union operator has much better results than PreAdd on the task of toxicity reduction (see Section 5.1) and sentiment control (see Appendix F). Finally, our extension of speculative sampling allows us to increase the speed of inference and can also be applied on top of PreAdd.

Reviewer comment (Experiments):
The comparison in Table 5, showing the toxicity reduction results, appears biased. PREADD is tested in a single ad-hoc configuration, whereas model arithmetic is assessed across six, potentially skewing fairness.

Author response:
**Q: Can you run prior work for more parameters, such that the results in Table 5 become unbiased?**

We extended the amount of parameters on which previous methods were evaluated and included updated values in Table 5. While we tried several parameters before, we now more thoroughly investigated the possible parameters and found that the parameters provided for SelfDebias and Fudge were already optimal, while the parameter for PreAdd could be increased slightly without sacrificing the perplexity of any model. We note that this does not change our conclusions and that our method still outperforms PreAdd.

Reviewer comment (Evaluation):
Lack of finer grained analysis of generation. Neither a human or an LLM-backed evaluation (e.g., with GPT-4) was done, only classifier and perplexity based evaluations (where perplexity is taken under the original vanilla language model). This means that text could be qualitatively strange in some capacity, and none of the evaluations would show it as long as the original model is not too surprised by it. I find this somewhat worrying, as the problem with many model control techniques is the fact that they collapse onto strange parts of the distribution that are low perplexity but still wrong, e.g. the repetition with greedy and beam search.

Author response:
**Q: Can you include GPT-4 or human evaluation?**

We included an evaluation by GPT-4 validating our results for both the toxicity reduction (see Table 6, Section 5.1) and sentiment control (see Table 10 and 12, Appendix F) tasks as a surrogate for human evaluation. We find that GPT-4 confirms our results and prefers our method over prior work. For further details, please see Q1 and Q2 of our main reply.

**Q: Can you provide the perplexity graphs for the results in Section 5.2 to ensure that results remain sensible?**

We included plots for the perplexity of the formulas in Section 5.2 in Appendix I.2. As expected, perplexity smoothly changes over different parameter values. Further, perplexity overall remains under the reasonable value of $6.2$ compared to $4.8$ by the highest prompted model. 

Manual inspection of the data generated by the extreme points shows that the generated text is still fluent and the unprompted model $M$ associates a slightly higher perplexity to the text due to the high presence of an uncommon attribute that would normally not appear in the generated text by $M$. The only exception here is the educational topic, which uses a classifier with a very high weight ($12.0$) for which we found that a word sometimes gets replaced by a nonsensical one.  We now also include several randomly selected examples of texts generated by the model under these high parameters in Appendix I.2. These examples show the various characters produced by the formulas and show that the generated output is still fluent.

Reviewer comment (Experiments):
Lack of different generative scenarios. While section 5.2 shows fine grained control over the generated topic, this is an extremely limited generation scenario where all the model has to do is speak about a specific topic. Evaluation on controlled generation, personalized summarization or translation, coherent essay writing, or any somewhat more defined generation scenario would have made evaluation more convincing.

Author response:
**Q: Can you further explore a more defined generation scenario, such as personalized translation?**

We included an extra evaluation section in Appendix F on sentiment control, a task commonly found in prior work (Pei et al. 2023, Liu et al. 2021, Krause et al. 2020). We find that model arithmetic significantly outperforms prior approaches for this task, as discussed in more detail in Q2 of our main reply.

Reviewer comment (Experiments):
Lack of exploration of the space of possible expressions. The authors present an extremely general method for constructing interesting distributions to sample from, but only test a handful. It would be disappointing if it turned out that this technique is extremely brittle to the specific Language Model Arithmetic expression that is used, which would simply have transported the complexity of prompting to the complexity of finding the right arithmetic expression. The results in the paper do not suggest this is the case, but evaluate only 4 different expressions for quality, with the rest of the results only changing the coefficient values or testing speed.

Author response:
**Q: Is it possible that the proposed technique is brittle to the specific language model arithmetic expressions used?**

We never encountered any brittle behavior while evaluating and testing on a wide variety of expressions, both in the examples presented in Figure 1, Table 2, Table 3, Table 4 and Table 17 and in the various expressions presented in Sections 5.1, 5.2, 5.3 and Appendix F. Only for the educational attribute in Section 5.2 we found a decreased fluency when using the classifier at very high strengths. We specifically evaluated the expressions from Sections 5.1, 5.2 and Appendix F for perplexity and attribute presence and always found that the formula acts as expected. Furthermore, manual inspection of data generated in all sections shows that generated results are still fluent and do not showcase any weird artifacts with the exception of the slight decrease in fluency for the classifier in Section 5.2.

**Q: How much tweaking do the model arithmetic expressions require?**

As discussed in Q4 of our main reply, we rely on Theorem 1 and the interpretation of coefficients as relative strengths to construct the formulas for all examples in the paper. This allowed us to easily choose the right ballpark for the examples shown in the paper without requiring any further tweaking. 

**Q: Does model arithmetic just shift the complexity of finding the right prompt to finding the right expression?**

As noted in the previous answer, finding the right expression is relatively simple. More importantly, we note that formulas are more powerful than prompts and therefore do not just shift the complexity to finding the right expression, as shown by our results on toxicity reduction (see Section 5.1) and sentiment control (see Appendix F), and by examples showcasing interesting possibilities such as overstressing a specific attribute by biasing away from the standard model output (see e.g., Table 2) and using a classifier to generate more human-like text (see e.g., Table 3).

Reviewer comment (Theory):
The proposed method needs heavy human design for specific tasks. The formulas for toxicity reduction (Table 5) and fine-grained control (Section 5.2) is not intuitive. I wonder whether there is a principle or a theoretically-supported method to choose operators and determine their coefficients for different tasks. If these are just empirical tries, the applicability of the proposed method may be largely degraded.

Author response:
**Q: Can [model arithmetic] operators and their coefficients be determined in a theoretically-backed manner?**

As discussed in Q4 of our main reply, we used Theorem 1 in our paper to choose parameters and terms in a theoretically grounded manner. This allowed us to quickly and intuitively construct formulas for all the examples presented in the entire paper, such as the ones in Figure 1, Table 2, Table 3, Table 4 and Table 17, without any finetuning of the coefficients being required. We note it is possible to determine the coefficients by optimizing them via gradient descent on a small number of samples. However, in the paper we focus on applications in the zero-shot setup and we found that a correct ballpark for all our examples was sufficient without requiring any finetuning.

Reviewer comment (Novelty):
The position of this paper needs to be further considered. In my view, the proposed method seems like a general form of existing works because the operators based on linear combinations and classifiers are not first proposed by this paper. They have been already studied in the line of work on controlled text generation. The current position may exaggerate the contribution and novelty of the proposed method.

Author response:
**Q: Can you clarify the contributions and novelty of the proposed method?**

Sure! Model arithmetic improves in several ways over prior work:

First, previous works (using log probabilities) lack theoretical foundations when it comes to model biasing, which often prevents the introduction of multiple attributes. In contrast, our work addresses this by framing CTG as a minimization problem, finding the correct normalization required to bias towards more than one attribute and providing a natural extension to prior work. This allows us to express a lot of existing methods, provides a theoretical foundation for their constructions and naturally extends CTG – for the first time – to more complicated and relevant examples, such as for the conversational styles discussed in Section 5.2. 

Second, our theoretical framework enables us to combine classifiers and linear operators and thereby allows for a higher flexibility with better results as shown in Section 5.1. Further, we introduce the novel operator, union, which has better results on the task of toxicity reduction and sentiment control and can be used on top of the other operators we use.

Third, our use of speculative sampling allows us to use model arithmetic at only a slightly increased cost compared to normal single-model generation. This approach can be used directly, also with prior work, and therefore speeds up any method that we can express.

We hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply.