Paper: SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking (ICLR-2024)

Reviewer comment (Experiments):
The experiment section is not so convincing. The method is evaluated mainly in two settings. In the first setting, the method shows a significant improvement on a subtask of the math dataset. But how about the other sub-tasks? We don’t know if the method can generalize well on other tasks. In the second method, the method has a slightly better mauve score, but the perplexity score drops by a large margin. How can we know that this method indeed improves the model generation?

Author response:
"__In the second method, the method has a slightly better mauve score, but the perplexity score drops by a large margin.__"
+ We agree that the behavior of the perplexity is counter-intuitive. However, in our setting the perplexity does not necessarily correspond to the probability of generating a sequence. We use the term perplexity in the usual way: to denote the exponential of the average negative log likelihood of the next action. Usually the perplexity on the train set directly corresponds to the probability of generating in-distribution sequences. However, in the case with a backspace token, this is not necessarily true, since the probability of generating a sequence must be computed by marginalizing over all the possible sequences of actions (including backtracking via the backspace) that could generate the sequence.
+ As an example, if we consider a very simple language with tokens ‘A’ and ‘B’, where the dataset consists only of sequences ‘AAAAA…’, we can generate an in-distribution sequence by following a policy of sampling A,B uniformly, and then choosing <backspace> if we last sampled a B. We will eventually always generate ‘AAAAA…’, but the perplexity on the dataset is not 1, as it would be for a simple autoregressive model that always generated ‘AAAAA..’. Indeed, a drawback of our method is that we lose the ability to compute the probability of generating a sequence. However, in practice these probabilities do not tend to be very useful for e.g. anomaly detection, so we think it is a good trade-off for the ability to backtrack.

Reviewer comment (Experiments):
The method relies on negative sampling, but there is no reasonable negative sampling strategy provided for open language tasks. This paper provides two negative sampling strategies: random noise and ground-truth noise. As the authors also mentioned in Sec. 6.1, the ground-truth noise needs to be manually constructed for different tasks. The random noise simply samples other tokens in the vocabulary. Both strategies look hard to generalize to open tasks, as it is almost impossible to design ground-truth noise for all the tasks in the world, and the vocabulary for open tasks is too large (the whole token codebook) for the random noise to get meaningful negative samples easily. Thinking about applying SeqenceMatch to the pretraining of LLMs like Llama2, what negative sampling strategy should be used?

Author response:
"__There is no reasonable negative sampling strategy provided for open language tasks.__"
+ One possible sampling strategy is the uniform random strategy. As seen in figure 3 and table 2, this has advantages over no added noise. Figure 3 also shows that problem-specific noise can be advantageous. For the problem of open language tasks, uniform noise would be the default, but potentially there could be ways to develop tailored noise, although these are speculative for the time being. For instance, if we assume access to an LLM of similar size, we could use it as an oracle to generate eg incorrect French words in a translation task. We could also use crowd workers to generate text and record the complete generative process, including any backspaces used by the workers.

"__Thinking about applying SeqenceMatch to the pretraining of LLMs like Llama2, what negative sampling strategy should be used?__"
+ As mentioned above, the random uniform sampling is a baseline approach which gives some advantage. Other approaches could be investigated, including using other models to generate plausible (but incorrect) continuation tokens.

Reviewer comment (Experiments):
Does the method affect the combinational generalization? Given the famous avocado armchair as an example. Assume we have a training dataset that contains avocados, and different types of armchairs but doesn’t contain avocado armchairs. Does the method discourage the model from combinationally generalizing to avocado armchairs, as it is not included in the training set?

Author response:
"__Does the method discourage the model from combinationally generalizing to avocado armchairs, as it is not included in the training set?__"
+ This is a great question. Unfortunately, we don’t have a way to answer it theoretically; at least, we’re unaware of a mathematical formalism of the combinatorial generalization phenomenon. Intuitively, we think it’s likely that the loss encourages more mode-concentrating behavior in exchange for increased probability of staying in-distribution, as discussed in the response to reviewer TvsS. This would lead to decreased levels of ‘combinatorial generalization’. Of course, in the setting of language models, we could argue that hallucinations are an example of incorrect generalization, where a model is happy to continue generating details of events that did not happen. Perhaps in this setting, or settings such as question answering, it would be better if the level of generalization was lower.

Reviewer comment (Novelty):
Please better identify what is the contribution with respect to Garg et al. 2021.

Author response:
Thanks for your thoughtful and comprehensive review. Your main question was about the relation of the two contributions, and the novelty of the approach compared to Garg [2021]. We hope to have answered this question in the global response.

__Please better identify what is the contribution w.r.t Garg et al. 2021__
+ See global response

Please let us know if there are any remaining questions or aspects of the questions we have not addressed, and we would be very happy to discuss them!

Reviewer comment (Experiments):
The method induces nonnegligible training overhead due to sampling trajectories and using a complex loss objective. Also, using backspace introduces some inference overhead -- is the case of rolling back a long segment very unlikely?

Author response:
"__The method induces nonnegligible training overhead due to sampling trajectories and using a complex loss objective__"
+ We give some numerical details on the training overhead in the appendix, section F. The overhead of the loss itself is mainly due to the loss of specialized cuda kernels for causally-masked attention. The overhead of the sampling is fundamentally unavoidable, however. It can be amortized by keeping a replay buffer of previously-generated samples and using those stale replays during training. Since the loss approach is off-policy, using stale replays is valid. If this approach is deployed at a large scale during pre-training, we could also parallelize the generation during training--one machine or set of GPUs could be used to continuously generate sequences, and the parameters for the generation machine updated periodically.

"__Also, using backspace introduces some inference overhead -- is the case of rolling back a long segment very unlikely?__"
+ Yes, it’s unlikely. In fact, even under high levels of noise, the generated sequences do not contain many backspaces. With 20% of tokens subject to additional noise, the rate of backspaces in the arithmetic task was around 1.5% of all actions for the SequenceMatch-trained model. It was rare to have more than one backspace in a row. On the other hand, we found that of the cases where backspace is used, it is around 90% accurate--it only removes a correct token 10% of the time.

Reviewer comment (Theory):
The proposed work provides an estimator but does not show any properties of the new estimator (consistency, unbiasedness, etc.). Historically, the MLE has been nice properties, but it is unclear whether Eq (4) has any nice properties with relation to the true divergence.

Author response:
"__Does not show any properties of the new estimator (consistency, unbiasedness, etc.)__"

+ Thank you for pointing this out. The estimator in equation 4 is a plug-in estimator for the population objective stated in the previous section. As such, under modest conditions it is unbiased and consistent, in the sense that the expected value of ${\hat J}(\ell_{\theta})$ is $J(\ell_{\theta})$ and ${\hat J}(\ell_{\theta})$ converges in probability to $J(\ell_\theta)$ as we increase the number of sequences used in the estimator. For simplicity in equation 4 we present the estimator using a single sequence. In practice we always average the loss over several sequences, with a batch size of 64 typical. Using conditions such as boundedness of the logits and $\phi = x^2 - x^2/4$ for the $\chi^2$-divergence, proof of unbiasedness follows from linearity of expectation, and consistency from the central limit theorem. We will include these proofs in the next version of the appendix.

Reviewer comment (Theory):
There is an implicit assumption in this work that the KL divergence is not a good metric because what we desire is for the model not to generate when the true data distribution has probability 0. The work does not discuss the opposite, however, where the data distribution has non-zero mass, KL minimization will encourage the model to always try to match the distribution — an explicit advantage of KL. Without minimizing KL my concern is that we do not cover the full data distribution. Explicitly, to address the main metric of interest in 2.1.1, implicitly the reverse KL would be a good choice to optimize. This can be a problem especially for a large model or small data.

Author response:
"__Without minimizing KL my concern is that we do not cover the full data distribution.__"
+ When we can’t match a distribution exactly, any choice of loss is implicitly specifying a preference of some (approximate) solutions over others. For a single conditional, the forward KL will certainly have more mode-covering behavior than the $\chi^2$ or reverse KL. However, as we discuss in section 2.1.1., the compounding error problem can have unwelcome interactions with the mode-covering behavior. 
+ As a simple example, consider a generative process for a sequence $x_1, x_2, \ldots, x_N$, where each $x_i \in \mathbb{R}$. The ground-truth conditional distribution $P(x_{i+1}|x_i)$ is a mixture of Gaussians with two modes. However, let’s say our parameterized conditional model is a unimodal Gaussian. Then, the forward KL will generically have mode-covering behavior with some density outside the bulk of the expert density. The reverse KL or $chi^2$ will generally have more mode-covering behavior. The problem arises when we use the forward-KL-based model to generate sequences, since the mode-covering model will sample points $x_1$ that are out-of-distribution and have been seen very infrequently during training. Meanwhile, the reverse KL trained model (as an extreme example) would have all samples in-distribution, although they would all be in the same mode (so lack diversity).
+ Certainly it would be better if we could cover the full distribution, but given the choice between covering the distribution to some limited extent, and getting coherent long sequences, many would prefer the coherent sequences, especially for a task such as question-answering, where receiving a coherent but non-diverse response is generally preferred to a set of diverse but incoherent results.