Paper: Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression (ICLR-2024)

Reviewer comment (Experiments):
Ablations regarding the design choice of $\gamma_t$ scheduling are missing. In the paper, a polynomial decay is used and ablated, but other schedules have also been used with EMA (like cosine). It would be nice to understand a bit better why this decay is important and how to design one which is tailored at the problem at hand.

Author response:
Anecdotally, we find that the effect of EMA on GVA is relatively insensitive to the specific decay schedule of the EMA.  We have not experimented with more complicated, non-monotonic decays, but will do this for one of the MuJoCo experiments and include the result in the final paper.

Reviewer comment (Evaluation):
In the context of autoregressive language generation: Validation perplexity perhaps does not show the existence of GVA problem here in the most clear way. Ideally, some equivalent to a metric of generation quality of horizon H autoregressive rollouts should have been used instead.

Author response:
We agree that changes in validation perplexity are orthogonal to GVA (and we have pointed this out in the paper, which we will attempt to further clarify in the revision). As a more direct parallel to the imitation learning experiments, we also studied full-sequence switching costs in argmax decoding. That GVA appears in perplexity is interesting in its own right, in that perplexity is not intrinsically autoregressive (evaluating the likelihood of entire text generation); note that this also occurs in the MuJoCo setup (see Figure 6b). We are excited to understand how GVA is so widespread as to affect even the 1-step loss, as well as developing more refined metrics, to future work. We emphasize that our LM results are preliminary and primarily illustrative of the phenomenon of error amplification when a 1-step objective is used as a proxy in closed-loop settings.

Reviewer comment (Writing):
In **Proposition 3.1**, I guess that $\Delta$ refers to two different smooth “error functions” in each of the two inequalities.

Author response:
Yes, in each inequality in the proposition, the Delta is allowed to depend on the precise parameters and thus may be different.  Note that this is a stronger statement than if they must be the same.