From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The reviewer consensus is to accept this paper. The paper is well-written and addresses an important problem. It is well-placed in the context of previous works on complex sequence generation. In the proposed CASR method, model-generated intermediate outputs are fed into the next step. For training, the model generated (possibly incorrect) output is fed into the next step, resulting in a matching training/inference procedure. Reviewers found the experimental results to be convincing, with a mix of real world and the Sudoku task especially suitable for this method.

Reviewers liked that the source code is available for their review as well.
justification_for_why_not_higher_score: no reviewers felt this was an especially strong paper. Given that the differences are usually not huge (except for Sudoku) and the tasks are not very competitive. 
justification_for_why_not_lower_score: Reviewer consensus
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper6141/Authors
title: Replies to Reviewer B3Ea
comment: Thank you for your suggestions.

CASR points out the challenge brought by the multi-hop dependency chain in L2R generation. CASR uses self-boost refinement. Although there are many other work using refinement/iteration, they are not designed to tackle the "**multi-hop dependency challenge**" (as discussed in the **general response**). Limited by our survey, we only select 6 baselines, as displayed in table 5 and figure 10. 

For example, progressive generation(Tan et al., 2021) designs heuristic intermediate sequences based on tf-idf, which benefits long sequence generation but cannot deal with the "dependency challenge" in complex sequence generation. Iteration-based methods with T=2 (like bidirectional decoder, Zhang et al., 2018) are not sufficient for the **example** in the **general response** (where **3 iterations** are necessary according to the assumption). 

About **the two suggested related work**. Compared to them, CASR tackles different challenges (multi-hop dependency vs. right-side dependency), works on different tasks (complex sequence generation vs. natural language generation), and uses different models (**pre-trained language model** like T5 vs. RNN/VAE). And we will include a discussion about them in the revised version.

From: ICLR.cc/2023/Conference/Paper6141/Authors
title: Replies to Reviewer rnaL
comment: Thank you for your suggestions. 

About **why left-to-right decoding doesn’t work and why refinement is necessary**. In the **general response**, we construct an example target sequence "12345" with its multi-hop dependency chain, **3<--4, 4<--2, 3<--2, 2<--5, 5<--1**. And we use this example to discuss how refinement helps in L2R generation in the general response, which decouples the dependency chain in L2R generation. 

| webqsp:f1 | mtop:temp-acc | mtop:match | kvret:bleu | kvret:micro-f1 | avg |
| - | - | - | - | - | - |
| 73.03 | 82.92 | 76.73 | 18.20 | 67.63 | 63.70 |

About **using one regular refinement model**. We try it, and get the results above. This row can be put in table 5. As we can see, it's better than fine-tuning only, but worse than CASR. That's because only one origin model and one refine model are used, rather than T models. 

About **enlarging beam size**. We will include this experiment in the revised version, but we think larger beam size still suffers from the "multi-hop dependency challenge". Here is the reason: Beam search allows correcting left positions with right ones, in the form of probability. Even if the probability of left positions is high, it could be filtered by multiplying a low $P(right|left)$. However, beam search (on a one-pass L2R model) and CASR have different objectives. As we can see in table 1, line 4 and 8, when $beamsize$ --> $infinity$, the decoded prediction $\hat{Y}$ --> $argmax_Y P(Y|X)$. However, $P(Y|X)$ is constructed by $\prod_i P(y_i|X,y_{<i})$, and $P(y_i|X,y_{<i})$ can hardly be precisely learned. For the **example** in the **general response**, $P(y_2=2|X,y_1=1)$ can hardly be precisely learned (unless the model is extremely clever), because "4<--2 and 3<--2" but "3" and "4" have not been addressed. The objective of CASR is $\prod_i P(y_i^t|X,\hat{Y}^{t-1},y_{<i}^t)$. In this example, when t=1, i=2, the item is $P(y_2^1=2|X,\hat{Y}^0=aa34a, y_1^1=1)$, which is kind of learnable, compared to $P(y2=2|X,y1=1)$, because "3" and "4" are addressed in $\hat{Y}^0$. 

About **sampling & re-ranking**. If $\prod_i P(y_i|X,y_{<i})$ can be precisely learned,$ P(Y=Y^*|X)$ will be large enough to help $Y^*$ easily sampled (let $Y^*$ denote the ground truth). Otherwise, let $P(.)$ denote the true probability, $P_M(.)$ denote the unprecise probability given by the converged model. In extreme case, if $P_M(Y=Y^*|X)$<<$P(Y=Y^*|X)$, $Y^*$ can hardly be sampled. 

About **Additional exposure bias**. It means, when training the model with heuristic intermediate sequence, the model takes in rule-based $Y^{t-1}$ in training (row 7 in table 1), but the previous prediction $\hat{Y}^{t-1}$ in inference. The difference in input between training and inference results in additional exposure bias. 

About **how to define "hard/easy" of a target sequence**. If the multi-hop dependency chain is given/labeled, the complexity can be defined as the number of connections in the chain or the minimal necessary iterations. When it's not given, we can indirectly measure it with different methods depending on tasks. In Sec 5.1, we try to empirically measure the **complexity** of sequences from different tasks. For WebQSP, MTOP, and KVRET, Table 6 assumes the more complex a sequence is, the longer it will be. For Sudoku, Table 7 defines the difficulty to fill each blank, and counts the minimal necessary iterations (named correct-solving castep in the paper) to correctly fill each blank. Table 6 shows that, CASR helps more on sequences with more complexity. 

We are sorry for the **reference** issue, and we will correct it in the revised version.

From: ICLR.cc/2023/Conference/Paper6141/Authors
title: Replies to Reviewer Nx7n
comment: Thank you for your suggestions. 

About **end2end training**. The model $M^t$ takes in $X$ and $\hat{Y}^{t−1}$ in both **training** and inference. Because $\hat{Y}^{t−1}$ is generated by $M^{t-1}$, $M^{t-1}$ should be trained before $M^t$, which makes CASR not end2end trainable. $\hat{Y}^0,...,\hat{Y}^{T-1}$ is the intermediate sequences discovered by CASR. (For the example in the **general response**, they are "aa34a", "a2345", "12345" ) Meanwhile, intermediate sequences of template-based models (like Tan et al., 2021)  are constructed by heuristic rules, so all models $M^0,...,M^{T-1}$ can be parallelly trained. If we can get the correct order (or the dependency chain, or the proper intermediate sequences) before training (we leave it as future work), the training process can be more efficient. 
 
About **density changes**. We are curious about density changes. In the revised version, we will enlarge T to see how density changes. Since we haven't seen the drop in performance or $D_{\hat{Y}^{t-1}}$ with the growth of t, according to current experiment results (table 3 and 11), if we enlarge T, we can infer that: 1) the performance of "Restart" should increase and converge, $D_{\hat{Y}^{t-1}}$ should increase and converge as well. 2) the performance of "Continue" may reach the top and then drop (although we haven't seen the drop, we suspect the drop because of overfitting); and when the performance increases, $D_{\hat{Y}^{t-1}}$ should increase as well; but when the performance drops, we can't tell how $D_{\hat{Y}^{t-1}}$ will change. 

About **"Restart" falls behind "Continue" on Sudoku**. For Sudoku task, models are trained from scratch (different from finetuning T5 on the other three tasks), which means "Restart" suffers more from the **cold start issue** compared to "Continue".

From: ICLR.cc/2023/Conference/Paper6141/Authors
title: General Response
comment: We sincerely appreciate the constructive suggestions from the reviewers. 

To the best of our knowledge, **CASR is the first work to point out the challenge brought by the multi-hop dependency chain in L2R generation.** Progressive Generation (Tan et al., 2021) is proposed to deal with long text generation, but it's a different challenge. Although complex sequences are usually long, long text doesn't necessarily have multi-hop dependency chain. Bidirectional Decoder (Zhang et al., 2018) tries to deal with right-side dependency, but cannot handle the multi-hop dependency. And there are many other work using refinement, but they are not designed to tackle the "multi-hop dependency challenge" (as displayed in table 5 and figure 10). 

To explain why refinement helps, let's construct the following **example**: 

Let the target sequence be "12345". "x<--y" denotes "y depends on x", assuming that, to address "y", "x" should be addressed before. (This is an **assumption**, because models may be clever enough to address "y" before "x", or too stupid to address "y" even if "x" is already addressed. ) We construct the multi-hop dependency chain of the example as: 

**3<--4, 4<--2, 3<--2, 2<--5, 5<--1** (It is an implicit property of the example, and is **unknown** to the model)

The first iteration generates from left to right. When trying to predict "1", it fails, because "5" is not addressed. Then, "2" fails because "3" and "4" are not addressed. "3" succeeds because "3" doesn't depend on anyone. "4" succeeds because "3" is addressed before. "5" fails because "2" fails. **In the first iteration, only "3" and "4" are addressed.** (As we can see, when decoding with one-pass left-to-right order, only "3" and "4" are convincing. Given the dependency chain, the best order to decode should be 3,4,2,5,1.)

The second iteration also generates from left to right. Similarly, "1" fails. "2" succeeds because "3" and "4" are addressed in the previous iteration. Skip "3" and "4". "5" succeeds because "2" is addressed. **In the second iteration, "2" and "5" are addressed.**

**In the third iteration, "1" is addressed.** Three iterations are necessary and enough. (As we can see, even if the model doesn't know the best order, it can address all positions simply with refinement. ) And the intermediate sequences of the three iterations may be **"aa34a", "a2345", "12345"** ("a" denotes the prediction of this position is not convincing. ) 

If Bidirectional Decoder (Zhang et al., 2018) is applied in this example, the backward decoder generates from right to left and produces "aa3aa", and the forward decoder generates from left to right and produces "aa34a", which cannot handle the multi-hop dependency. 

If the dependency chain looks like: 1<--2<--3<--4<--5, one iteration is enough. When the dependency chain is any other directed acyclic graph (DAG), the upper bound of iterations is the sequence length n, because at least one position will be addressed in each iteration.  When the chain is not a DAG (has rings in it), the upper bound becomes infinite (when models are clever enough to address "y" before "x" (given "x<--y") and break the rings, the upper bound becomes finite again).

From: ICLR.cc/2023/Conference/Paper6141/Reviewer_Nx7n
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a novel approach [CASR] for sequence generation where multiple language models are connected in an iterative manner and predictions of a previous timestep are taken as an input to the model at future timestep [different from vanilla RNN]. The rationale is that sequence generation problems only specify “difficult” FINAL sequences which need to be solved, and the intermediate order to solve “simpler sequences” is defined by heuristics [Tan et Al]. 


strength_and_weaknesses: **Strengths**

- The authors demonstrate that a network can itself learn the simpler sequences it needs to solve, and that there is a natural order of progression from solving “simpler” sequences to complex ones. Furthermore, they empirically reveal that this progression is different from the simple left-to-right solving order that sequence generation usually takes. 

- The manuscript is well crafted, with convincing experiments and detailed ablations. Authors claim SOTA [70%] on KVRET F1 metric. 

**Weaknesses**

- The basic training process [Algo 2] has a limitation that the language model at each timestep needs to be trained fully [line 3] before moving on to the next timestep. This prevents the network to be E2E trainable, leaving CASR prone to parameter-effficient tuning. [3.3].

- Authors define the notion of density measuring the attention magnitude at each sequence token [5.2]. Increasing density at each recurrent step, seems to show that the prediction made by the model becomes more informative. I am curious as to whether the density becomes maximum when the network converges to optimal solution. Please clarify this.

- On sudoku, the restart strategy lags behind Continue strategy by a huge difference ~20% (table 4). But we don’t observe this trend in other three datasets, well recognized in the community (Table 3).  Could the authors please mention their intuitions on these observations?

clarity,_quality,_novelty_and_reproducibility: Authors have released the code for this work anonymously. 
summary_of_the_review: This work makes a SINGLE minor change to the mathematical formulation of Tan et Al [Figure 1], but still improves empirical results on only 1 out of 3 datasets studied. [Sudoku is a toy dataset from Kaggle which authors don’t show other methods against]. Due to lack of novelty in technical contribution, but still beating SOTA on 1 dataset, this work has incremental contributions. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6141/Reviewer_rnaL
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper deals with tasks where left-to-right generation is not the best strategy. The authors propose a refinement framework named CASR – which models p(i-th-timestep output token at t-th refinement step | input to task, output of model at the (t-1)-th refinement step, the prefix/history of output tokens at t-th refinement step). So when training CASR, the input contains the models’ predictions at the previous refinement step as well as the input to the task, and the output is the reference answer. At test-time, iterative refinement is applied. 

Experiments are done on WebQSP, MTOP, KVRET, Sudoku (see Section 2.2 for details). The authors have also done ablation studies on architecture choices (e.g., how to encode the input to the CASR model, how to initialize weights of the model at each refinement step, fine-tuning vs. using adapters, etc.). CASR is effective, and achieves SoTA results on KVRET. 

strength_and_weaknesses: The comparison of the authors’ approach vs. iterative refinement and progressive generation is quite clear in Table 1. The table helps understanding greatly. 

Ablation studies are quite comprehensive and inspiring. The analysis on attention is interesting. 

Reproducibility is great, given that the code is provided. 



I’m not convinced at why using the previous refinement round’s output as input to the CASR model is necessary. Experiments on iterative refinement using a regular refinement model would be helpful. Regular refinement: train regular refinement model (inputs: generated output from original model; outputs: reference output); then, do one-round refinement after the original model and use these outputs as the input to the refinement model; then, do one-round refinement using this updated model (after the original model), and use these outputs as input to the refinement model; etc. Please correct me if I’m missing important details. (*)

The second paragraph on page 2 (starting with “however”) is lacking. The authors believe that if there are right-side dependencies “which together lead to multi-hop dependency chains,” then “left-to-right is not the best order for generation.” There needs to be citation or empirical proof. First, left-to-right generation with multiple refinement steps could lead to good results. Second, if we have a perfect model that scores each sequence, and a good decoding strategy (e.g., sampling many candidates and then reranking), then left-to-right generation can solve the tasks like Sudoku and WebQSP. The issue may be that our model is not perfect at scoring sequences, so we may need to rely on other strategies. Anyhow, the authors’ writing that “because there are multi-hop dependency chains, left-to-right is not the best order” should be more nuanced and in depth. 

Related to the above: Have the authors attempted better decoding strategies like sampling 100 candidates and rerank, or beam search with large beam size? It’s not clear whether the authors have definite proof that left-to-right decoding doesn’t work and refinement is necessary. (*)

clarity,_quality,_novelty_and_reproducibility: Line 5 of the third paragraph on page 2: Sun et al. should be cited using citet instead of citep.

The authors use Wu et al. (2016) as L2R’s citation, which is a bit weird because left-to-right generation existed many decades ago (e.g., Hochreiter, 1998). 

What do the authors mean by “additional exposure bias” at the end of the fourth paragraph on page 2? Additional to what? 

It’s not clear how hard/easy is defined on the second line of the first paragraph of page 3.   


Reproducibility is great, given that the code is provided. 
summary_of_the_review: The idea is interesting. Results on the chosen tasks (though quite specific) are great. Ablation studies are comprehensive and inspiring. I would want to see two more baselines -- labeled (*) above; or have the authors justify why the two baselines are not necessary. Currently I'm leaning toward weak accept. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6141/Reviewer_B3Ea
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes CASR: a framework that generates complex sequences by iteratively editing previously generated sequences. The problem is well-motivated: in complex scenarios, we might need multiple rounds of editing to generate correct sequences. CASR achieves this goal by feeding a transformer model with input X and prediction from last iteration. The model achieves good performance over both conventional AR and NAR baselines. Compared to template-based methods, CSAR is more flexible and avoids additional exposure bias 
strength_and_weaknesses: Strength:

1. Well-motivated problem and clear presentation.
2. The proposed CSAR is effective compared to vanilla AR and NAR models.
3. The evaluation and analysis are somewhat comprehensive. 

Weakness:

The major concern I have regarding this submission is that some critical related works are missing.

In particular, this line of related work, according to my knowledge, started from Deliberation Networks 2016[1], which is similar to CSAR except that it only focuses on the decoder (the previous prediction will go through the decoder multiple times to be refined). Another recent model that I found similar to CSAR is Draft and Edit 2020 [2], which is basically a CASR with T=2, and uses VAE as the backbone. Many other similar models can be found in Deliberation Networks' citation. 
In short, I believe this line of research is highly correlated with this submission and has to be discussed. I can see that there are still differences between CSAR and these works, but those have to be thoroughly discussed in the paper. 

Furthermore, although the existing evaluation is already quite comprehensive, compared to related works above is necessary. Such that we can perceive what is the advantage of CSAR over other iterative models. 
 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written.
The evaluation is solid.

The major novelty of CASR, as discussed in the Introduction is 1) we no longer need templates/rules to generate intermediate sequences 2) we will not introduce exposure bias because of (1).
However, if we consider those iterative methods(see Weakness), they have those advantages too. In terms of model structure, CASR is also quite similar to previous works. From this perspective, this submission's novelty is somewhat marginal. 
summary_of_the_review: Given those discussed in Weakness and Novelty, I think this submission is not ready for acceptance yet, but I am happy to raise my score if my concerns are addressed.  
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold