Paper: On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methods (ICLR-2024)

Reviewer comment (Theory):
Although ForgetNet perfectly possesses the Markov property of algorithm execution, the newly proposed G-ForgetNet still utilizes historical embeddings but outperforms ForgetNet. Is there an inherent drawback of neural algorithmic reasoning in possessing the Markov property?

Author response:
The drawback of ForgetNet is not due to the Markov property, in fact, the Markov property is ForgetNet's biggest strength. The drawback of ForgetNet is the training difficulties that stem from removing the connections between layers. We have tried to clarify this point in Sections 4.1 and 4.2 of our revised manuscript. Additionally, motivated by comments from reviewers, we added a loss penalty to G-ForgetNet which better aligns G-ForgetNet with the Markov property. We demonstrate that, with the new loss penalty, the gate norm goes to 0 during training, *i.e.,* the gate becomes entirely closed. We observe that enforcing the gate to be closed significantly improves the performance of G-ForgetNet, as it now outperforms the baseline on all 30/30 algorithms. As such, we further show the power of aligning neural algorithmic reasoners with the important Markov property. We refer the reviewer to the revised Section 5 to see these new results.

Reviewer comment (Evaluation):
For certain tasks, G-ForgetNet performs worse than ForgetNet, or ForgetNet performs worse than the baseline. Some task-specific explanations regarding this should be discussed.

Author response:
Generally ForgetNet struggles on algorithms where the $\texttt{hints}$ time series is very long, such as Jarvis' March, Articulation Points, or Heapsort, where the training time series lengths are 204, 163, and 91, respectively. This aligns with our expectations as it is incredibly difficult for the model to backpropagate signals through 100+ sequentially applied processor layers when the predictions early in the time series are inaccurate. We believe that ForgetNet underperforms on algorithms such as Naive String Matcher or Floyd-Warshall because the update step between consecutive $\texttt{hints}$ is non-trivial. E.g. in Floyd-Warshall, each step requires the model to simulate a double-nested for-loop, which is generally difficult to learn, and these difficulties are then compounded by the previously discussed training issues in ForgetNet. 

Conversely, ForgetNet performs best on algorithms with very simple update steps between hints such as in DFS.

Reviewer comment (Evaluation):
In most algorithms ForgetNet beats the baseline (Figure 3). Is there any intuition or hypothesis around which particular algorithms the baseline is better? For example, for Naive String Matcher the baseline looks considerably better. Why might that be?

Author response:
Generally ForgetNet struggles on algorithms where the $\texttt{hints}$ time series is very long, such as Jarvis' March, Articulation Points, or Heapsort, where the training time series lengths are 204, 163, and 91, respectively. This aligns with our expectations as it is incredibly difficult for the model to backpropagate signals through 100+ sequentially applied processor layers when the predictions early in the time series are inaccurate. We believe that ForgetNet underperforms on algorithms such as Naive String Matcher or Floyd-Warshall because the update step between consecutive $\texttt{hints}$ is non-trivial. E.g. in Floyd-Warshall, each step requires the model to simulate a double-nested for-loop, which is generally difficult to learn, and these difficulties are then compounded by the previously discussed training issues in ForgetNet.

Reviewer comment (Presentation):
The reviewer likes the visualization with bars in Figure 1 and suggests that adding G-ForgetNet to that graphic or otherwise including a plot with the G-ForgetNet results would help. Can the authors provide such a visualization?

Author response:
We have added this visualization as Figure 6 in Appendix A.1.

Reviewer comment (Writing):
The discussion about the motivation could be improved.

Author response:
Prior works such as [1] have shown the importance of aligning the computational graph of neural networks with the structure of the underlying task. This motivates us to restructure the computational graph of neural algorithmic reasoners to be better aligned with the Markov nature of the algorithms in the CLRS-30 benchmark. In our revised manuscript we have clarified the motivation for our G-ForgetNet model. We refer the reviewer to Sections 4.1 and 4.2 to see our refined motivation for G-ForgetNet, including the mentioned connections with residual links.

Reviewer comment (Theory):
About the motivation that "algorithmic reasoning tasks are Markov": it should be clarified that this work is about algorithmic reasoning tasks that can be modeled by finite-state automata. In general, whether the process is Markov or not depends on the definition of the state space.

Author response:
Firstly, thank you for raising an interesting perspective on finite-state automata, which may broaden this direction in the future. Indeed, the algorithms in the CLRS-30 *can* be modeled by finite automata under a slightly irregular definition of the state space because the graphs used for training and testing are of fixed size. We leave further theoretical analysis of algorithmic reasoning in the context of finite automata or other models of computation to future work.

Reviewer comment (Writing):
Even though the Markov observation motivates to remove history information, history information proves to be important for stabilizing training (hence the proposal of G-ForgetNet). This is similar to the effect of residual links, which can be discussed more in the paper.

Author response:
We appreciate the insightful comments about the similarities with residual links. We have added some discussion on these similarities in Section 4.1.

Reviewer comment (Experiments):
Fig 4 (b): what are execution steps, and how are the labels acquired? Since there are labels to compute the per-step loss, can we also compare with per-step teacher forcing?

Author response:
This part of Figure 4 was moved to Appendix B.2 in our updated manuscript in order to make room for the rest of our updates. However, the execution steps in this figure are the time series of $\texttt{hints}$ for each intermediate step in the algorithm process. Per your suggestion, we evaluated ForgetNet with 100% teacher forcing. Allthough the losses for ForgetNet with teacher forcing are incredibly low, the model fails to generalize to the test data for most algorithms. The figure with the training losses for ForgetNet trained with teacher forcing on the Floyd-Warshall algorithm can be sound in the Supplementary Material, and the performance for ForgetNet with teacher forcing on each algorithm can be found below. Allthough the overall average performance of ForgetNet with teacher forcing is significantly below the baseline, it outperforms our best model, G-ForgetNet, on several algorithms, including naïve string matcher, where it achieves a remarkable 100% test accuracy. We believe teacher forcing shows significant potential for future works, however further study is outside the scope of our paper.

Reviewer comment (Evaluation):
Fig 5: for better comparison, could you also provide the norm of the residual branch in the baseline?

Author response:
Yes, per your suggestion, we have included the norm of the residual branch in the baseline in Figure 5 as well as in Figure 7 in Appendix A.2. In order to provide a fair comparison between G-ForgetNet and the baseline, we now report the norm of the gated hidden states instead of the norm of the gate itself.

Reviewer comment (Experiments):
I wonder if one can anneal the gating, i.e. gradually reducing the amount of the history (e.g. by controlling the gating value).

Author response:
Firstly, I believe was have addressed the concerns here by adding the gate penalty to G-ForgetNet and showing that is does in fact go to ~0 at the end of training, thereby enforcing the Markov property. Additionally, there are no longer any cases where G-ForgetNet is not better than the baseline. 
Secondly, we had previously tried a manually annealed schedule and found that it underperformed ForgetNet, performing about the same as the baseline, so we did not include these results in the paper. The performance on each algorithm for the annealed model is also reported in the table below.

### Performance of requested Teacher Forcing and Annealed models
Results outperforming G-ForgetNet are highlighted in bold
|Algorithm| ForgetNet - Teacher Forcing | Manual Annealing |
|--|--|--|
| Activity Selector | 91.89±2.20% | 95.99±1.02% |
| Articulation Points | 10.75±3.17 | 94.37±0.84% |
| Bellman-Ford | 95.96±0.30% | 98.71±0.08% |
|BFS | **99.98±0.02%** | 99.95±0.02% |
|Binary Search | 65.42±1.60% | 71.40±2.22% |
| Bridges | 22.34±3.93% | 96.25±1.41% |
| Bubble Sort | 9.28±0.86% | 77.53±3.20% |
| DAG Shortest Paths | 60.64±1.08% | 98.45±0.17% |
| DFS | 19.00±0.80% | 62.23±7.53% |
| Dijkstra | 64.74±4.20% | 98.23±0.06% |
| Find Max. Subarray | **80.97±1.03%** | 42.89±0.98% |
| Floyd-Warshal | 5.91±0.13% | 33.77±1.15% |
| Graham Scan | 66.82±2.23% | 92.17±1.41% |
| Heapsort | 6.01±0.61% | 37.26±3.89% |
| Insertion Sort | 64.13±1.57% | 87.52±0.93% |
| Jarvis' March | **95.97±0.38%** | 82.45±1.45% |
| Knuth-Morris-Pratt | 14.43±1.05% | **23.79±2.96%** |
| LCS Length | 71.87±8.48% | 84.49±0.60% |
| Matrix Chain Order | 78.72±0.59% | 92.94±1.08% |
| Minimum | 99.01±0.14% | 97.73±0.40% |
| MST-Kruskal | 15.60 ±0.93% | 85.73±0.72% |
| MST-Prim | 69.20±2.34% | 92.75±0.53% |
| Naïve String Matcher | **100.00±0.00%** | 27.32±1.39% |
| Optimal BST | 36.60±5.64% | 78.52±0.73% |
| Quickselect | 1.99±0.12% | 3.54±0.90% |
| Quicksort | 8.47±0.80% | 49.56±5.09% |
| Segments Intersect | 99.01±0.09% | 98.47±0.06% |
| SCC | 8.94±1.90% | 41.90±2.54% |
| Task Scheduling | 79.98±3.72% | 81.05±0.44% |
| Topological Sort | 3.65±0.29% | 91.48±3.68% |
| Overall Average | 50.47±1.97% | 73.95±1.58% |

Reviewer comment (Evaluation):
One motivation for annealing the gating is that in Table 1, when G-ForgetNet is not better than the baseline, its performance is very close to the baseline. This suggests that G-ForgetNet falls back to the baseline in these cases, hence ensuring the history embeddings are actually not being used may help with performance.

Author response:
Firstly, I believe was have addressed the concerns here by adding the gate penalty to G-ForgetNet and showing that is does in fact go to ~0 at the end of training, thereby enforcing the Markov property. Additionally, there are no longer any cases where G-ForgetNet is not better than the baseline. 
Secondly, we had previously tried a manually annealed schedule and found that it underperformed ForgetNet, performing about the same as the baseline, so we did not include these results in the paper. The performance on each algorithm for the annealed model is also reported in the table below.

### Performance of requested Teacher Forcing and Annealed models
Results outperforming G-ForgetNet are highlighted in bold
|Algorithm| ForgetNet - Teacher Forcing | Manual Annealing |
|--|--|--|
| Activity Selector | 91.89±2.20% | 95.99±1.02% |
| Articulation Points | 10.75±3.17 | 94.37±0.84% |
| Bellman-Ford | 95.96±0.30% | 98.71±0.08% |
|BFS | **99.98±0.02%** | 99.95±0.02% |
|Binary Search | 65.42±1.60% | 71.40±2.22% |
| Bridges | 22.34±3.93% | 96.25±1.41% |
| Bubble Sort | 9.28±0.86% | 77.53±3.20% |
| DAG Shortest Paths | 60.64±1.08% | 98.45±0.17% |
| DFS | 19.00±0.80% | 62.23±7.53% |
| Dijkstra | 64.74±4.20% | 98.23±0.06% |
| Find Max. Subarray | **80.97±1.03%** | 42.89±0.98% |
| Floyd-Warshal | 5.91±0.13% | 33.77±1.15% |
| Graham Scan | 66.82±2.23% | 92.17±1.41% |
| Heapsort | 6.01±0.61% | 37.26±3.89% |
| Insertion Sort | 64.13±1.57% | 87.52±0.93% |
| Jarvis' March | **95.97±0.38%** | 82.45±1.45% |
| Knuth-Morris-Pratt | 14.43±1.05% | **23.79±2.96%** |
| LCS Length | 71.87±8.48% | 84.49±0.60% |
| Matrix Chain Order | 78.72±0.59% | 92.94±1.08% |
| Minimum | 99.01±0.14% | 97.73±0.40% |
| MST-Kruskal | 15.60 ±0.93% | 85.73±0.72% |
| MST-Prim | 69.20±2.34% | 92.75±0.53% |
| Naïve String Matcher | **100.00±0.00%** | 27.32±1.39% |
| Optimal BST | 36.60±5.64% | 78.52±0.73% |
| Quickselect | 1.99±0.12% | 3.54±0.90% |
| Quicksort | 8.47±0.80% | 49.56±5.09% |
| Segments Intersect | 99.01±0.09% | 98.47±0.06% |
| SCC | 8.94±1.90% | 41.90±2.54% |
| Task Scheduling | 79.98±3.72% | 81.05±0.44% |
| Topological Sort | 3.65±0.29% | 91.48±3.68% |
| Overall Average | 50.47±1.97% | 73.95±1.58% |

Reviewer comment (Evaluation):
Another reason for annealing the gating is that according to Fig 5, the gate value is ~0.38 even at the end of training. This means the history embeddings are still helpful, which weakens the claim that the task should be Markov, unless the history embeddings are effectively removed.

Author response:
Firstly, I believe was have addressed the concerns here by adding the gate penalty to G-ForgetNet and showing that is does in fact go to ~0 at the end of training, thereby enforcing the Markov property. Additionally, there are no longer any cases where G-ForgetNet is not better than the baseline. 
Secondly, we had previously tried a manually annealed schedule and found that it underperformed ForgetNet, performing about the same as the baseline, so we did not include these results in the paper. The performance on each algorithm for the annealed model is also reported in the table below.

### Performance of requested Teacher Forcing and Annealed models
Results outperforming G-ForgetNet are highlighted in bold
|Algorithm| ForgetNet - Teacher Forcing | Manual Annealing |
|--|--|--|
| Activity Selector | 91.89±2.20% | 95.99±1.02% |
| Articulation Points | 10.75±3.17 | 94.37±0.84% |
| Bellman-Ford | 95.96±0.30% | 98.71±0.08% |
|BFS | **99.98±0.02%** | 99.95±0.02% |
|Binary Search | 65.42±1.60% | 71.40±2.22% |
| Bridges | 22.34±3.93% | 96.25±1.41% |
| Bubble Sort | 9.28±0.86% | 77.53±3.20% |
| DAG Shortest Paths | 60.64±1.08% | 98.45±0.17% |
| DFS | 19.00±0.80% | 62.23±7.53% |
| Dijkstra | 64.74±4.20% | 98.23±0.06% |
| Find Max. Subarray | **80.97±1.03%** | 42.89±0.98% |
| Floyd-Warshal | 5.91±0.13% | 33.77±1.15% |
| Graham Scan | 66.82±2.23% | 92.17±1.41% |
| Heapsort | 6.01±0.61% | 37.26±3.89% |
| Insertion Sort | 64.13±1.57% | 87.52±0.93% |
| Jarvis' March | **95.97±0.38%** | 82.45±1.45% |
| Knuth-Morris-Pratt | 14.43±1.05% | **23.79±2.96%** |
| LCS Length | 71.87±8.48% | 84.49±0.60% |
| Matrix Chain Order | 78.72±0.59% | 92.94±1.08% |
| Minimum | 99.01±0.14% | 97.73±0.40% |
| MST-Kruskal | 15.60 ±0.93% | 85.73±0.72% |
| MST-Prim | 69.20±2.34% | 92.75±0.53% |
| Naïve String Matcher | **100.00±0.00%** | 27.32±1.39% |
| Optimal BST | 36.60±5.64% | 78.52±0.73% |
| Quickselect | 1.99±0.12% | 3.54±0.90% |
| Quicksort | 8.47±0.80% | 49.56±5.09% |
| Segments Intersect | 99.01±0.09% | 98.47±0.06% |
| SCC | 8.94±1.90% | 41.90±2.54% |
| Task Scheduling | 79.98±3.72% | 81.05±0.44% |
| Topological Sort | 3.65±0.29% | 91.48±3.68% |
| Overall Average | 50.47±1.97% | 73.95±1.58% |

Reviewer comment (Experiments):
The paper proposes one solution, gating, but some questions remain — why is gating better than the NN more generally learning how to combine historical and current embeddings, as in the baseline?

Author response:
In response to your comments and those of other reviewers, we have added a regularization term to the loss function which penalizes the magnitude of the gate norm, ensuring that the gate is aligned with the Markov property as desired. However, to answer your original question, the G-ForgetNet model *without* this penalty performs better than the baseline because, while it still may fit some historical noise, it is less reliant on historical embeddings than the baseline model, and therefore closer aligned with the Markov property. We further analyze your question in Appendix A.3 of our revised manuscript.

Reviewer comment (Experiments):
What are other solutions, possibly not involving learning?

Author response:
We previously tried to use a manual annealing schedule for the gate in which the gate is initially fully open, and is linearly decayed until the end of training, where it would be entirely closed, thus explicitly enforcing the Markov property. However, we found that this did not significantly improve the performance over the baseline. The results of this experiment for each algorithm are given in the table below. 

Results outperforming G-ForgetNet are highlighted in bold

|Algorithm | Manual Annealing |
|--|--|
| Activity Selector | 95.99±1.02% |
| Articulation Points | 94.37±0.84% |
| Bellman-Ford | 98.71±0.08% |
|BFS |  99.95±0.02% |
|Binary Search | 71.40±2.22% |
| Bridges | 96.25±1.41% |
| Bubble Sort | 77.53±3.20% |
| DAG Shortest Paths | 98.45±0.17% |
| DFS |  62.23±7.53% |
| Dijkstra | 98.23±0.06% |
| Find Max. Subarray | 42.89±0.98% |
| Floyd-Warshal | 33.77±1.15% |
| Graham Scan | 92.17±1.41% |
| Heapsort | 37.26±3.89% |
| Insertion Sort | 87.52±0.93% |
| Jarvis' March | 82.45±1.45% |
| Knuth-Morris-Pratt | **23.79±2.96%** |
| LCS Length |  84.49±0.60% |
| Matrix Chain Order | 92.94±1.08% |
| Minimum | 97.73±0.40% |
| MST-Kruskal | 85.73±0.72% |
| MST-Prim | 92.75±0.53% |
| Naïve String Matcher | 27.32±1.39% |
| Optimal BST | 78.52±0.73% |
| Quickselect | 3.54±0.90% |
| Quicksort | 49.56±5.09% |
| Segments Intersect | 98.47±0.06% |
| SCC | 41.90±2.54% |
| Task Scheduling | 81.05±0.44% |
| Topological Sort | 91.48±3.68% |
| Overall Average | 73.95±1.58% |

Reviewer comment (Novelty):
The TripletGMPNN includes a gating layer in the single-task setup. A discussion on how this differs from the gating in G-ForgetNet should be included.

Author response:
The gate in Triplet-GMPNN is placed ahead of the decoder and is meant to bias the model towards only updating a subset of the hidden states at each step, while the remaining hidden states are left unchanged. Our gate is placed ahead of the processor and is meant to bias the model towards not including *any* hidden states at all, which we further enforce using a loss penalty. We have added further clarification about the difference between our gates in Section 5 of the revised manuscript.