Paper: Stability Analysis of Various Symbolic Rule Extraction Methods from Recurrent Neural Network (ICLR-2024)

Reviewer comment (Theory):
I think the mathematical formulation of the problems considered in this paper can be more precise and more detailed, to make it easier to understand.

Author response:
Dear Reviewer AfSv,

We thank you for your insightful review and comments.

With the help of your feedback, we have updated our paper with mathematical formulations for the following sections:
1. Stability Analysis
2. Formulation of R1 and R2 founded in the mathematics of stability
3. Updated mathematical formulation of L* algorithm (in Appendix)
4. Rephrased our conclusion to show that results are well in the mathematical formulations of R1 and R2, and our observations have theoretical backing.

**All updates in the paper are reflected in blue font color**

We have also provided a paper overview with mathematical formulations of **R1** and **R2** in the comments above.

Thank You !

Reviewer comment (Writing):
It would enhance the paper if the authors could furnish additional insights or intuition regarding the significance of exploring the stability of rule extraction strategies. Alternatively, a clearer definition of stability in this context would also be helpful. In the current work, stability appears to be synonymous with the variance of performance.

Author response:
Dear Reviewer nZtd,

We thank you for your insightful reviews and comments.

We base our notion of stability of DFA extraction on Brouwers's Fixed Point Theorem and build on previous theoretical works on the stability of state estimation by RNNs. 

---
**Theorem 1**
BROUWER’S FIXED POINT THEOREM: For any continuous mapping $f:Z \rightarrow Z$, where Z is a compact, non-empty convex set, $\exists \ z_f$ s.t. $f(z_f) \rightarrow z_f$

**Definition 1**
Let $\mathcal{E}$ be an estimator of fixed point $z_f$ for mapping $f:Z \rightarrow Z$, where $Z$ is a metric space. $\mathcal{E}$ is a stable estimator iff estimated fixed point $\hat z_f$ is in immediate vicinity of $z_f$, i.e. $ | \hat z_f - z_f | < \epsilon, $ for arbitrarily small $\epsilon$.

**Corollary 1**
Stability of estimator $\mathcal{E}$ can be shown by iteratively computing $z^{(t+1)} = \mathcal{E_f} (z^{(t)})$ with $z^{(0)}$ in the neighborhood of $z_f$. For stable estimator $\mathcal{E}$, $\lim_{t\rightarrow \infty}z^{(t)}  = z_f$. Neighborhood of $z_f$, $N_{z_f}$ is a set of points $z_n \in N_{z_f}$ near $z_f$ s.t. $|z_n - z_f| < \delta $.  

Omlin and Giles, 1996,  use this idea of stability to show that for provably stable second-order RNNs, the sigmoidal discriminant function should drive neuron values to near saturation. Similarly, Stogin and Mali, 2020 use the fixed point analysis to prove the stability of differentiable stack and differentiable tape with RNNs. 

**Definition 2**
Let $\mathcal{E^*}$ be a stochastic fixed point estimator for $f$. The estimated $RMS$ error for fixed point estimation  $\hat z_{f;i} = \mathcal{E^*}(f)$ can be defined as:
$$
    e_{rms} = \sqrt{\frac{1}{n}\sum_{i=0}^{n} || \hat z_{f;i} - z_f ||^2}
$$

Note that when true $z_f$ is unknown, the law of large numbers allows us to use the estimated mean $\overline{z_f}$ instead. Also the error estimate $e_{rms}$ is the standard deviation from $z_f$

**Stability of States**
Let states $Q$ of DFA ($\Sigma, Q, \delta, q_0, F$) be embedded in the metric space of $\mathcal{R}^n$ and mapping $\delta : Q \times \Sigma \rightarrow Q$ be the transition function. Consider transitions for state $q \in Q$ s.t. $\delta(q, a) = q$ for some $a \in \Sigma$. By Theorem 1 and Definition 2, the error in the stability of states can be estimated. The stability of states directly influences the stability of predictions.

---

**We have updated our paper with this stability analysis and updated the Conclusion section to reflect that. All changes made in the paper are highlighted in blue font color**

---

With the help of your feedback, we have grounded our motivation for this paper into two research questions, R1 and R2, as mentioned in the Paper Overview commented above, and updated our Conclusion accordingly. In the conclusion section, we show that the empirical results for DFA stability are well within the theoretical framework of stable state estimation and answers R1 and R2. 

Thank You !

Reviewer comment (Evaluation):
A more thorough justification for the conclusion would be beneficial. Instead of solely benchmarking several methods across various settings, delving into the rationale behind the conclusions drawn would strengthen the paper and provide a more comprehensive perspective.

Author response:
Dear Reviewer nZtd,

We thank you for your insightful reviews and comments.

We base our notion of stability of DFA extraction on Brouwers's Fixed Point Theorem and build on previous theoretical works on the stability of state estimation by RNNs. 

---
**Theorem 1**
BROUWER’S FIXED POINT THEOREM: For any continuous mapping $f:Z \rightarrow Z$, where Z is a compact, non-empty convex set, $\exists \ z_f$ s.t. $f(z_f) \rightarrow z_f$

**Definition 1**
Let $\mathcal{E}$ be an estimator of fixed point $z_f$ for mapping $f:Z \rightarrow Z$, where $Z$ is a metric space. $\mathcal{E}$ is a stable estimator iff estimated fixed point $\hat z_f$ is in immediate vicinity of $z_f$, i.e. $ | \hat z_f - z_f | < \epsilon, $ for arbitrarily small $\epsilon$.

**Corollary 1**
Stability of estimator $\mathcal{E}$ can be shown by iteratively computing $z^{(t+1)} = \mathcal{E_f} (z^{(t)})$ with $z^{(0)}$ in the neighborhood of $z_f$. For stable estimator $\mathcal{E}$, $\lim_{t\rightarrow \infty}z^{(t)}  = z_f$. Neighborhood of $z_f$, $N_{z_f}$ is a set of points $z_n \in N_{z_f}$ near $z_f$ s.t. $|z_n - z_f| < \delta $.  

Omlin and Giles, 1996,  use this idea of stability to show that for provably stable second-order RNNs, the sigmoidal discriminant function should drive neuron values to near saturation. Similarly, Stogin and Mali, 2020 use the fixed point analysis to prove the stability of differentiable stack and differentiable tape with RNNs. 

**Definition 2**
Let $\mathcal{E^*}$ be a stochastic fixed point estimator for $f$. The estimated $RMS$ error for fixed point estimation  $\hat z_{f;i} = \mathcal{E^*}(f)$ can be defined as:
$$
    e_{rms} = \sqrt{\frac{1}{n}\sum_{i=0}^{n} || \hat z_{f;i} - z_f ||^2}
$$

Note that when true $z_f$ is unknown, the law of large numbers allows us to use the estimated mean $\overline{z_f}$ instead. Also the error estimate $e_{rms}$ is the standard deviation from $z_f$

**Stability of States**
Let states $Q$ of DFA ($\Sigma, Q, \delta, q_0, F$) be embedded in the metric space of $\mathcal{R}^n$ and mapping $\delta : Q \times \Sigma \rightarrow Q$ be the transition function. Consider transitions for state $q \in Q$ s.t. $\delta(q, a) = q$ for some $a \in \Sigma$. By Theorem 1 and Definition 2, the error in the stability of states can be estimated. The stability of states directly influences the stability of predictions.

---

**We have updated our paper with this stability analysis and updated the Conclusion section to reflect that. All changes made in the paper are highlighted in blue font color**

---

With the help of your feedback, we have grounded our motivation for this paper into two research questions, R1 and R2, as mentioned in the Paper Overview commented above, and updated our Conclusion accordingly. In the conclusion section, we show that the empirical results for DFA stability are well within the theoretical framework of stable state estimation and answers R1 and R2. 

Thank You !

Reviewer comment (Writing):
The way this paper motivates the reader, and how the background is introduced, is somewhat too narrowly focused such that only researchers working in the most vicinity of the DFA extraction area could appreciate it.

Author response:
We understand that the community of researchers working on DFA extraction is relatively small. However, this methodology is well accepted in the community working towards explainable AI. Since the dawn of computing, DFAs have been a fundamental structure that has guided hardware and software designs. Chomsky’s hierarchy provides us with a framework that allows us to classify problems based on their complexity and design machines to solve them. In the case of neural networks, Delétang, et. al. 2022, have shown the performance of RNN, memory-augmented NNs, and Transformer networks on a variety of tasks classified into different categories according to Chomsky’s hierarchy. In neural networks, RNNs are the only networks that can model state transitions directly in their architecture. Other networks like Transformers are shown to solve stateful problems, but **state extraction from Transformers has remained an open research question**. On the contrary, there have been a lot of works that demonstrate the extraction of states from RNNs and create a DFA. However, RNNs are stochastic systems. Omlin and Giles, 1996,  have shown the fixed point stability of RNN can be achieved using constraints on discriminant function. However, ours is the first empirical study showing the stability of DFAs extracted from 1st and 2nd-order RNN architectures and contrasting the performance of quantization and equivalence query mechanisms. We show that 2nd-order RNNs are better at state estimation, and quantization methods can stably extract these states.

**Why it is important**: Equivalence with DFA helps in the verification of NNs (Wang et al., 2020) and formal verification (Alur et al., 2009). Extracting automata from RNNs is termed interpretable by extraction (Marzouk et al., 2020). Thus, the model can be termed interpretable if it shows stable rule extraction behavior.  Third, O2RNN can refine and insert rules (Giles and Omlin, 93), an essential direction towards neuro-symbolic and XAI. Thus if we can demonstrate the stability of these networks, then we can use them in various high-risk domains. For instance, automata are used in network and cyber security, DNA, health, controls, etc. Thus, this study will open the door to other fields where explainability and verification are essential.

**Why it is relevant to the ICLR audience**: We believe the ICLR community will greatly benefit from this empirical study and theoretical foundation of RNNs, showing the stability.

Reviewer comment (Writing):
Even though this paper aims to provide empirical validation of some prior theoretical results, its exposition of the theoretical background is a little incomplete and not well organized (for example, what is  $\mathcal{E}_f$ and what's its relation with $\mathcal{E}$ and $\mathcal{E}^*(f)$? What does the subscript $|_i$ means in notation $\mathcal{E}^*(f)|_i$?) I think the paper would benefit hugely from a rewriting of these sections, to fill the gap between rough discussions of prior works and actual mathematical backgrounds and the question this paper is tackling.

Author response:
We greatly appreciate your suggestions on notations. Here is more clarification on the notations:

+ $\mathcal{E}$ is an arbitrary fixed point estimator
+ $\mathcal{E}_f$ is fixed point estimator for mapping $f : Z \rightarrow Z$
+ $\mathcal{E}^*$ is a stochastic arbitrary fixed point estimator
+ $\mathcal{E}^*(f)$ is a stochastic arbitrary fixed point estimator  for mapping $f : Z \rightarrow Z$. We aim to change it to $\mathcal{E}^*_f$ for clarity and consistency.
+ $z_f$ is the true fixed point for mapping $f$.
+ $\hat{z}_f$ is the estimated fixed point for mapping $f$.
+ Since fixed point estimation is done in an iterative way, $\hat{z_{f;i}}$ is the fixed point estimated in $i^{th}$ iteration. For clarity and consistency we aim to change this to  $\hat{z}_{f}^{(i)}$ 
+ The expression $\hat{z}_{f;i} = \mathcal{E}^*(f)|_i$ represents the stochastic estimation of fixed point at $i^{th}$ iteration. For clarity and consistency we aim to change this to $\hat{z}_f^{(i)} = \mathcal{E}^*_f(\hat{z}_f^{(i-1)})$. This will represent the iterative nature of the stochastic estimator in a better way. 

We will update the paper with notations and better explanations upon acceptance.

Reviewer comment (Evaluation):
The importance of the key evaluation metrics is not well motivated, and corresponding results are not well explained. For example, this paper should motivate more on why the number of extracted states can be used to evaluate DFA extraction strategies, and why lower numbers are preferred (yes the conclusion section contains better forms of your research questions but still why aren't they presented earlier?).

Author response:
In Table 2, we have shown the number of minimal states in Tomita grammars. The goal is to extract DFA with the number of states close to the optimal number of states required for a given language (minimalistic or minimal DFA). If the extraction process produces more states, that can happen for any of the following reasons:
+ RNN could not converge to the actual states.
+ The extraction process could not partition state space optimally.

It is well known that converting any DFA (N) to minimal DFA (M`) can be achieved in polynomial time. However, it is important to note that NNs are stochastic thus, initially, state machines learned from RNN can be considered as non-deterministic or probabilistic. In this scenario, if we assume language learned by RNN (L1) = NFA (L1) and try to minimize NFA, then the problem becomes NP-complete or PSPACE-complete problem (Malcher et al., 2004). Furthermore, **the decidability problem for first-order RNN with probabilistic FA/ Weighted FA/non-deterministic FA is undecidable (marzouk et al., 2020)**. Furthermore, the equivalence between two DFAs is also NP-complete (Martens et al., 2023). 

Thus, in both cases mentioned above, even if we get a state machine or DFA, it would not be possible to extract the minimal and correct DFA in polynomial time. Thus it becomes very difficult to show empirically that two languages accepted by source DFA and RNN DFA are same. Given equivalence is NP-complete. 
To this end it’s important we have RNNs that could extract minimal DFA without any bell and whistle and we show second order RNN can stably achieve this.

We motivate our paper and aim to showcase the empirical results in our Introduction. We provided R1 and R2 in a more theoretical perspective only after introducing the underlying mathematical basis and formulations in the Background and Methodology section.

Reviewer comment (Novelty):
The overall significance of the results in this paper is limited by the paper's choice to study only RNNs and rule extraction approaches, as they do not imply any further empirical messages, as the paper has not pointed to any other potential implications.

Author response:
The significance of this paper is in the context of the stability of rules extraction and embedding or RNN verification and explainability. We compare extraction methods for 1st and 2nd order RNNs. This paper encourages the research community to opt for RNN architectures that can estimate states more stably and use stable state extraction mechanisms for rule verification. This is an essential step in stable and explainable AI. We have covered the importance of this work in the first point.