Paper: Complexity of Formal Explainability for Sequential  Models (ICLR-2024)

Reviewer comment (Experiments):
This is a purely theoretical paper with a list of propositions and proof sketches but with absolutely no practical side of things. No experimental evidence is provided either. As a result, whether or not FXAI is usable in the context of such models is left unclear. (Recall that AXp's and CXp's are known to be in general intractable to compute for non-sequential models; so no big deal.) This makes me wonder why this line of work can be deemed relevant for a conference like ICLR. (KR would be a nice fit but ICLR is hard to justify.)

Author response:
Q: How are they related to machine learning, FXAI, and ICLR? 
A: We thank you for your questions and comments. We would like to convince you that our results are of interests to those working in FXAI and ML. In the revised version, we have placed a stronger emphasis on the connection to model generation (learning) and explanation generation. Note that our results concern the latter, so we mention relevant existing results on the former to put the results in contexts. We summarize the connection to ML below.

Our work was actually very much motivated by the machine learning work on the extraction of DFA from RNN using automata learning algorithms (e.g. Angluin's L*), e.g., see the ICML paper by Weiss et al. (2018) and the AAAI paper by Okudono et al. (2020). This work hinges on the assumption that DFA is *more interpretable* than RNN, but it was never really clear what this means. Following the work for non-sequential models, we associate interpretability with the computational tractability (following standard complexity theoretic assumptions) to compute abductive and counterfactual explanations. In this technical sense, we showed that DFA is more interpretable than RNN and transformers.

So, a fuller working (F)XAI schema for could be as follows: (1) train RNN R, (2) extract DFA A out of R (e.g. using Weiss et al.), and (3) use our algorithm on A to generate explanations. 

Note that (1) could perhaps be replaced directly by an algorithm that extracts a DFA A directly from the datasets. Note also that although we deal with decision problems, the polynomial-time algorithms can be used to extract an explanation (we have clarified this in the new version).

The aforementioned concentrates on a finite alphabet. Since time series (e.g. stock price chart) is ubiquitous in applications, we went one step further to look at sequential models that may take *data sequences* as input (i.e. sequences of rationals). Two prominent extensions of DFA in this setting are interval automata and k-deterministic register automata, for both of which automata learning algorithms (e.g. Moerman et al. (2017); Howar et al. (2012); Cassel et al. (2016); Drews & D’Antoni (2017); Argyros & D’Antoni (2018)) are available. In the new version, we have also expanded our small examples in the previous version on stock price signals, and how they can be captured by Interval Automata and 2-DRA. Finally, our results are a fundamental building block that enables a similar working schema to the aforementioned case of finite alphabet.

Reviewer comment (Presentation):
Overall presentation could and should be significantly improved. Definitions of abductive and contrastive explanations in the context of sequential models are given in a somewhat lazy way. (The models are defined similarly.) Also, the paper gives no examples whatsoever that would demonstrate how these explanations look like for sequential models. As this paper claims to be the first of its kind, such examples should be provided.

Author response:
Q: a clear example of a model with AXp and CXp explanations?

A: In the new version, we have expanded our small examples in the previous version (Example 1 to Example 3). They concern a simple document identification task (using DFA over finite alphabets) and a simple stock trading signal (using interval automata/k-DRA). We show how counterfactual/abductive explanations mean in this context.

Reviewer comment (Novelty):
No algorithms to be used for computing such explanations are proposed (even in theory). This exacerbates the issue of the venue choice above.

Author response:
Q: No algorithms to be used for computing such explanations are proposed (even in theory). 
A: Thank you for bringing up the question of computing explanations. We have stated our positive results as polynomial-time algorithms for decision problems. However, the algorithms can actually generate (counterfactual) explanations in polynomial-time. We apologize for not making this explicit, which we have corrected in the revised version.

Reviewer comment (Theory):
Minor: on p9 there is a sentence: "This type of explainability falls within the category of post-hoc local model-agnostic explanations". No, it does not. Both abductive and contrastive explanations are global. None of them are model-agnostic as you have to reason about the model's behavior formally and hence require access to its original representation.

Author response:
Q: Are your explanations local and model agnostic?

A: Local explanations refer to less complex solution subspaces (e.g. specific inputs), e.g., see survey https://arxiv.org/abs/1910.10045. We believe that both abductive and counterfactual explanations are local explanations since they refer to a specific input instance. The same claim was also made in previous works on such explanations (e.g. see the citation Barcelo et al. (2020) in our paper).

Model-agnostic explanations treat the model as a blackbox, as far as the explanations. See the survey https://www.frontiersin.org/articles/10.3389/fdata.2021.688969/full. They explicitly pinpoint that counterfactual explanations are model-agnostic.

Reviewer comment (Presentation):
Can you show a clear example of an RNN model, some of the instances it takes as input and also exemplify abductive and contrastive explanations for some of the model's decisions?

Author response:
Q: a clear example of a model with AXp and CXp explanations?

A: In the new version, we have expanded our small examples in the previous version (Example 1 to Example 3). They concern a simple document identification task (using DFA over finite alphabets) and a simple stock trading signal (using interval automata/k-DRA). We show how counterfactual/abductive explanations mean in this context.

Reviewer comment (Reproducibility):
Are the algorithms originally proposed for AXp and CXp computation are directly usable in this setting? If not, how should one compute them?

Author response:
Q: Are the algorithms originally proposed for AXp and CXp computation are directly usable in this setting? If not, how should one compute them?

A: Existing algorithms for AXp and CXp concern non-sequential models, so cannot be used in our setting. Our polynomial-time algorithms can generate the counterfactuals in polynomial-time. We made this explicit in the revised version.

Reviewer comment (Theory):
Can you summarize the differences between the complexity of these problems for non-sequential and sequential models? This way, can you contrast your results with what is already known to hold in general?

Author response:
Q: Differences between the complexity of these problems for non-sequential and sequential models? 

A: From existing works (e.g. by Izza et al. and Barcelo et al.), the complexity of explainability is typically decidable (within PSPACE) for non-sequential models. In contrast, the same problems are undecidable in the sequential setting, owing to Turing-completeness of RNN and Transformers.

Reviewer comment (Presentation):
In terms of presentation I think a small table summarizing all results would be helpful; the complexity landscape of the different queries is spread through the text right now.

Author response:
Thanks for the excellent suggestion. We will add this in the final version.

Reviewer comment (Experiments):
The counter to this point is that the presented extensions to DFAs seem like an interesting direction inspired by the complexity analysis. Nonetheless, there is no experimental evidence showing that they can be useful in practical prediction tasks, or directions towards making them more practical.

Author response:
We would like to convince you that our results can in fact provide insights into the models. We have stated our positive results as polynomial-time algorithms for decision problems. However, the algorithms can actually generate (counterfactual) explanations in polynomial-time. We have made this more explicit in the revised version. 

Although the focus of our current paper is on classifying explainability of different non-sequential models and, we do agree that providing experimental evidence of polynomial-time algorithms in the paper is an important next step. Another interesting research direction is how to realize the aforementioned (F)XAI schema for sequential data, by leveraging existing automata learning algorithms for DFA, interval automata, and k-DRA.

Reviewer comment (Novelty):
Perhaps something that could help the evaluation of the paper is some justification for this work being fit for the venue; at least superficially it seems of a slightly different nature than most ICLR work.

Author response:
We thank you for your comment and question. In the new version, we have placed a stronger emphasis on the connection to model generation (learning) and explanation generation. Note that our results concern the latter, so we mention relevant existing results on the former to put the results in contexts. We summarize the connection to ICLR work below.

Our work was actually very much motivated by the machine learning work on the extraction of DFA from RNN using automata learning algorithms (e.g. Angluin's L*), e.g., see the ICML paper by Weiss et al. (2018) and the AAAI paper by Okudono et al. (2020). This work hinges on the assumption that DFA is "more interpretable" than RNN, but it was never really clear what this means. Following the work for non-sequential models, we associate interpretability with the computational tractability (following standard complexity theoretic assumptions) to compute abductive and counterfactual explanations. In this technical sense, we showed that DFA is more interpretable than RNN and transformers.

So, a fuller working (F)XAI schema for could be as follows: (1) train RNN R, (2) extract DFA A out of R (e.g. using Weiss et al.), and (3) use our algorithm on A to generate explanations. 

Note that (1) could perhaps be replaced directly by an algorithm that extracts a DFA A directly from the datasets. Note also that although we deal with decision problems, the polynomial-time algorithms can be used to extract an explanation (we have clarified this in the new version).

The aforementioned work so far concentrates over strings (over a finite alphabet). Since time series (e.g. stock price chart) is ubiquitous in multiple applications, we went one step further to look at sequential models that may take "data sequences" as input (i.e. sequences of rationals). Two prominent extensions of DFA in this setting are interval automata and k-deterministic register automata, for both of which automata learning algorithms (e.g. Moerman et al. (2017); Howar et al. (2012); Cassel et al. (2016); Drews & D’Antoni (2017); Argyros & D’Antoni (2018)) have been developed in the last 10 years. In the new version, we have also expanded our small examples in the previous version on stock price signals, and how they can be captured by Interval Automata and 2-DRA. Finally, our results are a fundamental building block that enables a similar working schema to the aforementioned case of finite alphabet.

In the revised version, we have also expanded our small examples in the previous version (Example 1 to Example 3). They concern a simple document identification task (using DFA over finite alphabets) and a simple stock trading signal (using interval automata/k-DRA). We show how counterfactual/abductive explanations mean in this context.

Reviewer comment (Theory):
Unless I'm missing something, the proof of Prop 3. seems incomplete! The sentence ends with "This is equivalent to checking". I imagine this was an oversight during submission.

Author response:
We thank you for pointing this out, and sincerely apologize for the confusion. There appeared to be a problem with a LaTeX macro that caused this part (Proposition 3 and incomplete proof) to be displayed when we switched into a submission mode. Certainly, this was not meant to be there, which we have deleted in the revised version.

Reviewer comment (Theory):
Regarding the proof Prop 2, isn't DFA intersection / minimization poly time -- intersection is at product construction and minimization is polytime due to Hopcroft's algorithm. So testing emptiness should be polytime. Why does the proof imply PSPACE-completeness?

Author response:
Indeed, checking emptiness of the intersection of two DFA is solvable in polynomial-time, but this problem becomes PSPACE-complete if we want to check emptiness of n DFAs (where n is non-fixed number). This was proven by Kozen in the FOCS'77 paper titled "Lower bounds for natural proof systems."

Reviewer comment (Theory):
Even if explanations could be generated for the purpose of checking them with formal tools, in what applications would it be helpful given the definitions of "abductive" and "counterfactual" explanations. The example on the top of page 2 does not convince me in the slightest, given that in modern applications features are basically never equipped with clean semantics as assumed here.

Author response:
Q: Features are never equipped with clearly defined semantics  
A: We agree with this. That said, we would like to convince you that regular languages (and interval automata, as well as k-DRA, in the case of data sequences) are still reasonable representations of features. Firstly, they are well-known to be sufficiently expressive to capture a variety of concepts in many applications. Secondly, a regular language representing a specific feature can itself be learned using an automata learning algorithm. For example, we have a DFA A that classifies whether an input is a spam. Features could be: (1) a DFA B that identifies if the email sender asks for money, and (2) a DFA C that identifies if there are dangerous links (a huge list of known dangerous links). Even when B does not have a clear semantics, it is possible to infer B from examples using a learning algorithm.

Reviewer comment (Novelty):
The paper starts with some very simple negative results for RNNs and Transformers and then focuses almost entirely on automata models (DFA, NFA, and Interval Automata) that have an unclear connection to most work in ICLR.

Author response:
Q: The paper focuses on DFAs and Interval Automata. How are they related to machine learning and ICLR?  

A: We thank you for your comment. In the new version, we have placed a stronger emphasis on the connection to model generation (learning) and explanation generation. Note that our results concern the latter, so we mention relevant existing results on the former to put the results in contexts. We summarize the connection below.

Our work was actually very much motivated by the machine learning work on the extraction of DFA from RNN using automata learning algorithms (e.g. Angluin's L*), e.g., see the ICML paper by Weiss et al. (2018) and the AAAI paper by Okudono et al. (2020). This work hinges on the assumption that DFA is "more interpretable" than RNN, but it was never really clear what this means. Following the work for non-sequential models, we associate interpretability with the computational tractability (following standard complexity theoretic assumptions) to compute abductive and counterfactual explanations. In this technical sense, we showed that DFA is more interpretable than RNN and transformers.

So, a fuller working (F)XAI schema for could be as follows: (1) train RNN R, (2) extract DFA A out of R (e.g. using Weiss et al.), and (3) use our algorithm on A to generate explanations. 

Note that (1) could perhaps be replaced directly by an algorithm that extracts a DFA A directly from the datasets. Note also that although we deal with decision problems, the polynomial-time algorithms can be used to extract an explanation (we have clarified this in the new version).

The aforementioned work so far concentrates over strings (over a finite alphabet). Since time series (e.g. stock price chart) is ubiquitous in multiple applications, we went one step further to look at sequential models that may take "data sequences" as input (i.e. sequences of rationals). Two prominent extensions of DFA in this setting are interval automata and k-deterministic register automata, for both of which automata learning algorithms (e.g. Moerman et al. (2017); Howar et al. (2012); Cassel et al. (2016); Drews & D’Antoni (2017); Argyros & D’Antoni (2018)) have been developed in the last 10 years. In the new version, we have also expanded our small examples in the previous version on stock price signals, and how they can be captured by Interval Automata and 2-DRA. Finally, our results are a fundamental building block that enables a similar working schema to the aforementioned case of finite alphabet.

Reviewer comment (Theory):
The negative results here seem to (at least partially) rely on the fact that weights are rational numbers. However, in modern models the exact value of these weights does not seem to matter very much, as can be seen by the success of quantization techniques. Hence, a much more interesting model of RNNs and Transformers could be one where weights are binary or chosen from a small range (e.g. a 4-bit representation). In terms of explanations: how hard is it to extract small formulas that represent the output, perhaps with PAC guarantees?

Author response:
Q: What about weights with a few bits? Extracting representations of the output with PAC guarantees?  

A: Thank you for this interesting idea for future work. In terms of explainability, the results from Barcelo et al. (2020) showed that small weights (with a fixed number of bits) are sufficient for obtaining NP-hardness for multilayer perceptrons (especially,  MinimumChangeRequired). This NP lower bound carries over to RNN and Transformers. As you correctly pointed out, the proofs of other negative results no longer hold (e.g. Turing-completeness of RNN and Transformers), so a better upper bound could be possible. In terms of a generation of a simplified model (or an explanation), the same working schema that we mentioned above (RNN -> Automata -> explanation) could still be used in this case, but indeed it would be interesting in the future to provide a more solid theoretical guarantee of such a method. [Incidentally, works on extraction of DFA from RNN using ideas of PAC-learning and Angluin's stochastic implementation of equivalence queries are available, e.g., Mayr and Yovine (2018,2021).]

Reviewer comment (Theory):
Proposition 3 seems to directly contradict Proposition 2: interval automata are more general than DFA, generating is harder than deciding the existence. In fact the proof of Proposition 3 stops in the middle, and I could not locate the proof in the appendix. This has to be fixed. This is why I put a soundness score of 2, but if it was only a mistake from the authors with their latex files during submission and is corrected the soundness score could go higher. This has been fixed in the updated version.

Author response:
Q: Propositon 3 seems to directly contradict Proposition 2 … interval automata are more general than DFA, generating is harder than deciding the existence... In fact the proof of Proposition 3 stops in the middle, and I could not locate the proof in the appendix **************** This has to be fixed. This is why I put a soundness score of 2, but if it was only a mistake from the authors with their latex files during submission and is corrected the soundness score could go higher

A: Thanks for noticing this. There appeared to be a problem with a LaTeX macro that caused this part (Proposition 3 and incompleted proof) to be displayed when we switched into a submission mode. Certainly, this was not meant to be there, which we have deleted in the revised version. We sincerely apologize for the confusion.

Reviewer comment (Theory):
Page 2, "the same problems were shown to be solvable in polynomial-time for decision trees and perceptrons Barceló et al. (2020)": This is misleading. According to Barceló et al. (2020), proposition 6, minimum sufficient reasons is NP-hard for decision trees.

Author response:
Q: p2 "the same problems were shown to be solvable in polynomial-time for decision trees and perceptrons Barceló et al. (2020)": This is misleading. According to Barceló et al. (2020), proposition 6, minimum sufficient reasons is NP-hard for decision trees.

A: We thank you for noticing this and apologize for the confusion. Indeed, minimum sufficient reasons are NP-hard for decision trees, whereas minimal sufficient reasons are polynomial-time solvable (which is the problem we deal with in the paper). We have corrected the citations in the revised version.

Reviewer comment (Theory):
Page 4, why do you forbid an abductive explanation to be empty? The definition seems broken and should be fixed as follows: it is simply a minimal (under inclusion) subset of features such that v satisfies all features and all completions are classified the same. If this set turns out to be empty, this just means that the model is constant, and so the "empty" explanation simply explains this.

Author response:
Q: p4 why do you forbid an abductive explanation to be empty? I think that your definition is broken and should be fixed as follows: it is simply a minimal (under inclusion) subset of features such that v satisfies all features and all completions are classified the same. If this set turns out to be empty, this just means that the model is constant, and so the "empty" explanation simply explains this.
A: We have incorporated your more general definition in the revised version, noting that an empty intersection results in \Sigma^*. The PSPACE-hardness result holds either way.

Reviewer comment (Theory):
Proposition 2, and in general: you could also consider the setting where the set of features is fixed, which would give PTIME for DFAs.

Author response:
Q: Proposition 2, and in general: you could also consider the setting where the set of features is fixed, which would give you PTIME for DFAs.

A: Thanks for the suggestion. We have added this statement in the text.

Reviewer comment (Writing):
Page 9, "RNN and Transformers are either difficult": grammar/English issue.

Author response:
Thanks for the "last comment" - we have implemented this correction in the new revised version.