Explainability Via Causal Self-Talk
Anonymous Author(s)
AffiliationAddress email
Abstract
Explaining the behavior of AI systems is an important problem that, in practice, isgenerally avoided.
While the XAI community has been developing an abundanceof techniques, most incur a set of costs that the wider deep learning communityhas been unwilling to pay in most situations.
We take a pragmatic view of theissue, and define a set of desiderata that capture both the ambitions of XAI and thepractical constraints of deep learning.
We describe an effective way to satisfy allthe desiderata: train the AI system to build a causal model of itself.
We developan instance of this solution for Deep RL agents: Causal Self-Talk.
CST operatesby training the agent to communicate with itself across time.
We implement thismethod in a simulated 3D environment, and show how it enables agents to generatefaithful and semantically-meaningful explanations of their own behavior.
Beyondexplanations, we also demonstrate that these learned models provide new ways ofbuilding semantic control interfaces to AI systems.
Introduction
As modern machine learning systems become more powerful and embedded in our lives, the needto have these systems explain their behavior becomes increasingly urgent.
Despite incredibleperformance in a variety of domains, almost all systems are completely unable to provide a satisfyinganswer to the simple question, "Why did you do that?".
While there have been innovations in particular explainability methods [e.g. 8, 55, 56, 66], andsome attempts to apply techniques to domains that demand it
[e.g. 23, 65], the vast majority of deeplearning applications do not engage with these techniques.
This is in part because there are fewgeneral methods, and most incur costs such as reduced performance and scalability; compoundingthis, existing techniques often fail to deliver a useful or credible account of the systems’ behavior.
We believe that the reason for this is that explanation systems tend to fail to satisfy at least one ofa number of key desiderata.
Two particular failure modes are most common: the addition of anexplanation system limits generality or hurts performance (e.g. when it requires explicitly structuringthe base system); or the explanations it produces cannot be g these pitfalls expla model
Our contributions are as follows.
In Section 2, we articulate five key desiderata for explanationsystems.
However, there is some tension between these.
In Section 3, we describe how commonhistorical XAI techniques resolve these trade-offs by making extreme choices between the desiderata.
In Section 4, we demonstrate that a balance can be found between the five desiderata by training AI systems to build self-models.
In particular, we introduce a solution concept for Deep RL agents:
Casual Self-Talk (CST).
In Sections 5-6 we show that CST allows model-free RL agents in anembodied 3D virtual environment to faithfully explain their behavior in terms of semanticallyaccessible beliefs about the state of the world.
Desiderata for explanatory models
Our goal is to build an explanatory model of an AI system, i.e. a mechanism that translates betweenits internal representations and/or computations, and some other representational form.
We set outfive key desiderata for such a model:
Grounded.
For the model to be useful to external users, it should deploy a representationlanguage that is semantically-grounded.
This may require additional data to train.
Flexible.
The method should be able to produce models of many different representationalforms (e.g. classes, graphs, strings).
Ideally it should also be sufficiently general to be agnosticto the base system’s input/output modalities, and possibly even its overall architecture.
Minimally-interfering.
Training and inference on the model should have as little impact aspossible on the training and inference of the base system.
Solutions which are disruptive toperformance are highly unlikely to be adopted in practice.
Scalable.
Once a solution is found, it should be straightforward to apply it to any network.
Faithful.
The model should give accurate, factual, and truthful descriptions of the underlyingsystem.
The gold standard is that it is a causal model of the agent, facilitating validationthrough interventions [38].
This is possibly the most challenging to achieve.
A number of past works have also articulated desirable properties for explanations [e.g. 28, 35, 36,38, 49, 53, 69].
Our desiderata focus less on the qualities which constitute a good explanation perse, but instead on what is desired of the method itself that generates the explanation.
Common historical solutions
We examine whether various existing explanation techniques satisfy each of these desiderata.
Attribution methods.
A set of popular techniques attempts to model a neural network’s decisionmaking at the input level, by estimating contributions to its output from input features [e.g. 18, 27, 56],or training examples [e.g. 6, 14, 42, 43, 46, 71], or by finding optimal inputs for a given output[e.g.
Such approaches benefit from being minimally-interfering (they are entirely post-hoc)and scalable (they can be automated).
However, some methods have been empirically shown to beunfaithful [2, 7], while methods which explicitly aim for faithfulness such as LIME [13] and Anchors[60] are only local in scope, and are expensive to scale.
Most importantly, all such methods sacrificeflexibility: one is constrained to produce explanations of a very particular form, e.g. expressed interms of the input modality, whose interpretation may be highly subjective.
Explicitly structuring networks.
A sub-field of deep learning has emerged wherein the intermediate computations are constrained to particular forms, and/or are grounded by external data[e.g.
As explanations of behavior, structured intermediates areattractive as they are both grounded and faithful by design.
However, this is achieved at the expenseof minimal-interference, as it requires fundamentally altering the base system to explicitly depend onthis structure.
It is also limited in flexibility and scalability, as the technique can also only be applied when sufficient domain knowledge or data is available, and only in the form dictated by these priorconstraints.
These issues are exacerbated by the fact that the representations required to optimizeperformance objectives are often at odds with those that would subserve explainability.
Fine-grained mechanistic interpretability.
One common approach is to allow a base system totrain and run au naturel , and to post-hoc dissect its representations or computations, often usingtechniques borrowed or inspired from neuroscience [e.g. 17, 21, 26, 57].
Such approaches championthe minimal-interference desideratum.
In the best case, they promise to uncover faithful descriptionsof the actual causal processes operating within the base system.
However, this comes at a huge cost toscalability.
Many human hours are typically expended in reverse engineering, often yielding resultsthat are limited to a handful of neurons or circuits seen in a single network.
Moreover, by makingstrong commitments to respect the internal representations of the base system, it is very difficult toground the models that arise, let alone to flexibly choose the form they take.
Decoders.
Decoders can be viewed as trained tools to map from the internal private language ofa base system to an interpretable representation space [e.g. 4, 11, 15, 31–33, 44, 52, 58, 74].
Thisapproach has many attractive qualities: it produces semantically-grounded representations of thebase system’s internal private language; is flexible in the choice of modality; scales well; and isminimally-interfering, insofar as decoders can be separated from the inference path of the maincomputational graph and gradients can be stopped from propagating to the base system if desired(though are often not to assist representation learning, [e.g. 3, 37, 40, 48, 54]).
This is all achieved,however, at the expense of faithfulness, as decoded outputs are off the causal path of the base system[47].
Decoders just report whether information is present in the source representation, not whetherthat information is actually being used for behavior [12, 20, 59].
Similar faithfulness issues arisewith post-hoc behavioral analyses [e.g. 19, 30, 39, 50, 66, 70].
Solution concept
The challenges faced by previous attempts at explainability show that the tensions between thedesiderata are considerable.
It is nonetheless possible to reason a way forward.
First, we want a description of the base system that is both grounded and flexible.
The internalsof the base system are neither.
While we could explicitly structure the base system to change this,this would violate the minimal interference desideratum.
Thus we need a model of the base system.
Second, the solution must be scalable.
This means we cannot rely on human labor to map fromthe base system’s representations to the model’s representations.
The only scalable approach is tolearn the model .
Third, the solution must be faithful.
Our gold standard of faithfulness requires theability to validate the model by intervening on it.
To do this, one must be able to map in the oppositedirection: from interventions on the model onto interventions on the base system.
Because both thebase system and the model are learned, so this mapping must be learned as well.
The most direct wayof learning this mapping is to feed the model’s outputs as inputs back into the base system.
A solution to these problems is thus a “self-model” which is intertwined with the base system.
It isbuilt of the base system , by the base system , and for the base system .
We develop an instance of this solution in a particular setting: a Deep RL agent, acting in an episodic
POMDP, which builds a model of itself in terms of a set of semantically-grounded beliefs about thestate of the world.
We consider how to expand the scope of this technique in Section 7.
We follow the standard construction of a POMDP.
We denote the agent’s observation at time t withinan episode as x t , and assume that it maintains an internal state m t , which we can take to be, forexample, a recurrent state (such as from an LSTM), or a variable-length array of embeddings ofprevious observations x 0: t (as used in a Transformer).
We denote the agent’s state update function asf θ , with learned parameters, θ , s.t. m t +
=
f θ ( x t +1 , m t ) .
We denote its policy as π t
= h θ ( m t ) .
Our construction starts with a decoder, shown in Figure 1a.
A decoder maps from the state m t toan output z t ∼ q ( z t ) , via a learned function g θ : m t → q ( z t ) .
This mapping is trained using targetdata ˆ z ∈ D z that lives in a semantically-meaningful space.
As we previously identified in Section 3,decoders benefit from being grounded, flexible, scalable, and minimally-interfering.
However, theirweakness is faithfulness.
This is because the output z t merely reflects what information is present inm t , and not whether that information is actually used in the computation of π t .
Correlational faithfulness: if the decoder reports a value of z t =
z , to what degree is the behavior (from the policy π ) consistent with this value
Causal faithfulness:if we were to counterfactually intervene on z t , changing it to a valueof z ′ , to what degree would this produce behavior consistent with this new value, z ′ , ratherthan its original value, z ?
Causal faithfulness is a much stronger property than correlational faithfulness.
It’s highly desirablebecause it allows one to validate the agent’s commitment to the value of z t at runtime by makinginterventions on it.
This validation mechanism is especially useful when out of distribution.
Re-routed decoders.
While it is possible for a decoder to have a degree of correlational faithfulness,it is structurally impossible for it to exhibit causal faithfulness.
A simple fix is to route the output ofthe decoder back into the internal representation (Figure 1b), e.g. by concatenating z t with x t +1 .
Thisamounts to incorporating the decoder output in the state update rule, with m t +1 =
f θ ([ x t +1 , z t ] , m t ) .
The difficulty with this simple solution is that there is no guarantee that f θ will actually "listen" to z twhen producing m t
+
This is because the information in z t is largely if not completely redundantwith m t .
This may be further exacerbated by the presence of any stochastic sampling in the generationof z t from m t .
Indeed, as shall be described in Section 6, our experiments find that re-routed decodersmake no measurable progress beyond ordinary decoders on causal faithfulness.
How do we render the function f θ sensitive to the value of z , when there is redundancy between z tand m t ?
It is useful to frame the problem in terms of communication.
The decoder pathway can beviewed as a (externally-grounded) communication channel between the internal state and itself, i.e.a form of "self-talk" or "inner speech" [67].
The problem therein is that the speaker ( m t ) and thelistener (also m t ) are identical, so there is no useful information in the message z t for the listener.
Causal self-talk.
To overcome this problem, we need to create an information asymmetry betweenspeaker and listener.
We introduce Causal Self-Talk (CST) : we encourage f θ to be sensitive tothe message z t by creating an augmented data distribution where the speaker and listener diverge.
Fortunately, we have a ready source of alternate listeners: internal states m t ′ for t ′ < t .
We thus provide the decoder output with an additional role: to compactly and effectively communicateinformation to earlier versions of the same agent (from the same episode), via a semanticallygrounded communication channel.
Formally, this takes the form of an intervention (Figure 1c):the internal state, m t , is replaced with a counterfactual one, m t ′ , before being integrated with theoriginal message ( z t ) and next observation ( x t +1 ), in order to produce a counterfactual internal state,m ( t ← t ′ ) t + = f θ ([ x t +1 , z t ] , m t ′ ) .
This yields a counterfactual policy, π ( t ← t ′ ) t +
[Equation]
This alone, however, is insufficient to fully specify a CST algorithm.
We must also stipulate thedesired outcome from enacting these interventions.
We describe three alternative choices:
CST-RL: The messages z t should allow earlier versions of the same agent to maximizereturn given the current environment state.
To train this, interventions must occur online (according to some schedule), and we must allow the agent to act according to the counterfactualpolicy following an intervention.
We then use the maximization of subsequent discountedreward as the desired outcome from the intervention, via reinforcement learning (RL).
CST-MR: The messages z t should allow earlier versions of the same agent to reconstruct thenext internal state, m t +1 .
Unlike CST-RL, this can be done entirely in replay, without requiringthe agent to ever act according to the counterfactual policy on a live environment.
We add aweighted memory reconstruction (MR) term to the overall loss: L MR = || m
[Equation]
[-+
 || 2 .-][Equation]
CST-PD: The messages z t should allow earlier versions of the same agent to recover thetrue current (and future) policy.
As with CST-MR, interventions can be simulated entirely inreplay.
We add a weighted policy distillation (PD) term to the overall loss, using discountingfunction γ ( · ) :
L PD = (cid:80)
[Equation]
CST and the five desiderata.
CST offers a different, less extreme tradeoff between the desiderata.
Like ordinary decoders, CST can achieve grounding by supervising the representations z against asignal of known semantics.
It is also equally flexible.
Relative to ordinary decoders, there is sometrade-off with minimal-interference, insofar as the decoder output z t is now an obligate input intothe next time step.
However, much like ordinary decoders, the effects on training can, in principle,be contained through appropriate gradient flow, e.g. by limiting updates to parameters of the stateupdate function, f θ . It may also be possible to fine-tune a pre-existing base system.
CST is highly scalable.
CST-RL is straightforward to implement; in its simplest incarnation, fixingt ′
= 0 reduces to dropout on the whole internal state, m t .
The use of RL, however, occurs at theexpense of additional data as it requires online execution of the counterfactual policy.
CST-PD and
CST-MR, in contrast, are entirely self-supervised: they make use of the existing trajectories generatedduring the training of the base system.
These methods augment the data in a manner similar tothe interchange intervention technique of [25], using self-imitation as the desired outcome of theintervention.
Depending on the degree of non-stationarity afforded by the particular environment(and its degree of partial observability), the size of the augmented dataset can scale quadraticallywith the length of each episode (from choosing both t and t ′ ); in some settings, it may be possible toexceed this by sampling counterfactual previous states m t ′ from other episodes.
Finally, unlike ordinary decoders, CST is explicitly trained to provide causal faithfulness, by facilitating counterfactual control.
We test the efficacy of this in the following section.
Experiments
Task.
We study variants of CST in a 3D virtual environment built in Unity [1, 68].
This features afixed-layout indoor space comprising five rooms, each with a different wall color.
The agent receivesas input a 1 st -person visual observation and a text observation.
The agent can freely navigate using a4D continuous action space ( a ∈ [ − 1 , 1] 4 ) consisting of in-plane movements and a 2-DOF rotation.
We developed a simple task in this environment, called DaxDucks (Figure 2).
We designed DaxDucksto: (1) be easy for an agent to learn a good policy; (2) require an agent to maintain and update beliefsabout the latent environment state in order to maximize reward; (3) provide a source of data to groundthe self-model; and (4) provide a means to evaluate the faithfulness of the learned self-model.
DaxDucks is an instruction-based, fast-binding search task, where the agent is rewarded for finding aduck that exhibits an instructed tag.
Each episode consists of a sequence of trials.
At the beginning ofeach trial, the agent’s avatar spawns in the center of the middle room facing a random direction.
Fouridentical ducks spawn at the center of the four outer rooms.
Each duck is randomly assigned one offour tags (“ dax ”, “ gavagai ”, “ kleeg ”, or “ plork ”).
The agent is instructed to collect (i.e. collidewith) the duck with a specified tag via the text observation channel, e.g. “ Collect a gavagai. ”
The agent may observe a duck’s tag by entering the duck’s room and orienting the center of its view [Figure] towards the duck.
This adds an additional string to the text observation, e.g. “ This is a kleeg. ”.
Colliding with any duck terminates the current trial, in which case the agent is respawned in the centerof the middle room again and a random new instruction is delivered; a reward is also delivered if theagent collided with the correct duck.
Finally, when a new trial starts, the ducks usually maintain theirtags, but with a small probability ( p sh = 0 . 1 ), the tags are randomly shuffled between the ducks.
Thusthe agent can only maximize reward by effectively retaining a belief state over which tag belongs towhich duck.
Each episode lasts for 5000 steps regardless of the number of trials completed.
Agent architecture.
For the base system, we used a standard model-free agent architecture. Weencode inputs with a ResNet (images) and an LSTM (text); we use an LSTM memory module and MLPs for the policy and value heads; and we trained with V-trace [22].
All agents achieved a similarlyhigh return after training.
Full details are in Appendix A.
We demonstrate the flexibility of grounding by considering two forms for the self-model:
One-hot. The message z t takes the form of a set of four one-hot vectors (one per tag),expressing which room the agent believes each tag to be in.
Language.
The message z t takes the form of a synthetic language string, expressing whichroom the agent believes a single chosen tag to be in.
For the one-hot form (Section 6), we parameterize q ( z t ) =
(cid:81) 4 τ =
q
( z τt ) as a product of independentcategorical distributions q ( z τ t ) for the four tags τ , with g θ (an MLP) outputting the parameters of eachq ( z τt ) at time t , and a sample z ∼ q ( z t ) concatenated with the encoded input at the next time step.
For the language form (Section 6), we use an LSTM for g θ , sampling z t directly.
Examples from asingle trajectory are shown in Figure 3a (one-hot) and Figure 4a (language).
We ground both formsof self-model by supervising z t against ground-truth values.
For the language form, this amounts toconstructing a target ˆ z t of the form: “ The <instructed tag> is in the <color> room. ”.
Given the simplicity of the environment, we implement all three CST algorithms using t ′ = 0throughout.
This means that CST aims to drive the decoder to communicate its beliefs to the earlierversion of the agent from the start of the episode.
Although this reduces the scale of data augmentationfrom quadratic to linear, and requires interventions on z to be accompanied by a reset of the memorystate to m 0 , in return it makes it easier to measure the effects of interventions on the agent’s behavior.
We used the following schedules for interventions at training time.
For CST-RL, interventions( m t ← m 0 ) occurred randomly, with probability p = 0 .
03 that an intervention would occur at anygiven timestep t .
For CST-MR, we simulated interventions in replay at every timestep.
For CST-PD,we simulated interventions in replay: we divided every trajectory into a sequence of blocks of variableduration, with a p = 0 .
03 probability that a new block would start at any time t ; we computed L PDonly up to a horizon of the end of the block, and with a constant discounting function, γ (∆ t )
= 1 .
[Figure]
O r d - D e c
RR - D e c C S T - R L C S T - M R C S T - P D  f r e q u e n c y correlational faithfulness
O r d - D e c
R R - D e c C S T - R L C S T - M R C S T - P D  f r e q u e n c y causal faithfulness
[Figure]  Results
One-hot self-talk.
We start by considering the one-hot form for the messages z (Figure 3a).
Wecompare CST against two baselines: the ordinary decoder ( Ord-Dec ; Figure 1a) and the re-routeddecoder ( RR-Dec ; Figure 1b).
We focus on the metrics of faithfulness previously introduced.
Correlational faithfulness.
When no interventions are taking place, is the agent’s physicalbehavior congruent with the output of its self-model?
To measure this, we filter to all timesin evaluation episodes when the agent’s avatar is in the central room.
We then determine thedegree to which the belief attested in q ( z t ) matches the next-visited room, r (whether or notthe instructed tag is actually in r ).
For the one-hot message form, this amounts to reading offthe value of q ( z τt = r ) for the instructed tag τ .
Causal faithfulness.
When we intervene on z t , is the agent’s subsequent physical behaviorcongruent with this new value?
To measure this, we run evaluation episodes where, at the startof each trial, we inject a message z ′ indicating that the instructed tag is in some random roomr ′ .
We then measure the probability that the agent actually visits room r ′
next.
CST maintains comparable levels of correlational faithfulness as the baseline decoders.
However,the decoder baselines failed to exhibit a meaningful degree of causal faithfulness.
This is indeedimpossible for Ord-Dec given the lack of any causal pathway from z t to m t +1 . Re-routing z back tothe internal state in RR-Dec does result in a small increase above chance, but the effect is relativelytiny. In contrast, all CST objectives substantially increase the degree of causal faithfulness (Figure 3b). Language-based self-talk.
Next, we trained agents to model their belief state using messagesz in a synthetic language form (e.g. Figure 4a).
We repeated the same set of evaluations on thelanguage-form self-models as we did on the one-hot forms.
For correlational faithfulness, we obtainanalogous values of q ( z τt = r ) by computing the likelihoods of generating the four messages “ The<instructed tag> is in the <color> room. ”, and normalizing these to sum to 1.
[Figure]
O r d - D e c R R - D e c
C S T - R L
C S T - M
R
C S T - P D  f r e q u e n c y correlational faithfulness
O r d - D e c
RR - D e c C S T - R L
C S T - M
R
C S T - P D  f r e q u e n c y
[Figure]
faithfulness
[Figure]
As with the one-hot form, we found that the greatest gain of CST was its ability to imbue theself-model with causal faithfulness.
The effect was most profound with CST-RL, and moderate with
CST-PD.
However, we found it difficult to get any traction with the memory reconstruction versionof the algorithm. Unlike all the other variants, CST-MR was often disruptive to learning the basepolicy, and ineffective at causal faithfulness on the few seeds where it was able to learn.
We note that the faithfulness results with the language form were overall weaker than for the one-hotform.
This may be because the language messages encode a room belief about a single tag, while theone-hot messages encode a belief about all four tags.
[Figure]
z ′ at the start of each trial, we can successfully guide the agent directly to the goal object under task conditions with p sh
= 1 where it would otherwise be forced to search.
Semantic control.
As a final test, we consider whether CST endows the self-modelwith sufficient causal power to act as aneffective semantic control interface for theagent.
To measure this, we determine howeffectively the agent can use injected information to obtain reward.
We start withtwo baselines: (1) the return on evaluationepisodes where p sh = 1 , i.e. where thetags are shuffled on every trial; and (2)the return on evaluation episodes wherep sh = 0 , i.e. where the tags are nevershuffled.
Respectively, these represent thebounds of rewards the agent naturally obtains when it can acquire (1) minimal or(2) maximal information about the locationof the instructed tag.
We compare these tothe reward obtained on evaluation episodeswhere information about the tag location isprovided through intervening on z .
Specifically, we set p sh = 1 , and inject a message z ′ at the startof each trial indicating the location of all the tags (for the one-hot message form), or the location ofthe instructed tag (for the language message form).
The results (Figure 5) show that CST not onlyfurnishes an agent with a semantically-grounded model that describes its behavior, but also one whichcan be used as a mechanism to guide it.
Discussion
We articulated five key desiderata of an ideal explanation system for an AI system: grounding,flexibility, minimal-interference, scalability, and faithfulness.
Unlike existing approaches, we seek tosatisfy all of these by training the base system to serve a causal model of itself.
We developed aninstance of this solution for Deep RL agents: Causal Self-Talk.
In CST, an agent learns to serve agrounded and faithful model of its internal state by learning to communicate with itself across time.
We demonstrated its effectiveness under various forms in a simple embodied task in a simulated 3Denvironment.
We also showed that CST enables a new mechanism for effective semantic control.
We presented three variants of the CST algorithm, based on different training objectives.
Ultimately,
CST-RL yielded the greatest degree of causal faithfulness and efficacy of semantic control.
This isachieved by intervening on the agent’s internal state at training time, forcing the agent to learn how touse the content of its self-model to maximize its reward.
Its success, however, may reflect features ofthe particular task setting we study: here, there is a close alignment between writing a faithful z andmaximizing reward.
This may not always be the case.
In contrast, CST-PD offers an attractive alternative for two reasons. First, it trains entirely in replay,and thus potentially interferes less with the base system relative to CST-RL. Second, as it primarilyseeks to enable imitation of the current behavior, the faithfulness of the self-model is targeted bydesign.
One challenge here is learning to capture the important features of behavior; rather thansimple behavioral cloning, as we have attempted, more impressive gains may be possible using richerimitation learning techniques [e.g. 34].
In principle, CST-MR, which tries to reconstruct the entirememory state, offers the greatest potential.
However, we had the least success with it. There also maybe theoretical obstacles to deploying CST-MR in some settings, e.g. when the dimensionality of theinternal state is very large, or is of variable length.
Limitations.
In our experiments, we only implemented a limited form of CST, where we exclusivelyrevert memory states m t to the state at time t ′ = 0 , rather than the more general case of t ′ < t .
Thisimparts two limitations: (1) it requires m t to be reverted to m 0 in order to effectively intervene onthe message z ; and (2) it requires z to contain all behaviorally-relevant information.
While sufficientfor the simple task at hand, this is unrealistic in more complex environments.
When z contains only apartial description of m t , it will be necessary for the agent to be able to re-integrate injected z valueswith its current memory state.
A viable path to achieving this would be the full solution of memoryreversion: sampling t ′ <
t rather than fixing t ′ = 0 .
Utility requires that the self-model be grounded in a semantically-meaningful form.
We have shownhow a ready source of data can be used to train the self-model.
How this method stands up in morelimited data regimes, or in OOD scenarios where the self-model hasn’t been supervised, remains anopen question.
It may be possible in some domains to leverage semi-supervised methods, and/or toimpose structure on the self-model as a helpful constraint.
Future opportunities.
We focused here on expressing the agent’s beliefs about the current state ofthe environment in the self-model.
A complete answer to the question, "Why did you do that?"
mightalso include an expression of an agent’s goals, plans, and model of the world [53].
The schema wehave developed here should be sufficiently flexible to extend in this direction, given the right trainingdata.
Ultimately, it would be of great value to determine how to apply these techniques beyond RLagents, repurposing them to model the computational processes occurring in generic Deep Learningsystems.
Our primary focus in this paper has been on building self-models to yield explanations.
However, thisis only one of several utilities that a faithful self-model might deliver.
For example, we demonstratethat causal self-models provide a semantic control mechanism.
This could be useful for guidingagent learning through human interaction [1], metacognitive reporting and control [9, 24, 29], or asan interface for safety-critical applications [41, 45, 51, 61].
By learning to express its internal statethrough a different representational schema, this may also open up avenues for new exploration orhierarchical reinforcement learning techniques.
Finally, training agents to constantly talk to themselves might prove to be a data-efficient way ofenabling agents to talk to others (and us!).
Ultimately, this may pave a way for us to invert of the flowof knowledge, allowing agents to not only answer, "Why did you do that?", but also "How?".