Explainability Via Causal Self-Talk
Anonymous Author(s)
AffiliationAddress email
Abstract
Explaining the behavior of AI systems is an important problem that, in practice, isgenerally avoided.
While the XAI community has been developing an abundanceof techniques, most incur a set of costs that the wider deep learning communityhas been unwilling to pay in most situations.
We take a pragmatic view of theissue, and define a set of desiderata that capture both the ambitions of XAI and thepractical constraints of deep learning.
We describe an effective way to satisfy allthe desiderata: train the AI system to build a causal model of itself.
We developan instance of this solution for Deep RL agents: Causal Self-Talk.
CST operatesby training the agent to communicate with itself across time.
We implement thismethod in a simulated 3D environment, and show how it enables agents to generatefaithful and semantically-meaningful explanations of their own behavior.
Beyondexplanations, we also demonstrate that these learned models provide new ways ofbuilding semantic control interfaces to AI systems
Introduction
As modern machine learning systems become more powerful and embedded in our lives, the needto have these systems explain their behavior becomes increasingly urgent.
Despite incredibleperformance in a variety of domains, almost all systems are completely unable to provide a satisfyinganswer to the simple question, "Why did you do that?".
While there have been innovations in particular explainability methods [e.g. 8, 55, 56, 66], andsome attempts to apply techniques to domains that demand it
[e.g. 23, 65], the vast majority of deeplearning applications do not engage with these techniques.
This is in part because there are fewgeneral methods, and most incur costs such as reduced performance and scalability; compoundingthis, existing techniques often fail to deliver a useful or credible account of the systems’ behavior.
We believe that the reason for this is that explanation systems tend to fail to satisfy at least one of fivekey desiderata discussed in detail below, namely: groundedness, flexibility, minimally-interfering,scalability, and faithfulness.
Two particular failure modes are most common: the addition of anexplanation system limits generality or hurts performance (e.g. when it requires explicitly structuringthe base system); or the explanations it produces cannot be trusted to be accurate (e.g. with manyexternal or post-hoc methods). An ideal explanation system would avoid both these pitfalls. In this paper, we propose one such solution. We take a pragmatic viewpoint that providing explanations of a system’s behavior is, in essence, a task of building a model of that system.
We arguethat the AI system itself is well-positioned to supply a model that fulfills all the aforementioneddesiderata.
We thus train the base system to supply a "self-model" alongside its main representations. Importantly, the base system and self-model are subject to different constraints: we want the basesystem to be primarily optimized for performance, while we want the self-model to take on a faithful,semantically-interpretable form. The different form allows the self-model to deliver supplementary utility that the base system does not, e.g. providing an interface to external users, and enabling themto understand, predict, and control the base system in convenient ways. Our contributions are as follows.
In Section 2, we articulate five key desiderata for explanationsystems.
However, there is some tension between these.
In Section 3, we describe how commonhistorical XAI techniques resolve these trade-offs by making extreme choices between the desiderata.
Section 4 marks a transition from discussing the desiderata in theory to exploring them in practice,as we demonstrate that a balance can be found between the five desiderata by training AI systemsto build self-models.
In particular, we introduce a solution concept for Deep RL agents: Casual
Self-Talk (CST), in which an agent must learn to use its own outputs as inputs.
In Sections 5-6 weshow that CST allows model-free RL agents in an embodied 3D virtual environment to faithfullyexplain their behavior in terms of semantically-accessible beliefs about the state of the world.
Desiderata for explanatory models
Our goal is to build an explanatory model of an AI system, i.e. a mechanism that translates betweenits internal representations and/or computations, and some other representational form.
We set outfive key desiderata for such a model.
We flesh these out in more detail in Section 3.
Grounded.
For the model to be useful to external users, it should deploy a representationlanguage that is semantically-grounded.
This may require additional data to train.
Flexible.
The method should be able to produce models of many different representationalforms (e.g. classes, graphs, strings).
Ideally it should also be sufficiently general to be agnosticto the base system’s input/output modalities, and possibly even its overall architecture.
Minimally-interfering.
Training and inference on the model should have as little impact aspossible on the training and inference of the base system.
Solutions which are disruptive toperformance are highly unlikely to be adopted in practice.
Scalable.
Once a solution is found, it is straightforward to apply it to any size network.
Faithful.
The model should provide accurate descriptions of the underlying system.
Thegold standard is a causal model of the agent, which facilitates validation of its accuracythrough interventions [38].
This is possibly the most challenging to achieve, and is discussedfurther in the specific context of decoders in Section 3.
A number of past works have also articulated desirable properties for explanations [e.g. 28, 35, 36,38, 49, 53, 69].
Our desiderata focus less on the qualities which constitute a good explanation perse, but instead on what is desired of the method itself that generates the explanation.
Common historical solutions
We examine whether various existing explanation techniques satisfy each of these desiderata.
Attribution methods.
A set of popular techniques attempts to model a neural network’s decisionmaking at the input level, by estimating contributions to its output from input features [e.g. 18, 27, 56],or training examples [e.g. 6, 14, 42, 43, 46, 71], or by finding optimal inputs for a given output[e.g.
Such approaches benefit from being minimally-interfering (they are entirely post-hoc)and scalable (they can be automated).
However, some methods have been empirically shown to beunfaithful [2, 7], while methods which explicitly aim for faithfulness such as LIME [13] and Anchors[60] are only local in scope, and are expensive to scale.
Most importantly, all such methods sacrificeflexibility: one is constrained to produce explanations of a very particular form, e.g. expressed interms of the input modality, whose interpretation may be highly subjective.
Explicitly structuring networks.
A sub-field of deep learning has emerged wherein the intermediate computations are constrained to particular forms, and/or are grounded by external data[e.g.
As explanations of behavior, structured intermediates areattractive as they are both grounded and faithful by design.
However, this is achieved at the expenseof minimal-interference, as it requires fundamentally altering the base system to explicitly depend onthis structure.
It is also limited in flexibility and scalability, as the technique can also only be appliedwhen sufficient domain knowledge or data is available, and only in the form dictated by these priorconstraints.
These issues are exacerbated by the fact that the representations required to optimizeperformance objectives are often at odds with those that would subserve explainability.
Fine-grained mechanistic interpretability.
One common approach is to allow a base system totrain and run without interference, and to post-hoc dissect its representations or computations, oftenusing techniques borrowed or inspired from neuroscience [e.g. 17, 21, 26, 57].
Such approacheschampion the minimal-interference desideratum.
In the best case, they promise to uncover faithfuldescriptions of the actual causal processes operating within the base system.
However, this comes ata huge cost to scalability.
Many human hours are typically expended in reverse engineering, oftenyielding results that are limited to a handful of neurons or circuits seen in a single network.
Moreover,by making strong commitments to respect the internal representations of the base system, it is verydifficult to ground the models that arise, let alone to flexibly choose the form they take.
Decoders.
Decoders can be viewed as trained tools to map from the internal private language ofa base system to an interpretable representation space [e.g. 4, 11, 15, 31–33, 44, 52, 58, 74].
Thisapproach has many attractive qualities: it produces semantically-grounded representations of thebase system’s internal private language; is flexible in the choice of modality; scales well; and isminimally-interfering, insofar as decoders can be separated from the inference path of the maincomputational graph and gradients can be stopped from propagating to the base system if desired(though are often not to assist representation learning, [e.g. 3, 37, 40, 48, 54]).
This is all achieved,however, at the expense of faithfulness, as decoded outputs are off the causal path of the base system[47].
Decoders just report whether information is present in the source representation, not whetherthat information is actually being used for behavior [12, 20, 59].
Similar faithfulness issues arisewith post-hoc behavioral analyses [e.g. 19, 30, 39, 50, 66, 70].
Solution concept
The challenges faced by previous attempts at explainability show that the tensions between thedesiderata are considerable.
It is nonetheless possible to reason a way forward.
Motivation.
First, we want a description of the base system that is both grounded and flexible.
Theinternals of the base system are neither.
While we could explicitly structure the base system to changethis, this would violate the minimal interference desideratum.
Thus we need a model of the basesystem.
Second, the solution must be scalable.
This means we cannot rely on human labor to mapfrom the base system’s representations to the model’s representations.
The only scalable approach isto learn the model .
Third, the solution must be faithful.
Our gold standard of faithfulness requires theability to validate the model by intervening on it.
To do this, one must be able to map in the oppositedirection: from interventions on the model onto interventions on the base system.
Because both thebase system and model are learned, this mapping between them must also be learned.
The most directway of learning this mapping from model to base system is to feed the model’s outputs as inputs tothe base system .
A solution to these problems is thus a “self-model” which is intertwined with the base system.
Wedevelop an instance of this solution in a particular setting: a Deep RL agent, acting in an episodic
POMDP, which builds a model of itself in terms of a set of semantically-grounded beliefs about thestate of the world.
We consider how to expand the scope of this technique in Section 7.
Agent construction.
We follow the standard construction of a POMDP.
We denote the agent’sobservation at time t within an episode as x t , and assume that it maintains an internal state m t , whichwe can take to be, for example, a recurrent state (such as from an LSTM), or a variable-length arrayof embeddings of previous observations x 0: t (as used in a Transformer).
We denote the agent’s state update function as f θ , with learned parameters, θ , s.t. m t +
=
f θ ( x t +1 , m t ) .
We denote its policy asπ t
= h θ ( m t ) .
counterfactual
[Figure] Our construction starts with a decoder, shown in Figure 1a.
decoder maps from the state m t toan output z t ∼ q ( z t ) , via a learned function g θ : m → q ( z
This mapping is trained using z ∈ D z that lives in a space.
As we previously identified in Section 3, benefit from being grounded, flexible, scalable, and minimally-interfering.
However, theirweakness is faithfulness.
This is because the output z merely reflects what information is present in and not whether that information is actually used in the computation of Defining faithfulness.
We now chart a path forward from decoders, with an aim of making themfaithful.
To do so, we first define two metrics for faithfulness, one weak, one strong:
Correlational faithfulness: if the decoder reports a value of z t =
z , to what degree is thebehavior (from the policy π ) consistent with this value, z ?
Causal faithfulness:if we were to counterfactually intervene on z t , changing it to a valueof z ′ , to what degree would this produce behavior consistent with this new value, z ′ , ratherthan its original value, z ?
Causal faithfulness is a much stronger property than correlational faithfulness.
It’s highly desirablebecause it allows one to validate the agent’s commitment to the value of z t at runtime by makinginterventions on it.
This validation mechanism is especially useful when out of distribution.
Re-routed decoders.
While it is possible for a decoder to have a degree of correlational faithfulness,it is structurally impossible for it to exhibit causal faithfulness.
A simple fix is to route the output ofthe decoder back into the internal representation (Figure 1b), e.g. by concatenating z t with x t +1 .
Thisamounts to incorporating the decoder output in the state update rule, with m t +1 =
f θ ([ x t +1 , z t ] , m t ) .
The difficulty with this simple solution is that there is no guarantee that f θ will actually "listen" to z twhen producing m t
+
This is because the information in z t is largely if not completely redundantwith m t .
This may be further exacerbated by the presence of any stochastic sampling in the generationof z t from m t .
Indeed, as shall be described in Section 6, our experiments find that re-routed decodersmake no measurable progress beyond ordinary decoders on causal faithfulness.
How do we render the function f θ sensitive to the value of z , when there is redundancy between z tand m t ?
It is useful to frame the problem in terms of communication.
The decoder pathway can beviewed as a (externally-grounded) communication channel between the internal state and itself, i.e.a form of "self-talk" or "inner speech" [67].
The problem therein is that the speaker ( m t ) and thelistener (also m t ) are identical, so there is no useful information in the message z t for the listener.
Causal self-talk.
To overcome this problem, we need to create an information asymmetry betweenspeaker and listener.
We introduce Causal Self-Talk (CST) : we encourage f θ to be sensitive tothe message z t by creating an augmented data distribution where the speaker and listener diverge.
Fortunately, we have a ready source of alternate listeners: internal states m t ′ for t ′ < t .
We thus provide the decoder output with an additional role: to compactly and effectively communicateinformation to earlier versions of the same agent (from the same episode), via a semanticallygrounded communication channel.
Formally, this takes the form of an intervention (Figs 1c, S1):the internal state, m t , is replaced with a counterfactual one, m t ′ , before being integrated with theoriginal message ( z t ) and next observation ( x t +1 ), in order to produce a counterfactual internal state, m ( t ← t ′ ) t + = f θ ([ x t +1 , z t ] , m t ′ ) .
This yields a counterfactual policy, π ( t ← t ′ ) t +
[Equation]
This alone, however, is insufficient to fully specify a CST algorithm.
We must also stipulate thedesired outcome from enacting these interventions.
We describe three alternative choices:
CST-RL: The messages z t should allow earlier versions of the same agent to maximizereturn given the current environment state.
To train this, interventions must occur online (according to some schedule), and we must allow the agent to act according to the counterfactualpolicy following an intervention.
We then use the maximization of subsequent discountedreward as the desired outcome from the intervention, via reinforcement learning (RL).
CST-MR: The messages z t should allow earlier versions of the same agent to reconstruct thenext internal state, m t +1 .
Unlike CST-RL, this can be done entirely in replay, without requiringthe agent to ever act according to the counterfactual policy on a live environment.
We add aweighted memory reconstruction (MR) term to the overall loss: L MR = || m
( t ← t ′ ) t + − m t
{++
 || 2 .+}
CST-PD: The messages z t should allow earlier versions of the same agent to recover thetrue current (and future) policy.
As with CST-MR, interventions can be simulated entirely inreplay.
We add a weighted policy distillation (PD) term to the overall loss, using discountingfunction γ ( · ) :
L PD = (cid:80) ∆ t> 0 γ (∆ t ) · D KL (cid:16) π t +∆ t
|| π ( t ← t ′ ) t +∆ t (cid:17) .
CST and the five desiderata.
CST offers a different, less extreme tradeoff between the desiderata.
By design, it inherits decoders’ resolution to several desiderata.
Like decoders, CST can achievegrounding by supervising the representations z against a signal of known semantics.
It is also equallyflexible.
Relative to decoders, there is some trade-off with minimal-interference as the decoder outputz t is now an obligate input into the next time step.
However, much like decoders, the effects ontraining can, in principle, be contained through appropriate gradient flow, e.g. by limiting updates toparameters of the state update function, f θ , and/or fine-tuning a pre-existing base system.
CST is highly scalable.
CST-RL is straightforward to implement; in its simplest incarnation, fixingt ′
= 0 reduces to dropout on the whole internal state, m t .
The use of RL, however, occurs at theexpense of additional data as it requires online execution of the counterfactual policy.
CST-PD and
CST-MR, in contrast, are entirely self-supervised: they make use of the existing trajectories generatedduring the training of the base system.
These methods augment the data in a manner similar tothe interchange intervention technique of [25], using self-imitation as the desired outcome of theintervention.
Depending on the degree of non-stationarity afforded by the particular environment(and its degree of partial observability), the size of the augmented dataset can scale quadraticallywith the length of each episode (from choosing both t and t ′ ); in some settings, it may be possible toexceed this by sampling counterfactual previous states m t ′ from other episodes.
Finally, unlike decoders, CST is explicitly trained to provide causal faithfulness, by facilitatingcounterfactual control.
Our goal in the following two sections is to evaluate how well it does this.
Experiments
Task.
We study variants of CST in a 3D virtual environment built in Unity [1, 68].
This features afixed-layout indoor space comprising five rooms, each with a different wall color.
The agent receivesas input a 1 st -person visual observation and a text observation.
The agent can freely navigate using a4D continuous action space ( a ∈ [ − 1 , 1] 4 ) consisting of in-plane movements and a 2-DOF rotation. [Figure]
We developed a simple task in this environment, called DaxDucks (Figure 2).
We designed DaxDucksto: (1) be easy for an agent to learn a good policy; (2) require an agent to maintain and update beliefsabout the latent environment state in order to maximize reward; (3) provide a source of data to groundthe self-model; and (4) provide a means to evaluate the faithfulness of the learned self-model.
DaxDucks is an instruction-based, fast-binding search task, where the agent is rewarded for finding aduck that exhibits an instructed tag.
Each episode consists of a sequence of trials.
At the beginning ofeach trial, the agent’s avatar spawns in the center of the middle room facing a random direction.
Fouridentical ducks spawn at the center of the four outer rooms.
Each duck is randomly assigned one offour tags (“ dax ”, “ gavagai ”, “ kleeg ”, or “ plork ”).
The agent is instructed to collect (i.e. collidewith) the duck with a specified tag via the text observation channel, e.g. “ Collect a gavagai. ”
The agent may observe a duck’s tag by entering the duck’s room and orienting the center of its viewtowards the duck.
This adds an additional string to the text observation, e.g. “ This is a kleeg. ”.
Colliding with any duck terminates the current trial, in which case the agent is respawned in the centerof the middle room again and a random new instruction is delivered; a reward is also delivered if theagent collided with the correct duck.
Finally, when a new trial starts, the ducks usually maintain theirtags, but with a small probability ( p sh = 0 . 1 ), the tags are randomly shuffled between the ducks.
Thusthe agent can only maximize reward by effectively retaining a belief state over which tag belongs towhich duck.
Each episode lasts for 5000 steps regardless of the number of trials completed.
Agent architecture.
For the base system, we used a standard model-free agent architecture.
Weencode inputs with a ResNet (images) and an LSTM (text); we use an LSTM memory module and
MLPs for the policy and value heads; and we trained with V-trace [22].
All agents achieved a similarlyhigh return after training.
Full details are in Appendix B.
We demonstrate the flexibility of grounding by considering two forms for the self-model:
One-hot.
The message z t takes the form of a set of four one-hot vectors (one per tag),expressing which room the agent believes each tag to be in.
Language.
The message z t takes the form of a synthetic language string, expressing whichroom the agent believes a single chosen tag to be in.
For the one-hot form (Section 6), we parameterize q ( z t ) =
(cid:81) 4 τ =
q
( z τt ) as a product of independentcategorical distributions q ( z τ t ) for the four tags τ , with g θ (an MLP) outputting the parameters of eachq ( z τt ) at time t , and a sample z ∼ q ( z t ) concatenated with the encoded input at the next time step.
For the language form (Section 6), we use an LSTM for g θ , sampling z t directly.
Examples from asingle trajectory are shown in Figure 3a (one-hot) and Figure 4a (language).
We ground both formsof self-model by supervising z t against ground-truth values.
For the language form, this amounts toconstructing a target ˆ z t of the form: “ The <instructed tag> is in the <color> room. ”. [Figure] O r d - D e c
RR - D e c C S T - R L C S T - M R C S T - P D  f r e q u e n c y correlational faithfulness
O r d - D e c
R R - D e c C S T - R L C S T - M R C S T - P D  f r e q u e n c y causal faithfulness
[Figure] Given the simplicity of the environment, we implement all three CST algorithms using t ′ = 0throughout.
This means that CST aims to drive the decoder to communicate its beliefs to the earlierversion of the agent from the start of the episode.
Although this reduces the scale of data augmentationfrom quadratic to linear, and requires interventions on z to be accompanied by a reset of the memorystate to m 0 , in return it makes it easier to measure the effects of interventions on the agent’s behavior.
We used the following schedules for interventions at training time.
For CST-RL, interventions( m t ← m 0 ) occurred randomly, with probability p = 0 .
03 that an intervention would occur at anygiven timestep t .
For CST-MR, we simulated interventions in replay at every timestep.
For CST-PD,we simulated interventions in replay: we divided every trajectory into a sequence of blocks of variableduration, with a p = 0 .
03 probability that a new block would start at any time t ; we computed L PDonly up to a horizon of the end of the block, and with a constant discounting function, γ (∆ t )
= 1 .
Results
One-hot self-talk.
We start by considering the one-hot form for the messages z (Figure 3a).
Wecompare CST against two baselines: the ordinary decoder ( Ord-Dec ; Figure 1a) and the re-routeddecoder ( RR-Dec ; Figure 1b).
We focus on the metrics of faithfulness previously introduced.
Correlational faithfulness.
When no interventions are taking place, is the agent’s physicalbehavior congruent with the output of its self-model?
To measure this, we filter to all timesin evaluation episodes when the agent’s avatar is in the central room.
We then determine thedegree to which the belief attested in q ( z t ) matches the next-visited room, r (whether or notthe instructed tag is actually in r ).
For the one-hot message form, this amounts to reading offthe value of q ( z τ t = r ) for the instructed tag τ .
Causal faithfulness.
When we intervene on z t , is the agent’s subsequent physical behaviorcongruent with this new value?
To measure this, we run evaluation episodes where, at the startof each trial, we inject a message z ′ indicating that the instructed tag is in some random roomr ′ .
We then measure the probability that the agent actually visits room r ′
next.
[Figure]
O r d - D e c R R - D e c
C S T - R L
C S T - M
R
C S T - P D  f r e q u e n c y correlational faithfulness
O r d - D e c
RR - D e c C S T - R L
C S T - M
R
C S T - P D  f r e q u e n c y
[Figure]
faithfulness
[Figure]
CST maintains comparable levels of correlational faithfulness as the baseline decoders.
However,the decoder baselines failed to exhibit a meaningful degree of causal faithfulness.
This is indeedimpossible for Ord-Dec given the lack of any causal pathway from z t to m t +1 .
Re-routing z back tothe internal state in RR-Dec does result in a small increase above chance, but the effect is relativelytiny.
In contrast, all CST objectives substantially increase the degree of causal faithfulness (Figure 3b). Language-based self-talk. Next, we trained agents to model their belief state using messagesz in a synthetic language form (e.g. Figure 4a). We repeated the same set of evaluations on thelanguage-form self-models as we did on the one-hot forms.
For correlational faithfulness, we obtainanalogous values of q ( z τt = r ) by computing the likelihoods of generating the four messages “ The<instructed tag> is in the <color> room. ”, and normalizing these to sum to 1.
As with the one-hot form, we found that the greatest gain of CST was its ability to imbue theself-model with causal faithfulness.
The effect was most profound with CST-RL, and moderate with
CST-PD.
However, we found it difficult to get any traction with the memory reconstruction versionof the algorithm.
Unlike all the other variants, CST-MR was often disruptive to learning the basepolicy, and ineffective at causal faithfulness on the few seeds where it was able to learn.
We note that the faithfulness results with the language form were overall weaker than for the one-hotform.
This may be because the language messages encode a room belief about a single tag, while theone-hot messages encode a belief about all four tags.
Semantic control.
As a final test, we consider whether CST endows the self-model with sufficientcausal power to act as an effective semantic control interface for the agent.
To measure this, wedetermine how effectively the agent can use injected information to obtain reward.
We start with twobaselines: (1) the return on evaluation episodes where p sh = 1 , i.e. where the tags are shuffled onevery trial; and (2) the return on evaluation episodes where p sh = 0 , i.e. where the tags are nevershuffled.
Respectively, these represent the bounds of rewards the agent naturally obtains when itcan acquire (1) minimal or (2) maximal information about the location of the instructed tag.
Wecompare these to the reward obtained on evaluation episodes where information about the tag locationis provided through intervening on z .
Specifically, we set p sh = 1 , and inject a message z ′ at the startof each trial indicating the location of all the tags (for the one-hot message form), or the location ofthe instructed tag (for the language message form).
The results (Figure 5) show that CST not only furnishes an agent with a semantically-grounded model that describes its behavior, but also one whichcan be used as a mechanism to guide it. [Figure]
Discussion
[Figure]
p s h =
O r d - D e c RR - D e c C S T - R L C S T - M R C S T - P D
O r d - D e c RR - D e c C S T - R L C S T - M R C S T - P D p s h =
[Figure] z ′ at the start of each trial, we can successfully guide the agent directly to the goal object under task conditions with p sh = 1 where it would otherwise be forced to search. We articulated five key desiderata ofan ideal explanation system for an AIsystem: grounding, flexibility, minimalinterference, scalability, and faithfulness.
Unlike existing approaches, we seek to satisfy all of these by training the base system to serve a causal model of itself.
Wedeveloped an instance of this solution for Deep RL agents: Causal Self-Talk.
In CST,an agent learns to serve a grounded andfaithful model of its internal state by learning to communicate with itself across time.
We demonstrated its effectiveness undervarious forms in a simple embodied taskin a simulated 3D environment.
We alsoshowed that CST enables a new mechanismfor effective semantic control.
We presented three variants of the CST algorithm, based on different training objectives.
Ultimately, CST-RL yielded the greatest degree of causal faithfulness and efficacy of semanticcontrol.
This is achieved by intervening on the agent’s internal state at training time, forcing theagent to learn how to use the content of its self-model to maximize its reward.
Its success, however,may reflect features of the particular task setting we study: here, there is a close alignment betweenwriting a faithful z and maximizing reward.
This may not always be the case.
In contrast, CST-PD offers an attractive alternative for two reasons.
First, it trains entirely in replay,and thus potentially interferes less with the base system relative to CST-RL.
Second, as it primarilyseeks to enable imitation of the current behavior, the faithfulness of the self-model is targeted bydesign.
One challenge here is learning to capture the important features of behavior; rather thansimple behavioral cloning, as we have attempted, more impressive gains may be possible using richerimitation learning techniques [e.g. 34].
In principle, CST-MR, which tries to reconstruct the entirememory state, offers the greatest potential.
However, we had the least success with it.
There also maybe theoretical obstacles to deploying CST-MR in some settings, e.g. when the dimensionality of theinternal state is very large, or is of variable length.
Limitations.
In our experiments, we use a limited form of CST, where we exclusively revertmemory states m t to the state at time t ′ = 0 , rather than the more general case of t ′ < t .
We do thisto simplify the presentation and evaluation of the technique, but it imparts two limitations: (1) itrequires m t to be reverted to m 0 in order to effectively intervene on the message z ; and (2) it requiresz to contain all behaviorally-relevant information.
While sufficient for the simple task at hand, thisbecomes unrealistic in more complex environments.
When z contains only a partial description ofm t , it will be necessary for the agent to be able to re-integrate injected z values with its currentmemory state.
A viable path to achieving this would be enabling full memory reversion: samplingt ′ <
t rather than fixing t ′ = 0 .
Utility requires that the self-model be grounded in a semantically-meaningful form.
We have shownhow a ready source of data can be used to train the self-model.
How this method stands up in morelimited data regimes, or in OOD scenarios where the self-model hasn’t been supervised, remains anopen question.
It may be possible in some domains to leverage semi-supervised methods, and/or toimpose structure on the self-model as a helpful constraint.
Future opportunities.
We focused here on expressing the agent’s beliefs about the current state ofthe environment in the self-model.
A complete answer to the question, "Why did you do that?"
mightalso include an expression of an agent’s goals, plans, and model of the world [53].
The schema we have developed here should be sufficiently flexible to extend in this direction, given the right trainingdata.
Ultimately, it would be of great value to determine how to apply these techniques beyond RLagents, repurposing them to model the computational processes occurring in generic Deep Learningsystems.
Our primary focus in this paper has been on building self-models to yield explanations.
However, thisis only one of several utilities that a faithful self-model might deliver.
For example, we demonstratethat causal self-models provide a semantic control mechanism.
This could be useful for guidingagent learning through human interaction [1], metacognitive reporting and control [9, 24, 29], or asan interface for safety-critical applications [41, 45, 51, 61].
By learning to express its internal statethrough a different representational schema, this may also open up avenues for new exploration orhierarchical reinforcement learning techniques.
Finally, training agents to constantly talk to themselves might prove to be a data-efficient way ofenabling agents to talk to others (and us!).
Ultimately, this may pave a way for us to invert of the flowof knowledge, allowing agents to not only answer, "Why did you do that?", but also "How?".