Original: Case-Based Off-Policy Evaluation Using Prototype Learning
Revised: Case-Based Off-Policy Evaluation Using Prototype Learning

Original: Abstract
Revised: Abstract

Original: Importance sampling (IS) is often used to perform off-policy evaluation but it is prone to several issues—especially when the behavior policy is unknown and must be estimated from data.
Revised: Importance sampling (IS) is often used to perform off-policy evaluation but it is prone to several issues—especially when the behavior policy is unknown and must be estimated from data.

Original: Significant differences between target and behavior policies can result in uncertain value estimates due to, for example, high variance.
Revised: Significant differences between target and behavior policies can result in uncertain value estimates due to, for example, high variance.

Original: Standard practices such as inspecting IS weights may be insufficient to diagnose such problems and determine for which type of inputs the policies differ in suggested actions and resulting values.
Revised: Standard practices such as inspecting IS weights may be insufficient to diagnose such problems and determine for which type of inputs the policies differ in suggested actions and resulting values.

Original: To address this, we propose estimating the behavior policy for IS using prototype learning.
Revised: To address this, we propose estimating the behavior policy for IS using prototype learning.

Original: The learned prototypes provide a condensed summary of the input-action space, which allows for describing differences between policies and assessing the support for evaluating a certain target policy.
Revised: The learned prototypes provide a condensed summary of the input-action space, which allows for describing differences between policies and assessing the support for evaluating a certain target policy.

Original: In addition, we can describe a value estimate in terms of prototypes to understand which parts of the target policy have the most impact on the estimate.
Revised: In addition, we can describe a value estimate in terms of prototypes to understand which parts of the target policy have the most impact on the estimate.

Original: We find that this provides new insights in the examination of a learned policy for sepsis management.
Revised: We find that this provides new insights in the examination of a learned policy for sepsis management.

Original: Moreover, we study the bias resulting from restricting models to use prototypes, how bias propagates to IS weights and estimated values and how this varies with history length.
Revised: Moreover, we study the bias resulting from restricting models to use prototypes, how bias propagates to IS weights and estimated values and how this varies with history length.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Historical data on decisions and outcomes provide opportunities for evaluating policies for future decision-making.
Revised: Historical data on decisions and outcomes provide opportunities for evaluating policies for future decision-making.

Original: For example, the prospect of using patient records to evaluate new policies for medication dosing in sepsis management has attracted recent attention [Komorowski et al., 2018,
Revised: For example, the prospect of using patient records to evaluate new policies for medication dosing in sepsis management has attracted recent attention [Komorowski et al., 2018,

Original: Gottesman et al., 2019].
Revised: Gottesman et al., 2019].

Original: An example of off-policy evaluation (OPE), this amounts to estimating the value of a target policy based on data gathered under a different so-called behavior policy; see e g., Thomas [2015] for an overview.
Revised: An example of off-policy evaluation (OPE), this amounts to estimating the value of a target policy based on data gathered under a different so-called behavior policy; see e g., Thomas [2015] for an overview.

Original: Importance sampling (IS) methods [Precup, 2000] perform
Revised: Importance sampling (IS) methods [Precup, 2000] perform

Original: OPE by weighting observed outcomes by the density ratio of the target policy and the behavior policy.
Revised: OPE by weighting observed outcomes by the density ratio of the target policy and the behavior policy.

Original: IS methods are often preferred over alternatives which rely on modeling outcomes or covariate transitions, due to their simplicity and the fact that behavior policies often are controllable or human made.
Revised: IS methods are often preferred over alternatives which rely on modeling outcomes or covariate transitions, due to their simplicity and the fact that behavior policies often are controllable or human made.

Original: Similarly, the equivalent strategy of inversepropensity weighting is fundamental to the study of causal effects [Rosenbaum and Rubin, 1983, Hirano et al., 2003].
Revised: Similarly, the equivalent strategy of inversepropensity weighting is fundamental to the study of causal effects [Rosenbaum and Rubin, 1983, Hirano et al., 2003].

Original: In practice, it is difficult to assess the quality of an IS value estimate.
Revised: In practice, it is difficult to assess the quality of an IS value estimate.

Original: When the behavior policy is unknown and must be estimated from data, conditions which guarantee good estimates are hard to meet and rely on untestable assumptions [Rosenbaum et al., 2010, Namkoong et al., 2020].
Revised: When the behavior policy is unknown and must be estimated from data, conditions which guarantee good estimates are hard to meet and rely on untestable assumptions [Rosenbaum et al., 2010, Namkoong et al., 2020].

Original: Standard practices of inspecting weights [Li et al., 2019] and removing outliers [Crump et al., 2009] give only aggregate or per-sample perspectives on potential issues and are often insufficient for domain experts to reason about the validity of the result.
Revised: Standard practices of inspecting weights [Li et al., 2019] and removing outliers [Crump et al., 2009] give only aggregate or per-sample perspectives on potential issues and are often insufficient for domain experts to reason about the validity of the result.

Original: There is a clear need to better inspect and diagnose importance sampling estimates.
Revised: There is a clear need to better inspect and diagnose importance sampling estimates.

Original: In this paper, we propose estimating the unknown behavior policy using prototype learning [Li et al., 2018b, Ming et al., 2019].
Revised: In this paper, we propose estimating the unknown behavior policy using prototype learning [Li et al., 2018b, Ming et al., 2019].

Original: The learned prototypes are selected cases from the input data, readily interpretable by a domain expert and representative of the input-action space.
Revised: The learned prototypes are selected cases from the input data, readily interpretable by a domain expert and representative of the input-action space.

Original: In healthcare applications, the prototypes are trajectories of former patients, and a prototype-based estimate of the behavior policy is analogous to how physicians use experience from previous patients to treat new ones.
Revised: In healthcare applications, the prototypes are trajectories of former patients, and a prototype-based estimate of the behavior policy is analogous to how physicians use experience from previous patients to treat new ones.

Original: While offering transparency, a prototype model is flexible enough to model behavior policies in large and/or sequential input spaces.
Revised: While offering transparency, a prototype model is flexible enough to model behavior policies in large and/or sequential input spaces.

Original: Our main contribution is to use learned prototypes as an
Revised: Our main contribution is to use learned prototypes as an

Original: OPE diagnostic tool.
Revised: OPE diagnostic tool.

Original: In addition to enabling interpretation of individual predictions, we show that (a) prototypes can be used to describe areas of similarities/dissimilarities between behavior and target policies; and (b) prototypes induce a soft clustering which can be used to explain differences in value for different policies.
Revised: In addition to enabling interpretation of individual predictions, we show that (a) prototypes can be used to describe areas of similarities/dissimilarities between behavior and target policies; and (b) prototypes induce a soft clustering which can be used to explain differences in value for different policies.

Original: We elaborate on this idea in Section 3.3 and demonstrate our method in Section 4.1 using an example of sepsis management.
Revised: We elaborate on this idea in Section 3.3 and demonstrate our method in Section 4.1 using an example of sepsis management.

Original: Further, we study the added bias of restricting the model class to use prototypes and how this bias propagates to the IS weights (Section 4.2).
Revised: Further, we study the added bias of restricting the model class to use prototypes and how this bias propagates to the IS weights in Section 4.2.

Original:  OFF-POLICY EVALUATION
Revised:  OFF-POLICY EVALUATION

Original: Policy evaluation refers to estimating the value V ( π ) of a target policy π ∈ Π , as defined below.
Revised: Policy evaluation refers to estimating the value V ( π ) of a target policy π ∈ Π , as defined below.

Original: We focus on the sequential case, where a policy is used to select an action A ∈
Revised: We focus on the sequential case, where a policy is used to select an action A ∈

Original: A = { 1 , . . .
Revised: A = { 1 , . . .

Original: , k } after a history H ∈ H , comprising a sequence of previous actions and contexts X ∈ X .
Revised: , k } after a history H ∈ H , comprising a sequence of previous actions and contexts X ∈ X .

Original: The history until time t is defined as H t :=
Revised: The history until time t is defined as H t :=

Original: ( X 0 , A 0 , X 1 , A 1 , . . .
Revised: ( X 0 , A 0 , X 1 , A 1 , . . .

Original: , X t ) , with H 0 =
Revised: , X t ) , with H 0 =

Original: X 0 .
Revised: X 0 .

Original: A policy π : H → ∆ A is a map from a history to a distribution over A .
Revised: A policy π : H → ∆ A is a map from a history to a distribution over A .

Original: In a medical example, a context X could correspond to information about a patient’s state, an action A to a medical intervention, and the target policy π to new clinical guidelines.
Revised: In a medical example, a context X could correspond to information about a patient’s state, an action A to a medical intervention, and the target policy π to new clinical guidelines.

Original: The value of a policy π is defined as the expectation of a reward or outcome R ∈ R , accumulated after acting according to π .
Revised: The value of a policy π is defined as the expectation of a reward or outcome R ∈ R , accumulated after acting according to π .

Original: Here, we study the special case where a single reward is awarded at the end of the sequence,
Revised: Here, we study the special case where a single reward is awarded at the end of the sequence,

Original: R = R T , but our results generalize to the case where rewards are given after every action.
Revised: R = R T , but our results generalize to the case where rewards are given after every action.

Original: Under the distribution p π ( X 0 , A 0 , . . .
Revised: Under the distribution p π ( X 0 , A 0 , . . .

Original: , X T , A T , R ) =
Revised: , X T , A T , R ) =

Original: p π ( H T , A T , R ) , induced by the policy π , the value is V ( π ) :=
Revised: p π ( H T , A T , R ) , induced by the policy π , the value is V ( π ) :=

Original: E π [ R ] .
Revised: E π [ R ] .

Original: Estimating V ( π ) is trivial given a large enough number of samples from the target policy p π .
Revised: Estimating V ( π ) is trivial given a large enough number of samples from the target policy p π .

Original: In off-policy evaluation (OPE), we have access to no such samples, but must estimate V ( π ) using an observational dataset of m samples
Revised: In off-policy evaluation (OPE), we have access to no such samples, but must estimate V ( π ) using an observational dataset of m samples

Original: D =
Revised: D =

Original: (( h 1 t 1 , a 1 t 1 , r 1 ) , . .
Revised: (( h 1 t 1 , a 1 t 1 , r 1 ) , . .

Original: , ( h mt m , a mt m , r m )) , drawn according to a distribution p µ ( H T , A T , R ) , controlled by a behavior policy µ ∈ Π .
Revised: , ( h mt m , a mt m , r m )) , drawn according to a distribution p µ ( H T , A T , R ) , controlled by a behavior policy µ ∈ Π .

Original: In the medical example, the behavior policy represents current clinical practice.
Revised: In the medical example, the behavior policy represents current clinical practice.

Original: In this work, the behavior policy µ is unknown and an estimate ˆ p µ ( A | H ) is learned from the samples D .
Revised: In this work, the behavior policy µ is unknown and an estimate ˆ p µ ( A | H ) is learned from the samples D .

Original: A common method for OPE is importance sampling (IS).
Revised: A common method for OPE is importance sampling (IS).

Original: The IS estimator uses an estimate ˆ p µ in a weighted average over the samples D [Hanna et al., 2019]:
Revised: The IS estimator uses an estimate ˆ p µ in a weighted average over the samples D [Hanna et al., 2019]:

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Sufficient conditions for the estimator ˆ V IS ( π ; ˆ µ ) to be an unbiased estimator of V ( π ) include (sequential) ignorability and overlap [Rosenbaum and Rubin, 1983, Robins, 1986].
Revised: Sufficient conditions for the estimator ˆ V IS ( π ; ˆ µ ) to be an unbiased estimator of V ( π ) include (sequential) ignorability and overlap [Rosenbaum and Rubin, 1983, Robins, 1986].

Original: In our setting, ignorability may be defined as for all t , the conditional distribution of R given A t and H t is the same under π and µ , i.e., ∀ t : p π ( R | H t , A t )
Revised: In our setting, ignorability may be defined as for all t , the conditional distribution of R given A t and H t is the same under π and µ , i.e., ∀ t : p π ( R | H t , A t )

Original: = p µ ( R | H t , A t ) .
Revised: = p µ ( R | H t , A t ) .

Original: Overlap is satisfied for a pair ( h, a ) if it being observable under π implies that it is observable under µ , p π ( A t = a |
Revised: Overlap is satisfied for a pair ( h, a ) if it being observable under π implies that it is observable under µ , p π ( A t = a |

Original: H t = h ) > 0 ⇒ p µ ( A t = a | H t = h ) > 0 .
Revised: H t = h ) > 0 ⇒ p µ ( A t = a | H t = h ) > 0 .

Original: We say that overlap is partially violated if this condition is violated for some pairs of histories and actions.
Revised: We say that overlap is partially violated if this condition is violated for some pairs of histories and actions.

Original: Even when ignorability and overlap are satisfied, if µ and π differ significantly, the estimator ˆ V IS ( π ; ˆ µ ) suffers from high variance.
Revised: Even when ignorability and overlap are satisfied, if µ and π differ significantly, the estimator ˆ V IS ( π ; ˆ µ ) suffers from high variance.

Original: The weighted importance sampling  estimator [Rubinstein and Kroese, 2016], ˆ V WIS ( π ; ˆ µ ) :=
Revised: The weighted importance sampling (WIS) estimator [Rubinstein and Kroese, 2016], ˆ V WIS ( π ; ˆ µ ) :=

Original: [Equation]
Revised: [Equation]

Original: 1 w i (cid:80) mi =1 w i r i , introduces bias, but often has less variance.
Revised: mi =1 w i (cid:80) mi =1 w i r i , introduces bias, but often has less variance.

Original: Under the Markov assumption, i.e., that context (or “state”) transitions, actions and rewards depend only on the most recent context-action pair, the history H t in (2) can be replaced by X t .
Revised: Under the Markov assumption, i.e., that context (or “state”) transitions, actions and rewards depend only on the most recent context-action pair, the history H t in (2) can be replaced by X t .

Original: We leave out the subscript t where clear.
Revised: We leave out the subscript t where clear.

Original: 2.1 CAN WE TRUST AN IS ESTIMATE?
Revised: 2.1 CAN WE TRUST AN IS ESTIMATE?

Original: A fundamental challenge with off-policy evaluation is that no ground truth value, or even samples of it, are available.
Revised: A fundamental challenge with off-policy evaluation is that no ground truth value, or even samples of it, is available.

Original: What is worse, the assumption of ignorability cannot be verified statistically [Rosenbaum et al., 2010] and the extent of overlap is unknown if µ is unknown.
Revised: What is worse, the assumption of ignorability cannot be verified statistically [Rosenbaum et al., 2010] and the extent of overlap is unknown if µ is unknown.

Original: As a result, assessing the quality of an estimate ˆ V IS inherently relies on domain expertise.
Revised: As a result, assessing the quality of an estimate ˆ V IS inherently relies on domain expertise.

Original: By examining importance weights { w i } mi =1 and estimated propensities ˆ p µ ( A t | H t ) , analysts can spot outliers with extremely large weights, and compute the effective sample size (ESS) [Gottesman et al., 2018, Owen, 2013, Chapter 9].
Revised: By examining importance weights { w i } mi =1 and estimated propensities ˆ p µ ( A t | H t ) , analysts can spot outliers with extremely large weights, and compute the effective sample size (ESS) [Gottesman et al., 2018, Owen, 2013, Chapter 9].

Original: These practices give a per-sample and an average view of potential issues with variance and the potential for removing samples with excessive weights [Crump et al., 2009, Stürmer et al., 2010].
Revised: These practices give a per-sample and an average view of potential issues with variance and the potential for removing samples with excessive weights [Crump et al., 2009, Stürmer et al., 2010].

Original: However, several questions remain regarding what replacing µ with π would imply in practice:
Revised: However, several questions remain regarding what replacing µ with π would imply in practice:

Original: • Where do π and µ differ?
Revised: • Where do π and µ differ?

Original: How can we describe the inputs for which the most probable actions under π differ from those under µ ?
Revised: How can we describe the inputs for which the most probable actions under π differ from those under µ ?

Original: • If ˆ V ( π ) >
Revised: • If ˆ V ( π ) >

Original: ˆ V ( µ ) , what gives π the edge?
Revised: ˆ V ( µ ) , what gives π the edge?

Original: In which situations does acting according to π result in higher rewards than acting according to µ ?
Revised: In which situations does acting according to π result in higher rewards than acting according to µ ?

Original: Inspecting weights and propensities in aggregate or on a persample basis is insufficient to answer these questions as they concern patterns in policy decisions, weights and rewards.
Revised: Inspecting weights and propensities in aggregate or on a persample basis is insufficient to answer these questions as they concern patterns in policy decisions, weights and rewards.

Original: For example, extreme weights and small ESS may indicate lack of overlap between π and µ but do not explain the cause of the problem.
Revised: For example, extreme weights and small ESS may indicate lack of overlap between π and µ but do not explain the cause of the problem.

Original: Next, we show that a case-based model of the behavior policy µ can help identify said patterns by inducing a soft clustering over the space of histories. 
Revised: Next, we show that a case-based model of the behavior policy µ can help identify said patterns by inducing a soft clustering over the space of histories. ℎ [Figure]

Original: ℎ !" ℎ #" ⋮ℎ $ ! "
Revised: ℎ [Figure] ⋮ℎ $ ! "

Revised: [Figure]

Original: 𝑝 𝑎 𝒵 ℎ$
Revised: 𝑝 𝑎 𝒵 ℎ

Revised: [Figure]

Original:  OFF-POLICY EVALUATION WITH PROTOTYPES
Revised:  OFF-POLICY EVALUATION WITH PROTOTYPES

Original: We propose performing off-policy evaluation using prototype learning [Li et al., 2018b, Ming et al., 2019].
Revised: We propose performing off-policy evaluation using prototype learning [Li et al., 2018b, Ming et al., 2019].

Original: The idea is to express the behavior policy p µ ( A | H ) by comparing the history H to a relatively small set of prototype histories from the training data, see Figure 1.
Revised: The idea is to express the behavior policy p µ ( A | H ) by comparing the history H to a relatively small set of prototype histories from the training data, see Figure 1.

Original: In a clinical setting, such a policy may correspond to physicians choosing treatment for a new patient based on their prior experience in treating similar patients.
Revised: In a clinical setting, such a policy may correspond to physicians choosing treatment for a new patient based on their prior experience in treating similar patients.

Original: For a domain expert, trained in interpreting such cases, a prototype-based estimate is transparent as long as the number of prototypes is small enough.
Revised: For a domain expert, trained in interpreting such cases, a prototype-based estimate is transparent as long as the number of prototypes is small enough.

Original: By examining how policy overlap and value estimates vary with prototypes, we can answer the questions raised above.
Revised: By examining how policy overlap and value estimates vary with prototypes, we can answer the questions raised above.

Original: 3.1 MODELING BEHAVIOR WITH PROTOTYPES
Revised: 3.1 MODELING BEHAVIOR WITH PROTOTYPES

Original: Let ˜ H
Revised: Let ˜ H

Original: =
Revised: =

Original: [˜ h 1 , . .
Revised: [˜ h 1 , . .

Original: ˜ h n ] ⊺ be a list of n prototype histories 2 .
Revised: ˜ h n ] ⊺ be a list of n prototype histories.

Original: Each prototype is a subsequence of an observed history, ˜ h
Revised: Each prototype is a subsequence of an observed history, ˜ h

Original: j = h it for h i ∈ D and t ≤ t i .
Revised: j = h it for h i ∈ D and t ≤ t i .

Original: We allow the prototypes to be subsequences of full-length histories since OPE requires evaluating the behavior policy at each time step.
Revised: We allow the prototypes to be subsequences of full-length histories since OPE requires evaluating the behavior policy at each time step.

Original: The behavior policy p µ ( A t | H t = h t ) is approximated based on the similarity between an observation h t and the prototypes in a learned representation.
Revised: The behavior policy p µ ( A t | H t = h t ) is approximated based on the similarity between an observation h t and the prototypes in a learned representation.

Original: The prototypes ˜ H are themselves selected by the learning algorithm.
Revised: The prototypes ˜ H are themselves selected by the learning algorithm.

Original: To learn ˜ H , we follow Li et al.
Revised: To learn ˜ H , we follow Li et al.

Original: [2018b], Ming et al.
Revised: [2018b], Ming et al.

Original: [2019] by first learning a set of latent prototypes as free parameters ˜
Revised: [2019] by first learning a set of latent prototypes as free parameters ˜

Original: Z =
Revised: Z =

Original: [˜ z 1 , . .
Revised: [˜ z 1 , . .

Original: ˜ z n ] ⊺ in an encoding space Z .
Revised: ˜ z n ] ⊺ in an encoding space Z .

Original: Given an encoder e : H → Z , for an arbitrary history h t , let S( ˜ Z, e ( h t
Revised: Given an encoder e : H → Z , for an arbitrary history h t , let S

Original: [Equation]
Revised: [Equation]

Original: = [ s (˜ z 1 , e ( h t )) , . . . , s (˜ z n , e (
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: t ))] ⊺ be the similarity vector for the encoding of h t comparing e ( h t ) to ˜ Z using a fixed function s :Z × Z
Revised: ⊺ be the similarity vector for the encoding of h t comparing e ( h t ) to ˜ Z using a fixed function s :

Original:  → R + .
Revised: Z × Z → R + .

Original: We use an RBF-kernel with unit bandwidth ( γ 
Revised: We use an RBF-kernel with unit bandwidth ( γ =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which takes values between 0 (no similarity) and 1 (full similarity).
Revised: which takes values between 0 (no similarity) and 1 (full similarity).

Original: With B ∈ R k × n , we estimate the behavior policy µ through logistic regression in the space induced by S , 
Revised: With B ∈ R k × n , we estimate the behavior policy µ through logistic regression in the space induced by S , [Equation] p

Original: [Equation]
Revised: [Equation]

Original: where f σ denotes the softmax function over rows and c ∈ R k is a bias term.
Revised: where f σ denotes the softmax function over rows and c ∈ R k is a bias term.

Original: Column j of B represents the coefficients determining the action probabilities associated with ˜ h j .
Revised: Column j of B represents the coefficients determining the action probabilities associated with ˜ h j .

Original: If the coefficient B ij is positive, higher similarity between h t and ˜ h j makes action i more probable for h t ; a negative coefficient makes action i less probable.
Revised: If the coefficient B ij is positive, higher similarity between h t and ˜ h j makes action i more probable for h t ; a negative coefficient makes action i less probable.

Original: The model parameters Θ = ( e, B, c, ˜ H ) , comprising the parameters of the encoder e , coefficients B , c and the set of prototypes ˜ H , are all unknown and must be learned from data.
Revised: The model parameters Θ = ( e, B, c, ˜ H ) , comprising the parameters of the encoder e , coefficients B , c and the set of prototypes ˜ H , are all unknown and must be learned from data.

Original: As encoder, we use either feedforward or recurrent neural networks.
Revised: As encoder, we use either feedforward or recurrent neural networks.

Original: Following Ming et al.
Revised: Following Ming et al.

Original: [2019], we learn Θ by minimizing the regularized negative log-likelihood 
Revised: [2019], we learn Θ by minimizing the regularized negative log-likelihood (NLL) [Equation]

Original: [Equation]
Revised: [Equation]

Original: using stochastic gradient descent.
Revised: using stochastic gradient descent.

Original: The regularization terms
Revised: The regularization terms

Original: R d (Θ) , R c (Θ) and R e (Θ) encourage diversity , clustering and evidence , respectively, and are defined in Appendix A.
Revised: R d (Θ) , R c (Θ) and R e (Θ) encourage diversity , clustering and evidence , respectively, and are defined in Appendix A.

Original: To make sure that prototypes represent real cases, i.e., to select ˜ H , latent prototypes are projected onto encodings of training samples at regular intervals between descent steps,
Revised: To make sure that prototypes represent real cases, i.e., to select ˜ H , latent prototypes are projected onto encodings of training samples at regular intervals between descent steps,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: with D the set of all subsequences of trajectories in D .
Revised: with D the set of all subsequences of trajectories in D .

Original: Is there a good prototype model?
Revised: Is there a good prototype model?

Original: Modeling the behavior policy using prototypes places additional restrictions on the functional form of estimates.
Revised: Modeling the behavior policy using prototypes places additional restrictions on the functional form of estimates.

Original: It is natural to ask:
Revised: It is natural to ask:

Original: Assuming that adjusting for the history H t is sufficient for unbiased policy evaluation, do there exist prototype histories ˜ H , an encoding e and a similarity function s such that evaluation using the prototype model is exact or accurate?
Revised: Assuming that adjusting for the history H t is sufficient for unbiased policy evaluation, do there exist prototype histories ˜ H , an encoding e and a similarity function s such that evaluation using the prototype model is exact or accurate?

Original: In Section 4, we study this question empirically.
Revised: In Section 4, we study this question empirically.

Original: Additionally, in Appendix A.2, we show constructively that there are indeed problems for which a prototype model exists that exactly describes the behavior policy µ .
Revised: Additionally, in Appendix A.2, we show constructively that there are indeed problems for which a prototype model exists that exactly describes the behavior policy µ .

Original: 3.2 PREDICTING WITH PROTOTYPES
Revised: 3.2 PREDICTING WITH PROTOTYPES

Original: When computing the estimated behavior policy (4) for a history h , the similarity vector S ( e ( ˜ H ) , e ( h )) determines how similar each of the n prototypes are to h .
Revised: When computing the estimated behavior policy (4) for a history h , the similarity vector S ( e ( ˜ H ) , e ( h )) determines how similar each of the n prototypes are to h .

Original: The number n is a hyperparameter.
Revised: The number n is a hyperparameter.

Original: The more prototypes are used, the greater the flexibility of the model, but a large n may result in S consisting of multiple elements close to 1 , making predictions difficult to interpret.
Revised: The more prototypes are used, the greater the flexibility of the model, but a large n may result in S consisting of multiple elements close to 1 , making predictions difficult to interpret.

Original: For example, if s ( e (˜ h j ) , e ( h )) ≈for more than 10 prototypes j , it may be difficult to reason about the policy decision after all.
Revised: For example, if s ( e (˜ h j ) , e ( h )) ≈for more than 10 prototypes j , it may be difficult to reason about the policy decision after all.

Original: To address this, we use only a limited number of q ≤ n prototypes—so-called prediction prototypes —when making predictions with the trained model.
Revised: To address this, we use only a limited number of q ≤ n prototypes—so-called prediction prototypes —when making predictions with the trained model.

Original: Let s q ( h ) be the similarity between e ( h ) and its q th most similar latent prototype.
Revised: Let s q ( h ) be the similarity between e ( h ) and its q th most similar latent prototype.

Original: For j = 1 , . . . , n , we truncate the similarity vector according to
Revised: For j = 1 , . . . , n , we truncate the similarity vector according to

Original: [Equation]
Revised: [Equation]

Original: As an example, with q = 2 the (sorted) similarity vector in
Revised: As an example, with q = 2 the (sorted) similarity vector in

Original: Figure 1 would become [0 . 9 , 0 . 6 , 0] ⊺ .
Revised: Figure 1 would become [0 . 9 , 0 . 6 , 0] ⊺ .

Original: We perform the truncation step independently for all contexts h .
Revised: We perform the truncation step independently for all contexts h .

Original: In Section 4.2, we study the resulting trade-off between transparency (small q and n ) and bias as we vary the number of (prediction) prototypes, ( q ) n .
Revised: In Section 4.2, we study the resulting trade-off between transparency (small q and n ) and bias as we vary the number of (prediction) prototypes, ( q ) n .

Original: Note that we optimize the regularization parameters λ d , λ c and λ e with respect to the choice of q .
Revised: Note that we optimize the regularization parameters λ d , λ c and λ e with respect to the choice of q .

Original: 3.3 USING PROTOTYPES FOR EVALUATION
Revised: 3.3 USING PROTOTYPES FOR EVALUATION

Original: Prototypes induce a soft clustering of the space of histories.
Revised: Prototypes induce a soft clustering of the space of histories.

Original: Each prototype represents a group of similar histories which can be associated with a certain distribution over actions.
Revised: Each prototype represents a group of similar histories which can be associated with a certain distribution over actions.

Original: In
Revised: In

Original: Figure 1, we see for example that the “green prototype”— representing the patients in the “green cluster”—is given a higher dose of the drug than the other prototypes.
Revised: Figure 1, we see for example that the “green prototype”— representing the patients in the “green cluster”—is given a higher dose of the drug than the other prototypes.

Original: Given characteristics of the “green prototype”, a domain expert should be able to explain why it receives this type of treatment.
Revised: Given characteristics of the “green prototype”, a domain expert should be able to explain why it receives this type of treatment.

Original: While it is possible to use other methods to cluster the space of histories, prototypes have the advantage of being based in cases and trained to describe groups of subjects who are treated differently under the behavior policy.
Revised: While it is possible to use other methods to cluster the space of histories, prototypes have the advantage of being based in cases and trained to describe groups of subjects who are treated differently under the behavior policy.

Original: We see in Section 4.2 that this is beneficial also for accuracy.
Revised: We see in Section 4.2 that this is beneficial also for accuracy.

Original: When modeling the behavior policy µ using prototypes, we can utilize the induced clustering structure to answer the questions raised in Section 2.1.
Revised: When modeling the behavior policy µ using prototypes, we can utilize the induced clustering structure to answer the questions raised in Section 2.1.

Original: First of all, the prototypes ˜ h
Revised: First of all, the prototypes ˜ h

Original: j and their action probabilities
Revised: j and their action probabilities

Original: [Equation]
Revised: [Equation]

Original: give an overview of the estimated behavior policy.
Revised: give an overview of the estimated behavior policy.

Original: By comparing the action probabilities ˆ p jµ ( a ) with the corresponding action probabilities under π ,
Revised: By comparing the action probabilities ˆ p jµ ( a ) with the corresponding action probabilities under π ,

Original: [Equation]
Revised: [Equation]

Original: we can explain input regions for which π and µ differ in their suggested actions.
Revised: we can explain input regions for which π and µ differ in their suggested actions.

Original: Domain experts can use this overview to assess how well the data supports evaluation of the target policy.
Revised: Domain experts can use this overview to assess how well the data supports evaluation of the target policy.

Original: For example, if π , for a certain prototype, suggests actions that are extremely rare under µ , there may not be enough data on these decisions to accurately estimate V ( π ) .
Revised: For example, if π , for a certain prototype, suggests actions that are extremely rare under µ , there may not be enough data on these decisions to accurately estimate V ( π ) .

Original: It is good practice to compare ˆ V ( π ) with ˆ V ( µ ) , i.e., the mean reward in data.
Revised: It is good practice to compare ˆ V ( π ) with ˆ V ( µ ) , i.e., the mean reward in data.

Original: If ˆ V ( π ) is different from ˆ V ( µ ) , we would like to know for which inputs π gain or lose performance in relation to µ .
Revised: If ˆ V ( π ) is different from ˆ V ( µ ) , we would like to know for which inputs π gain or lose performance in relation to µ .

Original: The prototypes allow us to divide the estimated values into prototype-based contributions and answer this question.
Revised: The prototypes allow us to divide the estimated values into prototype-based contributions and answer this question.

Original: To make the implicit clustering explicit, we define J t to be a random variable with values in { 1 , . . . , n } , representing an assignment of a history H t to prototype j at time t .
Revised: To make the implicit clustering explicit, we define J t to be a random variable with values in { 1 , . . . , n } , representing an assignment of a history H t to prototype j at time t .

Original: We let the probability of being assigned to prototype j be proportional to the similarity s ,
Revised: We let the probability of being assigned to prototype j be proportional to the similarity s ,

Original: [Equation]
Revised: [Equation]

Original: Now, we define the value V j,t ( π ) of prototype j at time t , obtained under a policy π , as the expected future reward under π given the assignment J t = j :
Revised: Now, we define the value V j,t ( π ) of prototype j at time t , obtained under a policy π , as the expected future reward under π given the assignment J t = j :

Original: [Equation]
Revised: [Equation]

Original: With p ( J t = j ) the marginal probability of being assigned to prototype j at time t , by the law of total expectation,
Revised: With p ( J t = j ) the marginal probability of being assigned to prototype j at time t , by the law of total expectation,

Original: V ( π ) =
Revised: V ( π ) =

Original: (cid:80) nj =1 V j,t [Equation] π ) p
Revised: (cid:80) nj =1 V j,t ( π ) p

Original: [Equation]
Revised: [Equation]

Original: J t = j ) for any t .
Revised: J t = j ) for any t .

Original: Each term
Revised: Each term

Original: [Equation]
Revised: [Equation]

Original: in the sum represents the contribution to the overall value
Revised: in the sum represents the contribution to the overall value

Original: V ( π ) from histories which are similar to prototype j at time t , effectively stratifying the value by types of situations.
Revised: V ( π ) from histories which are similar to prototype j at time t , effectively stratifying the value by types of situations.

Original: Note that we can compute V j,t ( µ ) in a similar way to compare the estimated values of π and µ from a prototype perspective.
Revised: Note that we can compute V j,t ( µ ) in a similar way to compare the estimated values of π and µ from a prototype perspective.

Original: We may express V j,t ( π ) as a weighted expectation under the behavior policy µ , with importance weights W ,
Revised: We may express V j,t ( π ) as a weighted expectation under the behavior policy µ , with importance weights W ,

Original: [Equation]
Revised: [Equation]

Original: where p π ( J t ) is found by importance-weighted marginalization over H t (see derivation in Appendix A.1).
Revised: where p π ( J t ) is found by importance-weighted marginalization over H t (see derivation in Appendix A.1).

Original: We use this strategy to estimate V j,t ( π ) from finite samples.
Revised: We use this strategy to estimate V j,t ( π ) from finite samples.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In Section 4.1, we illustrate our method by examining an example of sepsis management.
Revised: In Section 4.1, we illustrate our method by examining an example of sepsis management.

Original: Using patient data from the
Revised: Using patient data from the

Original: MIMIC-III database [Johnson et al., 2016], we evaluate a replication of the so-called AI Clinician implemented by
Revised: MIMIC-III database [Johnson et al., 2016], we evaluate a replication of the so-called AI Clinician implemented by

Original: Komorowski et al.
Revised: Komorowski et al.

Original: The data are coded as multidimensional time series with a discrete time step of 4 hours.
Revised: [2018], see below for details. In Section 4.2, we inspect the prototype model in more detail.

Original: [Figure] −
Revised: We study the trade-off between transparency and bias and compare the model to several baseline estimators. By utilizing a sepsis simulator, we also investigate the bias induced by prototypes as a function of the sequence length.

Original: [-[Figure] P C      
-]
Revised: [-[Figure] P C      
-]AI Clinician.

Original: Action(f,v) (0, 0) (0, 4) (1, 3) (2, 2) (3, 1) (4, 0) (4, 4)
Revised: The AI Clinician is a clinical decision support model for sepsis management [Komorowski et al., 2018].

Original: Prototype
Revised: The model is learned from data of sepsis patients extracted from the MIMIC-III database [Johnson et al., 2016]. The patient data (e.g., demographics, vital signs and laboratory values) are coded as multidimensional time series with a discrete time step of 4 hours.

Original: [Figure] No the action (dose of IV fluids  and vasopressors) taken by the physicians. The prototypes are numbered 1–10. Note that prototype learning affects the structure of the encoding space.
Revised: There are two treatment variables: the total volume of intravenous (IV) fluids ( f ) and maximum dose of vasopressors ( v ) administered over each 4-hour period.

Original: Post-hoc clustering of the latent space of a model trained without prototypes gives a substantially worse approximation of the behavior policy, see Table 1. In short,  AI Clinician is learned by clustering the data into 750 states, discretizing the combinations of intravenous (IV) fluids ( f ) and vasopressors ( v ) into 25 possible actions ( f, v ) ∈ { 0 , 1 , 2 , 3 , 4 } 2 , and solving the corresponding 
Revised: In short, the AI Clinician is built by clustering the data into 750 states, discretizing the combinations of treatment doses into 25 possible actions ( f, v ) ∈ { 0 , 1 , 2 , 3 , 4 } 2 , and solving the corresponding Markov decision process using value iteration.

Original: Markov decision process using value iteration with rewards r i = ± 100 based on the survival of the patients.
Revised: The final reward r i is +100 if the patient survived and − 100 if the patient died.

Original: This process is repeated 500 times, each time with a new train-test split, and the policy that performs best on the test set is taken as the target policy, π AIC .
Revised: This process is repeated 500 times, each time with a new train-test split, and the model with the highest WIS value estimate on the test set is taken as the target policy, π AIC .

Original: We use the  split associated with π AIC in our experiments.
Revised: We use the data split associated with π AIC in our experiments.

Original: In Section 4.2, we inspect the prototype model in more detail.
Revised: Experimental setup. We consider two types of encoders for the prototype framework: a feedforward neural network (FNN) and a recurrent neural network (RNN). Both encoders have two layers of size 64 with ReLu and tanh, respectively, as activation function. We name these models ProNet and ProSeNet, respectively.

Original: Westudy the trade-off between transparency and bias and compare the model to several baseline estimators: a logistic regression classifier (LR), a random forest classifier (RF), a feedforward neural network (FNN), a recurrent neural network (RNN) and a model based on post-hoc clustering of encoded histories.
Revised: We compare the prototype framework to several baseline models: a logistic regression classifier (LR), a random forest classifier (RF), a vanilla FNN, a vanilla RNN, and a model based on post-hoc clustering of RNN-encoded histories.

Revised: The neural network baselines have the same structure as the corresponding prototype encoder.

Revised: In the main sepsis experiment, we train all neural networks over 400 epochs, using a batch size offor RNN and ProSeNet, and 1024 for FNN and ProNet.

Original: By utilizing a sepsis simulator, we also investigate the bias induced by prototypes as a function of the sequence length.
Revised: We use the Adam algorithm for optimization with learning rate 0.001, weight decay 0.001 and otherwise default parameters. Furthermore, we use the NLL loss when training the vanilla neural networks. All models are calibrated using sigmoid calibration on a held-out validation set ( 25 % of the training data). Further details, including hyperparameter selection, are provided in Appendix B.

Original: 4.1 DEMONSTRATING THE FRAMEWORK
Revised: 4.1 DEMONSTRATING THE FRAMEWORK

Original: To demonstrate the benefit of learning prototypes in OPE, we estimate the behavior policy µ , i.e., the policy followedby clinicians in the MIMIC-III data, using a prototype model with n
Revised: To demonstrate the benefit of learning prototypes in OPE, we estimate the behavior policy µ , i.e., the policy followed

Original:  = 10 prototypes, q = 2 prediction prototypes and an
Revised: [Figure] IV and vasopressors) hoc clustering of the latent space of a model trained without prototypes gives a substantially worse proximation of the behavior policy by clinicians in the MIMIC-III data, using a prototype model with n = 10 prototypes, q = 2 prediction prototypes and an

Original: RNN encoder (see Appendix B for details).
Revised: RNN encoder (i.e., a ProSeNet model).

Original: As an overview of the relationship between prototypes, a PCA plot of encoded training data is shown in Figure 2.
Revised: As an overview of the relationship between prototypes, a PCA plot of encoded training data is shown in Figure 2.

Original: The latent prototypes are numbered 1–10 and the colors indicate action chosen by µ in the data.
Revised: The latent prototypes are numbered 1–10 and the colors indicate action chosen by µ in the data.

Original: Note that the figure is intended to help the reader to orient him/herself in this section; we do not rely on this projection in itself.[Figure]
Revised: Note that the figure is intended to help the reader to orient him/herself in this section; we do not rely on this projection in itself.

Original: We can interpret the prototypes by visualizing trajectories of the corresponding patients.
Revised: We can interpret the prototypes by visualizing trajectories of the corresponding patients.

Original: In Figure 3, we take a closer look at prototypes 5, 7 and 8, which represent each of the major clusters in Figure 2.
Revised: In Figure 3, we take a closer look at prototypes 5, 7 and 8, which represent each of the major clusters in Figure 2.

Original: By plotting three key features—heart rate (HR), mean blood pressure (BP) and SOFA score—and the treatment variables against time, we immediately get a sense of which type of patients the prototypes represent.
Revised: By plotting three key features—heart rate (HR), mean blood pressure (BP) and SOFA score—and the treatment variables against time, we immediately get a sense of which type of patients the prototypes represent.

Original: For example, the patient corresponding to prototype 5 has high heart rate, low blood pressure and high SOFA score—signs of severe sepsis—and receives an aggressive treatment.
Revised: For example, the patient corresponding to prototype 5 has high heart rate, low blood pressure and high SOFA score—signs of severe sepsis—and receives an aggressive treatment.

Original: The prototype 7 patient, who has lower heart rate, higher blood pressure and lower SOFA score, receives low doses of IV fluids and vasopressors.
Revised: The prototype 7 patient, who has lower heart rate, higher blood pressure and lower SOFA score, receives low doses of IV fluids and vasopressors.

Original: We understand that trajectories that are most similar to prototype 7 in the latent space belong to patients who are currently relatively healthy, which they are more likely to be at an early stage of the course.
Revised: We understand that trajectories that are most similar to prototype 7 in the latent space belong to patients who are currently relatively healthy, which they are more likely to be at an early stage of the course.

Original: Interestingly, we observe that all encoded histories until time t = 0 are most similar to either prototype 7 or prototype 9.
Revised: Interestingly, we observe that all encoded histories until time t = 0 are most similar to either prototype 7 or prototype 9.

Original: Is it therefore relevant to ask the question: If we were to follow the AI Clinician instead of the behavior policy, how would the treatment strategychange in an initial stage? That is, how do µ and π AIC differ
Revised: Is it therefore relevant to ask the question: If we were to follow the AI Clinician instead of the behavior policy, how would the treatment strategy

Original: P r o b a b ili t y
Revised: 100 H R

Original: Prototype 7 Physicians ( µ )
Revised: Prototype Prototype

Original: AI Clinician ( π ) ( 0 ,) ( 1 ,) ( 2 ,) ( 3 ,) ( 4 ,) Action (f, v) 0.
Revised: Prototype

Original: P r o b a b ili t y
Revised: M e a n B P S O F

Original: [Figure]

Original: AI
Revised: A

Original: [Figure]{+I V ﬂu i d s 0 2 
 Time step V a s o p r e ss o r s change in an initial stage?+}
Revised: {+I V ﬂu i d s 0 2 
 Time step V a s o p r e ss o r s change in an initial stage?+}

Original: [Figure] for prototypes 7 and 9?
Revised: That is, how do µ and π AIC differ for prototypes 7 and 9?

Original: We can answer this question by comparing the distributions of actions taken under µ and π AIC for these prototypes, see
Revised: We can answer this question by comparing the distributions of actions taken under µ and π AIC for these prototypes, see

Original: Figure 4.
Revised: Figure 4.

Original: For prototype 7, we see that the most likely treatment under both µ and π AIC is to not give any IV fluids or vasopressors.
Revised: For prototype 7, we see that the most likely treatment under both µ and π AIC is to not give any IV fluids or vasopressors.

Original: However, under π AIC , there is also a relatively high probability of increasing the dose of IV fluids—a rare action under µ .
Revised: However, under π AIC , there is also a relatively high probability of increasing the dose of IV fluids—a rare action under µ .

Original: The differences are even greater for prototype 9, where π AIC has a nonzero probability of giving an aggressive treatment with combinations of IV fluids and vasopressors.
Revised: The differences are even greater for prototype 9, where π AIC has a nonzero probability of giving an aggressive treatment with combinations of IV fluids and vasopressors.

Original: Under µ , these treatments have almost zero probability.
Revised: Under µ , these treatments have almost zero probability.

Original: A domain expert can reason about the validity of π AIC : given characteristics of the patient corresponding to prototype 9, would it be medically sound to treat this type of patient as suggested by π AIC in Figure 4?
Revised: A domain expert can reason about the validity of π AIC : given characteristics of the patient corresponding to prototype 9, would it be medically sound to treat this type of patient as suggested by π AIC in Figure 4?

Original: From an OPE perspective, the initial differences between µ and π AIC makes it difficult to accurately estimate V ( π AIC ) .
Revised: From an OPE perspective, the initial differences between µ and π AIC make it difficult to accurately estimate V ( π AIC ) .

Original: A known problem with importance sampling is that the variance of the weights w i can grow exponentially with the sequence length Liu et al.
Revised: A known problem with importance sampling is that the variance of the weights w i can grow exponentially with the sequence length [Liu et al., 2018].

Original: Here, the assumption of overlap is potentially partially violated already at the first time step, and regardless of model of the behavior policy, we observe high variance in the IS weights (ranging from ≪ 1 to the order of 10 3 ) and an extremely small effective sample size ( < 10 ).
Revised: Here, the assumption of overlap is potentially partially violated already at the first time step, and regardless of model of the behavior policy, 0 .0 .0 .0 . P r o b a b ili t y Prototype 7 Physicians ( µ )

Original: To reduce variance, we instead evaluate a different target policy where we follow π AIC in the first time step and then follow µ until the end of the sequence.
Revised: AI Clinician ( π ) [Figure]

Original: For comparison, we do the same with a zero-drug policy πwhich suggests leaving patients untreated.
Revised: P r o b a b ili t y Prototype 9 Physicians ( µ ) AI Clinician ( π )

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: we observe high variance in the IS weights (ranging from ≪ 1 to the order of 10 3 ) and an extremely small effective sample size ( < 10 ). To reduce variance, we instead evaluate a different target policy where we follow π AIC in the first time step and then follow µ until the end of the sequence. For comparison, we do the same with a zero-drug policy πwhich suggests leaving patients untreated.

Original: Figure 5 shows 100 bootstrap estimates of the policy values using five different estimators of µ : the baselines LR, RF,
Revised: Figure 5 shows 100 bootstrap estimates of the policy values using five different estimators of µ : the baselines LR, RF,

Original: FNN and RNN, and our prototype model.
Revised: FNN and RNN, and our prototype model.

Revised: For all models except the RNN baseline and the prototype model, we make

Revised: Prototype ( j )    ˆ V j , 0 ( π ) ˆ p (

Revised: J= j )

Original: For all models except the RNN baseline and the prototype model, we make the Markov assumption and model p ( A | H ) using only the last context-action pair of the history.
Revised: Initial policy AI Clinician Zero-drug Physicians the Markov assumption and model p ( A | H ) using only the last context-action pair of the history.

Original: We note that the results are consistent across estimators.
Revised: We note that the results are consistent across estimators.

Original: In comparison with the estimated value of µ , ˆ V ( µ ) , the results indicate that it could be beneficial to avoid giving drugs to patients at the initial time step, while it seems less favorable to follow π AIC .
Revised: In comparison with the estimated value of µ , ˆ V ( µ ) , the results indicate that it could be beneficial to avoid giving drugs to patients at the initial time step, while it seems less favorable to follow π AIC .

Original: The prototypes allow us to break down the result and answer the question:
Revised: The prototypes allow us to break down the result and answer the question:

Original: If ˆ V ( π ) ̸
Revised: If ˆ V ( π ) ̸

Original: = ˆ V ( µ ) , where does π gain or lose performance?
Revised: = ˆ V ( µ ) , where does π gain or lose performance?

Original: In Figure 6, we show 100 bootstrap estimates of the contribution to the overall value, see (10), for prototypes 1, 7 and 9 at time t = 0 .
Revised: In Figure 6, we show 100 bootstrap estimates of the contribution to the overall value, see (10), for prototypes 1, 7 and 9 at time t = 0 .

Original: These prototypes define the bottom right cluster of Figure 2 where all encoded histories until time t = 0 belong.
Revised: These prototypes define the bottom right cluster of Figure 2 where all encoded histories until time t = 0 belong.

Original: As expected, prototypeand 9 contribute the most to the overall value.Note that the
Revised: As expected, prototypeand 9 contribute the most to the overall value.

Original: # Prediction prototypes ( q ) 0 .0 .0 .

Original: A cc u r a c y [Figure] n n = difference in variance between these prototypes is explained by Figure 4 where π AIC and µ differ more for prototypethan for prototype 7.
Revised: Note that the difference in variance between these prototypes is explained by Figure 4 where π AIC and µ differ more for prototypethan for prototype 7.

Original: Also note that the trend in Figure 5 is repeated here: following the zero-drug policy at the initial time step is generally better than following the AI Clinician.
Revised: Also note that the trend in Figure 5 is repeated here: following the zero-drug policy at the initial time step is generally better than following the AI Clinician.

Original: 4.2 PERFORMANCE OF THE PROTOTYPE MODEL
Revised: 4.2 PERFORMANCE OF THE PROTOTYPE MODEL

Original: While increasing transparency, the use of prototypes imposes restrictions on the model, possibly increasing the approximation error.
Revised: While increasing transparency, the use of prototypes imposes restrictions on the model, possibly increasing the approximation error.

Original: In Figure 7, we show the accuracy of two different prototype models—ProNet and ProSeNet—in approximating p µ ( A | H t ) on the sepsis test data for a varying number of prototypes n and prediction prototypes q (see
Revised: In Figure 7, we show the accuracy of two different prototype models—ProNet and ProSeNet—in approximating p µ ( A | H t ) on the sepsis test data for a varying number of prototypes n and prediction prototypes q (see

Original: Section 3.2).
Revised: Section 3.2).

Original: As encoder, the models use an FNN (ProNet) and an RNN (ProSeNet), respectively.
Revised: As encoder, the models use an FNN (ProNet) and an RNN (ProSeNet), respectively.

Original: The sequential model, making use of the entire history H t , performs the best, especially for q = 1 and q ≥ 4 .
Revised: The sequential model, making use of the entire history H t , performs the best, especially for q = 1 and q ≥ 4 .

Original: Interestingly, the effect of increasing the number of prototypes from 10 to 50 or even  100 is small.
Revised: Interestingly, the effect of increasing the number of prototypes from 10 to 50 or even [Figure] [Figure] 100 is small.

Original: Using only two prediction prototypes works well for this dataset.
Revised: Using only two prediction prototypes works well for this dataset.

Original: In Table 1, we compare the prototype models with n =and q = 2 to the baseline estimators in approximating p µ ( A | H t ) .
Revised: In Table 1, we compare the prototype models with n =and q = 2 to the baseline estimators in approximating p µ ( A | H t ) .

Original: Here, we report accuracy, the area under the ROC curve (AUC) and the static calibration error (SCE) [Nixon et al., 2019], a multiclass extension of the expected calibration error.
Revised: Here, we report accuracy, the area under the ROC curve (AUC) and the static calibration error (SCE) [Nixon et al., 2019], a multiclass extension of the expected calibration error.

Original: The prototype models are superior to the (regularized) LR
Revised: The prototype models are superior to the (regularized) LR

Original: but they perform slightly worse than the black-box models RF, FNN and RNN.
Revised: but they perform slightly worse than the black-box models RF, FNN and RNN.

Original: However, as we see in Figure 7, with increased number of prototypes, ProSeNet has the capacity to approach the performance of these models, at least in terms of accuracy.
Revised: However, as we see in Figure 7, with increased number of prototypes, ProSeNet has the capacity to approach the performance of these models, at least in terms of accuracy.

Original: Finally, we note that the prototype models are superior to a model where posthoc clustering of the RNN encodings are used to identify [Figure]
Revised: Finally, we note that the prototype models are superior to a model where post-hoc clustering of the RNN encodings are used to identify “prototypes”, showing the power of learning prototypes in a supervised manner.

Original: Sequence
Revised: In practice, the trade-off between transparency and bias is likely less of a problem. In the process of diagnosing policy value estimates, we may sacrifice some accuracy in favor of interpretability.

Original: [Figure] “prototypes”, showing the power of learning prototypes in a supervised manner.
Revised: That is, we can learn a model with few prototypes to reason about the target and behavior policies.

Original: We refer to Appendix B.1 for details.
Revised: Then, if the initial analysis indicates that the data supports evaluation of the target policy, we can learn a more complex model for the actual policy evaluation.

Original: 4.2.1 Bias Due to Increased Sequence Length
Revised: 4.2.1 Bias Due to Increased Sequence Length

Original: If the use of prototypes introduces a bias in the estimated propensity, it is natural to ask what it means for the sequential setting, where multiple propensities are multiplied together to form the importance weights.
Revised: If the use of prototypes introduces a bias in the estimated propensity, it is natural to ask what it means for the sequential setting, where multiple propensities are multiplied together to form the importance weights.

Original: To quantity this effect, we consider the synthetic environment of sepsis management provided by Oberst and Sontag [2019].
Revised: To quantity this effect, we consider the synthetic environment of sepsis management provided by Oberst and Sontag [2019].

Original: By sampling a large amount of data from the environment, we estimate the true parameters of the underlying Markov decision process.
Revised: By sampling a large amount of data from the environment, we estimate the true parameters of the underlying Markov decision process.

Original: We then learn an optimal behavior policy using policy iteration.
Revised: We then learn an optimal behavior policy using policy iteration.

Original: We refer to Appendix B.2 for details.
Revised: We refer to Appendix B.2 for details.

Original: We collect trajectories of the behavior policy of various lengths, from 5 to 30 time steps.
Revised: We collect trajectories of the behavior policy of various lengths, from 5 to 30 time steps.

Original: For each trajectory length, we use the data to (a) estimate the behavior policy µ using an FNN and different prototype models, respectively; (b) learn a target policy π using policy iteration; and (c) estimate the value of π using both the true behavior policy µ and its estimators ˆ µ .
Revised: For each trajectory length, we use the data to (a) estimate the behavior policy µ using a vanilla FNN and FNN-based prototype models with varying number of (prediction) prototypes, respectively; (b) learn a target policy π using policy iteration; and (c) estimate the value of π using both the true behavior policy µ and its estimators ˆ µ .

Original: Note that any difference in the estimated values stems from a difference in the importance weights w .
Revised: Note that any difference in the estimated values stems from a difference in the importance weights w .

Original: In Figure 8, we plot the relative error of estimated weights against the trajectory length for four estimators of µ .
Revised: In Figure 8, we plot the relative error of estimated weights against the trajectory length for four estimators of µ .

Original: 4 Averaging over 100 iterations of the sampling and learning process, we see that the ratio w ˆ µ /w µ generally differs from 1 for all estimators and that modeling with prototypes induces larger bias than using an FNN.
Revised: 5 Averaging over 100 iterations of the sampling and learning process, we see that the ratio w ˆ µ /w µ generally differs from 1 for all estimators and that modeling with prototypes induces larger bias than using an FNN.

Original: For longer sequences, the number of prediction prototypes q becomes critical.[Table]
Revised: For longer sequences, the number of prediction prototypes q becomes critical.

Original: [Table] Finally [Table] we quantify the absolute effect prototypes have on the value estimate for sequences of length 15, which is close to the average sequence length in the MIMIC-III data.
Revised: Finally, we quantify the absolute effect prototypes have on the value estimate for sequences of length 15, which is close to the average sequence length in the MIMIC-III data.

Original: We compute the true value of π by running it in the simulator and compare this value to weighted IS estimates using the estimators in Figure 8.
Revised: We compute the true value of π by running it in the simulator and compare this value to weighted IS estimates using the estimators in Figure 8.

Original: We observe final rewards r i = ± 1 if a simulated patient is discharged or dies; otherwise r i = 0 .
Revised: We observe final rewards r i = ± 1 if a simulated patient is discharged or dies; otherwise r i = 0 .

Original: Averaging over 100 iterations, the estimated value has an absolute difference from the true value that amounts to 0.39 for the FNN (standard deviation 0.27), 0.46 (0.33) for ProNet with n = 10 and q = 2 , 0.50 (0.34) for ProNet with n = 10 and q = 5 and 0.41 (0.31) for ProNet with n = 100 and q = 5 .
Revised: Averaging over 100 iterations, the estimated value has an absolute difference from the true value that amounts to 0.39 for the FNN (standard deviation 0.27), 0.46 (0.33) for ProNet with n = 10 and q = 2 , 0.50 (0.34) for ProNet with n = 10 and q = 5 and 0.41 (0.31) for ProNet with n = 100 and q = 5 .

Original: These results should be compared to the average value of π (0.06 (0.10)) and the WIS estimate using the true behavior policy (0.38 (0.27)).
Revised: These results should be compared to the average value of π (0.06 (0.10)) and the WIS estimate using the true behavior policy (0.38 (0.27)).

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Issues with importance sampling methods for OPE are well known.
Revised: Issues with importance sampling methods for OPE are well known.

Original: Several works aim at describing issues related to high variance [Gottesman et al., 2019], or mitigating them using methodological advances [Precup, 2000, Thomas and
Revised: Several works aim at describing issues related to high variance [Gottesman et al., 2019], or mitigating them using methodological advances [Precup, 2000, Thomas and

Original: Brunskill, 2016, Jiang and Li, 2016, Schneeweiss et al., 2009, Swaminathan and Joachims, 2015].
Revised: Brunskill, 2016, Jiang and Li, 2016, Schneeweiss et al., 2009, Swaminathan and Joachims, 2015].

Original: Others aim to use weights to identify a new study population for which the policy’s value can be more efficiently estimated [Li et al., 2018a, Fogarty et al., 2016].
Revised: Others aim to use weights to identify a new study population for which the policy’s value can be more efficiently estimated [Li et al., 2018a, Fogarty et al., 2016].

Original: Oberst et al.
Revised: Oberst et al.

Original: [2020] emphasize the value of interpretability in this endeavour to communicate the generalizability of the estimate.
Revised: [2020] emphasize the value of interpretability in this endeavour to communicate the generalizability of the estimate.

Original: Our method is compatible with all three approaches, allowing for transparent descriptions of variance issues, identifying new study populations and for use as plug-in estimates.
Revised: Our method is compatible with all three approaches, allowing for transparent descriptions of variance issues, identifying new study populations and for use as plug-in estimates.

Original: Interpretability is a an important component of learning systems deployed in increasingly critical functions [Rudin, 2019, Lipton, 2018].
Revised: Interpretability is a an important component of learning systems deployed in increasingly critical functions [Rudin, 2019, Lipton, 2018].

Original: Rule-based estimators, such as rule list [Wang and Rudin, 2015] and decision trees, are often favored for their short descriptions but generalize poorly to sequential inputs which are the focus of this work.
Revised: Rule-based estimators, such as rule list [Wang and Rudin, 2015] and decision trees, are often favored for their short descriptions but generalize poorly to sequential inputs which are the focus of this work.

Original: Gottesman et al.
Revised: Gottesman et al.

Original: [2020] proposed an approach for interpretable
Revised: [2020] proposed an approach for interpretable

Original: OPE which highlights transitions in data whose removal would have a large impact on the estimate.
Revised: OPE which highlights transitions in data whose removal would have a large impact on the estimate.

Original: This approach is related to ours but answers a different set of questions.
Revised: This approach is related to ours but answers a different set of questions.

Original: Evaluating policies using direct sample-to-sample comparison has a long tradition in policy evaluation through the use of matching estimators of causal effects, see e.g., [Rosenbaum and Rubin, 1983, Rubin, 2006, Kallus, 2020].
Revised: Evaluating policies using direct sample-to-sample comparison has a long tradition in policy evaluation through the use of matching estimators of causal effects, see e.g., [Rosenbaum and Rubin, 1983, Rubin, 2006, Kallus, 2020].

Original: While favored for its transparency, this approach is typically only used to compare two deterministic policies such as “treat all” or “treat none”.
Revised: While favored for its transparency, this approach is typically only used to compare two deterministic policies such as “treat all” or “treat none”.

Original: Matching often relies either on specifying a similarity function in advance or on an estimate of the behavior policy.
Revised: Matching often relies either on specifying a similarity function in advance or on an estimate of the behavior policy.

Original: In high-dimensional settings, this often leads to bias or lost interpretability.
Revised: In high-dimensional settings, this often leads to bias or lost interpretability.

Original: Our approach aims to combine the transparency of matching estimators with the flexibility of representation learning methods.
Revised: Our approach aims to combine the transparency of matching estimators with the flexibility of representation learning methods.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: In this work, we have studied off-policy evaluation (OPE) using importance sampling (IS) in the case where the behavior policy µ is unknown and must be estimated from data.
Revised: In this work, we have studied off-policy evaluation (OPE) using importance sampling (IS) in the case where the behavior policy µ is unknown and must be estimated from data.

Original: While IS is a popular OPE method, it may be difficult to assess the quality of an IS value estimate; standard practices, such as inspecting importance weights, provide only an average or a per-sample view of potential issues.
Revised: While IS is a popular OPE method, it may be difficult to assess the quality of an IS value estimate. Standard practices, such as inspecting importance weights, provide only an average or a per-sample view of potential issues.

Original: To address this issue, we proposed estimating the behavior policy for IS using prototype learning to better explain patterns in policy  decisions and value estimates.
Revised: To address this issue, we proposed estimating the behavior policy for IS using prototype learning to better explain patterns in policy [Table] [Table] decisions and value estimates.

Original: We demonstrated our idea using a real-world example of sepsis management.
Revised: We demonstrated our idea using a real-world example of sepsis management.

Original: While the use of prototypes increases the approximation error, we find that prototype models have the capacity to perform similarly to plain neural networks.
Revised: While the use of prototypes increases the approximation error, we found that prototype models have the capacity to perform similarly to plain neural networks.

Original: We hope to provide theoretical results for the approximation error in future work.

Original: When reflecting upon the results of the sepsis study it may seem strange that it would be advantageous to never treat patients at the onset of sepsis.
Revised: When reflecting upon the results of the sepsis study it may seem strange that it would be advantageous to never treat patients at the onset of sepsis.

Original: Even though prototypes serve as a tool for inspecting policies and value estimates, they do not answer all questions.
Revised: Even though prototypes serve as a tool for inspecting policies and value estimates, they do not answer all questions.

Original: For example, there may be variables affecting both the treatment and the outcome that are not present in the observed data.
Revised: For example, there may be variables affecting both the treatment and the outcome that are not present in the observed data.

Original: In such a case, the ignorability assumption defined in Section 2 no longer holds.A THE PROTOTYPE MODEL
Revised: In such a case, the ignorability assumption defined in Section 2 no longer holds.

Original: In Section 3.1, we only briefly described the objective function

Original: [Equation]

Original: where NLL( D ; Θ) is the negative log-likelihood, R d (Θ) , R c (Θ) and R e (Θ) are regularization terms and λ d , λ c and λ e are regularization parameters. Θ denotes the set of model parameters, i.e., the parameters of the encoding network e , the weights B, c and the prototypes ˜ H . For a given dataset
Revised: Failure to detect such an issue is not a limitation of prototypes; ignorability cannot be verified by statistical means.

Original: D =

Original: (( h 1 t 1 , a 1 t 1 ) , . .

Original: , ( h mt m , a mt m )) , drawn according to a distribution p µ , the NLL loss of the estimate ˆ p µ , parameterized in Θ , is defined as

Original: [Equation]

Original: Furthermore, the regularization terms are defined as follows (see Ming et al.

Original: [2019] for further details):

Original: • The diversity regularization

Original: [Equation]

Original: where d ( z, z ′ )

Original: =

Original: ∥ z − z ′ ∥ 2 , penalizes latent prototypes that are too close to each other. The parameter d min is a tunable hyperparameter in our experiments. • The clustering regularization R
Revised: A limitation of our analysis is that it does not separate different types of errors introduced by prototype learning.

Original: [Equation]

Original: encourages the encoded histories to approach the most similar latent prototypes, which creates a clustering structure in the latent space.

Original: • The evidence regularization

Original: [Equation] encourages the latent prototypes to approach the encodings that are most similar to them. A.1 PROTOTYPE VALUE In Section 3.3, we showed how the prototypes can be used to compute the value of a policy π for prototype j at time t ,
Revised: We conjecture that a model with fewer prototypes is more likely to overestimate overlap between behavior and target policies, rather than underestimate it, due to increased smoothness in the estimated behavior policy.

Original: V j,t ( π ) .

Original: Below, we derive a statistical estimator for V j,t ( π ) using observations under µ .

Original: First, we have

Original: [Equation] This equation follows from the fact that J t is conditionally independent of all other variables given H t . Now, with W importance weightsfor π and µ , [Equation]
Revised: This will likely result in less extreme importance weights and lower variance, potentially at the cost of increased bias.

Original: Following standard definitions,

Original: [Equation]

Original: which may be identified using importance sampling, p

Original: [Equation]

Original: [-with W t = (cid:81) tt ′ =
-]
Revised: [-with W t = (cid:81) tt ′ =
-]

Original: p π ( A t ′ | H t ′ )

Original: p µ ( A t ′ | H t ′ ) .

Original: Hence, we may estimate

Original: [Equation]

Original: For trajectories i which end before t , we let ˆ p ( J t =

Original: j |

Original: H t = h it )

Original: = 0 . A.2 IS THERE A GOOD PROTOTYPE MODEL? In Section 3.1, we asked the question: Assuming that adjusting for the history H t is sufficient for unbiased policy evaluation, do there exist prototype histories ˜ H , an encoding e and asimilarity function s such that evaluation using the prototype model is accurate?
Revised: We hope to provide analysis which more precisely characterizes the approximation error as a function of the number of prototypes in future work.

Original: Here follows an example when this is provably the case.

Original: [-Consider the history at the first time step, H 
-]
Revised: [-Consider the history at the first time step, H 
-]

Original: = X 0 ∈ R d .

Original: Assume that for each action a , the distribution of histories in which action a is taken is isotropic Gaussian, ( H 0 | A 0 = a ) ∼ N ( µ a , γ 2 ) .

Original: Then, the joint distribution of ( H 0 , A 0 ) is a Gaussian mixture model (GMM) with components identified by the actions, a = 1 , . . .

Original: , k .

Original: We have by the definition of the GMM that

Original: [Equation]

Original: where s is defined as in (3).

Original: As a result, with the component means ˜ X = [ µ 1 , ..., µ k ] as prototypes and S ( ˜ X, h )
Revised: Finally, we have used subsequences of all variables in patient histories as prototypes.

Original: =

Original: [-[ s ( µ 1 , h ) , ..., s ( µ k , h )] ⊺ , the behavior policy is given by p ( A 0 | H 
-]
Revised: [-[ s ( µ 1 , h ) , ..., s ( µ k , h )] ⊺ , the behavior policy is given by p ( A 0 | H 
-]

Original: = h ) =

Original: S ( ˜ X, h ) /

Original: ( (cid:80) a s (˜ x a , h )) , which matches (4) with B = 1 / ( (cid:80) a s (˜ x a , h )) and c = 0 , up to the application of the softmax function.

Original: Furthermore, the prototype assignment probability in (8) is equal to the behavior policy.

Original: We state a generalization below.

Original: Theorem 1.

Original: Assume that there exists a bijective, differentiable encoding function e : H → Z such that ∀ t : ( e ( H t ) , A t ) ∼ GMM with stationary component means { µ a } ka =1 and variance γ 2 .

Original: Then, with prototypes ˜

Original: [Equation]

Original: [Equation]

Original: ] ⊺ , and S as defined above,

Original: [Equation]

Original: Proof.

Original: Due to bijectivity, p ( A t | H t = h ) =

Original: p

Original: A t | e ( H t ) =

Original: e ( h )) .

Original: The final result follows from the same argument as for the special case of H 0 above. Theorem 1 shows that there are indeed problems for which a prototype model that exactly describes the behavior policy exists . This also implies that there exists a prototype estimate of the value V ( π ) which isunbiased. However, it does notgive guarantees for recovering such a model from data, or that the training set contains samples which act well as prototypes.
Revised: This choice is aligned with the literature on sequence prototypes but is not the only option.

Original: Learning encoding functions e which satisfy the conditions of Theorem 1 has been studied in the context of normalizing flows [Kong and Chaudhuri, 2020, Rezende and Mohamed, 2015].

Original: B EXPERIMENTAL DETAILS

Original: The prototype approach for OPE was evaluated on an realworld sepsis data extracted from the MIMIC-III database [Johnson et al., 2016].

Original: In addition, a synthetic environment for sepsis management, provided by Oberst and Sontag [2019], was used to study the bias induced by prototypes as a function of the trajectory length.

Original: In this section, we give details about the experiments.

Original: To produce the results presented in this paper, we needed about 750 core-hours of computational time. The neural networks were implemented in PyTorch and trained on GPU (Nvidia Tesla T4). Other models were implemented using scikit-learn. B.1 USING DATA FROM MIMIC-III
Revised: For example, in applications of prototype learning to image classification, parts of images were used as prototypes [Li et al., 2018b], not entire images from the training set.

Original: We extracted the dataset of patients suffering from sepsis from the MIMIC-III database using the code provided by

Original: Komorowski et al.

Original: This dataset contains the features listed in Supplementary Table 2 in Komorowski et al.

Original: [2018] as well as the total fluid intake and the total urine output for each patient.

Original: We learned the target policy, the so-called AI Clinician, using the Matlab code provided by

Original: Komorowski et al.

Original: To evaluate the 500 candidate policies, we used only the MIMIC-III test data and not data from the eICU Research Institute Database.

Original: We used the train-test split associated with the best performing candidate in our experiments.

Original: We trained the estimators of the behavior policy using a subset of the available features: heart rate, systolic blood pressure, diastolic blood pressure, mean blood pressure, shock index, hemoglobin, BUN, creatine, urine output over 4 hours, pH, base excess, bicarbonate, lactate, PaO 2 / FiOratio, age, Elixhauser index and SOFA score.

Original: In addition, we included the treatment dose of vasopressors and IV fluids, respectively, over the previous 4 hours.

Original: These values were set to 0 at the initial time steps. We estimated the value of the policies performing weighted importance sampling of the test set trajectories. For all estimators of  the behavior policy, we excluded a small number of sequences whose IS weight exceeded 100. B.1.1 Model Parameters
Revised: To simplify description further and improve interpretability in policy evaluation, we may define prototypes as sequences of only variables which are important for the behavior policy.

Original: We considered two types of encoders for the prototype framework: a feedforward neural network (FNN) and a recurrent neural network (RNN). We named these models ProNet (FNN encoder) and ProSeNet (RNN encoder), respectively. Both encoders had two hidden layers, each of size 64, and we used ReLu and tanh, respectively, as activation function.
Revised: Acknowledgements

Original: We compared the prototype models to two neural network baselines—a vanilla FNN and a vanilla

Original: RNN—with architectures identical to the encoders. A linear layer was added to the baselines to obtain predictions of the desired shape. We trained all neural networks over 400 epochs, using a batch size of 64 for RNN and ProSeNet, and 1024 for FNN and ProNet. For optimization, the Adam algorithm was used with default parameters, learning rate 0.001 and weight decay 0.001.
Revised: We would like to thank Patrick Royer for insightful discussions regarding the sepsis management policy evaluation.

Original: The NLL was used as loss function for FNN and

Original: RNN.

Original: For ProNet and ProSeNet, we selected parameters of the diversity regularization ( d min , λ d ) by performing 3-fold cross-validation over a grid of points in the parameter space { 1 , 2 , 3 , 4 , 5 } × { 0 . 00001 , 0 . 0001 , 0 .

Original: These parameters were optimized for each combination of prototypes n and prediction prototypes q in our experiments.

Original: The parameters λ c and λ e were set to 0.001, and we performed the projection step, see (6), every fifth epoch.

Original: We also compared the prototype models to a logistic regression (LR) classifier and a random forest (RF) classifier.

Original: For LR and RF, we searched for optimal models using 3fold cross-validation, considering the following parameter values:

Original: • LR: regularization: { L1 , L2 } ; regularization strength: 10 numbers spaced evenly on a log-scale from 1 × 10 − 4 t o 1 × 10 4 ; • RF: maximum tree depth: { 5 , 10 , 15 , 20 , None } .

Original: Finally, we compared the prototype models to the a model based on post-hoc clustering of encodings. More specifically, we performed K-means clustering to cluster the encodings of the converged RNN model into 10 clusters. We used the encodings that had the shortest Euclidean distance to the cluster centroids as “prototypes” and trained a logistic regression model on the similarity vector between encodings and “prototypes”. All models were calibrated using sigmoid calibration on a held-out validation set ( 25 % of the training data).
Revised: Furthermore, we would like to thank Devdatt Dubhashi,

Original: B.2 USING THE SEPSIS SIMULATOR

Original: To estimate the bias induced by prototypes as a function of trajectory length, we utilized the sepsis simulator provided by Oberst and Sontag [2019].

Original: 6 We used the full state representation consisting of five discrete variables—a binary diabetes indicator and four ordinal-valued vital signs (heart rate, systolic blood pressure, blood glucose level and blood oxygen level)—as well as the previously administered treatments There is a probability of 0.2 that a randomly initialized patient has diabetes. Furthermore, there are three binary treatment variables—antibiotics, vasopressors and ventilation—resulting in an 8-dimensional action space.
Revised: Emil Carlsson, Morteza Haghir Chehreghani and Adam

Original: A simulated patient is discharged if the patient has normal vitals and is not given any treatments; discharge results in a positive reward.

Original: Death is associated with a negative reward and occurs if at least three vitals are abnormal. If neither discharge nor death has occurred at the end of the sequence, the reward is zero. We refer to the code for details about, for example, the levels of the vitals and the transition probabilities of the Markov decision process (MDP). For the experiment presented in Figure 8, we used the notebook learn_mdp_parameters provided by Oberst and
Revised: Breitholtz for valuable feedback on this work.

Original: Sontag [2019] to estimate the true parameters of the MPD.

Original: We then learned an optimal behavior policy using policy iteration. The policy was softened so that all actions had a nonzero probability of being chosen in each state. We generated trajectories of varying length from this policy to estimate the effect of bias. Some trajectories ended prematurely due to discharge or death and we therefore ensured that the number of collected state-action pairs were roughly the same for all sequence lengths.
Revised: This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Original: Specifically, for each sequence length, we generated 20 000 s tate-action pairs for training and sigmoid calibration of the estimators ˆ µ , respectively.

Original: The training set was also used to estimate the MDP parameters and learn a target policy π using policy iteration.

Original: Also this policy was softened to avoid zero probabilities.

Original: We generated an additional evaluation set of the same size as the training and calibration sets. We repeated the process of data collection and learning 100 times. To produce Figure 8, we removed a small fraction of samples for which the weight ratio exceeded 10 3 . B.2.1 Model Parameters
Revised: The computations in this work were enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at Chalmers Centre for Computational Science and Engineering (C3SE) partially funded by the Swedish

Original: Since we used the full state representation, we made the

Original: Markov assumption and estimated the behavior policy using a vanilla FNN and FNN-based prototype models—ProNet models—with different settings for the number of (prediction) prototypes.

Original: The FNN-coder had two layers, each of size 64, and ReLu as activation function.

Original: The FNN baseline used the same architecture with an additional linear layer to obtain predictions of the right shape We trained all models over 30 epochs, using a batch size of 128. For optimization, the Adam algorithm was used with default parameters, learning rate 0.001 and weight decay 0.001. We selected parameters of the diversity regularization ( d min , λ d ) by performing 3-fold cross-validation over a grid of points in the parameter space { 1 , 2 , 3 , 4 , 5 } × { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } .
Revised: Research Council through grant agreement no. 2018-05973.

Original: The parameters λ c and λ e were set to 0.001, and we performed the projection step every fifth epoch.