Original: M ODEL A GNOSTIC I NTERPRETABILITY FOR M ULTIPLE I NSTANCE L EARNING
Revised: M ODEL A GNOSTIC I NTERPRETABILITY FOR M ULTIPLE I NSTANCE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In Multiple Instance Learning (MIL), models are trained using bags of instances, where only a single label is provided for each bag.
Revised: In Multiple Instance Learning (MIL), models are trained using bags of instances, where only a single label is provided for each bag.

Original: A bag label is often only determined by a handful of key instances within a bag, making it difficult to interpret what information a classifier is using to make decisions.
Revised: A bag label is often only determined by a handful of key instances within a bag, making it difficult to interpret what information a classifier is using to make decisions.

Original: In this work, we establish the key requirements for interpreting MIL models.
Revised: In this work, we establish the key requirements for interpreting MIL models.

Original: We then go on to develop several model-agnostic approaches that meet these requirements.
Revised: We then go on to develop several model-agnostic approaches that meet these requirements.

Original: Our methods are compared against existing inherently interpretable MIL models on several datasets, and achieve an increase in interpretability accuracy of up to 30%.
Revised: Our methods are compared against existing inherently interpretable MIL models on several datasets, and achieve an increase in interpretability accuracy of up to 30%.

Original: We also examine the ability of the methods to identify interactions between instances and scale to larger datasets, improving their applicability to real-world problems.
Revised: We also examine the ability of the methods to identify interactions between instances and scale to larger datasets, improving their applicability to real-world problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In Multiple Instance Learning (MIL), data is organised into bags of instances, and each bag is given a single label.
Revised: In Multiple Instance Learning (MIL), data is organised into bags of instances, and each bag is given a single label.

Original: This reduces the burden of labelling, as each instance does not have to be assigned a label, making it useful in applications where labelling is expensive, such as healthcare (Carbonneau et al., 2018).
Revised: This reduces the burden of labelling, as each instance does not have to be assigned a label, making it useful in applications where labelling is expensive, such as healthcare (Carbonneau et al., 2018).

Original: However, there are often only a few key instances within a bag that determine the bag label, so it is necessary to identify these key instances in order to interpret a model’s decision-making (Liu et al., 2012).
Revised: However, there are often only a few key instances within a bag that determine the bag label, so it is necessary to identify these key instances in order to interpret a model’s decision-making (Liu et al., 2012).

Original: Directly interpreting the decision-making process of machine learning methods is difficult due to the complexity of the models and the scale of the data on which they trained, so there is a need for methods that allow insights into the decision-making processes (Gilpin et al., 2018).
Revised: Directly interpreting the decision-making process of machine learning methods is difficult due to the complexity of the models and the scale of the data on which they trained, so there is a need for methods that allow insights into the decision-making processes (Gilpin et al., 2018).

Original: In MIL problems, only some of the instances in each bag will be discriminatory, i.e., a significant number of the instances in a bag will be non-discriminatory or even related to other bag classes (Amores, 2013).
Revised: In MIL problems, only some of the instances in each bag will be discriminatory, i.e., a significant number of the instances in a bag will be non-discriminatory or even related to other bag classes (Amores, 2013).

Original: Identifying the important instances and presenting those as the interpretation of model decision-making filters out the non-discriminatory information and provides the explainee with a reduced and relevant interpretation, which will avoid overloading the user (Li et al., 2015).
Revised: Identifying the important instances and presenting those as the interpretation of model decision-making filters out the non-discriminatory information and provides the explainee with a reduced and relevant interpretation, which will avoid overloading the user (Li et al., 2015).

Original: Our work provides the following novel contributions:
Revised: Our work provides the following novel contributions:

Original: MIL interpretability definition We identify two questions that interpretability methods for MIL should be able to answer: 1) Which are the key instances for a bag? 2)
Revised: MIL interpretability definition We identify two questions that interpretability methods for MIL should be able to answer: 1) Which are the key instances for a bag? 2)

Original: What outcome (class/value) does each key instance support?
Revised: What outcome (class/value) does each key instance support?

Original: In the rest of this work, we will refer to these two questions as the which and the what questions respectively.
Revised: In the rest of this work, we will refer to these two questions as the which and the what questions respectively.

Original: As we explore in Section 2, existing interpretability methods are able to answer which questions with varying degrees of accuracy, but can only answer what questions under certain assumptions.
Revised: As we explore in Section 2, existing interpretability methods are able to answer which questions with varying degrees of accuracy, but can only answer what questions under certain assumptions.

Original: MIL model-agnostic interpretability methods Existing interpretability methods are often model-specific, i.e., they can only be applied to certain types of MIL models.
Revised: MIL model-agnostic interpretability methods Existing interpretability methods are often model-specific, i.e., they can only be applied to certain types of MIL models.

Original: To this end, we build upon the state-of-the-art MIL inherent interpretability methods by developing model-agnostic interpretability methods that are up to 30% more more accurate.
Revised: To this end, we build upon the state-of-the-art MIL inherent interpretability methods by developing model-agnostic interpretability methods that are up to 30% more more accurate.

Original: The model-agnostic interpretability methods that we propose can be applied to any MIL model, and are able to answer both which and what questions.
Revised: The model-agnostic interpretability methods that we propose can be applied to any MIL model, and are able to answer both which and what questions.

Original: MIL interpretability method comparison We compare existing model-specific methods with our model-agnostic methods on several MIL datasets.
Revised: MIL interpretability method comparison We compare existing model-specific methods with our model-agnostic methods on several MIL datasets.

Original: Our experiments are also carried out on four types of MIL model, giving a comprehensive comparison of interpretability performance.
Revised: Our experiments are also carried out on four types of MIL model, giving a comprehensive comparison of interpretability performance.

Original: To the best of the authors’ knowledge, this is one of the first studies to compare different interpretability methods for MIL.
Revised: To the best of the authors’ knowledge, this is one of the first studies to compare different interpretability methods for MIL.

Original: The remainder of this work is laid out as follows.
Revised: The remainder of this work is laid out as follows.

Original: Section 2 provides relevant background knowledge and reviews existing MIL interpretability methods.
Revised: Section 2 provides relevant background knowledge and reviews existing MIL interpretability methods.

Original: Section 3 outlines the requirements for MIL interpretability and details our approaches that meet these requirements.
Revised: Section 3 outlines the requirements for MIL interpretability and details our approaches that meet these requirements.

Original: Next, Section 4 provides our results and experiments.
Revised: Next, Section 4 provides our results and experiments.

Original: Section 5 discusses our findings, and Section 6 concludes.
Revised: Section 5 discusses our findings, and Section 6 concludes.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: The standard MIL assumption (SMIL) is a binary problem with positive and negative bags (Dietterich et al., 1997; Maron & Lozano-P´erez, 1998).
Revised: The standard MIL assumption (SMIL) is a binary problem with positive and negative bags (Dietterich et al., 1997; Maron & Lozano-P´erez, 1998).

Original: A bag is positive if any of its instances are positive, otherwise it is negative.
Revised: A bag is positive if any of its instances are positive, otherwise it is negative.

Original: The assumptions of SMIL can be relaxed to allow more generalised versions of MIL, e.g., through extensions that include additional positive classes (Scott et al., 2005; Weidmann et al., 2003).
Revised: The assumptions of SMIL can be relaxed to allow more generalised versions of MIL, e.g., through extensions that include additional positive classes (Scott et al., 2005; Weidmann et al., 2003).

Original: SMIL also assumes there is no interaction between instances.
Revised: SMIL also assumes there is no interaction between instances.

Original: However, recent work has highlighted that modelling relationships between instances is beneficial for performance (Tu et al., 2019;
Revised: However, recent work has highlighted that modelling relationships between instances is beneficial for performance (Tu et al., 2019;

Original: Zhou et al., 2009).
Revised: Zhou et al., 2009).

Original: Existing methods for interpreting MIL models often rely on the SMIL assumption, so cannot generalise to other MIL problems.
Revised: Existing methods for interpreting MIL models often rely on the SMIL assumption, so cannot generalise to other MIL problems.

Original: In addition, existing methods are often model-specific, i.e., they only work for certain types of MIL models, which constrains the choice of model (Ribeiro et al., 2016a).
Revised: In addition, existing methods are often model-specific, i.e., they only work for certain types of MIL models, which constrains the choice of model (Ribeiro et al., 2016a).

Original: Identifying the key instances in MIL bags is a form of local interpretability (Molnar, 2020), as the key instances are detected for a particular input to the model.
Revised: Identifying the key instances in MIL bags is a form of local interpretability (Molnar, 2020), as the key instances are detected for a particular input to the model.

Original: Two of the key motivators for interpreting the decision-making process of a MIL system are reliability and trust — identifying the key instances allows an evaluation of the reliability of the system, which increases trust as the decision-making process becomes more transparent.
Revised: Two of the key motivators for interpreting the decision-making process of a MIL system are reliability and trust — identifying the key instances allows an evaluation of the reliability of the system, which increases trust as the decision-making process becomes more transparent.

Original: In this work, we use the term interpretability rather than explainability to convey that the analysis remains tied to the models, i.e., these methods do not provide non-technical explanations in human terms.
Revised: In this work, we use the term interpretability rather than explainability to convey that the analysis remains tied to the models, i.e., these methods do not provide non-technical explanations in human terms.

Original: Under the SMIL assumption, which and what questions are equivalent — there are only two classes (positive and negative), and the only key instances are the instances that are positive, therefore once the key instances are identified, it is also known what outcome they support.
Revised: Under the SMIL assumption, which and what questions are equivalent — there are only two classes (positive and negative), and the only key instances are the instances that are positive, therefore once the key instances are identified, it is also known what outcome they support.

Original: However, when there are multiple positive classes, if instances from different classes co-occur in the same bags, answering which and what questions becomes two distinct problems.
Revised: However, when there are multiple positive classes, if instances from different classes co-occur in the same bags, answering which and what questions becomes two distinct problems.

Original: For example, some key instances will support one positive class, and some key instances will support another.
Revised: For example, some key instances will support one positive class, and some key instances will support another.

Original: Therefore, solely identifying which are the key instances does not answer the second question of what class they support.
Revised: Therefore, solely identifying which are the key instances does not answer the second question of what class they support.

Original: Existing methods, such as key instance detection (Liu et al., 2012), MIL attention (Ilse et al., 2018), and MIL graph neural networks (GNNs; Tu et al.
Revised: Existing methods, such as key instance detection (Liu et al., 2012), MIL attention (Ilse et al., 2018), and MIL graph neural networks (GNNs; Tu et al.

Original: (2019)) do not condition their output on a particular class, so it is not apparent what class each instance supports, i.e., they can only answer which questions.
Revised: (2019)) do not condition their output on a particular class, so it is not apparent what class each instance supports, i.e., they can only answer which questions.

Original: One existing method that can answer what questions is mi-Net (Wang et al., 2018), as it produces instance-level predictions as part of its processing.
Revised: One existing method that can answer what questions is mi-Net (Wang et al., 2018), as it produces instance-level predictions as part of its processing.

Original: However, these instance-level predictions do not take account of interactions between the instances, so are often inaccurate.
Revised: However, these instance-level predictions do not take account of interactions between the instances, so are often inaccurate.

Original: A related piece of work on MIL interpretability is Tibo et al.
Revised: A related piece of work on MIL interpretability is Tibo et al.

Original: (2020), which considers interpretability within the scope of multi-multi-instance learning (MMIL; Tibo et al.
Revised: (2020), which considers interpretability within the scope of multi-multi-instance learning (MMIL; Tibo et al.

Original: ; Fuster et al.
Revised: ; Fuster et al.

Original: In MMIL, the instances within a bag are arranged into into further bags, giving a hierarchical bags-of-bags structure.
Revised: In MMIL, the instances within a bag are arranged into into further bags, giving a hierarchical bags-of-bags structure.

Original: The interpretability techniques presented by Tibo et al.
Revised: The interpretability techniques presented by Tibo et al.

Original: (2020) are model-specific as they are only designed for MMIL networks.
Revised: (2020) are model-specific as they are only designed for MMIL networks.

Original: In this work, we aim to overcome the limitations of existing methods by developing model-agnostic methods that can answer both which and what questions.
Revised: In this work, we aim to overcome the limitations of existing methods by developing model-agnostic methods that can answer both which and what questions.

Original: In single instance supervised learning, model-agnostic techniques have been developed to interpret models.
Revised: In single instance supervised learning, model-agnostic techniques have been developed to interpret models.

Original: Post-hoc local interpretability methods, such as Local Interpretable Model-agnostic Explanations (LIME Ribeiro et al.
Revised: Post-hoc local interpretability methods, such as Local Interpretable Model-agnostic Explanations (LIME; Ribeiro et al.

Original: (2016b))
Revised: (2016b))

Original: and SHapley Additive exPlanations (SHAP; Lundberg & Lee (2017)), work by approximating the original predictive model with a locally faithful surrogate model that is inherently interpretable.
Revised: and SHapley Additive exPlanations (SHAP; Lundberg & Lee (2017)), work by approximating the original predictive model with a locally faithful surrogate model that is inherently interpretable.

Original: The surrogate model learns from simplified inputs that represent perturbations of the original input that is being analysed.
Revised: The surrogate model learns from simplified inputs that represent perturbations of the original input that is being analysed.

Original: In this work, we propose a MIL-specific version of this approach.
Revised: In this work, one of our proposed methods is a MIL-specific version of this approach.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section we outline the requirements for MIL interpretability (Section 3.1). 
Revised: At the start of this section, we outline the requirements for MIL interpretability (Section 3.1). In

Original: In Section 3.2, we propose three methods that meet these requirements under the assumption that there are no interactions between instances (independent-instance methods).
Revised: Section 3.2, we propose three methods that meet these requirements under the assumption that there are no interactions between instances (independent-instance methods).

Original: In Section 3.3 we remove this assumption and propose our  model-agnosticpost-hoc interpretability method for MIL.
Revised: In Section 3.3 we remove this assumption and propose our local surrogate model-agnostic interpretability method for MIL.

Original: 3. MIL INTERPRETABILITY REQUIREMENTS
Revised: 3. MIL INTERPRETABILITY REQUIREMENTS

Original: A general MIL classification problem has C possible classes, with one class being negative and the rest being positive.
Revised: A general MIL classification problem has C possible classes, with one class being negative and the rest being positive.

Original: This is a generalisation of the SMIL assumption, which is a special case when
Revised: This is a generalisation of the SMIL assumption, which is a special case when

Original: C = 2 .
Revised: C = 2 .

Original: An interpretability method that can only provide the general importance of an instance (without associating it with a particular class) can only answer which questions.
Revised: An interpretability method that can only provide the general importance of an instance (without associating it with a particular class) can only answer which questions.

Original: In order to answer what questions, the method needs to state which classes each instance supports and refutes.
Revised: In order to answer what questions, the method needs to state which classes each instance supports and refutes.

Original: Formally, for a bag of instances X = { x 1 , . . .
Revised: Formally, for a bag of instances X = { x 1 , . . .

Original: , x k } and a bag classification function F , we want to assign a value to each instance that represents whether it supports or refutes a class c :
Revised: , x k } and a bag classification function F , we want to assign a value to each instance that represents whether it supports or refutes a class c :

Original: [Equation]
Revised: [Equation]

Original: where I is the interpretability function and ϕ i ∈ R is the interpretability value for instance i with respect to class c .
Revised: where I is the interpretability function and ϕ i ∈ R is the interpretability value for instance i with respect to class c .

Original: Here, we assume that ϕ i > 0 means instance i supports class c , ϕ i < 0 implies instance i refutes class c , and the greater | ϕ i | , the greater the importance of instance i .
Revised: Here, we assume that ϕ i > 0 means instance i supports class c , ϕ i < 0 implies instance i refutes class c , and the greater | ϕ i | , the greater the importance of instance i .

Original: For some existing methods, such as attention, ϕ i is the same for all classes, and ϕ i ≥ 0 in all cases, so these requirements are not met.
Revised: For some existing methods, such as attention, ϕ i is the same for all classes, and ϕ i ≥ 0 in all cases, so these requirements are not met.

Original: We later demonstrate these limitations in Section 5.
Revised: We later demonstrate these limitations in Section 5.

Original: In the next two sections we propose several model-agnostic methods that satisfy these requirements.
Revised: In the next two sections we propose several model-agnostic methods that satisfy these requirements.

Original: 3. D ETERMINING INSTANCE ATTRIBUTIONS
Revised: 3. D ETERMINING INSTANCE ATTRIBUTIONS

Original: In this section, we propose three model-agnostic methods for interpreting MIL models under the assumption that the instances are independent.
Revised: In this section, we propose three model-agnostic methods for interpreting MIL models under the assumption that the instances are independent.

Original: This means we can observe the effects of each instance in isolation without worrying about interactions between the instances.
Revised: This means we can observe the effects of each instance in isolation without worrying about interactions between the instances.

Original: We exploit a property of MIL models: the ability to deal with different sized bags.
Revised: We exploit a property of MIL models: the ability to deal with different sized bags.

Original: As MIL models are able to process bags of different sizes, it is possible to remove instances from the bags and observe any changes in prediction, allowing us to understand what instances are responsible for the model’s prediction.
Revised: As MIL models are able to process bags of different sizes, it is possible to remove instances from the bags and observe any changes in prediction, allowing us to understand what instances are responsible for the model’s prediction.

Original: Below, we propose three methods that use this property to interpret a model’s decision making.
Revised: Below, we propose three methods that use this property to interpret a model’s decision making.

Original: Singleinstance Given a bag of instances X = { x 1 , . . .
Revised: Single Given a bag of instances X = { x 1 , . . .

Original: , x k } and a bag classification function F , we can take each instance in turn and form a single instance bag: X i = {
Revised: , x k } and a bag classification function F , we can take each instance in turn and form a single instance bag: X i = {

Original: x i } for i ∈ { 1 , . . .
Revised: x i } for i ∈ { 1 , . . .

Original: , k } .
Revised: , k } .

Original: For a particular class c , we can then observe the model’s prediction on each single instance bag ϕ i
Revised: We then observe the model’s prediction on each single instance bag ϕ i

Original: = F c ( X i ) , where F c is the output of F for class c .
Revised: = F c ( X i ) , where F c is the output of F for class c (i.e., the c th entry in the output vector of F ( X i ) ).

Original: A large value for ϕ i suggests instance x i supports c , and a value close to zero suggest it has no effect with respect to class c .
Revised: A large value for ϕ i suggests instance x i supports c , and a value close to zero suggest it has no effect with respect to class c .

Original: If we repeat this over all instances and all classes, we can build a picture of the classes that each instance supports, allowing us to answer what questions.
Revised: If we repeat this over all instances and all classes, we can build a picture of the classes that each instance supports, allowing us to answer what questions.

Original: This gives the same outcome as the inherent interpretability performed by mi-Net (Wang et al., 2018), but here it is a model-agnostic rather than a model-specific method.However, this method cannot refute classes (i.e., ϕ i ≥ 0 in all cases).
Revised: However, this method cannot refute classes (i.e., ϕ i ≥ 0 in all cases). It should be noted that this method gives the same outputs as the inherent interpretability of mi-Net (Wang et al., 2018), but here it is a model-agnostic rather than a model-specific method.

Original: Leave one out A natural counterpart to the single instance method is the leave one out method, where each instance is removed from the complete bag in turn, i.e., we form bags X i =
Revised: One Removed A natural counterpart to the Single method is the One Removed method, where each instance is removed from the complete bag in turn, i.e., we form bags X i =

Original: X \{ x i } .
Revised: X \ { x i } .

Original: For a particular class c , we can then observe the change in the model’s prediction caused by removing x
Revised: For a particular class c , we can then observe the change in the model’s prediction caused by removing x

Original: i from the bag: ϕ i =
Revised: i from the bag: ϕ i =

Original: F c ( X ) − F c ( X i ) .
Revised: F c ( X ) − F c ( X i ) .

Original: If the prediction decreases, x i supports c , and if it increases, x i refutes c , i.e., this method is able to both support and refute different classes.
Revised: If the prediction decreases, x i supports c , and if it increases, x i refutes c , i.e., this method is able to both support and refute different classes.

Original: However, if there are other instances in the bag that support or refute class c , we may not observe a change in prediction when x i is removed, even if x i is a key instance.
Revised: However, if there are other instances in the bag that support or refute class c , we may not observe a change in prediction when x i is removed, even if x i is a key instance.

Original: Combined In order to access the benefits of both the single instance and leave one out methods, we can combine their outputs.
Revised: Combined In order to access the benefits of both the Single and One Removed methods, we can combine their outputs.

Original: A simple approach is to take the mean, i.e., ϕ i = 12 [ F c ( { x i } )
Revised: A simple approach is to take the mean, i.e., ϕ i = 12 [ F c ( { x i } )

Original: + F c ( X ) − F c ( X \ { x i } )] .
Revised: + F c ( X ) − F c ( X \{ x i } )] .

Original: This method can identify the important instances revealed by the single instance method, and also refute outcomes as revealed by the leave one out method.
Revised: This method can identify the important instances revealed by the Single method, and also refute outcomes as revealed by the One Removed method.

Original: With these three methods, it is assumed that there are no interactions between the instances.
Revised: With these three methods, it is assumed that there are no interactions between the instances.

Original: This is not true for all datasets, therefore, in the next section, we remove this assumption and propose a further method that is able to deal with the interactions between instances.
Revised: This is not true for all datasets, therefore, in the next section, we remove this assumption and propose a further method that is able to deal with the interactions between instances.

Original: 3. D EALING WITH INSTANCE INTERACTIONS
Revised: 3. D EALING WITH INSTANCE INTERACTIONS

Original: In order to calculate instance attributions whilst considering interactions between instances, we have to consider the effect of each instance within the context of the bag.
Revised: In order to calculate instance attributions whilst accounting for interactions between instances, we have to consider the effect of each instance within the context of the bag.

Original: With instance interactions, the co-occurrence of two (or more) instances changes the bag label from what it would be if the two instances were observed independently.
Revised: With instance interactions, the co-occurrence of two (or more) instances changes the bag label from what it would be if the two instances were observed independently.

Original: The instances have different meanings depending on the context of the bag, i.e., on their own they mean something different to what they mean when observed together.
Revised: The instances have different meanings depending on the context of the bag, i.e., on their own they mean something different to what they mean when observed together.

Original: One way to uncover these instance interactions is to perturb the original input to the model and observe the outcome.
Revised: One way to uncover these instance interactions is to perturb the original input to the model and observe the outcome.

Original: In the case of MIL, these perturbations can take the form of removing instances from the original bag.
Revised: In the case of MIL, these perturbations can take the form of removing instances from the original bag.

Original: By sampling coalitions of instances, and fitting a weighted linear regression model against the coalitions and their respective model predictions, it is possible to construct a surrogate locally faithful interpretable model that accounts for the instance interactions in the original bag.
Revised: By sampling coalitions of instances, and fitting a weighted linear regression model against the coalitions and their respective model predictions, it is possible to construct a surrogate locally faithful interpretable model that accounts for the instance interactions in the original bag.

Original: A coalition is a binary vector z ∈ { 0 , 1 } k that represents a subset
Revised: A coalition is a binary vector z ∈ { 0 , 1 } k that represents a subset

Original: S =
Revised: S =

Original: { x i |
Revised: { x i |

Original: z i = 1 } , so the number of ones in the coalition | z | is equal to the length of the S .
Revised: z i = 1 } , so the number of ones in the coalition | z | is equal to the length of the S .

Original: The surrogate model g c takes the form
Revised: The surrogate model g c takes the form

Original: [Equation]
Revised: [Equation]

Original: where each coefficient ϕ i ∈ { ϕ 1 , . .
Revised: where each coefficient ϕ i ∈ { ϕ 1 , . .

Original: . , ϕ k } is the importance attribution for each instance x i ∈ X with respect to class c .
Revised: . , ϕ k } is the importance attribution for each instance x i ∈ X with respect to class c .

Original: Given a collection of n coalitions Z , minimising the loss function
Revised: Given a collection of n coalitions Z , minimising the loss function

Original: [Equation]
Revised: [Equation]

Original: means g c is a locally faithful approximation of the original model F for class c .
Revised: means g c is a locally faithful approximation of the original model F for class c .

Original: The loss function is weighted by a kernel π , which determines how important it is for g c to be faithful for each individual coalition.
Revised: The loss function is weighted by a kernel π , which determines how important it is for g c to be faithful for each individual coalition.

Original: Here, g c only approximates F for class c , i.e., to produce interpretations for a bag with respect to all classes, a surrogate model needs to be fit for each class.
Revised: Here, g c only approximates F for class c , i.e., to produce interpretations for a bag with respect to all classes, a surrogate model needs to be fit for each class.

Original: This means the complexity of generating a complete set of interpretations for a bag is O

Original: ( Cnk 2 ) . Similar approaches have been applied in single instance supervised learning in methods such as LIME and KernelSHAP.
Revised: Similar approaches have been applied in single instance supervised learning in methods such as LIME and KernelSHAP.

Original: Both use different choices for π : LIME employs l 2 or cosine distance, and KernelSHAP uses a weighting scheme that approximates Shapley values (Shapley, 1953).
Revised: Both use different choices for π : LIME employs l 2 or cosine distance, and KernelSHAP uses a weighting scheme that approximates Shapley values (Shapley, 1953).

Original: For single instance supervised learning models, when perturbing the inputs, it is not possible to simply ‘remove’ a feature from an input as the models expect fixed-size inputs — either a new model has to be re-trained without that particular feature, or appropriate sampling from other data has to be undertaken, which can lead to unrealistic synthetic data.
Revised: For single instance supervised learning models, when perturbing the inputs, it is not possible to simply ‘remove’ a feature from an input as the models expect fixed-size inputs — either a new model has to be re-trained without that particular feature, or appropriate sampling from other data has to be undertaken, which can lead to unrealistic synthetic data.

Original: In MIL, as models are able to deal with different size bags, instances can be removed from the bags without the need for re-training or sampling from other data, meaning these issues do not occur in our setting.
Revised: In MIL, as models are able to deal with different size bags, instances can be removed from the bags without the need for re-training or sampling from other data, meaning these issues do not occur in our setting.

Original: While it is possible utilise the weight kernels from LIME and KernelSHAP for MIL, we identify a significant drawback with both methods.
Revised: While it is possible utilise the weight kernels from LIME and KernelSHAP for MIL, we identify a significant drawback with both methods.

Original: Their choice for π weights all coalitions of the same size equally, i.e., they do not consider the content of the coalitions, only their size.
Revised: Their choice for π weights all coalitions of the same size equally, i.e., they do not consider the content of the coalitions, only their size.

Original: Just because two coalitions are of the same size does not mean it is equally important that the surrogate model is faithful to both of them.
Revised: Just because two coalitions are of the same size does not mean it is equally important that the surrogate model is faithful to both of them.

Original: Furthermore, the sampling strategies of both approaches lead to very large ( | z | close to 1) or very small coalitions ( | z | close to 0).
Revised: Furthermore, the sampling strategies of both approaches lead to very large ( | z | close to 1) or very small coalitions ( | z | close to 0).

Original: Unless the number of samples n is very large, samples of average size ( | z | close to 0.5) will not be chosen.
Revised: Unless the number of samples n is very large, samples of average size ( | z | close to 0.5) will not be chosen.

Original: As we see in Section 4.4, this approach to sampling is appropriate for some datasets, but not for others.
Revised: As we see in Section 4.4, this approach to sampling is appropriate for some datasets, but not for others.

Original: We aim to overcome both of these drawbacks with our own MIL-specific choice of weight kernel and sampling approach.
Revised: We aim to overcome both of these drawbacks with our own MIL-specific choice of weight kernel and sampling approach.

Original: Below, we propose Multiple Instance Learning Local Interpretations (MILLI).
Revised: Below, we propose Multiple Instance Learning Local Interpretations (MILLI).

Original: At the core of our approach is a new method for weighting coalitions based on an initial ranking of instance importances r i ∈ { r 1 , . . .
Revised: At the core of our approach is a new method for weighting coalitions based on an initial ranking of instance importances r i ∈ { r 1 , . . .

Original: , r k } .
Revised: , r k } .

Original: For a coalition z and ranking of instance importances r , we define our weight kernel π M as:
Revised: For a coalition z and ranking of instance importances r , we define our weight kernel π M as:

Original: [Equation]
Revised: [Equation]

Original: where π R ( r i)
Revised: where π R ( r i

Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k ) e − λr i + 1 − α, if λ ≥ 0 , (1 − 2 α )(1 + r

Original: i − k k ) e | λ | ( r i − k )

Original: + α, otherwise, (4) and λ = (cid:26) β, if α < 0 .

Original: 5 , − β, otherwise.

Original: π R is a function that weights instances based on their order in the ranking.
Revised: π R is a function that weights instances based on their order in the ranking.

Original: Its two hyperparameters, α ∈ [0 , 1] and β ∈ [ −∞ , + ∞ ] define the shape of the function, and ultimately determine the kernel π M .
Revised: Its two hyperparameters, α ∈ [0 , 1] and β ∈ [ −∞ , + ∞ ] define the shape of the function, and ultimately determine the kernel π M .

Original: The value of α dictates whether the sampling should be biased towards instances that are highly ranked or not: α > 0 .
Revised: The value of α dictates whether the sampling should be biased towards instances that are highly ranked or not: α > 0 .

Original: 5 means π R is biased towards instances higher in ordering, and α < 0 .
Revised: 5 means π R is biased towards instances higher in ordering, and α < 0 .

Original: 5 means π R is biased towards values lower in the ordering.
Revised: 5 means π R is biased towards values lower in the ordering.

Original: The value of β is discussed below, as it becomes important when sampling coalitions.
Revised: The value of β is discussed below, as it becomes important when sampling coalitions.

Original: We provide an illustration of π R in Figure 1.
Revised: We provide an illustration of π R in Figure 1.

Original: In π M , the coalition z is weighted on its content rather than its length, i.e., the distance between a subset and a bag is no longer determined simply by the number of instances removed.
Revised: In π M , the coalition z is weighted on its content rather than its length, i.e., the distance between a subset and a bag is no longer determined simply by the number of instances removed.

Original: This is beneficial, as removing many non-discriminatory instances from the bag likely means the label of the bag remains the same, despite being different in size.
Revised: This is beneficial, as removing many non-discriminatory instances from the bag likely means the label of the bag remains the same, despite being different in size.

Original: Conversely, removing one discriminatory instance could change the bag label, even though the bag is only one instance smaller.
Revised: Conversely, removing one discriminatory instance could change the bag label, even though the bag is only one instance smaller.

Original: As well as using π R in the weight kernel, we also utilise it to sample coalitions.
Revised: As well as using π R in the weight kernel, we also utilise it to sample coalitions.

Original: We can consider the problem of sampling as a repeated coin toss, where P ( z i = 1) =
Revised: We can consider the problem of sampling as a repeated coin toss, where P ( z i = 1) =

Original: p i and P ( z i = 0)
Revised: p i and P ( z i = 0)

Original: = 1 − p i .
Revised: = 1 − p i .

Original: In fair random sampling,  p i = 0 .
Revised: In equal random sampling, every value p i ∈ { p 1 , . . .

Original: 5 for each instance, meaning E [ | z | ] = 0 .
Revised: , p k } is equal to 0.5, meaning E [ | z | ] = 0 .

Original: 5 k .
Revised: 5 k .

Original: However, it is possible to improve upon  random sampling by changing the value of pi for each instance in bag, i.e., we can change the likelihood of each instance being involved in a coalition.
Revised: However, it is possible to improve upon equal random sampling by changing the value of p for each instance in bag, i.e., we can change the likelihood of each instance being involved in a coalition.

Original: To sample more informative coalitions, we set p i =
Revised: To sample more informative coalitions, we set p i =

Original: [Equation] , meaning 
Revised: π R ( r i ) , meaning E [ | z | ]

Original: [Equation]
Revised: = (cid:82) k

Revised: π R dr :

Original: [Equation]
Revised: [Equation]

Original: If β < 0 , the sampling is biased towards smaller coalitions, and if β > 0 , the sampling is biased towards larger coalitions.
Revised: If β < 0 , the sampling is biased towards smaller coalitions, and if β > 0 , the sampling is biased towards larger coalitions.

Original: The maximum and minimum E [ | z | ] is controlled by α , and when α = 0 .
Revised: The maximum and minimum E [ | z | ] is controlled by α . When α = 0 .

Original: 5  , every valueof p i = 0 .
Revised: 5 or β = 0 , every value p i ∈ { p 1 , . .

Original: 5 ,  meaning we have fair random sampling as described above.
Revised: . , p k } is equal to 0.5, meaning we have equal random sampling as described above.

Original: We provide an illustration of how E [ | z | ] changes in Figure 2.
Revised: We provide an illustration of how E [ | z | ] changes in Figure 2.

Original: The final part of MILLI is how to determine the initial ranking of importances.
Revised: The final part of MILLI is how to determine the initial ranking of importances.

Original: For this, we can use the single instance method from Section 3.2, which produces values { ϕ 1 , . . .
Revised: For this, we can use the Single method from Section 3.2, which produces values { ϕ 1 , . . .

Original: , ϕ k } , and we convert this to an instance ranking: the new values { r 1 , . . .
Revised: , ϕ k } , and we convert this to an instance ranking: the new values { r 1 , . . .

Original: , r k } represent the position for instance i in { ϕ 1 , . . .
Revised: , r k } represent the position for instance i in { ϕ 1 , . . .

Original: , ϕ k } (e.g., the instance with the greatest value for ϕ has r = 0 ).
Revised: , ϕ k } (e.g., the instance with the greatest value for ϕ has r = 0 ).

Original: MILLI is more expensive to compute that the methods outlined in Section 3.2: O ( Cnk 2 ) compared with O ( Ck ) .
Revised: MILLI is more expensive to compute that the methods outlined in Section 3.2: O ( Cnk 2 ) compared with O ( Ck ) .

Original: However, when there are interactions between the instances, this extra complexity is required in order to build a better understanding of the classes that each instance supports and refutes, allowing more accurate answering of what questions.
Revised: However, when there are interactions between the instances, this extra complexity is required in order to build a better understanding of the classes that each instance supports and refutes, allowing more accurate answering of what questions.

Original: It is important to note that MILLI is indeed model agnostic — it only requires access to the bag classification function F , and makes no assumptions about the underlying MIL model.
Revised: It is important to note that MILLI is indeed model agnostic — it only requires access to the bag classification function F , and makes no assumptions about the underlying MIL model.

Original: As we demonstrate in the next section, this means we can apply it to any MIL model.
Revised: As we demonstrate in the next section, this means we can apply it to any MIL model.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We apply our model-agnostic methods to seven MIL datasets.
Revised: We apply our model-agnostic methods to seven MIL datasets.

Original: In this section, we detail the evaluation strategy (Section 4.1), the datasets (Section 4.2), models (Section 4.3), and results (Section 4.4).
Revised: In this section, we detail the evaluation strategy (Section 4.1), the datasets (Section 4.2), models (Section 4.3), and results (Section 4.4).

Original: For implementation details, see Appendix A.1.
Revised: For implementation details, see Appendix A.1.

Original: 4. E VALUATION STRATEGY
Revised: 4. E VALUATION STRATEGY

Original: For our evaluation, we do not assume thateach of the interpretability methods has a consistent output domain; we only assume a larger value implies larger support.
Revised: For our evaluation, we do not assume that the interpretability methods have consistent output domains; we only assume a larger value implies larger support.

Original: Therefore, the best approach for evaluating the interpretability methods is to use ranking metrics — rather than looking at the absolute values produced by the methods, we compare the relative orderings they produce.
Revised: Therefore, the best approach for evaluating the interpretability methods is to use ranking metrics — rather than looking at the absolute values produced by the methods, we compare the relative orderings they produce.

Original: As noted by Carbonneau et al.
Revised: As noted by Carbonneau et al.

Original: (2018), although a large number of benchmark MIL datasets exist, many do not have instance labels.
Revised: (2018), although a large number of benchmark MIL datasets exist, many do not have instance labels.

Original: Without instance labels, evaluating interpretability is challenging, as we do not have a ground truth instance ordering to compare to.
Revised: Without instance labels, evaluating interpretability is challenging, as we do not have a ground truth instance ordering to compare to.

Original: We identify two appropriate evaluation metrics: one that requires instance labels, and one that does not. For datasets with instance labels, we propose the use of normalised discounted cumulative gain at n (NDCG@n), as it prioritises instances higher in the produced ranking by using a logarithmic weighting, so methods are heavily penalised if they mistake refuting instances forsupporting instances. For datasets without instance labels, we propose the use of area under the perturbation curve compared with random orderings (AOPC-R; Samek et al.
Revised: We identify two appropriate evaluation metrics: for datasets with instance labels, we propose the use of normalised discounted cumulative gain at n (NDCG@n), and for datasets without instance labels, we propose the use of area under the perturbation curve compared with random orderings (AOPC-R; Samek et al.

Original: (2016)), which measures how much the model prediction changes when successively removing instances according to the importance ordering. While AOPC-R does not require instance labels, it is very expensive to compute, therefore we chose NDCG@n as our primary metric.
Revised: While AOPCR does not require instance labels, it is very expensive to compute. A significant difference between the two metrics is the way they weight the importance ordering: NDCG@n prioritises performance at the start of the ordering, whereas AOPC-R equally prioritises performance across the entire ordering.

Original: We are the first to propose both of these metrics for use in evaluating MIL interpretability, and further details discussing the advantages of both metricsare given in Appendix A.3.
Revised: We are the first to propose both of these metrics for use in evaluating MIL interpretability, and further discuss the advantages of both metrics in Appendix A.3.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: Below we detail the three main datasets on which we can evaluate our interpretability methods.
Revised: Below we detail the three main datasets on which we can evaluate our interpretability methods.

Original: These datasets were selected as they have instance labels, so we can evaluate them using both
Revised: These datasets were selected as they have instance labels, so we can evaluate them using both

Original: NDCG@n and AOPC-R.
Revised: NDCG@n and AOPC-R.

Original: However, we provide further results on the classical MIL datasets Musk,
Revised: However, we provide further results on the classical MIL datasets Musk,

Original: Tiger, Elephant and Fox in Appendix A.

Revised: Tiger, Elephant and Fox in Appendix A.

Original: (Dietterich et al., 1997; Andrews et al., 2002).
Revised: (Dietterich et al., 1997; Andrews et al., 2002).

Original: SIVAL The Spatially Independent, Variable Area, and Lighting (SIVAL; Rahmani etal.
Revised: SIVAL The Spatially Independent, Variable Area, and Lighting (SIVAL; Rahmani et

Original:  dataset consists of 25 classes of complex objects photographed in different environments, where each class contains 60 images.
Revised: [Table] dataset consists of 25 classes of complex objects photographed in different environments, where each class contains 60 images.

Original: Each image has been segmented into approximately 30 segments, and each segment is represented by a 30-dimensional feature vector that encodes information such as the segment’s colour and texture.
Revised: Each image has been segmented into approximately 30 segments, and each segment is represented by a 30-dimensional feature vector that encodes information such as the segment’s colour and texture.

Original: The segments are labelled as containing the object or containing background.
Revised: The segments are labelled as containing the object or containing background.

Original: We selected 12 of the 25 classes to be positive classes, and randomly sampled from the other 13 classes to form the negative class.
Revised: We selected 12 of the 25 classes to be positive classes, and randomly sampled from the other 13 classes to form the negative class.

Original: For additional details see Appendix A.6.
Revised: For additional details see Appendix A.6.

Original: Four-class MNIST-Bags The SIVAL dataset has no co-occurrence of instances from different classes, i.e., each bag only contains one class of positive instances.
Revised: Four-class MNIST-Bags The SIVAL dataset has no co-occurrence of instances from different classes, i.e., each bag only contains one class of positive instances.

Original: Therefore,from an interpretability perspective, the which and what questions are the same — if we can identify the key instances, then by the nature of the dataset we know which class they support.
Revised: Therefore, the which and what questions are the same.

Original: In order to explore what happens when there are different classes of positive instances in the same bag, we propose an extension of the MNIST-Bags dataset introduced by Ilse et al.
Revised: In order to explore what happens when there are different classes of positive instances in the same bag, we propose an extension of the MNIST-Bags dataset introduced by Ilse et al.

Original: Our extension, four-class MNIST-Bags (4-MNIST-Bags) is setup as follows: class 1 if 8 in bag, class 2 if 9 in bag, class 3 in 8 and 9 in bag, and class 0 otherwise 1 .
Revised: Our extension, four-class MNIST-Bags (4-MNIST-Bags) is setup as follows: class 1 if 8 in bag, class 2 if 9 in bag, class 3 in 8 and 9 in bag, and class 0 otherwise 1 .

Original: Note that a bag can only have one label.

Original: In this dataset, answering what questions goes beyond answering which questions, e.g., the existence of an 8 supports classes one and three, but refutes classes zero and two.
Revised: In this dataset, answering what questions goes beyond answering which questions, e.g., the existence of an 8 supports classes one and three, but refutes classes zero and two.

Original: For further details see Appendix A
Revised: For further details see Appendix A.7.

Original: Colon Cancer To test the applicability of the interpretability methods on larger bag sizes, we apply it to colorectal cancer tissue classification.
Revised: Colon Cancer To test the applicability of the interpretability methods on larger bag sizes, we apply it to colorectal cancer tissue classification.

Original: The ColoRectal Cancer (CRC) dataset  is a collection of microscopy imagesannotated with four classes of nuclei: epithelial, inflammatory, fibroblast, and miscellaneous (Sirinukunwattana et al., 2016).
Revised: The ColoRectal Cancer (CRC) dataset (Sirinukunwattana et al., 2016) is a collection of microscopy images with annotated nuclei.

Original: We follow the same setup as Ilse et al.
Revised: We follow the same setup as Ilse et al.

Original: (2018), in which a bag is positive if it contains one or more nuclei from the epithelial class.
Revised: (2018), in which a bag is positive if it contains one or more nuclei from the epithelial class.

Original: This means the problem conforms to the SMIL assumption, and there are no interactions between instances.
Revised: This means the problem conforms to the SMIL assumption, and there are no interactions between instances.

Original: Each microscopy image is 500 x 500 pixels, and was split into 27 x 27 pixel patches to give a maximum of 324 patches per slide (patches were discarded if they contained mostly slide background).
Revised: Each microscopy image is 500 x 500 pixels, and was split into 27 x 27 pixel patches to give a maximum of 324 patches per slide (patches were discarded if they contained mostly slide background).

Original: Not all of the instances are labelled, so we tailor our assessment  to only consider labelled instances, demonstrating how these techniques can still be evaluated without exhaustive instance labelling.
Revised: Not all of the instances are labelled, so we tailor our assessment using NDCG@n to only consider labelled instances.

Original: For additional details see Appendix A.8.
Revised: For additional details see Appendix A.8.

Original: 4. M ODELS AND M ETHODS
Revised: 4. M ODELS AND M ETHODS

Original: To highlight the model-agnostic abilities of the proposed methods, for each dataset, we trained four different types of multi-class MIL model: an embedding-based multiple instance neural network (MI-Net; Wang et al.
Revised: To highlight the model-agnostic abilities of the proposed methods, for each dataset, we trained four different types of multi-class MIL model: an embedding-based multiple instance neural network (MI-Net; Wang et al.

Original: (2018)), an instance-based multiple instance network (mi-Net; Wang et al.
Revised: (2018)), an instance-based multiple instance network (mi-Net; Wang et al.

Original: (2018)), an attention-based model (MI-Attn; Ilse et al.
Revised: (2018)), an attention-based model (MI-Attn; Ilse et al.

Original: (2018)), and a graph neural network model (MI-GNN; Tu et al.
Revised: (2018)), and a graph neural network model (MI-GNN; Tu et al.

Original: Aside from MI-Net, each of these models provide their own inherent interpretability method that we can compare our methods against.
Revised: Aside from MI-Net, each of these models provide their own inherent interpretability method that we can compare our methods against.

Original: Additional details on the models are given in Appendix A.2.
Revised: Additional details on the models are given in Appendix A.2.

Original: As well as evaluating our instance-independent methods and MILLI, we also compare against LIME and SHAP using both random and guided sampling (choosing coalitions that maximise the weight kernel).
Revised: As well as evaluating our independent-instance methods and MILLI, we also compare against LIME and SHAP using both random and guided sampling (choosing coalitions that maximise the weight kernel).

Original: This means that in total we are evaluating nine different interpretability methods: inherent interpretability, the three independent-instance methods (Section 3.2), two LIME methods, two SHAP methods, and MILLI (Section 3.3).
Revised: This means that in total we are evaluating nine different interpretability methods: inherent interpretability, the three independent-instance methods (Section 3.2), two LIME methods, two SHAP methods, and MILLI (Section 3.3).

Original: Note that, aside from the inherent interpretability methods, these methods are either novel (independent-instance methods and
Revised: Note that, aside from the inherent interpretability methods, these methods are either novel (independent-instance methods and

Original: MILLI) or are applied to MIL for the time in this study (LIME and SHAP).
Revised: MILLI) or are applied to MIL for the time in this study (LIME and SHAP).

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: For each dataset, we measured the performance of each interpretability method on each of the four
Revised: For each dataset, we measured the performance of each interpretability method on each of the four

Original: MIL models.
Revised: MIL models.

Original: For the SIVAL dataset we measured the performance on only the negative class and the bag’s true class, but for all the other datasets we evaluated the interpretability over every class.
Revised: For the SIVAL dataset we measured the performance on only the negative class and the bag’s true class, but for all the other datasets we evaluated the interpretability over every class.

Original: This distinction was made as we know that each SIVAL bag can only contain instances from one positive class, therefore it is not necessary to evaluate over all possible classes.
Revised: This distinction was made as we know that each SIVAL bag can only contain instances from one positive class, therefore it is not necessary to evaluate over all possible classes.

Original: We present the interpretability results run against the test set for the SIVAL, 4-MNIST-Bags, and CRC datasets in
Revised: We present the interpretability results run against the test set for the SIVAL, 4-MNIST-Bags, and CRC datasets in

Original: Tables 1, 2 and 3 respectively.
Revised: Tables 1, 2 and 3 respectively.

Original: The results are averaged over ten repeat trainings of each model, and the standard error of the mean is given in each case. We also provide the test accuracy of each of the underlying MIL models.
Revised: The results are averaged over ten repeat trainings of each model, and we also give the test accuracy of each of the underlying MIL models.

Original: The results of the same experiments using the AOPC-R metric are given in

Original: Appendix A.4, and we discuss our choice of hyperparameters in Appendix A.5.[Table]
Revised: We discuss our choice of hyperparameters in Appendix A.5.

Original: [Table] [Table] [Table] By analysing the NDCG@n interpretability results across these three datasets, we find that MILLI performs best with an average of 0.85, followed by the GuidedSHAP and Combined methods (both with an average of 0.81).
Revised: By analysing the NDCG@n interpretability results across these three datasets, we find that MILLI performs best with an average of 0.85, followed by the GuidedSHAP and Combined methods (both with an average of 0.81).

Revised: For the average AOPC-R results (including the results on the classical MIL datasets, see Appendix A.4), we find that Combined, GuidedSHAP, RandomLIME, and

Revised: MILLI are the best performing methods, however the overall the difference in performance between the methods is much less than what we observe for the NDCG@n metric.

Revised: For the 4-MNIST-Bags dataset, the difference in performance of MILLI on NDCG@n vs AOPC-R is due to the difference in weighting between the metrics: MILLI achieves a better ordering for the most important instances (outperforms other methods on NDCG@n), but gives the same ordering as other methods for less important instances (equal performance on AOPC-R).

Original: In the majority of cases, all of our proposed model-agnostic methods outperform the inherent interpretability methods, and are relatively consistent in performance across all models.
Revised: In the majority of cases, all of our proposed model-agnostic methods outperform the inherent interpretability methods, and are relatively consistent in performance across all models.

Original: For the SIVAL dataset, the independent-instance methods perform well, which is expected as the instances are independent.
Revised: For the SIVAL dataset, the independent-instance methods perform well, which is expected as the instances are independent.

Original: However, on the 4-MNIST-Bags dataset, MILLI excels as it samples informative coalitions that capture the instance interactions.
Revised: However, on the 4-MNIST-Bags [Table] Combined 0.828 / [Table]

Revised: RandomSHAP 0.801 / 0.284 0.807 / 0.300 0.766 / 0.322 0.784 / 0.250 0.789 / 0.

Revised: GuidedSHAP 0.826 / 0.291 0.819 / 0.290 0.790 / 0.313 0.765 / 0.235 0.800 / 0.

Revised: RandomLIME 0.809 / 0.295 0.815 / 0.310 0.776 / 0.335 0.793 / 0.258 0.

Revised: [Table]

Revised: GuidedLIME 0.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: MILLI 0.823 / 0.283 0.827 / 0.307 0.794 / 0.307 0.790 / 0.239 0.808 / 0.

Revised: [Table]

Revised: [Table]

Revised: dataset, MILLI excels as it samples informative coalitions that capture the instance interactions.

Original: On the CRC dataset, methods that are able to isolate individual instances, (i.e., the Single, Combined, and GuidedSHAP methods) perform well due to instance independence in this dataset.
Revised: On the CRC dataset [Table] methods that are able to isolate individual instances, (i.e., the Single, Combined, and GuidedSHAP methods) perform well due to instance independence in this dataset.

Original: Furthermore, if we consider the witness rate (WR; the proportion of key instances in each bag; Carbonneau et al.
Revised: Furthermore, if we consider the witness rate (WR; the proportion of key instances in each bag; Carbonneau et al.

Original: (2018)) of the datasets
Revised: (2018)) of the datasets

Original: , we find that the CRC dataset has a higher WR (27.47%) than SIVAL (15.28%) and 4-MNIST-Bags (8.04%).
Revised: , we find that the CRC dataset has a higher WR (27.47%) than SIVAL (15.28%) and 4-MNIST-Bags (8.04%).

Original: This means, with larger coalitions, it becomes more difficult to isolate the contributions of individual instances, which is why the One Removed and Random sampling methods struggle.
Revised: This means, with larger coalitions, it becomes more difficult to isolate the contributions of individual instances, which is why the One Removed and Random sampling methods struggle.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: To demonstrate how MILLI captures instance interactions, and to show the limitations of existing methods that cannot condition their output for a particular class, we compare the MILLI interpretations with the attention interpretations on the 4-MNIST-Bags dataset (Figure 3).
Revised: To demonstrate how MILLI captures instance interactions, and to show the limitations of existing methods that cannot condition their output for a particular class, we compare the MILLI interpretations with the attention interpretations on the 4-MNIST-Bags dataset (Figure 3).

Original: As this bag contains an 8 and a 9 , the correct label for it is class three.
Revised: As this bag contains an 8 and a 9 , the correct label for it is class three.

Original: Both the MILLI and attention methods have identified the 8 and 9 instances as key instances, i.e., they have both answered the which question.
Revised: Both the MILLI and attention methods have identified the 8 and 9 instances as key instances, i.e., they have both answered the which question.

Original: However, only MILLI correctly identifies that the 8 refutes class two and that the 9 refutes class one, answering the what question, something that the attention values do not do.
Revised: However, only MILLI correctly identifies that the 8 refutes class two and that the 9 refutes class one, answering the what question, something that the attention values do not do.

Original: We provide further examples
Revised: We provide further examples, including interpretability outputs for the SIVAL and CRC datasets, in Appendix A.10.

Original: LIME, SHAP, and MILLI all require the number of sampled coalitions to be selected.
Revised: LIME, SHAP, and MILLI all require the number of sampled coalitions to be selected.

Original: A greater number of coalitions means there is more data to fit the surrogate model on, therefore it is more likely to faithful to the underlying model.
Revised: A greater number of coalitions means there is more data to fit the surrogate model on, therefore it is more likely to faithful to the underlying model.

Original: However, the more samples that are taken, the more expensive the computation.
Revised: However, the more samples that are taken, the more expensive the computation.

Original: As shown in Figure 4, MILLI is the most consistent in terms of performance and efficiency.
Revised: As shown in Figure 4, MILLI is the most consistent in terms of performance and efficiency.

Original: For all methods, it is a case of diminishing returns, where additional samples only lead to a small increase in performance.
Revised: For all methods, it is a case of diminishing returns, where additional samples only lead to a small increase in performance.

Original: The different sampling approaches used in this work have distinct advantages.
Revised: The different sampling approaches used in this work have distinct advantages.

Original: When there are interactions between instances, RandomSHAP and RandomLIME are better than GuidedSHAP and
Revised: When there are interactions between instances, RandomSHAP and RandomLIME are better than GuidedSHAP and

Original: GuidedLIME as they form larger coalitions that are more likely to capture the instance interactions.
Revised: GuidedLIME as they form larger coalitions that are more likely to capture the instance interactions.

Original: However, for the CRC dataset, where there are a large number of independent instances and a high WR, RandomSHAP, RandomLIME and GuidedLIME struggle as they cannot form small coalitions.
Revised: However, for the CRC dataset, where there are a large number of independent instances and a high WR, RandomSHAP, RandomLIME and GuidedLIME struggle as they cannot form small coalitions.

Original: MILLI performs relatively well across all datasets as the size of its sampled coalitions can be adapted depending on the dataset, demonstrating its generalisability.
Revised: MILLI performs relatively well across all datasets as the size of its sampled coalitions can be adapted depending on the dataset, demonstrating its generalisability.

Original: However, it is still outperformed by GuidedSHAP on the CRC dataset.
Revised: However, it is still outperformed by GuidedSHAP on the CRC dataset.

Original: One possible explanation for this is that MILLI is limited to sampling only smaller or larger coalitions, whereas GuidedSHAP samples both large and small coalitions.
Revised: One possible explanation for this is that MILLI is limited to sampling only smaller or larger coalitions, whereas GuidedSHAP samples both large and small coalitions.

Original: One approach for improving MILLLI would be to incorporate paired sampling (Covert & Lee, 2020),
Revised: One approach for improving MILLLI would be to incorporate paired sampling (Covert & Lee, 2020),

Original: where for every sampled coalition z
Revised: where for every sampled coalition z

Original: i , we also sample its complement coalition 1 − z i .
Revised: i , we also sample its complement coalition 1 − z i .

Original: Following the advice of Carbonneau et al.
Revised: Following the advice of Carbonneau et al.

Original: (2018), further MIL studies on more complex datasets, such as Pascal VOC (Everingham et al., 2010), could also be insightful for evaluating MIL interpretability methods.
Revised: (2018), further MIL studies on more complex datasets, such as Pascal VOC (Everingham et al., 2010), could also be insightful for evaluating MIL interpretability methods.

Original: It would also be beneficial to examine if these techniques are applicable to
Revised: It would also be beneficial to examine if these techniques are applicable to

Original: MIL domains beyond classification, e.g. MIL regression as in Wang et al.
Revised: MIL domains beyond classification, e.g. MIL regression as in Wang et al.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have discussed the process of model-agnostic interpretability for MIL.
Revised: In this work, we have discussed the process of model-agnostic interpretability for MIL.

Original: Along with defining the requirements for MIL interpretability, we have presented our own approaches and compared them to existing inherently interpretable MIL models.
Revised: Along with defining the requirements for MIL interpretability, we have presented our own approaches and compared them to existing inherently interpretable MIL models.

Original: By analysing the methods across several datasets, we have shown that independent-instance methods can be effective, but local surrogate methods are required when there are interactions between the instances.
Revised: By analysing the methods across several datasets, we have shown that independent-instance methods can be effective, but local surrogate methods are required when there are interactions between the instances.

Original: All of our proposed methods are more effective than existing inherently interpretable models, and are able to not only identify which are the key instances, but also say what classes they support and refute.
Revised: All of our proposed methods are more effective than existing inherently interpretable models, and are able to not only identify which are the key instances, but also say what classes they support and refute.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For our work, the main details for the methodology are detailed in Section 3.
Revised: For our work, the main details for the methodology are detailed in Section 3.

Original: In addition, we provide details that will aid reproducibility in the Appendix.
Revised: In addition, we provide details that will aid reproducibility in the Appendix.

Original: The dataset sources and a general overview of our implementation is given in Appendix A.1.
Revised: The dataset sources and a general overview of our implementation is given in Appendix A.1.

Original: Further information on the MIL models used in this work can be found in Appendix A.2, and specific details on their architectures and training can be found in Appendices A.6, A. and A.8 for the SIVAL, 4-MNIST-Bags, and CRC datasets respectively.
Revised: Further information on the MIL models used in this work can be found in Appendix A.2, and specific details on their architectures and training can be found in Appendices A.6, A. and A.8 for the SIVAL, 4-MNIST-Bags, and CRC datasets respectively.

Original: The complete codebase is included as supplementary material for this submission and will also be made publicly available on GitHub at the time of publication
Revised: The complete codebase is included as supplementary material for this submission and will also be made publicly available on GitHub at the time of publication

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: We used SciencePlots (Garrett, 2021) for formatting our Matplotlib figures
Revised: We used SciencePlots (Garrett, 2021) for formatting our Matplotlib figures

Original: A A PPENDIX A. I MPLEMENTATION DETAILS
Revised: A A PPENDIX A. I MPLEMENTATION DETAILS

Original: All code for this work was implemented in Python 3.8, using the PyTorch library for the machine learning functionality.
Revised: All code for this work was implemented in Python 3.8, using the PyTorch library for the machine learning functionality.

Original: Hyperparameter tuning was carried out using the Optuna libary.
Revised: Hyperparameter tuning was carried out using the Optuna libary.

Original: Some local experiments were carried out on a Dell XPS Windows laptop, utilising a GeForce GTXgraphics card with 4GB of VRAM.
Revised: Some local experiments were carried out on a Dell XPS Windows laptop, utilising a GeForce GTXgraphics card with 4GB of VRAM.

Original: GPU support for machine learning was enabled through CUDA v11.0.
Revised: GPU support for machine learning was enabled through CUDA v11.0.

Original: Other longer running experiments, such as hyperparameter tuning, were carried out on a remote GPU node utilising a Volta V100 Enterprise Compute GPU with 16GB of VRAM.
Revised: Other longer running experiments, such as hyperparameter tuning, were carried out on a remote GPU node utilising a Volta V100 Enterprise Compute GPU with 16GB of VRAM.

Original: The longest model to train was the GNN for the CRC dataset, which took just under half an hour.
Revised: The longest model to train was the GNN for the CRC dataset, which took just under half an hour.

Original: The longest running experiment was the hyperparameter analysis for the CRC dataset at just under 40 hours.
Revised: The longest running experiment was the hyperparameter analysis for the CRC dataset at just under 40 hours.

Original: This was due to high number of samples for the local surrogate methods and the fact that the results were average over 10 different models.
Revised: This was due to high number of samples for the local surrogate methods and the fact that the results were average over 10 different models.

Original: The randomisation of data splits was fixed using seeding; the seed values are provided in the code for each experiment.
Revised: The randomisation of data splits was fixed using seeding; the seed values are provided in the code for each experiment.

Original: We use the following sources for our data:
Revised: We use the following sources for our data:

Original: • The annotated SIVAL dataset was downloaded from the publicly accessible page http://pages.cs.wisc.edu/˜bsettles/data/ .
Revised: • The annotated SIVAL dataset was downloaded from the publicly accessible page: http://pages.cs.wisc.edu/ ˜ bsettles/data/ .

Original: • The MNIST dataset was access directly from the PyTorch Python library.
Revised: • The MNIST dataset was access directly from the PyTorch Python library: https://pytorch.org/vision/stable/datasets.html#mnist .

Original: • The CRC dataset was downloaded from the publicly accessible page https://warwick.ac.uk/fac/cross_fac/tia/data/crchistolabelednucleihe/ .
Revised: • The CRC dataset was downloaded from the publicly accessible page: https://warwick.ac.uk/fac/cross_fac/tia/data/crchistolabelednucleihe/ .

Original: The Musk dataset was downloaded from the publicly accesible page https://archive.ics.uci.edu/ml/datasets/Musk+%28Version+2% • The Tiger, Elephant and Fox datasets were downloaded from the publicly accesible page https://archive.ics.uci.edu/ml/datasets/Musk+%28Version+2%
Revised: The Musk dataset was downloaded from the publicly accessible page: https://archive.ics.uci.edu/ml/datasets/Musk+%28Version+2% • The Tiger, Elephant and Fox datasets were downloaded from the publicly accessible page: https://archive.ics.uci.edu/ml/datasets/Musk+%28Version+2%

Original: A. M ODELS
Revised: A. M ODELS

Original: In this work, we trained four different models for each dataset.
Revised: In this work, we trained four different models for each dataset.

Original: In this section, we go into more detail about each of the models.
Revised: In this section, we provide further details on each of the models.

Original: Each model was tuned independently for each dataset, so in the later sections we provide details of the specific architectures for each dataset.
Revised: Each model was tuned independently for each dataset, so in the later sections we provide details of the specific architectures for each dataset.

Original: We also tuned the learning rate, weight decay, and dropout for each of the models independently for every dataset.
Revised: We also tuned the learning rate, weight decay, and dropout for each of the models independently for every dataset.

Original: Again, these are given in the later sections for each specific dataset.
Revised: Again, these are given in the later sections for each specific dataset.

Original: MI-Net
Revised: MI-

Original:  Each instance is embedded to a fixed size, and then these embeddings are aggregated to give a single bag embedding.
Revised: Net Each instance is embedded to a fixed size, and then these embeddings are aggregated to give a single bag embedding.

Original: This aggregation can either take the mean (mil-mean) or max (milmax).
Revised: This aggregation can either take the mean (mil-mean) or max (milmax) of the instance embeddings.

Original: The bag embedding is then classified to give an overall bag prediction.
Revised: The bag embedding is then classified to give an overall bag prediction.

Original: For this model, we tuned the number and size of fully connected (FC) layers, as well as the choice of aggregation function.
Revised: For this model, we tuned the number and size of fully connected (FC) layers, as well as the choice of aggregation function.

Original: mi-Net
Revised: mi-

Original:  A prediction is made for each individual instance, and then these are aggregated to a single outcome for the bag as a whole.
Revised: Net A prediction is made for each individual instance, and then these are aggregated to a single outcome for the bag as a whole.

Original: Similar to the MI-Net model, we tuned the FC layers and the aggregation function.
Revised: Similar to the MI-Net model, we tuned the FC layers and the aggregation function.

Original: MI-Attn
Revised: MI

Original:  Each instance is embedded to a fixed size, and then the mil-attn block produces an attention value for each instance embedding.
Revised: Attn Each instance is embedded to a fixed size, and then the mil-attn block produces an attention value for each instance embedding.

Original: The instance embeddings are then aggregated to a single bag embedding by performing a weighted sum based on the attention values.
Revised: The instance embeddings are then aggregated to a single bag embedding by performing a weighted sum based on the attention values.

Original: The bag embedding is then classified to give an overall bag prediction.
Revised: The bag embedding is then classified to give an overall bag prediction.

Original: The mil-attn block is the same as per the MIL attention mechanism proposed by Ilse et al.
Revised: The mil-attn block is the same as per the MIL attention mechanism proposed by Ilse et al.

Original: (2018), i.e., using a single hidden layer.
Revised: (2018), i.e., using a single hidden layer.

Original: We tuned the the number and size of the FC layers, as well as the size of the hidden attention layer.
Revised: We tuned the the number and size of the FC layers, as well as the size of the hidden attention layer.

Original: MI-GNN The GNN model treats the bag as a fully connected graph and uses graph convolutions to propagate information between instances.
Revised: MI GNN The GNN model treats the bag as a fully connected graph and uses graph convolutions to propagate information between instances.

Original: Initially, the original instances are embedded to a fixed size.
Revised: Initially, the original instances are embedded to a fixed size.

Original: These instance embeddings are then passed onto a GNN embed block and a GNN cluster block, the output of which is used to reduce the graph representation down into a single embedding using differentiable pooling (gnn-pool).
Revised: These instance embeddings are then passed onto a GNN embed block and a GNN cluster block, the output of which is used to reduce the graph representation down into a single embedding using differentiable pooling (gnn-pool).

Original: The final bag representation is then classified to give an overall bag prediction.
Revised: The final bag representation is then classified to give an overall bag prediction.

Original: For more details see Tu et al.
Revised: For more details see Tu et al.

Original: We tuned the number and 
Revised: We tuned the number and size of the embedding, GNN, and classifier layers.

Original: A. E VALUATION METRICS
Revised: A. E VALUATION METRICS

Original: In this section, we provide further details on how we use normalised discounted cumulative gain at n (NDCG@n) to evaluate the interpretability methods for datasets with instance labels, and how we use area under the perturbation curve with random orderings (AOPC-R) to evaluate the interpretability methods for datasets without instance labels.
Revised: In this section, we provide further details on how we use normalised discounted cumulative gain at n (NDCG@n) to evaluate the interpretability methods for datasets with instance labels, and how we use area under the perturbation curve with random orderings (AOPC-R) to evaluate the interpretability methods for datasets without instance labels.

Original: NDCG@n To use NDCG@n, we first need a ground truth ordering to compare to.
Revised: NDCG@n To use NDCG@n, we first need a ground truth ordering to compare to.

Original: For a particular class, we can place each instance into one of three groups based on their ground truth instance labels: supporting instances, neutral instances, and refuting instances.
Revised: For a particular class, we can place each instance into one of three groups based on their ground truth instance labels: supporting instances, neutral instances, and refuting instances.

Original: The ideal instance ordering for that class would then have all of the supporting instances at the beginning, followed by all the neutral instances, and then all of the refuting instances at the end.
Revised: The ideal instance ordering for that class would then have all of the supporting instances at the beginning, followed by all the neutral instances, and then all of the refuting instances at the end.

Original: Then, given interpretability outputs { ϕ 1 , . . .
Revised: Then, given interpretability outputs { ϕ 1 , . . .

Original: , ϕ k } for a particular class, we rank the outputs from highest to lowest, i.e., in order of how much they support that class.
Revised: , ϕ k } for a particular class, we rank the outputs from highest to lowest, i.e., in order of how much they support that class.

Original: This importance ordering is then compared to the ground truth ordering using the follow metric:
Revised: This importance ordering is then compared to the ground truth ordering using the follow metric:

Original: [Equation]
Revised: [Equation]

Original: where IDCG = n (cid:88) i =  log 2 ( i + 1) .
Revised: where IDCG = n (cid:88) i =  log 2 ( i + 1) .

Original: IDCG is the ideal discounted cumulative gain that normalises the scores across different values of n .
Revised: IDCG is the ideal discounted cumulative gain that normalises the scores across different values of n .

Original: The relevance function rel ( i ) is as follows:
Revised: The relevance function rel ( i ) is as follows:

Original: [Equation]
Revised: [Equation]

Original:   if the i th instance in the ranking supports the class, − 1 if the i th instance in the ranking refutes the class, otherwise.
Revised:   if the i th instance in the ranking supports the class, − 1 if the i th instance in the ranking refutes the class, otherwise.

Original: AOPC-R
Revised: AOPC-R

Original: Although originally designed for single instance supervised learning, here we adapt
Revised: Although originally designed for single instance supervised learning, here we adapt

Original: AOPC-R for MIL.
Revised: AOPC-R for MIL.

Original: Given an importance ordering, AOPC successively removes the most relevant instances (i.e., those at the start of the given importance ordering), and measures the change in prediction.
Revised: Given an importance ordering, AOPC successively removes the most relevant instances (i.e., those at the start of the given importance ordering), and measures the change in prediction.

Original: A better ordering will show a more rapid decrease in prediction, as instances that are more supportive will be removed first.
Revised: A better ordering will show a more rapid decrease in prediction, as instances that are more supportive will be removed first.

Original: This rate of decrease in prediction with respect to class c for a classifier F and bag X = { x 1 , . . .
Revised: This rate of decrease in prediction with respect to class c for a classifier F and bag X = { x 1 , . . .

Original: , x k } is measured as follows: given the ordered bag
Revised: , x k } is measured as follows: given the ordered bag

Original: O X
Revised: O X

Original: = { o 1 , . . .
Revised: = { o 1 , . . .

Original: o k } (which is just X ordered by instance importance, with the most important instances at the start),
Revised: o k } (which is just X ordered by instance importance, with the most important instances at the start),

Original: [Equation]
Revised: [Equation]

Original: where X
Revised: where X

Original: [Equation]
Revised: [Equation]

Original: MoRF =
Revised: MoRF =

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: and X
Revised: and X

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: To normalise the results, one approach is to measure the average difference in AOPC for the given ordering to the AOPC of several random orderings:
Revised: To normalise the results, one approach is to measure the average difference in AOPC for the given ordering to the AOPC of several random orderings:

Original: [Equation]
Revised: [Equation]

Original: where r is the number of random orderings, and O R i is the i th random ordering of X .
Revised: where r is the number of random orderings, and O R i is the i th random ordering of X .

Original: The repeated calls to F c make AOPC-R very expensive to compute.
Revised: The repeated calls to F c make AOPC-R very expensive to compute.

Original: This can be reduced by examining the p first perturbations rather than all k − 1 possible perturbations, but that was not something we investigated in this work (we kept p = k and r = 10 ).
Revised: This can be reduced by examining the p first perturbations rather than all k − 1 possible perturbations, but that was not something we investigated in this work (we kept p = k and r = 10 ).

Original: Furthermore, due to the use of random orderings, this evaluation metric is inherently stochastic, meaning not only do we have variance in the orderings produced in the methods (e.g., from random sampling), we also have variance due to measurement.
Revised: Furthermore, due to the use of random orderings, this evaluation metric is inherently stochastic, meaning not only do we have variance in the orderings produced in the methods (e.g., from random sampling), we also have variance due to measurement.

Original: NDCG@n does not have either of the issues (i.e., its cheap to compute and deterministic), however it requires (at least some) instance labels.
Revised: NDCG@n does not have either of the issues (i.e., its cheap to compute and deterministic), however it requires (at least some) instance labels.

Original: A. A DDITIONAL RESULTS
Revised: A. A DDITIONAL RESULTS

Original: In this section

Original: [Table]

Original: we detail our additional results.

Original: First, we give the AOPC-R results for the SIVAL and 4-MNIST-Bags datasets (AOPC-R vs NDCG@n).

Original: Second, we give results on the Musk, Tiger, Elephant and Fox MIL datasets (AOPC-R on classical MIL datasets) [Table]
Revised: In this section, we detail our additional results on the Musk and Tiger, Elephant & Fox (TEF) classical MIL datasets.

Original: AOPC-R vs NDCG@n Although we have instance labels for the SIVAL, 4-MNIST-Bags, and CRC datasets (which allows us to use NDCG@n to evaluate the interpretability methods), it is useful to compare our results when using the AOPC-R metric. The CRC dataset has much larger bag sizes (264 instances per bag on average), therefore it is infeasible to compute the AOPC-R results on the CRC dataset (see Appendix A.3)
Revised: Although these datasets do not have instance labels, since they are widely used

Original: [Table]

Original: so we only provide AOPC-R results on the SIVAL (Table A1) and 4-MNIST-Bags (Table A2) datasets.

Original: [Table]

Original: [Table] For the AOPC-R results, we observe some similar trends to the NDCG@n results. Firstly, the instance independent methods performequally well as the local surrogate methods on the SIVAL dataset, but slightly worse on the 4-MNIST-Bags dataset. Secondly, MILLI performs better relative to the other methods on the 4-MNIST-Bags dataset compared to the SIVAL dataset, although its gain in performance is much less (to the extent where it has the same performance as GuidedSHAP and
Revised: MIL datasets, it is useful to see how our interpretability methods perform on them.

Original: RandomLIME).

Original: The LIME methods perform much better when evaluated using AOPC-R than they do with NDCG@n, and overall the difference in performance between the methods is much less than what we observe for the NDCG@n metric.

Original: A significant difference between the two metrics is the way they weight the importance ordering: in NDCG@n, the ordering is logarithmically weighed, whereas in AOPC-R, the ordering is equally weighted, i.e. NDCG@n prioritises performance at the start of the ordering, whereas AOPC-R equally prioritises performance across the entire ordering.

Original: This suggests MILLI achieves a better ordering for the most important instances, but gives the same ordering as other methods for less important instances.

Original: We consider the ordering for the important instances to be more crucial: these are the instances that are most significant in dictating the bag label, therefore it is more critical to order them correctly than it is for less important instances (hence our selection of NDCG@n as a ranking metric).

Original: AOPC-R on classical MIL datasets Although the Musk and Tiger, Elephant & Fox (TEF) datasets do not have instance labels, since they are fundamental MIL datasets, it is useful to see how our interpretability methods perform on them.

Original: Each of these datasets are instance independent, and as they have a low number of instances per bag, we adapted our local surrogate methods to allow sampling with repeats (otherwise we cannot sample enough coalitions).
Revised: Each of these datasets are instance independent, and as they have a low number of instances per bag, we adapted our local surrogate methods to allow sampling with repeats (otherwise we cannot sample enough coalitions).

Original: In our previous experiments, we restricted the local surrogate methods to sampling without coalitions, which was not an issue with the larger bag sizes in the SIVAL, 4-MNIST-Bags, and CRC datasets.
Revised: In our previous experiments, we restricted the local surrogate methods to sampling without repeats, which was not an issue with the larger bag sizes in the SIVAL, 4-MNIST-Bags, and CRC datasets.

Original: For each of these classic datasets, we used the same model hyperparameters that we used for SIVAL (see Appendix A.6), i.e., we didn’t retune the training parameters or model architectures.
Revised: For each of these classic datasets, we used the same model hyperparameters that we used for SIVAL (see Appendix A.6), i.e., we didn’t retune the training parameters or model architectures.

Original: However, we did tune the parameters for the interpretability methods, which we discuss in Appendix A.5.
Revised: However, we did tune the parameters for the interpretability methods, which we discuss in Appendix A.5.

Original: We give the interpretability results as well as the model performance for Musk (Table A3), Tiger (Table A4), Elephant (Table A5), and Fox (Table A6) below.
Revised: We give the interpretability results as well as the model performance for Musk (Table A1), Tiger (Table A2), Elephant (Table A3), and Fox (Table A4) below.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We find that there is very little difference in interpretability performance for each of our proposed methods across these four datasets.
Revised: We find that there is very little difference in interpretability performance for each of our proposed methods across these four datasets.

Original: This is to be expected, as the instances are independent, therefore the instance independent methods as well as the local surrogate methods are able to indentify the important instances, i.e., there is little to be gained by sampling coalitions when each instance can be understood in isolation.
Revised: This is to be expected, as the instances are independent, therefore the independent-instance methods as well as the local surrogate methods are able to identify the important instances, i.e., there is little to be gained by sampling coalitions when each instance can be understood in isolation.

Original: However, this reinforces the applicability of all of our proposed methods.
Revised: However, this reinforces the applicability of all of our proposed methods.

Original: We note that the attention and GNN inherent interpretability methods perform poorly in these experiments — this is because they are unable to condition their outputs on a specific class (i.e., they can only answer which questions, not what questions).
Revised: We note that the attention and GNN inherent interpretability methods perform poorly in these experiments — this is because they are unable to condition their outputs on a specific class (i [Table] e., they can only answer which questions, not what questions).

Original: We also note that the performance is much worse on the Fox dataset.
Revised: We also note that the performance is much worse on the Fox dataset.

Original: Here, the underlying MIL models perform poorly, so it is unsurprising that the interpretability methods also perform poorly.
Revised: Here, the underlying MIL models perform poorly, so it is unsurprising that the interpretability methods also perform poorly.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. I NTERPRETABILITY METHOD HYPERPARAMETER SELECTION
Revised: A. I NTERPRETABILITY METHOD HYPERPARAMETER SELECTION

Original: In this section we discuss our method for hyperparameter selection in the interpretability methods, and discuss the hyperparameters that were found to be most effective.
Revised: In this section we discuss our method for hyperparameter selection in the interpretability methods, and detail the hyperparameters that we found to be most effective.

Original: An advantage of the inherent interpretability and instance independent methods is that they do not have hyperparameters, i.e., we only had to select hyperparameters for the local surrogate interpretability methods.
Revised: An advantage of the inherent interpretability and independent-instance methods is that they do not have hyperparameters, i.e., we only had to select hyperparameters for the local surrogate interpretability methods.

Original: First, we discuss our choice of hyperparameters for LIME, and then our choice of hyperparameters for MILLI.
Revised: First, we discuss our choice of hyperparameters for LIME, and then our choice of hyperparameters for MILLI.

Original: LIME hyperparameters When using the LIME weight kernel, there are two hyperparameters to tune.
Revised: LIME hyperparameters When using the LIME weight kernel, there are two hyperparameters to tune.

Original: Firstly, the distance measures that are commonly used are L2 and cosine distance.
Revised: Firstly, the distance measures that are commonly used are L2 and cosine distance.

Original: In our experiments, we found very little difference between each of these distance measures, therefore we arbitrarily chose to use L2 distance in all of our experiments.
Revised: In our experiments, we found very little difference between each of these distance measures, therefore we arbitrarily chose to use L2 distance in all of our experiments.

Original: The second hyperparameter is the kernel width, which determines the weighting of coalitions, i.e., a large kernel width means all coalitions are weighted more evenly, and a small kernel width prioritises larger coalitions.
Revised: The second hyperparameter is the kernel width, which determines the weighting of coalitions, i.e., a large kernel width means all coalitions are weighted more evenly, and a small kernel width prioritises larger coalitions.

Original: We chose to use a kernel width for all of our experiments that is determined by the average bag size, such that the half coalition (i.e., | z | = 0 .
Revised: We chose to use a kernel width for all of our experiments that is determined by the average bag size, such that the half coalition (i.e., | z | = 0 .

Original: 5 k ) is weighted at 0.5.
Revised: 5 k ) is weighted at 0.5.

Original: MILLI hyperparameters For MILLI, there were three hyperparameters to tune: the number of samples, α , and β .
Revised: MILLI hyperparameters For MILLI, there were three hyperparameters to tune: the number of samples, α , and β .

Original: For the number of samples, we generated sample size plots such as Figure(also see Appendix A.9), and chose the number of samples to be at the point where all the methods had reasonably converged.
Revised: For the number of samples, we generated sample size plots such as Figure(also see Appendix A.9), and chose the number of samples to be at the point where all the methods had reasonably converged.

Original: For α and β we ran a grid search over the possible values, and chose the best performing pair of parameters.
Revised: For α and β we ran a grid search over the possible values, and chose the best performing pair of parameters.

Original: The hyperparameters were tuned for each dataset, except for the TEF datasets, in which we only tuned on Tiger and then used the same hyperparameters across all three datasets.
Revised: The hyperparameters were tuned for each dataset, except for the TEF datasets, in which we only tuned on Tiger and then used the same hyperparameters across all three datasets.

Original: In Table A7, we provide the chosen hyperparameters for each dataset, as well as the expected coalition size E [ | z | ] determined by these samples.
Revised: In Table A5, we provide the chosen hyperparameters for each dataset, as well as the expected coalition size E [ | z | ] determined by α and β .

Original: For the CRC, Musk and TEF datasets, the chosen parameters produce small values for E [ | z | ] , i.e., the sampling is heavily biased towards smaller coalitions, which is expected as these are instance independent datasets.
Revised: For the CRC, Musk and TEF datasets, the chosen parameters produce small values for E [ | z | ] , i.e., the sampling is heavily biased towards smaller coalitions, which is expected as these are instance independent datasets.

Original: Conversely, the parameters for the 4-MNIST-Bags focus on sampling larger coalitions that are able to capture the instance interactions in the dataset.
Revised: Conversely, the parameters for the 4-MNIST-Bags focus on sampling larger coalitions that are able to capture the instance interactions in the dataset.

Original: Although the SIVAL dataset is instance independent, the parameters also focus on sampling larger coalitions.
Revised: Although the SIVAL dataset is instance independent, the parameters also focus on sampling larger coalitions.

Original: This could be because, although the instances are independent, it may be difficult to classify an object from just one instance, i.e., several instances are needed to make the correct decision.
Revised: This could be because, although the instances are independent, it may be difficult to classify an object from just one instance, i.e., several instances are needed to make the correct decision.

Original: We also note that, in all cases, α < 0 . 5 , i.e., the sampling is biased towards instances ranked lower in the initial importance ordering used by MILLI.
Revised: We also note that, in all cases, α < 0 . 5 , i.e., the sampling is biased towards instances ranked lower in the initial importance ordering used by MILLI.

Original: Our explanation for this is that it is easy to understand the contribution of discriminatory instances, as they have a large effect on the model prediction, but it is more difficult to understand the contribution of non-discriminatory instances, as they have much less of an effect.
Revised: Our explanation for this is that it is easy to understand the contribution of discriminatory instances, as they have a large effect on the model prediction, but it is more difficult to understand the contribution of non-discriminatory instances, as they have much less of an effect.

Original: Therefore, more samples containing non-discriminatory instances are required to properly understand their effect, hence biasing the sampling towards them.
Revised: Therefore, more samples containing non-discriminatory instances are required to properly understand their effect, hence biasing the sampling towards them.

Original: [Table]
Revised: [Table]

Original: A. SIVAL EXPERIMENT DETAILS
Revised: A. SIVAL EXPERIMENT DETAILS

Original: Dataset For the SIVAL dataset, each instance is represented by a 30-dimensional feature vector, and there are around 30 instances per bag.
Revised: Dataset For the SIVAL dataset, each instance is represented by a 30-dimensional feature vector, and there are around 30 instances per bag.

Original: We chose 12 of the 25 original classes to be the positive classes, and randomly selected 30 images from each of the other 13 classes to form the negative class, meaning overall we had 13 classes (12 positive and one negative).
Revised: We chose 12 of the 25 original classes to be the positive classes, and randomly selected 30 images from each of the other 13 classes to form the negative class, meaning overall we had 13 classes (12 positive and one negative).

Original: In total, we had 60 bags for each of the 12 positive classes, and 390 bags for the single negative class, meaning the class distribution was ≈ 5 .
Revised: In total, we had 60 bags for each of the 12 positive classes, and 390 bags for the single negative class, meaning the class distribution was ≈ 5 .

Original: 4% for each positive class and ≈ 35 .
Revised: 4% for each positive class and ≈ 35 .

Original: 1% for the negative class.
Revised: 1% for the negative class.

Original: The arbitrarily chosen positive classes were: apple , banana , checkeredscarf , cokecan , dataminingbook , goldmedal , largespoon , rapbook , smileyfacedoll , spritecan , translucentbowl , and wd40can .
Revised: The arbitrarily chosen positive classes were: apple , banana , checkeredscarf , cokecan , dataminingbook , goldmedal , largespoon , rapbook , smileyfacedoll , spritecan , translucentbowl , and wd40can .

Original: We normalised each instance according to the dataset mean and standard deviation.
Revised: We normalised each instance according to the dataset mean and standard deviation.

Original: No other data augmentation was used.
Revised: No other data augmentation was used.

Original: The dataset was separated into train, validation, and test data using an 80/10/10 split.
Revised: The dataset was separated into train, validation, and test data using an 80/10/10 split.

Original: This was done with stratified sampling in order to maintain the same data distribution across all splits.
Revised: This was done with stratified sampling in order to maintain the same data distribution across all splits.

Original: Training When training models against the SIVAL dataset, we used a batch size of one, i.e., a single bag of, on average, 30 instances.
Revised: Training When training models against the SIVAL dataset, we used a batch size of one, i.e., a single bag of, on average, 30 instances.

Original: We trained the models to minimise cross entropy loss using the Adam optimiser; the hyperparamater details for learning rate (LR), weight decay (WD) and dropout (DO) are given in Table A8.
Revised: We trained the models to minimise cross entropy loss using the Adam optimiser; the hyperparamater details for learning rate (LR), weight decay (WD) and dropout (DO) are given in Table A6.

Original: We utilised early stopping based on validation loss — if the validation loss had not decreased for 10 epochs then we terminated the training procedure and used the model that caused the last decrease.
Revised: We utilised early stopping based on validation loss — if the validation loss had not decreased for 10 epochs then we terminated the training procedure and reset the model to the point at which it caused the last decrease in validation loss.

Original: Otherwise, the maximum number of epochs was 100.
Revised: Otherwise, the maximum number of epochs was 100.

Original: The tuned architectures for each model are given in Tables A9 to A12, and the results for each model are comapred in Table A13.
Revised: The tuned architectures for each model are given in Tables A7 to A10, and the results for each model are comapred in Table A11.

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: + DO

Original: FC + ReLU
Revised: FC + ReLU

Original: + DO  mil-max
Revised: + DO  mil-max

Original: FC
Revised: FC

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. 4-MNIST-B AGS EXPERIMENT DETAILS
Revised: A. 4-MNIST-B AGS EXPERIMENT DETAILS

Original: Dataset In the 4-MNIST-Bags experiments, the bag sizes were draw from a normal distribution, with a mean of 30 and a variance of 2.
Revised: Dataset In the 4-MNIST-Bags experiments, the bag sizes were draw from a normal distribution, with a mean of 30 and a variance of 2.

Original: We used 2500 training bags, 1000 validation bags, andtest bags.
Revised: We used 2500 training bags, 1000 validation bags, andtest bags.

Original: The instances in the training bags were only drawn from the original MNIST training split, and the instances in the validation and test bags were only drawn from the original MNIST test split, i.e., there was no overlap between training, validation, and test instances.
Revised: The instances in the training bags were only drawn from the original MNIST training split, and the instances in the validation and test bags were only drawn from the original MNIST test split, i.e., there was no overlap between training, validation, and test instances.

Original: The classes were balanced, so, on average, there were 625 bags per class in the training data, and 250 bags per class in the validation and test data.
Revised: The classes were balanced, so, on average, there were 625 bags per class in the training data, and 250 bags per class in the validation and test data.

Original: We normalised the MNIST images using the PyTorch normalise transformation, with a mean of 0.1307 and a stand deviation of 0.3081.
Revised: We normalised the MNIST images using the PyTorch normalise transformation, with a mean of 0.1307 and a stand deviation of 0.3081.

Original: No other data augmentation was carried out.
Revised: No other data augmentation was carried out.

Original: Training The training procedure was the same as for the SIVAL experiments: a batch size of one, early stopping with a patience of ten, and a maximum of 100 epochs.
Revised: Training The training procedure was the same as for the SIVAL experiments: a batch size of one, early stopping with a patience of ten, and a maximum of 100 epochs.

Original: The training hyperparamater details are given in Table A14.
Revised: The training hyperparamater details are given in Table A12.

Original: For each model, we first passed the MNIST instances through a convolutional architecture to produce initial instance embeddings.
Revised: For each model, we first passed the MNIST instances through a convolutional architecture to produce initial instance embeddings.

Original: This encoder was not tuned for each model (i.e., the architecture was fixed, but the weights were learnt).
Revised: This encoder was not tuned for each model (i.e., the architecture was fixed, but the weights were learnt).

Original: The architecture for this encoder is given in Table A15, and then the model architectures are given in Tables A16 to A19.
Revised: The architecture for this encoder is given in Table A13, and then the model architectures are given in Tables A14 to A17.

Original: The encoder produces features vectors with 800 features, therefore the input size to each of the models is 800.
Revised: The encoder produces features vectors with 800 features, therefore the input size to each of the models is 800.

Original: The model results are given in Table A20.
Revised: The model results are given in Table A18.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. CRC EXPERIMENT DETAILS
Revised: A. CRC EXPERIMENT DETAILS

Revised: Dataset In the CRC dataset, there are 100 microscopy images, and each image is annotated with four classes of nuclei: epithelial, inflammatory, fibroblast, and miscellaneous.

Original: Dataset For the CRC dataset, we have 100 images,with 50  in the negative class (no epithelial nuclei), and 50  in the positive class (at least one epithelial nuclei).
Revised: Of these 100 images, 50 are in the negative class (no epithelial nuclei), and 50 are in the positive class (at least one epithelial nuclei).

Original: Each image is 500 x 500 pixels, and was split into 27 x 27 pixel patches to give 324 patches per slide.
Revised: Each image is 500 x 500 pixels, and was split into 27 x 27 pixel patches to give 324 patches per slide.

Original: The patches were created by applying a non-overlapping grid over the image, where each cell of the grid was a 27 x 27 pixel region.
Revised: The patches were created by applying a non-overlapping grid over the image, where each cell of the grid was a 27 x 27 pixel region.

Original: The alternative would be to extract pages centred on each marked nuclei (as done by Ilse et al.
Revised: The alternative would be to extract patches centred on each marked nuclei (as done by Ilse et al.

Original: , however this method then requires the nuclei to be marked for unseen data.
Revised: , however this method then requires the nuclei to be marked for unseen data.

Original: This accounts for the difference between our results and the results of Ilse et al.
Revised: This accounts for the difference between our results and the results of Ilse et al.

Original: (2018) — extracting the patches using a fixed grid will include patches that do not contain any marked nuclei, increasing the amount of non-discriminatory data in each bag and thus making the problem harder to learn.
Revised: (2018) — extracting the patches using a fixed grid will include patches that do not contain any marked nuclei, increasing the amount of non-discriminatory data in each bag and thus making the problem harder to learn.

Original: We removed slide background patches by using a brightness threshold — any patch with an average pixel value above 230 (using pixel values 0 to 255) was discarded.
Revised: We removed slide background patches by using a brightness threshold — any patch with an average pixel value above 230 (using pixel values 0 to 255) was discarded.

Original: This left an average of 264 patches per image, and one image was discarded as it had zero foreground patches (this was also manually verified).
Revised: This left an average of 264 patches per image, and one image was discarded as it had zero foreground patches (this was also manually verified).

Original: The images were normalised using the dataset mean (0.8035, 0.6499, 0.8348) and standard deviation (0.0858, 0.1079, 0.0731).
Revised: The images were normalised using the dataset mean (0.8035, 0.6499, 0.8348) and standard deviation (0.0858, 0.1079, 0.0731).

Original: During training, we also applied three transformations to patches at random: horizontal flips, vertical flips, and 90 degree rotations.
Revised: During training, we also applied three transformations to patches at random: horizontal flips, vertical flips, and 90 degree rotations.

Original: The dataset was separated into train, validation, and test data using a 60/20/20 split.
Revised: The dataset was separated into train, validation, and test data using a 60/20/20 split.

Original: This was done with stratified sampling in order to maintain the same data distribution across all splits.
Revised: This was done with stratified sampling in order to maintain the same data distribution across all splits.

Original: Training
Revised: Training

Original: The training procedure was the same as for the SIVAL and 4-MNIST-Bags experiments: a batch size of one, early stopping with a patience of ten, and a maximum of 100 epochs.
Revised: The training procedure was the same as for the SIVAL and 4-MNIST-Bags experiments: a batch size of one, early stopping with a patience of ten, and a maximum of 100 epochs.

Original: The training hyperparamater details are given in Table A21.
Revised: The training hyperparamater details are given in Table A19.

Original: Following the same procedure as the 4-MNIST-Bags experiments, for each model, we first passed the patches through a convolutional architecture to produce initial instance embeddings.
Revised: Following the same procedure as the 4-MNIST-Bags experiments, for each model, we first passed the patches through a convolutional architecture to produce initial instance embeddings.

Original: The architecture for this encoder is given in Table A22, and then the model architectures are given in Tables A23 to A26.
Revised: The architecture for this encoder is given in Table A20, and then the model architectures are given in Tables A21 to A24.

Original: The encoder produces features vectors with 1200 features, therefore the input size to each of the models is 1200.
Revised: The encoder produces features vectors with 1200 features, therefore the input size to each of the models is 1200.

Original: The model results are given in Table A27.
Revised: The model results are given in Table A25.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A DDITIONAL EXPERIMENTS
Revised: A. A DDITIONAL EXPERIMENTS

Original: In this section, we provide additional sample size experiments for the SIVAL, 4-MNIST-Bags
Revised: In this section, we provide additional sample size experiments for the SIVAL, 4-MNIST-Bags

Original: [Table]
Revised: [Table]

Original: and
Revised: and

Original: CRC datasets for MI-Net (Figure A1), mi-Net (Figure A2)
Revised: CRC datasets for MI-Net (Figure A1), mi-Net (Figure A2)

Original: [Table]
Revised: [Table]

Original: and MI-GNN (Figure A3).
Revised: and MI-GNN (Figure A3).

Original: We find the trends are relatively consistent across all the models.
Revised: We find the trends are relatively consistent across all the models.

Original: GuidedSHAP and MILLI are the most consistent and well performing methods
Revised: GuidedSHAP and MILLI are the most consistent and well performing methods

Original: [Table]
Revised: [Table]

Original: and MILLI is particularly effective on the 4-MNIST-Bags dataset.
Revised: and MILLI is particularly effective on the 4-MNIST-Bags dataset.

Original: One considerable difference is the performance of RandomLIME and GuidedLIME on the
Revised: One considerable difference is the performance of RandomLIME and GuidedLIME on the

Original: CRC dataset for the MI-GNN model; for all other models, both LIME methods perform poorly on the CRC dataset.
Revised: CRC dataset for the MI-GNN model; for all other models, both LIME methods perform poorly on the CRC dataset.

Original: Further investigation is required to understand why this happens.
Revised: Further investigation is required to understand why this happens.

Original: However, the
Revised: However, the

Original: LIME methods are still outperformed by MILLI and GuidedSHAP on the CRC dataset, even for the
Revised: LIME methods are still outperformed by MILLI and GuidedSHAP on the CRC dataset, even for the

Original: MI-GNN model.
Revised: MI-GNN model.

Original: A. A DDITIONAL OUTPUTS
Revised: A. A DDITIONAL OUTPUTS

Original: In this section, we provide additional interpretability outputs from our studies, similar to the one in Section 5.
Revised: In this section, we provide additional interpretability outputs from our studies, similar to the one in Section 5.

Original: First, we provide additional outputs for the 4-MNIST-Bags dataset.
Revised: First, we provide additional outputs for the 4-MNIST-Bags dataset.

Original: Then, we show the interpretability outputs for the SIVAL and CRC datasets.
Revised: Then, we show the interpretability outputs for the SIVAL and CRC datasets.

Original: The SIVAL outputs are created by ranking the instances in a bag according to some interpretability function (i.e., by using MILLI), and then selecting the top n , where n is the known number of key instances in the bag.
Revised: The SIVAL outputs are created by ranking the instances in a bag according to some interpretability function (i.e., by using MILLI), and then selecting the top n , where n is the known number of key instances in the bag.

Original: Then, the corresponding segment for each instance is weighted by the function log 2 ( i + 1) − 1 , where i is the instance’s position in the ranking (this is the same scaling used in NDCG@n).
Revised: Then, the corresponding segment for each instance is weighted by the function log 2 ( i + 1) − 1 , where i is the instance’s position in the ranking (this is the same scaling used in NDCG@n).

Original: The other instances all received a weighting of zero.
Revised: The other instances all received a weighting of zero.

Original: The brightness of each segment in the output image then corresponds with its relative importance according to the interpretability method.
Revised: The brightness of each segment in the output image then corresponds with its relative importance according to the interpretability method.

Original: For the CRC interpretability outputs, the important patches are found by using an interpretability method to output the top n patches that support a particular class, where n is the number of known important instances for that class.
Revised: For the CRC interpretability outputs, the important patches are found by using an interpretability method to output the top n patches that support a particular class, where n is the number of known important instances for that class.

Original: We then highlight these patches while dimming the other patches to produce an interpretation of the model’s decision-making.
Revised: We then highlight these patches while dimming the other patches to produce an interpretation of the model’s decision-making.

Original: Here the  ignoring  medal  the red ribbon, i e., if it were shown the medal without the ribbon, it would likely misclassify it
Revised: Here, the the is ignoring the gold medal on the red ribbon, i e., if it were shown the medal without the ribbon, it would likely misclassify it

Original: Here, the interpretations show the model is predominantly focusing on the strong yellow colour at the top of the can, and sometimes picks out the letters and red cap.
Revised: Here, the interpretations show the model is predominantly focusing on the strong yellow colour at the top of the can, and sometimes picks out the letters and red cap.

Original: The rest of the bottle is largely ignored, meaning if the top half were obscured, the model would likely misclassify it
Revised: The rest of the bottle is largely ignored, meaning if the top half were obscured, the model would likely misclassify it

Original: GuidedSHAP has identified labelled was not may contain nuclei without labelled is apparent that the model is using in further into the of nuclei in patches would reveal more information about how the makes its would not be