Original: Beyond Protected Attributes: Disciplined Detection of Systematic Deviations in Data
Revised: Beyond Protected Attributes: Disciplined Detection of Systematic Deviations in Data

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Finding systematic deviations of an outcome of interest in data and models isan important goal of trustworthy and socially responsible AI.
Revised: Finding systematic deviations of an outcome of interest in data and models is an important goal of trustworthy and socially responsible AI.

Original: To understand systematic deviations at a subgroup level, it is important to look beyond predefinedgroups and consider all possible subgroups for analysis.
Revised: To understand systematic deviations at a subgroup level, it is important to look beyond predefined groups and consider all possible subgroups for analysis.

Original: Of course this exhaustiveenumeration is not possible and there needs to be a balance of exploratory andconfirmation analysis in socially-responsible AI.
Revised: Of course this exhaustive enumeration is not possible and there needs to be a balance of exploratory and confirmation analysis in socially-responsible AI.

Original: In this paper we compare recentlyproposed methods for detecting systematic deviations in an outcome of interestat the subgroup level across three socially-relevant data sets.
Revised: In this paper we compare recently proposed methods for detecting systematic deviations in an outcome of interest at the subgroup level across three socially-relevant data sets.

Original: Furthermore, weshow the importance of looking through all possible subgroups for systematicdeviations by comparing detected patterns using only protected attributes againstpatterns detected using the entire search space.
Revised: Furthermore, we show the importance of looking through all possible subgroups for systematic deviations by comparing detected patterns using only protected attributes against patterns detected using the entire search space.

Original: One interesting pattern found inthe OULAD dataset is that while having a high course load and not being fromthe highest socio-economic decile of UK regions makes students 2.3 times morelikely to fail or withdraw from courses, being from Ireland or Wales mitigates thisrisk by 37%.
Revised: One interesting pattern found in the OULAD dataset is that while having a high course load and not being from the highest socio-economic decile of UK regions makes students 2.3 times more likely to fail or withdraw from courses, being from Ireland or Wales mitigates this risk by 37%.

Original: This pattern may have been missed if we focused our analysis onthe protected groups of gender and disability only.
Revised: This pattern may have been missed if we focused our analysis on the protected groups of gender and disability only.

Original: Python code for all methods,including the most recently proposed "AutoStrat" are available on open-sourcedcode repositories.
Revised: Python code for all methods, including the most recently proposed "AutoStrat" are available on open-sourced code repositories.

Original:  Introduction
Revised:  Introduction

Original: Understanding data at a subgroup level is an important part of building trustworthy, reliable andsocially responsible AI-based systems [1].
Revised: Understanding data at a subgroup level is an important part of building trustworthy, reliable, and socially responsible AI-based systems [1, 2].

Original: Systematic deviations in data can help reveal importantissues such as predictive bias [2], distribution shifts [3] and under/over representation in data [4, 5],among others.
Revised: Systematic deviations in data can help reveal important issues such as predictive bias [3], distribution shifts [4], and under/over representation in data [5, 6], among others.

Original: Most of the works in Trustworthy AI have involved analysis of unfair or unequaltreatment of an individual (or group) based on certain characteristics such as gender, race, maritalstatus, etc., that should not be used for making decisions, usually known as protected attributes [6, 7].
Revised: Most of the works in Trustworthy AI have involved analysis of unfair or unequal treatment of an individual (or group) based on certain characteristics such as gender, race, marital status, etc., that should not be used for making decisions, usually known as protected attributes [7, 8].

Original: However, focusing on some predefined attributes for systematic deviations can lead to fairnessgerrymandering where relevant and important patterns that do not appear at a group or individuallevel become visible at the subgroup level [8, 9].
Revised: However, focusing on some predefined attributes for systematic deviations can lead to fairness gerrymandering where relevant and important patterns that do not appear at a group or individual level become visible at the subgroup level [9, 10].

Original: For example, in a dataset with binary definitions ofrace and gender, a classifier that assigns an outcome to black men or white women independently anduniformly at random in a population will appear equitable when one considers the protected groupsof gender and race separately but is actually inequitable to black women, and white men subgroupsin the dataset [8].
Revised: In a dataset with binary definitions of race and gender, outcomes assigned to black men or white women independently and uniformly at random in a population will appear equitable when one considers the protected groups of gender and race separately but is actually inequitable to black women, and white men subgroups in the data [9].

Original: While it might be possible to enumerate all the possible subgroups from a handful of pre-determinedprotected attributes, we could still be missing out on more relevant and important patterns.
Revised: While it might be possible to enumerate all the possible subgroups from a handful of pre-determined protected attributes, we could still be missing out on more relevant and important patterns.

Original: Forexample, a classifier could assign outcomes  such that it would be fair to the four possible protectedsubgroups involving race and gender but may be unfair to a subgroup such as young married people or black women without a college degree when we add the unprotected attributes of age and educationlevel.
Revised: For example, outcomes could be assigned such that it would be fair to the four possible protected subgroups involving race and gender but may be unfair to a subgroup such as young married people or black women without a college degree when we add the unprotected attributes of age and education level.

Original: We need to consider all the possible subgroups in the data to advance towards a more trustworthyand responsible analysis.
Revised: We need to consider all the possible subgroups in the data to advance towards a more trustworthy and responsible analysis.

Original: However, there are two major obstacles to performing the analysis.
Revised: However, there are three major obstacles to performing the analysis.

Original: First,there are exponentially-many subgroups to consider, which means we cannot perform a brute-forcesearch.
Revised: First, there are exponentially-many subgroups to consider, which means we cannot perform a brute-force search.

Original: This is why many analyses focus on protected groups only, as it makes the discovery processmore tractable.
Revised: This is why many analyses focus on protected groups only, as it makes the discovery process more tractable.

Original: Furthermore, a heuristic-based search, such as a beam search, might miss out onimportant patterns.
Revised: Furthermore, a heuristic-based search, such as a beam search, might miss out on important patterns.

Original: Second, since there are a lot of subgroups and, by extension, hypotheses to test,we may run into the problem of a high false discovery rate due to multiple hypothesis testing [10, 11].
Revised: Second, since there are a lot of subgroups and, by extension, hypotheses to test, we may run into the problem of a high false discovery rate due to multiple hypothesis testing [11, 12].

Revised: Third, given several subgroups with statistically significant deviations, there is still the important question of how to rank them correctly (see Appendix A).

Original: To address these concerns, we use a technique that leverages mathematical properties of commonlyused methods of divergence to perform this search using standard compute resources [12, 13].
Revised: To address these concerns, we use a technique that leverages mathematical properties of commonly used methods of divergence to perform this search using standard compute resources [13, 14].

Original: Insummary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We describe Automatic Stratification (AutoStrat) - an efficient algorithm to automaticallydiscover systematic deviations in data in a more trustworthy and responsible way by goingbeyond protected subgroups.•
Revised: • We describe Automatic Stratification (AutoStrat) - an efficient algorithm to automatically discover systematic deviations in data in a more trustworthy and responsible way by going beyond protected subgroups.

Original:  We discuss the notion of divergence of a subgroup that automatically balances the sizeandsignificance of  systematic deviations.•
Revised: • We discuss the notion of divergence of a subgroup that automatically balances the size of the subgroup and the extremity of its systematic deviation.

Original:  We show the importance of disciplined subgroup discovery for three use cases - under/overrepresentation in data and model risk assessment.
Revised: • We show the importance of disciplined subgroup discovery for three use cases - under/over representation in data and model risk assessment.

Original: • We finally compare the subgroups discovered by our approach to the subgroups discovered in other subgroup discovery algorithms, showing the utility of allowing for more than onefeature value per feature in the subgroup description.
Revised: • We finally compare the subgroups discovered by our approach to the subgroups discovered in other subgroup discovery algorithms, showing the utility of allowing for more than one feature value per feature in the subgroup description.

Original:  Related Work
Revised:  Related Work

Original: There have been several attempts to understand interactions present in data used for machine learningpurposes [14, 15, 16].
Revised: There have been several attempts to understand interactions present in data used for machine learning purposes [15, 16, 17].

Original: However, this analysis is usually limited to two or three features, and itrequires domain knowledge which makes it difficult to identify critical but previously unknownsub-populations.
Revised: However, this analysis is usually limited to two or three features, and it requires domain knowledge which makes it difficult to identify critical but previously unknown sub-populations.

Original: Other works like [17, 18] have focused on the explainability of black-box models atan instance or global level of the data in order to uncover bias in how the model works or understandwhy the model is making certain predictions.
Revised: Other works like [18, 19] have focused on the explainability of black-box models at an instance or global level of the data in order to uncover bias in how the model works or understand why the model is making certain predictions.

Original: On the other hand, feature selection methods [19, 20]try to identify relevant features for prediction tasks to help the model generalize better to unseen data.
Revised: On the other hand, feature selection methods [20, 21] try to identify relevant features for prediction tasks to help the model generalize better to unseen data.

Original: More recent works [21, 22, 23] have tried to detect when unseen data deviates significantly from thetraining set.
Revised: More recent works [22, 23, 24] have tried to detect when unseen data deviates significantly from the training set.

Original: While all these methods are useful, they do not answer the critical question of how dowe automatically detect and characterize systematic deviations in data at a subgroup level for theexponentially many subgroups.
Revised: While all these methods are useful, they do not answer the critical question of how do we automatically detect and characterize systematic deviations in data at a subgroup level for the exponentially many subgroups.

Original: Subgroup discovery aims at automatically extracting subgroups with much higher (or lower) ratesof the target outcome in a given data [24, 25].
Revised: Subgroup discovery aims at automatically extracting subgroups with much higher (or lower) rates of the target outcome in a given data [25, 26].

Original: Pysubgroup [25] is an open-source implementationof several subgroup discovery algorithms where subgroups are a conjunction of simple predicates(logical ANDs only).
Revised: Pysubgroup [26] is an open-source implementation of several subgroup discovery algorithms where subgroups are a conjunction of simple predicates (logical ANDs only).

Original: Two recently proposed model validation algorithms, Slice Finder [26] and
Revised: Two recently proposed model validation algorithms, Slice Finder [27] and

Original: DivExplorer [27], could be applied to the subgroup discovery task(see Appendix A).
Revised: DivExplorer [28], could be applied to the subgroup discovery task(see Appendix B).

Original: Table 1 showsan overview comparison of these methods with our approach, AutoStrat.
Revised: Table 1 shows an overview comparison of these methods with our approach, AutoStrat.

Original: Unlike the existing methods,which only support logical AND combinations of feature values, AutoStrat also supports logical ORcombinations, resulting in a more flexible description of the identified subset.
Revised: Unlike the existing methods, which only support logical AND combinations of feature values, AutoStrat also supports logical OR combinations, resulting in a more flexible description of the identified subset (see Appendix C).

Original: Moreover, AutoStratoptimises for the mathematically correct balance of both the size and severity of the identifiedsubgroup, thereby reducing the likelihood of false discoveries.[Table] Pysubgroup [25] ✓ ✓ ✓ ✓ Slice Finder [26] ✓ ✓
Revised: Finally, AutoStrat optimises for the mathematically correct balance of both the size and severity of the deviation of identified subgroup, thereby reducing the likelihood of false discoveries.

Original: Div-Explorer [27] ✓ ✓

Original: AutoStrat (Ours) ✓ ✓ ✓ ✓

Original:  Approach: AutoStrat
Revised:  Approach: AutoStrat

Original: Multi-dimensional Subset Scan (MDScan) was originally proposed in a spatial-temporal setting withapplications in bio-surveillance and situational awareness [12, 28].
Revised: Multi-dimensional Subset Scan (MDScan) was originally proposed in a spatial-temporal setting with applications in bio-surveillance and situational awareness [13, 29].

Original: More recently, MDScan wasre-framed on tabular data and used to detect predictive bias in black box classifiers [29].
Revised: More recently, MDScan was re-framed on tabular data and used to detect predictive bias in black box classifiers [30]. Our current [Table]

Original: Our currentwork, AutoStrat, builds on top of MDScan with tabular data but without the focus on predictive biasin classifiers.
Revised: Pysubgroup [26] ✓ ✓ ✓ ✓ Slice Finder [27] ✓ ✓ Div-Explorer [28] ✓ ✓ AutoStrat (Ours) ✓ ✓ ✓ ✓ work, AutoStrat, builds on top of MDScan with tabular data but without the focus on predictive bias in classifiers.

Original: Rather, our goal is to identify anomalous subgroups that have higher-than-averageoutcomes as compared to the global mean.
Revised: Rather, our goal is to identify anomalous subgroups that have higher-than-average outcomes as compared to the global mean.

Original: Given a dataset D and a search space λ , we describe a subgroup, S as a conjuction of literals suchthat we have logical ANDs at the feature level and logical ORs at the feature value level for featuresin λ , e.g., Country = {Kenya OR Nigeria} AND Marital Status =
Revised: Given a dataset D and a search space λ , we describe a subgroup, S as a conjuction of literals such that we have logical ANDs at the feature level and logical ORs at the feature value level for features in λ , e.g., Country = {Kenya OR Nigeria} AND Marital Status =

Original: {Single OR Married} .
Revised: {Single OR Married} .

Original: This can alsobe referred to as a subspace of the Cartesian product of categorical features and their values.
Revised: This can also be referred to as a subspace of the Cartesian product of categorical features and their values.

Original: If λ islimited to just the protected features, then the subgroup discovered is called a protected subgroup(PS).
Revised: If λ is limited to just the protected features, then the subgroup discovered is called a protected subgroup (PS).

Original: Otherwise, if λ includes features beyond the protected attributes then the anomalous subgroupis a Beyond-protected subgroup (BPS).
Revised: Otherwise, if λ includes features beyond the protected attributes then the anomalous subgroup is a Beyond-protected subgroup (BPS).

Original: MDScan is an iterative ascent procedure (see Algorithm 1 for pseudocode) where each step is efficientand exact due to the Additive Linear-time Subset Scanning property (ALTSS) [30, 31].
Revised: MDScan is an iterative ascent procedure (see Algorithm 1 for pseudocode) where each step is efficient and exact due to the Additive Linear-time Subset Scanning property (ALTSS) [31, 32].

Original: While afeature containing k unique values has exponentially many ( O (2 k ) )
Revised: While a feature containing k unique values has exponentially many ( O (2 k ) )

Original: subgroups, a scoring functionthat satisfies the ALTSS property guarantees that the most anomalous (highest scoring) subgroup willbe one of only linearly-many ( O (2 K ) ) subgroups.
Revised: subgroups, a scoring function that satisfies the ALTSS property guarantees that the most anomalous (highest scoring) subgroup will be one of only linearly-many ( O (2 K ) ) subgroups.

Original: This property drastically reduces the search spaceand makes logical ORs between feature values tractable.
Revised: This property drastically reduces the search space and makes logical ORs between feature values tractable.

Original: Using this property, Bias Scan [29] searchesfor a subset S that maximizes the divergence between the observed and expected outcomes in Swhere y i is a binary outcome of interest for record i and e i is probability assigned to the outcome ofinterest for record i by a predictive model.
Revised: Using this property, Bias Scan [30] searches for a subset S that maximizes the divergence between the observed and expected outcomes in S where y i is a binary outcome of interest for record i and e i is probability assigned to the outcome of interest for record i by a predictive model.

Original: While Bias Scan focuses on identifying significant predictive bias in classifiers, AutoStrat pivotsaway from predictive models by replacing e i with the proportion of the target class y .
Revised: While Bias Scan focuses on identifying significant predictive bias in classifiers, AutoStrat pivots away from predictive models by replacing e i with the proportion of the target class y .

Original: This enables usto perform subgroup discovery by maximizing Equation 1.
Revised: This enables us to perform subgroup discovery by maximizing Equation 1.

Original: Given a subgroup S , the scoring function,Γ( S ) , is
Revised: Given a subgroup S , the scoring function, Γ( S ) , is

Original: [Equation]
Revised: [Equation]

Original: Where y i is observed outcome for the i th record, e i is the expected outcome and the proportion ofthe target class in y , and q is the multiplicative odds increase of the subgroup, S .
Revised: Where y i is observed outcome for the i th record, e i is the expected outcome and the proportion of the target class in y , and q is the multiplicative odds increase of the subgroup, S .

Original: This is also referredto as the "Bias score" of a subset [29].
Revised: This is also referred to as the "Bias score" of a subset [30].

Original: This scoring function can be derived from a likelihood ratiostatistic based on two Bernoulli distributions.
Revised: This scoring function can be derived from a likelihood ratio statistic based on two Bernoulli distributions (see [33]).

Original: The denominator (null hypothesis) is that the odds ofoutcome y
Revised: The denominator (null hypothesis) is that the odds of outcome y

Original: i are e i 1 − e i for all i .
Revised: i are e i 1 − e i for all i .

Original: The numerator (alternative hypothesis) is that the odds of y i have beenincreased by a multiplicative factor, q for records in some subset, S .
Revised: The numerator (alternative hypothesis) is that the odds of y i have been increased by a multiplicative factor, q for records in some subset, S .

Original: By maximizing this statisticover all subsets, we are finding the subset that shows the most evidence that q > 1 .
Revised: By maximizing this statistic over all subsets, we are finding the subset that shows the most evidence that q > 1 .

Original: Significance testing To ascertain the statistical significance of the found subgroup, randomizationbased hypothesis testing is used [32, 33].
Revised: Significance testing To ascertain the statistical significance of the found subgroup, randomizationbased hypothesis testing is used [34, 35].

Revised: We use randomization-based hypothesis testing because of the exponential nature of the number of the hypothesis we are testing during the discovery process.

Revised: Correction mechanisms such as the Benferroni correction [36] and the Benjamini-Hochberg correction [37] are too conservative and would incorrectly flag many subgroups discovered as false discoveries.

Original: We consider K = 100 iterations, and a replica of theobserved outcome is generated from the assumption of the null hypothesis for each iteration, k .
Revised: To carry out randomization-based hypothesis testing, we consider K = 100 iterations, and a replica of the observed outcome is generated from the assumption of the null hypothesis for each iteration, k .

Original: Scanning for the anomalous subset is applied to each replica of the outcome and the divergence scoresfrom all the replicas ( Γ( S k ), k = 1 , 2 , · · · , 100) are compared with the true divergence score ( Γ( S ∗ ) ).
Revised: Scanning for the anomalous subset is applied to each replica of the outcome and the divergence scores from all the replicas ( Γ( S k ), k = 1 , 2 , · · · , 100) are compared with the true divergence score ( Γ( S ∗ ) ).

Original: An empirical p -value ( p ) is computed as p =
Revised: An empirical p -value ( p ) is computed as p =

Original: ( r ( S ∗ )
Revised: ( r ( S ∗ )

Original: + 1) /
Revised: + 1) /

Original: ( K + 1) , where r ( S ∗ )
Revised: ( K + 1) , where r ( S ∗ )

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =1 ζ k ( S ∗ ) ,and ζ k ( S ∗ )
Revised: Kk =1 ζ k ( S ∗ ) , and ζ k ( S ∗ )

Original: = 1 if Γ( S k ) ≥ Γ( S ∗ ) else ζ k ( S ∗ )
Revised: = 1 if Γ( S k ) ≥ Γ( S ∗ ) else ζ k ( S ∗ )

Original: = 0 .
Revised: = 0 .

Original:  Experimental Setup
Revised:  Experimental Setup

Original: Datasets We use three commonly used datasets for Trustworthy AI research - Compas [2], Creditcard clients [34], and OULAD education data [35], covering three important use cases - under/overrepresentation in data and model risk assessment.
Revised: Datasets We use three commonly used datasets for Trustworthy AI research - Compas [3], Credit card clients [38], and OULAD education data [39], covering three important use cases - under/over representation in data and model risk assessment.

Original: The analysis for the OULAD dataset focuseson under representation of pass/distinction in the data while that of the Credit card client datasetfocus on the over representation of default payment in the next month.
Revised: The analysis for the OULAD dataset focuses on under representation of pass/distinction in the data while that of the Credit card client dataset focus on the over representation of default payment in the next month.

Original: For the Compas dataset, the[Table] outcome is the risk of violence as assigned by the Compas tool.
Revised: For the Compas dataset, the outcome is the risk of violence as assigned by the Compas tool.

Original: By focusing on a predictive model’spredictions without the ground truth , we are performing a model risk assessment [18, 36] to checkthe consistency of the model across subgroups rather than how it performs in relation to the groundtruth (model validation) or why it makes its predictions (model explainability).
Revised: By focusing on a predictive model’s predictions without the ground truth , we are performing a model risk assessment [19, 40] to check the consistency of the model across subgroups rather than how it performs in relation to the ground truth (model validation) or why it makes its predictions (model explainability).

Original: Table 2 provides anoverview of each dataset, including the number of records, search space size, protected attributes, etc.
Revised: Table 2 provides an overview of each dataset, including the number of records, search space size, protected attributes, etc.

Original: A full description of the datasets and their features can be found in [37].
Revised: A full description of the datasets and their features can be found in [41].

Original: Baseline methods We compare AutoStrat against four other recently proposed subgroup discoveryalgorithms - Beam Search and Apriori (as implemented by Pysubgroup [25]), Fp-growth algorithm(as implemented by DivExplorer [27]), and Slice Finder [26].
Revised: Baseline methods We compare AutoStrat against four other recently proposed subgroup discovery algorithms - Beam Search and Apriori (as implemented by Pysubgroup [26]), Fp-growth algorithm (as implemented by DivExplorer [28]), and Slice Finder [27].

Original: More details on these methods areprovided in the appendix.
Revised: More details on these methods are provided in the appendix.

Original: Evaluation metrics We evaluate the anomalousness of an identified subgroup ( S ) using the
Revised: Evaluation metrics We evaluate the anomalousness of an identified subgroup ( S ) using the

Original: Bernoulli likelihood ratio statistic, Γ( S ) .
Revised: Bernoulli likelihood ratio statistic, Γ( S ) .

Original: Given p 1 as the proportion of outcomes in the recordsbelonging to S , p as the proportion of outcomes in the entire dataset, n ′ as the number of recordsin S , n as the total of records in the dataset, and p 0 as proportion of outcomes in the records notbelonging to S , we report four other statistics - Lift 2, Support 3, Odds Ratio ( OR ) 4, andWeighted
Revised: Given p 1 as the proportion of outcomes in the records belonging to S , p as the proportion of outcomes in the entire dataset, n ′ as the number of records in S , n as the total of records in the dataset, and p 0 as proportion of outcomes in the records not belonging to S , we report four other statistics - Lift 2, Support 3, Odds Ratio ( OR ) 4, and

Original:  Relative Accuracy ( ϕ ( S ) ) [-
-] in addition to the anomalous score, Γ( S ) .
Revised: Weighted Relative Accuracy ( ϕ ( S ) ) [-
-]5 in addition to the anomalous score, Γ( S ) .

Revised: While Lift tells us how much more likely the target outcome in a subgroup is in comparison to the average, Support tells us the relative size of the subgroup to the entire data.

Revised: The odds ratio and weighted relative accuracy both measure how extreme the deviation of the subgroup is.

Original: Finally, we include runtimesfor all the experiments in seconds.
Revised: Finally, we include runtimes for all the experiments in seconds.

Revised: Code for all the experiments and analysis can be found at https://ibm.biz/BPS-Autostrat .

Original: [Equation]
Revised: [Equation]

Original:  Preliminary Results and Discussion 
Revised:  Preliminary Results and Discussion 5.1 Comparison between protected subgroups and non-protected subgroups

Original: 5.1 Comparison between protected subgroups and non-protected subgroups

Original: We ran experiments to obtain the subgroup with the most systematic deviation over the entire searchspace (Beyond Protected Subgroups, BPS) from AutoStrat and over only the protected attributes (PS)as defined in Table 2.
Revised: We ran experiments to obtain the subgroup with the most systematic deviation over the entire search space (Beyond Protected Subgroups, BPS) from AutoStrat and over only the protected attributes (PS) as defined in Table 2.

Original: We also report the empirical p -values as obtained using randomization-basedsignificance testing 3.
Revised: We also report the empirical p -values as obtained using randomization-based significance testing 3.

Original: Table 3 presents the results of the comparison between protected subgroups and beyond protectedsubgroups across the three datasets.
Revised: Table 3 presents the results of the comparison between protected subgroups and beyond protected subgroups across the three datasets.

Original: As reported across several research [2, 37], the patterns found inprotected attributes are statistically significant ( p -value < 0 . 05 ) for both the Compas and OULADdatasets but not significant for the Credit card dataset.
Revised: As reported across several research [3, 41], the patterns found in protected attributes are statistically significant ( p -value < 0 . 05 ) for both the Compas and OULAD datasets but not significant for the Credit card dataset.

Original: While the age_cat is less than 25 subgroupcould have been found by other techniques (e.g. a logistic regression analysison a one-hot encoded
Revised: While the age_cat is less than 25 subgroup could have been found by other techniques (e.g. a logistic regression analysis

Original: [Table]
Revised: [Table]

Original: credits =

Original: [Table]
Revised: [Table]

Original:  version of the dataset [2]) because of its relative simplicity, this significant pattern would have beenmissed if we only focused on protected attributes.
Revised: on a one-hot encoded version of the dataset [3]) because of its relative simplicity [Table] this significant pattern would have been missed if we only focused on protected attributes.

Original: In the OULAD dataset, we see that while theprotected subgroup has a significant deviation, when we maximize Eq.
Revised: In the OULAD dataset, we see that while the protected subgroup has a significant deviation [Table] when we maximize Eq.

Original: 1 over all features beyondjust the protected ones, we uncover a socially relevant group: students who are not from the highestsocio-economic decile regions of the UK excluding Ireland and Wales that study between 90 - 655credits units.
Revised: 1 over all features beyond just the protected ones, we uncover a socially relevant group: students who are not from the highest socio-economic decile regions of the UK excluding Ireland and Wales that study between 90 - 655 credits units.

Original: This group shows the most evidence that pass or distinction rates are lower than average.
Revised: This group shows the most evidence that pass or distinction rates are lower than average.

Original: It is interesting that while not being from the highest socio-economic decile regions of the UKcoupled with a high course workload makes a student more likely to fail or withdraw from courses,students from Ireland and Wales seem to be protected against this pattern as the odds ratio dropsby 33% from 2.26 to 1.42.
Revised: It is interesting that while not being from the highest socio-economic decile regions of the UK coupled with a high course workload makes a student more likely to fail or withdraw from courses, students from Ireland and Wales seem to be protected against this pattern as the odds ratio drops by 33% from 2.26 to 1.42.

Original: Similarly, while focusing on just protected attributes may have giveninvestigators a false sense of security regarding the extent of systematic deviations in the Credit carddataset, AutoStrat showed that a high systematic deviation exists in this data through an efficientsearch of an exponentially large space.
Revised: Similarly, while focusing on just protected attributes may have given investigators a false sense of security regarding the extent of systematic deviations in the Credit card dataset, AutoStrat showed that a high systematic deviation exists in this data through an efficient search of an exponentially large space.

Original: 5.2 Comparison with other recently proposed subgroup discovery algorithms
Revised: 5.2 Comparison with other recently proposed subgroup discovery algorithms

Original: Following the experimental results from section 5.1 that showed the importance of not limiting oursearch space to only some pre-determined protected attributes, we ran experiments to extract beyondprotected subgroups from the baseline methods (see section 4) and then compare the metrics ofthe extracted subgroups from these algorithms to the extracted subgroups from AutoStrat.Table 4presents
Revised: Following the experimental results from section 5.1 that showed the importance of not limiting our search space to only some pre-determined protected attributes, we ran experiments to extract beyond protected subgroups from the baseline methods (see section 4) and then compare the metrics of the extracted subgroups from these algorithms to the extracted subgroups from AutoStrat.

Original:  the results of the comparative analysis of AutoStrat when compared to these other subgroupdiscovery algorithms.
Revised: Tablepresents the results of the comparative analysis of AutoStrat when compared to these other subgroup discovery algorithms.

Original: There are three key takeaways from these results.
Revised: There are three key takeaways from these results.

Original: First, AutoStrat is ableto consistently find deviations that are the largest across all the datasets, whether that deviation ismeasured by the AutoStrat score, the Weighted Relative Accuracy, or the Odds Ratio (with oneexception).
Revised: First, AutoStrat is able to consistently find deviations that are the largest across all the datasets, whether that deviation is measured by the AutoStrat score, the Weighted Relative Accuracy, or the Odds Ratio (with one exception).

Original: This exception occurred because AutoStrat correctly balances the fact that 20% of thedataset having an odds ratio of 2.26 is a bigger systematic deviation than 0.191% of the datasethaving an odd ratio of 3.05 in the OULAD dataset.

Original: Second, AutoStrat is able to find subsets that thealternative methods would not find because of its ability to support more than one feature value perfeature (logical ORs) in its subgroup description.

Original: [-This is effectively shown in the results for the Creditcard dataset, where the other methods all find the same pretty significant deviation ( PAY _ 2 = 2 ) butmissed a significantly larger deviation that had almost 2x the odds ratio because they did not take intoaccount the OR relationships ( PAY _ 
-]
Revised: [-This is effectively shown in the results for the Creditcard dataset, where the other methods all find the same pretty significant deviation ( PAY _ 2 = 2 ) butmissed a significantly larger deviation that had almost 2x the odds ratio because they did not take intoaccount the OR relationships ( PAY _ 
-]

Original: = 2 OR 3 OR 4 ).

Original: Finally, despite searching through a muchbigger search space (see Table 2, AutoStrat is still competitive in terms of runtime 1 . Unsurprisingly,the heuristic beam search is the fastest. However, it has to be noted that it is not guaranteed to findthe largest deviations. [Table]
Revised: This exception occurred because AutoStrat correctly balances the fact that 20% of the

Original: [Table]

Original: OULAD

Original: [Table]
Revised: [Table]

Original: imd_band=0%
Revised: imd_band=0%

Original: [Table]
Revised: [Table]

Original: 90% AND studied_credits =
Revised: 90% AND studied_credits =

Original: 90OR 120 - 655 AND region =
Revised: 90OR 120 - 655 AND region =

Original: NOT(Ireland OR Wales) 1.
Revised: NOT(Ireland OR Wales) [Table]

Original:  Beam search [25] studied_credits=901.
Revised: [Table] [Table] [Table] Beam search [26] studied_credits=901.

Original: Apriori [25] studied_credits=901.
Revised: Apriori [26] studied_credits=901.

Original: Fp-growth [27] num_prev_attempts=0 AND studied_credits=901. 
Revised: Fp-growth [28] num_prev_attempts=0 AND studied_credits=901. [Table]

Original: Slice Finder [26] region=North Western AND num_prev_attempts= 1.
Revised: Slice Finder [27] region= North Western AND num_prev_attempts= 1.

Original: 1.91E3.
Revised: 1.91E-

Revised: [Table]

Original: Conclusion
Revised: dataset having an odds ratio of 2.26 is a bigger systematic deviation than 0.191% of the dataset having an odd ratio of 3.05 in the OULAD dataset. Second, AutoStrat is able to find subsets that the alternative methods would not find because of its ability to support more than one feature value per feature (logical ORs) in its subgroup description. This is effectively shown in the results for the Credit card dataset, where the other methods all find the same pretty significant deviation ( PAY _ 2 = 2 ) but missed a significantly larger deviation that had almost 2x the odds ratio because they did not take into account the OR relationships ( PAY _ 0 = 2 OR 3 OR 4 ). Finally, despite searching through a much bigger search space (see Table 2), AutoStrat is still competitive in terms of runtime 1 .

Revised: Unsurprisingly, the heuristic beam search is the fastest.

Original:  In this paper, we described AutoStrat - an efficient algorithm for divergent subgroup discovery ina way that correctly balances the significance of the divergence and the size of the subgroup whilesupporting the addition of more than one feature value for a feature in the subgroup description.
Revised: However, it has to be noted that it is not guaranteed to find the largest deviations.  Conclusion In this paper we described AutoStrat - an efficient algorithm for divergent subgroup discovery in a way that correctly balances the significance of the divergence and the size of the subgroup while supporting the addition of more than one feature value for a feature in the subgroup description.

Original: Wethen showed how this approach enables us to find more divergent subgroups across different tasks,some of which we would have missed out on if we only focused on subgroups involving the protectedgroups.
Revised: We then showed how this approach enables us to find more divergent subgroups across different tasks, some of which we would have missed out on if we only focused on subgroups involving the protected groups.

Original: We also compared the results discovered by the described algorithm with other recentlyproposed algorithms.
Revised: We also compared the results discovered by the described algorithm with other recently proposed algorithms.

Original: One limitation of AutoStrat, like the other subgroup discovery algorithms, isthe need to bin continuous features.
Revised: One limitation of AutoStrat, like the other subgroup discovery algorithms, is the need to bin continuous features.

Original: Future works include supporting continuous variables directly.
Revised: Future works include supporting continuous variables directly and applying this approach beyond tabular datasets to unstructured data such as images and text.

Original: Also, while we only focused on the most divergent subgroup in this paper, we would be extending theanalysis to  multiplereturned subgroups in future works.
Revised: Also, while we only focused on the most divergent subgroup in this paper, we would be extending the analysis to include multiple subgroups in future works

Original: Appendix A Baselines
Revised: Appendix A How Size and Extremity of Deviation Trade-off Affects Ranking

Revised: Consider five subgroups - A, B, C, D, and E as described in Table 5. To rank these subgroups, we have to take into account their relative sizes and the extremity of their deviations (i.e. the proportion of target outcome present in the subgroup relative to its size). Subgroup A contains 20% of the data and 30% of the target outcome.

Revised: Subgroup B contains 5% of the data and 15% of the target outcome.

Revised: The relative difference between the proportion of target outcome and relative size is equal for these subgroups, however this does not mean they are equally anomalous because when one takes a look at the relative ratio rather than the relative difference, subgroup B seems to be more extreme.

Original: A.1 Pysubgroup
Revised: Similarly, subgroup D has a higher relative difference and larger size when compared to subgroups B and C but they both have a higher relative ratio. Furthermore, subgroup E has a much higher relative ratio than the other subgroups but it is a very small proportion of the data with a small relative difference. It is not directly clear how to balance all these different factors to rank these subgroups. Thus, there must be a correct mathematical balance between the size of the found subgroup and the extremity of its deviation to correctly rank these subgroups.

Original:  Pysubgroup implements several subgroup discovery search mechanism such as beam Search, Apriori,
Revised: [Table] Appendix B Baselines B.1 Pysubgroup Pysubgroup implements several subgroup discovery search mechanism such as Beam Search, Apriori,

Original: Depth First Search, Best First Search, among others.
Revised: Depth First Search, Best First Search, among others.

Original: There are also several task-specific qualityfunctions also implemented.
Revised: There are also several task-specific quality functions also implemented.

Original: While it automatically balances the size and extent of divergence throughits quality score, the user is still required to pass in a depth parameter that controls how many possiblelogical ANDs can exist in the returned subgroups.
Revised: While it automatically balances the size and extent of divergence through its quality score, the user is still required to pass in a depth parameter that controls how many possible logical ANDs can exist in the returned subgroups.

Original: However, as shown in table 1, it does not supportlogical ORs.
Revised: However, as shown in table 1, it does not support logical ORs.

Original: In this analysis, the depth parameter was set at 5 and the quality function was weightedrelative accuracy (Eq. 5).
Revised: In this analysis, the depth parameter was set at 5 and the quality function was weighted relative accuracy (Eq. 5).

Original: While Eq.
Revised: While Eq.

Original: 5 does try to automatically balance size and significance of the systematic deviation, it islimited by the fact that the simple difference in proportion of outcomes in the subgroup and entirepopulation does not fully correctly capture the deviation.
Revised: 5 does try to automatically balance size and significance of the systematic deviation, it is limited by the fact that the simple difference in proportion of outcomes in the subgroup and entire population does not fully correctly capture the deviation.

Original: For example, consider two subgroups fromtwo datasets with a relative size of 20% but for subgroup 1, the subgroup proportion is 25% vs 5% inthe entire dataset while for subgroup 2, the subgroup proportion is 40% vs 25% in the entire dataset.
Revised: For example, consider two subgroups from two datasets with a relative size of 20% but for subgroup 1, the subgroup proportion is 25% vs 5% in the entire dataset while for subgroup 2, the subgroup proportion is 40% vs 25% in the entire dataset.

Original: Eq.
Revised: Eq.

Original: 5 would assign these two subgroups similar scores despite the fact that subgroup 1 had a 3xincrease in odds ratio vs 2x increase in subgroup 
Revised: 5 would assign these two subgroups the same score despite the fact that subgroup 1 had a 3x increase in odds ratio vs 2x increase in subgroup 2.

Original: A.2 DivExplorer
Revised: B.2 DivExplorer

Original: DivExplorer defines subgroups as a conjuction of logical ANDs.
Revised: DivExplorer defines subgroups as a conjuction of logical ANDs.

Original: The quality score measuresdifferences in metrics such as false positive rate, false negative rate, error rate, and accuracy forsubgroup the entire dataset.
Revised: The quality score measures differences in metrics such as false positive rate, false negative rate, error rate, and accuracy for subgroup the entire dataset.

Original: Candidates are generated using frequent pattern mining techniquesto identify itemsets and thus is focused only on classification tasks.
Revised: Candidates are generated using frequent pattern mining techniques to identify itemsets and thus is focused only on classification tasks.

Original: Continuous values are alsorequired to be binned for candidates to be generated.
Revised: Continuous values are also required to be binned for candidates to be generated.

Original: It does not adopt early stopping heuristics ofstopping search but has a post-processing pruning stage where items whose marginal contributionto divergence of an itemset is lower than some threshold are pruned.
Revised: It does not adopt early stopping heuristics of stopping search but has a post-processing pruning stage where items whose marginal contribution to divergence of an itemset is lower than some threshold are pruned.

Original: User must define a minimum
Revised: User must define a minimum

Original: Support threshold as it does not automatically balance significance and size of divergence.
Revised: Support threshold as it does not automatically balance significance and size of divergence.

Original: In thisanalysis, the minimum Support was set as 0.1 and the metric was accuracy.
Revised: In this analysis, the minimum Support was set as 0.1 and the metric was accuracy.

Original: DivExplorer’s accuracymetric is an unweighted of Eq.
Revised: DivExplorer’s accuracy metric is an unweighted of Eq.

Original: 5 (i.e. we do not multiply by the relative size of the subgroup).
Revised: 5 (i.e. we do not multiply by the relative size of the subgroup).

Original: Thus,
Revised: Thus,

Original: DivExplorer will return subgroups where the relative proportion in the subgroup is much higher thanthe average given that the subgroup has a relative size greater than the minimum Support threshold.
Revised: DivExplorer will return subgroups where the relative proportion in the subgroup is much higher than the average given that the subgroup has a relative size greater than the minimum Support threshold.

Original: A.3 Slice Finder
Revised: B.3 Slice Finder

Original: Slice finder focuses on more interpretable subsets i.e subsets with fewer number of literals.
Revised: Slice finder focuses on more interpretable subsets i.e subsets with fewer number of literals.

Original: Theseslices, are a conjuction of feature-value pairs (i.e logical ANDS only).
Revised: These slices, are a conjuction of feature-value pairs (i.e logical ANDS only).

Original: Continuous values are expectedto be discretized to help numeric feature values to be more sizable and meaningful.
Revised: Continuous values are expected to be discretized to help numeric feature values to be more sizable and meaningful.

Original: Slice finder usesan exhaustive lattice search algorithm in breadth-first mode to search for interesting slices, terminatingsearch when either k significant slices are found or there are no more slices to explore.
Revised: Slice finder uses an exhaustive lattice search algorithm in breadth-first mode to search for interesting slices, terminating search when either k significant slices are found or there are no more slices to explore.

Original: The qualityfunction is an effect size that test the extent of the differences in the loss distributions of a slice andits complement where complement is all records in the dataset that is not in the subgroup(Eq.
Revised: The quality function is an effect size that test the extent of the differences in the loss distributions of a slice and its complement where complement is all records in the dataset that is not in the subgroup(Eq.

Original: Fora classification task, the loss function in Eq.
Revised: For a classification task, the loss function in Eq.

Original: 6 is the logloss and the difference is the KL-divergence.
Revised: 6 is the logloss and the difference is the KL-divergence.

Original: The effect size during subgroup discovery measures the how much the distribution of outcomes inthe subgroup differs from the mean proportion in the dataset.
Revised: The effect size during subgroup discovery measures the how much the distribution of outcomes in the subgroup differs from the mean proportion in the dataset.

Original: The effect size is also used as an earlystopping and post-processing heuristic.
Revised: The effect size is also used as an early stopping and post-processing heuristic.

Original: Thus, the user has to pass in the effect size as a parameter.
Revised: Thus, the user has to pass in the effect size as a parameter.

Original: Additionally, a degree parameter controls the maximum logical ANDs the returned slice can contain is also required.
Revised: Additionally, a degree parameter controls the maximum logical ANDs the returned slice can contain is also required.

Original: In this analysis, the effect size and degree were 0.5 and 5 respectively.
Revised: In this analysis, the effect size and degree were 0.5 and 5 respectively.

Original: Because ofruntime issues, only 10% of the datasets was used for slice finder.
Revised: Because of runtime issues, only 10% of each dataset was used for slice finder.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Where:ψ
Revised: Where:

Original:  S, h ) − ψ
Revised: ψ ( S, h ) − ψ

Original:  S ′ , h ) is difference in loss function between the subgroup and its complement  s , σ 2 s  are variances of losses in  respectively
Revised: ( S ′ , h ) is difference in loss function between the subgroup and its complement σ 2 s , σ 2 s ′ are variances of losses in subgroup & its complement respectively

Revised: Appendix C Importance of Logical ORs

Revised: Supporting logical OR combinations ensures that a subgroup discovery method does not miss important subgroups that could be excluded because of size or extremity of deviation.

Revised: For example, given two subgroups, A and B, with relative sizes of 7% and 5% and descriptions red cats and black cats respectively, these subgroups will be excluded completely in the discovery process if the minimum relative size is set to 10% (DivExplorer requires a minimum relative size to be set) or might not have a deviation high enough to be considered (Slice Finder requires minimum effect size to be set).

Revised: However, a subgroup C that is equivalent to the combination of subgroup A and B through the use of Logical ORs (i.e red or black cats) might be an important subgroup with a high relative size and extremity.

Original:  Appendix B MDScan Pseudo-code
Revised: This type of subset would be missed completely even if multiple rules are returned and then post-processed to include Logical OR combinations as they would not have been returned as interesting subgroups. It is important that Logical OR combinations are a part of the discovery process for these types of subgroups. Similarly, Logical OR combinations generally give more sizeable subgroups which ensures that we find subgroups that are not just statistically significant but are also meaningfully substantial as they are large enough to be considered important for real world use. Appendix D MDScan Pseudo-code

Revised: Algorithm 1: Pseudo-code for Multi-dimensional subset scanning (MDScan) 1 # Input and output definition ; input : Dataset: D = { ( x i , y i ) | i = 1 , 2 , · · · , N } , Set of features:

Original: F =  f 1 , f 2 , · · · , f m , · · · , f M ] 
Revised: F = [ f 1 , f 2 , · · · , f m , · · · , f M ] output : AnomSubset , AnomScore 2 # Initialization ; 3 AnomSubset ← { ; 4 AnomScore ← −∞ UnCheckedF ←

Revised: #Iterate until while