Original: Distribution Preserving Bayesian Coresets using Set Constraints
Revised: Distribution Preserving Bayesian Coresets using Set Constraints

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Bayesian coresets have become of increasing interest recently for providing atheoretically sound, scalable approach to Bayesian inference.
Revised: Bayesian coresets have become of increasing interest recently for providing a theoretically sound, scalable approach to Bayesian inference.

Original: In brief, a coreset isa (weighted)subsample sample of a dataset that approximates the original datasetunder some metric.
Revised: In brief, a coreset is a (weighted) sample of a dataset that approximates the original dataset under some metric.

Original: Bayesian coresets speciﬁcally focus on approximations thatapproximate the posterior distribution.
Revised: Bayesian coresets speciﬁcally focus on approximations that approximate the posterior distribution.

Original: Unfortunately, existing Bayesian coresetapproaches can signiﬁcantly undersample minority subpopulations, leading to alack of distributional robustness.
Revised: Unfortunately, existing Bayesian coreset approaches can signiﬁcantly undersample minority subpopulations, leading to a lack of distributional robustness.

Original: As a remedy, this work extends existing Bayesiancoresets from enforcing sparsity constraints to group-wise sparsity constraints.
Revised: As a remedy, this work extends existing Bayesian coresets from enforcing sparsity constraints to group-wise sparsity constraints.

Original: Weexplore how this approach helps to mitigate distributional vulnerability.
Revised: We explore how this approach helps to mitigate distributional vulnerability.

Original: We furthergeneralize the group constraints to Bayesian coresets with matroid constraints,which may be of independent interest.
Revised: We further generalize the group constraints to Bayesian coresets with matroid constraints, which may be of independent interest.

Original: We present an optimization analysis of theproposed approach, along with an empirical evaluation on benchmark datasets thatsupport our claims.
Revised: We present an optimization analysis of the proposed approach, along with an empirical evaluation on benchmark datasets that support our claims.

Original:  Introduction
Revised:  Introduction

Original: Bayesian coresets have become of interest recently in the artiﬁcial intelligence community forproviding a theoretically sound, scalable approach to Bayesian inference.
Revised: Bayesian coresets have become of interest recently in the artiﬁcial intelligence community for providing a theoretically sound, scalable approach to Bayesian inference.

Original: The main idea of a Bayesiancoresets is to select a weighted subset of the original dataset such that the posterior inference usingthe weighted subset is a close approximation to the posterior inference using the entire dataset.
Revised: The main idea of a Bayesian coresets is to select a weighted subset of the original dataset such that the posterior inference using the weighted subset is a close approximation to the posterior inference using the entire dataset.

Original: If thedesired cardinality of the subset is known beforehand, then we can formulate the task of choosing thesubset as a constrained optimization problem, where the objective function measures the quality ofthe approximation of the subset, and the constraints enforce the cardinality of the subset is as desired.
Revised: If the desired cardinality of the subset is known beforehand, then we can formulate the task of choosing the subset as a constrained optimization problem where the objective function measures the quality of the approximation of the subset and the constraints enforce the cardinality of the subset is as desired.

Original: The sparsity constraints on the weighted subset are not convex, which poses a problem for exactoptimization.
Revised: The sparsity constraints on the weighted subset are not convex, which poses a problem for exact optimization.

Original: Proposed solutions by Campbell & Broderick [3, 2] include a convex relaxation of thesparsity constraint to the l 1 norm so iterative schemes like Frank-Wolfe can be used in a blackboxfashion, and using a local greedy selection to build a sparse solution vector.
Revised: Proposed solutions by Campbell & Broderick [3, 2] include a convex relaxation of the sparsity constraint to the l 1 norm so iterative schemes like Frank-Wolfe can be used in a blackbox fashion, and using a local greedy selection to build a sparse solution vector.

Original: Our work builds on is theresult of Zhang et.
Revised: Our work builds on is the result of Zhang et.

Original: al [7], which tackles the non-convex sparsity constraint directly via an iterativehard thresholding scheme devised by Blumensath and Davies [1].
Revised: al [7], which tackles the non-convex sparsity constraint directly via an iterative hard thresholding scheme devised by Blumensath and Davies [1].

Original: All aforementioned previous works only consider a uniform sparsity constraint, i.e., selecting asubset of size k from the original dataset of size n .
Revised: All aforementioned previous works only consider a uniform sparsity constraint, i.e., selecting a subset of size k from the original dataset of size n .

Original: Regarding distribution shifts, uniform sparsityconstraints have no knowledge of the distribution of the original dataset.
Revised: Regarding distribution shifts, uniform sparsity constraints have no knowledge of the distribution of the original dataset.

Original: This can lead to scenarioswhere the sample distribution is very different from the distribution of the original dataset.
Revised: This can lead to scenarios where the sample distribution is very different from the distribution of the original dataset.

Original: Forexample, say the full dataset consists of disjoint groups with known variability across groups (e.g.,demographic attributes such as age groups).
Revised: For example, say the full dataset consists of disjoint groups with known variability across groups (e.g., demographic attributes such as age groups).

Original: In a data summarization task, one may want the relativeproportions of these groups in the coreset to be roughly equal to the relative proportions of thesegroups in the original dataset.
Revised: In a data summarization task, one may want the relative proportions of these groups in the coreset to be roughly equal to the relative proportions of these groups in the original dataset.

Original: Still, the uniform sparsity constraint does not ensure this, and indeedin some instances, this can lead to undersampling of minority subgroups in the coreset.
Revised: Still, the uniform sparsity constraint does not ensure this, and indeed in some instances, this can lead to undersampling of minority subgroups in the coreset.

Original: Experimentalexamples of this are given in section 5.
Revised: Experimental examples of this are given in section 5.

Original: Contributions.
Revised: Contributions.

Original: More granular control over the selected elements in the representative subset isrequired to solve this sampling distribution shift issue.
Revised: More granular control over the selected elements in the representative subset is required to solve this sampling distribution shift issue.

Original: This is accomplished via more complex setconstraints on the sampled subset.
Revised: This is accomplished via more complex set constraints on the sampled subset.

Original: Speciﬁcally, we can encode the desired sample groups as a setof partition constraints (further details given in section 3).
Revised: Speciﬁcally, we can encode the desired sample groups as a set of partition constraints (further details given in section 3).

Original: Solving the new optimization problemwith the partition constraints is novel contribution.
Revised: Solving the new optimization problem with the partition constraints is novel contribution.

Original: This work further generalizes this result to anysparsity constraint encoded as a matroid – proposing an efﬁcient (weighted) selection procedure usingiterated hard thresholding, which may be of independent interest.
Revised: This work further generalizes this result to any sparsity constraint encoded as a matroid – proposing an efﬁcient (weighted) selection procedure using iterated hard thresholding, which may be of independent interest.

Original: Finally, we show how the ability togeneralize the sparsity constraints can help address sampling distribution shifts, with applications inalgorithmic fairness.
Revised: Finally, we show how the ability to generalize the sparsity constraints can help address sampling distribution shifts, with applications in algorithmic fairness.

Original:  Preliminaries
Revised:  Preliminaries

Original: Given a probability mass function p , which depends on parameters θ , the liklihood of a randomvariable x is the probability x was sampled from p with parameters θ .
Revised: Given a probability mass function p , which depends on parameters θ , the liklihood of a random variable x is the probability x was sampled from p with parameters θ .

Original: We will assume we aregiven n samples, where L i ( θ ) is the log-liklihood of the i -th sample parameterized by θ .
Revised: We will assume we are given n samples, where L i ( θ ) is the log-liklihood of the i -th sample parameterized by θ .

Original: Assumingconditional independence with respect to θ , one can represent the log-likliehood of observing allsamples as a summation of the individual log-liklihoods, i.e., L ( θ )
Revised: Assuming conditional independence with respect to θ , one can represent the log-likliehood of observing all samples as a summation of the individual log-liklihoods, i.e., L ( θ )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =

Revised: =

Original: L i ( θ ) .
Revised: L i ( θ ) .

Original: The goal of a
Revised: The goal of a

Original: Bayesian coreset is to approximate the full liklihood of all observations with a weighted sample,i.e., L w = (cid:80) ni
Revised: Bayesian coreset is to approximate the full liklihood of all observations with a weighted sample, i.e., L w = (cid:80) ni

Original: =1 w i L i , where w ∈ R n + is a non-negative sparse vector.
Revised: =1 w i L i , where w ∈ R n + is a non-negative sparse vector.

Original: We will use Dist ( . (cid:107) . ) torepresent the distance functional that measures the deviation between L and L w and we will mostlyconsider the case where Dist ( .
Revised: We will use Dist ( . (cid:107) . ) to represent the distance functional that measures the deviation between L and L w and we will mostly consider the case where Dist ( .

Original: (cid:107) . )
Revised: (cid:107) . )

Original: is the 2 -norm deﬁned in the function space.
Revised: is the 2 -norm deﬁned in the function space.

Original: In this setting, it isuseful to view L , L i , and L w as functions in a Hilbert space.
Revised: In this setting, it is useful to view L , L i , and L w as functions in a Hilbert space.

Original: Additionally, we will use the notation[ n ] as a shorthand for { 1 , 2 , 3 , ..., n } , the set of integers from 1 to n .
Revised: Additionally, we will use the notation [ n ] as a shorthand for { 1 , 2 , 3 , ..., n } , the set of integers from 1 to n .

Original: 2.1 Existing Approach: Standard Sparsity Problem
Revised: 2.1 Existing Approach: Standard Sparsity Problem

Original: As deﬁned by [7], the sparsity constrained Bayesian Coreset problem can be formulated as followsargmin
Revised: As deﬁned by [7], the sparsity constrained Bayesian Coreset problem can be formulated as follows

Original: [Equation]
Revised: [Equation]

Original: subject to || w || 0 ≤ k w i ≥∀ i ∈ [ n ] .

Original: Where L = (cid:80) ni
Revised: Where L = (cid:80) ni

Original: =1 L i , i.e., the sum of the log likelihoods of our n samples and L w = (cid:80) ni
Revised: =1 L i , i.e., the sum of the log likelihoods of our n samples and L w = (cid:80) ni

Original: =1 w i L i ,the likelihood of the sample with respect to w .
Revised: =1 w i L i , the likelihood of the sample with respect to w .

Original: Note the k selected points need not be representativeof the distribution of full dataset, which leads to the sampling induced distribution shifts.
Revised: Note the k selected points need not be representative of the distribution of full dataset, which leads to the sampling induced distribution shifts.

Original: We willrefer to this problem formulation as the “standard sparsity problem” for the remainder of this work.
Revised: We will refer to this problem formulation as the “standard sparsity problem” for the remainder of this work.

Original: 2.2 Finite Sample Estimation
Revised: 2.2 Finite Sample Estimation

Original: Given that Dist ( .
Revised: Given that Dist ( .

Original: (cid:107) . )
Revised: (cid:107) . )

Original: is the 2 -norm deﬁned in the function space, the objective function can beexpanded as follows
Revised: is the 2 -norm deﬁned in the function space, the objective function can be expanded as follows

Original: [Equation]
Revised: [Equation]

Original: Where π is the distribution of the parameters θ .
Revised: Where π is the distribution of the parameters θ .

Original: To compute this expectation exactly, one wouldneed to integrate over all possible parameter settings, which is clearly intractable.
Revised: To compute this expectation exactly, one would need to integrate over all possible parameter settings, which is clearly intractable.

Original: Luckily, theexpectation can be approximated by a ﬁnite dimensional L 2 norm by replacing the functions L and
Revised: Luckily, the expectation can be approximated by a ﬁnite dimensional L 2 norm by replacing the functions L and

Original: L w with vectors of sampled evaluations, essentially computing a Monte Carlo approximation of theexpectation.
Revised: L w with vectors of sampled evaluations, essentially computing a Monte Carlo approximation of the expectation.

Original: Further details are given in Zhang et.
Revised: Further details are given in Zhang et.

Original: al
Revised: al

Original: 2.3 Solution to Standard Sparsity Problem
Revised: 2.3 Solution to Standard Sparsity Problem

Original: Notice that in both problem formulations, the sparsity constraint causes the overall optimizationproblem to become non-convex.
Revised: Notice that in both problem formulations, the sparsity constraint causes the overall optimization problem to become non-convex.

Original: Therefore, it is not unreasonable to shift one’s focus from ananalytical solution to an approximation computed via some iterative method.
Revised: Therefore, it is not unreasonable to shift one’s focus from an analytical solution to an approximation computed via some iterative method.

Original: This is the solutionpresented by Zhang et.
Revised: This is the solution presented by Zhang et.

Original: al [7], where an iterative approach to approximate the optimal vector w is proposed.
Revised: al [7], where an iterative approach to approximate the optimal vector w is proposed.

Original: Speciﬁcally, the authors adapt accelerated iterative hard thresholding (IHT) schemes to the
Revised: Speciﬁcally, the authors adapt accelerated iterative hard thresholding (IHT) schemes to the

Original: Bayesian coreset problem.
Revised: Bayesian coreset problem.

Original: The classical IHT algorithm of Blumensath and Davies [1] is given below
Revised: The classical IHT algorithm of Blumensath and Davies [1] is given below

Original: Algorithm 
Revised: Algorithm 1: Vanilla IHT

Revised: Input : Objective f : R n → R ; sparsity k , step size µ 1 Initialize w 2 repeat w ← Π C k ∩ R + ( w − µ ∇ f ( w )) ; 4 until stop criteria

Original: Zhang et.
Revised: Zhang et.

Original: al [7] modify the vanilla IHT algorithm by adding a line search for picking the step size µin each iteration, as well as adding a momentum term to accelerate the rate of convergence, but thesechanges are more directed at improving the empirical performance of the algorithm, theoreticallyapplying the vanilla IHT algorithm to the standard sparsity problem produces an arbitrarily goodapproximation, under certain regularity conditions.
Revised: al [7] modify the vanilla IHT algorithm by adding a line search for picking the step size µ in each iteration, as well as adding a momentum term to accelerate the rate of convergence, but these changes are more directed at improving the empirical performance of the algorithm, theoretically applying the vanilla IHT algorithm to the standard sparsity problem produces an arbitrarily good approximation, under certain regularity conditions.

Original:  Proposed Approach: Group Sparsity Problem
Revised:  Proposed Approach: Group Sparsity Problem

Original: To make our sampling more robust to distribution shifts across groups, we encode the distributionof the full dataset as a set of partition constraints on the set of feasible samples.
Revised: To make our sampling more robust to distribution shifts across groups, we encode the distribution of the full dataset as a set of partition constraints on the set of feasible samples.

Original: Speciﬁcally, let usconsider the set of input samples D to be a collection of m disjoint sets, such that D = (cid:83) mi
Revised: Speciﬁcally, let us consider the set of input samples D to be a collection of m disjoint sets, such that D = (cid:83) mi

Original: =1 D m and
Revised: =1 D m and

Original: D i (cid:84)
Revised: D i (cid:84)

Original: D j = ∅ ∀ i, j ∈ [ m ] .
Revised: D j = ∅ ∀ i, j ∈ [ m ] .

Original: Since we want our sample of to be representative of this initial structureto avoid a sample induced distribution shift, we add some additional constraints to ensure the sampledoes not exceed a certain number of points from each subset.
Revised: Since we want our sample of to be representative of this initial structure to avoid a sample induced distribution shift, we add some additional constraints to ensure the sample does not exceed a certain number of points from each subset.

Original: Therefore we can modify the originalconstrained optimization problem to the following, which will be the main focus of this work.
Revised: Therefore we can modify the original constrained optimization problem to the following, which will be the main focus of this work.

Original: [Equation]
Revised: [Equation]

Original: Where w j ∈ R | D j | , and w is the concatenation of { w 1 , w 2 , ... , w m } .
Revised: Where w j ∈ R | D j | , and w is the concatenation of { w 1 , w 2 , ... , w m } .

Original: In other words w j is the vectorthat corresponds to the sampling of points from D j .
Revised: In other words w j is the vector that corresponds to the sampling of points from D j .

Original: Also, note that we are assuming (cid:80) 
Revised: Also, note that we are assuming (cid:80) mj

Original: [Equation] ,so the constraint in the original formulation of || w || 0 ≤ k is redundant since || w j || 0 ≤ k j ∀ j ∈
Revised: =1 k j = k , so the constraint in the original formulation of || w || 0 ≤ k is redundant since || w j || 0 ≤ k j ∀ j ∈

Original: [ m ]implies the former constraint.
Revised: [ m ] implies the former constraint.

Original: We will refer to this problem formulation as the “group sparsityproblem” for the remainder of this work.
Revised: We will refer to this problem formulation as the “group sparsity problem” for the remainder of this work.

Original: 3.1 Projection Step
Revised: 3.1 Projection Step

Original: A key step in the IHT algorithm is the projection step (step 3 of Algorithm 1 ), in which the currentiterate is projected onto the subspace spanned by the constraint.
Revised: A key step in the IHT algorithm is the projection step (step 3 of Algorithm 1 ), in which the current iterate is projected onto the subspace spanned by the constraint.

Original: We show that for the case of groupsparsity constraints, the projection step can be computed exactly in polynomial time.
Revised: We show that for the case of group sparsity constraints, the projection step can be computed exactly in polynomial time.

Original: Further detailsand proof are given in the Appendix A, and the projection is computed by greedily selecting thelargest non-negative k j entries for each j ∈ [ m ] .
Revised: Further details and proof are given in the Appendix A, and the projection is computed by greedily selecting the largest non-negative k j entries for each j ∈ [ m ] .

Original: It is interesting to note that the modiﬁcation of  the 
Revised: We will now consider more general setting where we can encode our constraints as a matroid ( N, E ) , where N is the ground set of indices corresponding to samples, and E are the subsets of indices which satisfy the given constraint. For example, in the original bayesian coreset problem on n samples,

Original: IHT projection step generalizes to any constraint that can be encoded as a matroid.
Revised: N = [ n ] and E = { S ⊂ N s.t | S | ≤ k } , i.e., the index set corresponding to the subsets of samples we could choose to satisfy the sparsity constraint.

Original: For the specialcase of the original bayesian coreset problem, the resulting matroid corresponds to the uniformmatroid, and in the aforementioned new problem setting where we have some initial structure fromthe input we would like to consider, the resulting matroid is the partition matroid.
Revised: For the special case of the original bayesian coreset problem, the resulting matroid corresponds to the uniform matroid, and in the aforementioned new problem setting where we have some initial structure from the input we would like to consider, the resulting matroid is the partition matroid.

Original: Further are detailsand proofs are given in the Appendix B, and a proof of convergence is given in the Appendix C.
Revised: Further are details and proofs are given in the Appendix B, and a proof of convergence is given in the Appendix C.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Given a matroid constraint ( N, E ) , where N is a ground set of indices correspondingto input samples, in the case of the Euclidean distance metric, the projection step of IHT can becomputed exactly in polynomial time via a greedy selection algorithm.
Revised: Given a matroid constraint ( N, E ) , where N is a ground set of indices corresponding to input samples, in the case of the Euclidean distance metric, the projection step of IHT can be computed exactly in polynomial time via a greedy selection algorithm.

Original:  Connections Between Algorithmic Fairness and Distribution Shifts
Revised:  Connections Between Algorithmic Fairness and Distribution Shifts

Original: Characterizing what exactly it means for an algorithm to be fair is an open and active area of researchwithout a general consensus, and in general a notion of fairness that makes sense in one situationmay not make sense in another.
Revised: Characterizing what exactly it means for an algorithm to be fair is an open and active area of research without a general consensus, and in general a notion of fairness that makes sense in one situation may not make sense in another.

Original: The notion of fairness we employ is the notion of balance [4].
Revised: The notion of fairness we employ is the notion of balance [4].

Original: Saywe have an initial dataset D of size n , and a sensitive feature with c possible value settings.
Revised: Say we have an initial dataset D of size n , and a sensitive feature with c possible value settings.

Original: For all i ∈ c , let D i be the elements from the dataset which have a value of i for the sensitive feature.
Revised: For all i ∈ c , let D i be the elements from the dataset which have a value of i for the sensitive feature.

Original: We will assume D i ∩ D j = ∅ ∀ i, j ∈ { c × c } , i.e., partitioning the dataset based on the sensitivefeature produces disjoint sets.
Revised: We will assume D i ∩ D j = ∅ ∀ i, j ∈ { c × c } , i.e., partitioning the dataset based on the sensitive feature produces disjoint sets.

Original: We say a sample of data points S is balanced with respect to a sensitivefeature if (cid:106)
Revised: We say a sample of data points S is balanced with respect to a sensitive feature if (cid:106)

Original: | D i | n (cid:107) ≤ | S
Revised: | D i | n (cid:107) ≤ | S

Original: ∩
Revised: ∩

Original: D
Revised: D

Original: i | | S | ≤ (cid:108) | D i | n (cid:109) holds for all i ∈ c .
Revised: i | | S | ≤ (cid:108) | D i | n (cid:109) holds for all i ∈ c .

Original: In other words, the relative proportions ofthe data with respect to the sensitive feature are preserved.
Revised: In other words, the relative proportions of the data with respect to the sensitive feature are preserved.

Original: It is easy to see that if the distribution ofthe sample with respect to the sensitive feature is not preserved, then the resulting sample S is notbalanced, and thus over represents certain subgroups, while under representing others.
Revised: It is easy to see that if the distribution of the sample with respect to the sensitive feature is not preserved, then the resulting sample S is not balanced, and thus over represents certain subgroups, while under representing others.

Original: Therefore,distribution robust sampling is inherently tied to algorithmic fairness via the notion of balance, and byusing the distribution preserving sampling methods proposed here, one can prevent bias in samplinginduced by distribution shifts.
Revised: Therefore, distribution robust sampling is inherently tied to algorithmic fairness via the notion of balance, and by using the distribution preserving sampling methods proposed here, one can prevent bias in sampling induced by distribution shifts.

Original:  Experimental Results
Revised:  Experimental Results

Original: We now move our attention to empirical performance metrics of our proposed algorithm.
Revised: We now move our attention to empirical performance metrics of our proposed algorithm.

Original: Theﬁrst experiment highlights a failure case using just the original uniform sparsity constraint, whichelucidates the distribution shifts that can be caused by such constraints.
Revised: The ﬁrst experiment highlights a failure case using just the original uniform sparsity constraint, which elucidates the distribution shifts that can be caused by such constraints.

Original: We construct a syntheticdataset of size N = 1000 , partitioned into 2 groups denoted by the orange and blue bars.
Revised: We construct a synthetic dataset of size N = 1000 , partitioned into 2 groups denoted by the orange and blue bars.

Original: Exactdetails on the construction of the synthetic dataset are given in the Appendix D. The left set of barsrepresent the true distribution of the dataset, the middle set of bars are the distribution of the sampleselected using the uniform sparsity constraint, and the right set of bars represent the distributionof the sample selected using the group sparsity constraint.
Revised: Exact details on the construction of the synthetic dataset are given in the Appendix D. The left set of bars represent the true distribution of the dataset, the middle set of bars are the distribution of the sample selected using the uniform sparsity constraint, and the right set of bars represent the distribution of the sample selected using the group sparsity constraint.

Original: Note that the distribution of the sampleselected using the group sparsity constraint matches the true distribution exactly, as the group sparsityconstraints allow us to control the sample distribution exactly.
Revised: Note that the distribution of the sample selected using the group sparsity constraint matches the true distribution exactly, as the group sparsity constraints allow us to control the sample distribution exactly.

Original: The results shown are averaged across10 trials and it is important to note that across all trials, none of the data in group 1 was selected to bethe sample using only the uniform sparsity constraint.
Revised: The results shown are averaged across 10 trials and it is important to note that across all trials, none of the data in group 1 was selected to be the sample using only the uniform sparsity constraint.

Original: [Figure]
Revised: [Figure]

Original: One might think of a hueristic method to approximate the solution to the group sparsity problem bysolving the uniform sparse selection for each group separately, and then concatenating the results toconstruct the ﬁnal sample - in fact this hueristic is widely used in practice when trying to enforcegroup sparsity constraints.
Revised: One might think of a hueristic method to approximate the solution to the group sparsity problem by solving the uniform sparse selection for each group separately, and then concatenating the results to construct the ﬁnal sample - in fact this hueristic is widely used in practice when trying to enforce group sparsity constraints.

Original: We  that  algorithm empirically outperforms  huerisitic 
Revised: We show that our algorithm empirically outperforms the huerisitic method by ﬁnding a solution with a lower objective value in the same number of iterations. The

Original:  Conclusion
Revised:  Conclusion

Original: In this work we have highlighted a particular type of distribution shift caused by sampling, and shownthat these types of distribution shifts relate to existing notion of algorithmic fairness.
Revised: In this work we have highlighted a particular type of distribution shift caused by sampling, and shown that these types of distribution shifts relate to existing notion of algorithmic fairness.

Original: We present asolution to the problem of sampling induced distribution shifts and show the algorithm is theoreticallysound, and performs well in practice.
Revised: We present a solution to the problem of sampling induced distribution shifts and show the algorithm is theoretically sound, and performs well in practice.

Original: This is an ongoing work, and we hope to show empirical resultson real-world datasets, as well as comparisons with other methods such as [5, 6] which also claim toimprove group robustness, with (in our case) applications to fairness.
Revised: This is an ongoing work, and we hope to show empirical results on real-world datasets, as well as comparisons with other methods such as [5, 6] which also claim to improve group robustness, with (in our case) applications to fairness.

Original: A Projection Proof
Revised: A Projection Proof

Original: Proof: A key step in the IHT algorithm is the projection step, in which the current iterate is projectedonto the subspace spanned by the constraint.
Revised: Proof: A key step in the IHT algorithm is the projection step, in which the current iterate is projected onto the subspace spanned by the constraint.

Original: Here we prove that for the case of group sparsityconstraints, the projection onto the subspace spanned by the constraints can be computed exactlyin polynomial time.
Revised: Here we prove that for the case of group sparsity constraints, the projection onto the subspace spanned by the constraints can be computed exactly in polynomial time.

Original: Let us denote this subspace as C k .
Revised: Let us denote this subspace as C k .

Original: Formally, let W j = { w j s.t (cid:107) w j (cid:107) 0 ≤ k j } .
Revised: Formally, let W j = { w j s.t (cid:107) w j (cid:107) 0 ≤ k j } .

Original: Then we can write C k
Revised: Then we can write C k

Original: = [ w 1 ∈ W 1 , w 2 ∈ W 2 , . . .
Revised: = [ w 1 ∈ W 1 , w 2 ∈ W 2 , . . .

Original: , w m ∈ W m ] , where w i is an arbitrary vector fromthe set W i .
Revised: , w m ∈ W m ] , where w i is an arbitrary vector from the set W i .

Original: Like the case in [7], we have the additional positivity constraint, so the subspace spannedby all of our constraints is actually ( C k ∩ R n + ) .
Revised: Like the case in [7], we have the additional positivity constraint, so the subspace spanned by all of our constraints is actually ( C k ∩ R n + ) .

Original: Unlike the case in [8], computing Π C k ∩ R n + ( w ) , theprojection of w onto the subspace spanned by the group sparsity constraints, is computationallyeasy in the case of the Euclidean distance metric.
Revised: Unlike the case in [8], computing Π C k ∩ R n + ( w ) , the projection of w onto the subspace spanned by the group sparsity constraints, is computationally easy in the case of the Euclidean distance metric.

Original: Take an arbitrary vector w (cid:48) ∈ ( C k ∩ R n + ) , and let
Revised: Take an arbitrary vector w (cid:48) ∈ ( C k ∩ R n + ) , and let

Original: S = supp ( w (cid:48) ) , i.e., the set of non-zero indices of w (cid:48) .
Revised: S = supp ( w (cid:48) ) , i.e., the set of non-zero indices of w (cid:48) .

Original: We can write (cid:107) w (cid:48) − w (cid:107) 

Revised: We can write (cid:107) w (cid:48) − w (cid:107)

Original: = (cid:107) w (cid:107) − (cid:80) i ∈
Revised: = (cid:107) w (cid:107) − (cid:80) i ∈

Original: S w 

Revised: S w

Original: i ,so to minimize this distance, we must maximize (cid:80) i ∈ S w 2 i , which can be done by simply picking Sto correspond to the largest non-negative k j entries for each j ∈ [ m ] .
Revised: i , so to minimize this distance, we must maximize (cid:80) i ∈ S w 2 i , which can be done by simply picking S to correspond to the largest non-negative k j entries for each j ∈ [ m ] .

Original: B Matroid Extension
Revised: B Matroid Extension

Original: We will now consider more general setting where we can encode our constraints as a matroid ( N, E ) ,where N is the ground set of indices corresponding to samples, and E are the subsets of indices whichsatisfy the given constraint.
Revised: We will now consider more general setting where we can encode our constraints as a matroid ( N, E ) , where N is the ground set of indices corresponding to samples, and E are the subsets of indices which satisfy the given constraint.

Original: For example, in the original bayesian coreset problem on n samples,
Revised: For example, in the original bayesian coreset problem on n samples,

Original: N = [ n ] and E = { S ⊂ N s.t | S | ≤ k } , i.e., the index set corresponding to the subsets of sampleswe could choose to satisfy the sparsity constraint.
Revised: N = [ n ] and E = { S ⊂ N s.t | S | ≤ k } , i.e., the index set corresponding to the subsets of samples we could choose to satisfy the sparsity constraint.

Original: For the special case of the original bayesian coresetproblem, the resulting matroid corresponds to the uniform matroid, and in the aforementioned newproblem setting where we have some initial structure from the input we would like to consider, theresulting matroid is the partition matroid.
Revised: For the special case of the original bayesian coreset problem, the resulting matroid corresponds to the uniform matroid, and in the aforementioned new problem setting where we have some initial structure from the input we would like to consider, the resulting matroid is the partition matroid.

Original: B.0.1 Proof of Lemma 3. 
Revised: B.0.1 Proof of Lemma 3.

Original: Proof :
Revised: Proof :

Original: In our setting, given a matroid ( N, E ) , we will consider the ground set N = [ n ] , where nis the total number of samples, and thus E is a set of subsets of indices, which correspond to inputsamples which satisfy some notion of independence.
Revised: In our setting, given a matroid ( N, E ) , we will consider the ground set N = [ n ] , where n is the total number of samples, and thus E is a set of subsets of indices, which correspond to input samples which satisfy some notion of independence.

Original: Consider an arbitrary vector w in the ambientspace, an arbitrary constraint e ∈ E , and a vector w (cid:48) which is the projection of w onto the constraintdenoted by e , so that supp ( w (cid:48) )
Revised: Consider an arbitrary vector w in the ambient space, an arbitrary constraint e ∈ E , and a vector w (cid:48) which is the projection of w onto the constraint denoted by e , so that supp ( w (cid:48) )

Original: =
Revised: =

Original: e .
Revised: e .

Original: Since e denotes some sparsity constraint, we know the optimalprojection can be computed as w (cid:48)
Revised: Since e denotes some sparsity constraint, we know the optimal projection can be computed as w (cid:48)

Original: i =
Revised: i =

Original: w i if i ∈ e and w i ≥ 0 , else w (cid:48)
Revised: w i if i ∈ e and w i ≥ 0 , else w (cid:48)

Original: i = 0 .
Revised: i = 0 .

Original: Under the Euclideanmetric, we can compute the projection cost as (cid:107) w (cid:48)
Revised: Under the Euclidean metric, we can compute the projection cost as (cid:107) w (cid:48)

Original: − w (cid:107) 

Revised: − w (cid:107)

Original: = (cid:107) w (cid:107) 22 − (cid:80) i ∈
Revised: = (cid:107) w (cid:107) 22 − (cid:80) i ∈

Original: e w 2 i , and since w is ﬁxed,in order to minimize the cost of the projection, we must maximize the second term in the projectioncost.
Revised: e w 2 i , and since w is ﬁxed, in order to minimize the cost of the projection, we must maximize the second term in the projection cost.

Original: Therefore, the projection step can be formulated as max e ∈ E (cid:80) i ∈
Revised: Therefore, the projection step can be formulated as max e ∈ E (cid:80) i ∈

Original: e w 2 i .
Revised: e w 2 i .

Original: In this form it is easy tosee that the projection step reduces to maximizing a modular function subject to a matroid constraint.
Revised: In this form it is easy to see that the projection step reduces to maximizing a modular function subject to a matroid constraint.

Original: It is known that the solution can be computed exactly via a greedy selection algorithm, since weare simply looking for the maximum cost base of the matroid.
Revised: It is known that the solution can be computed exactly via a greedy selection algorithm, since we are simply looking for the maximum cost base of the matroid.

Original: We simply choose the index whichgives the maximum gain in the objective function, while still satisfying the independence constraints,and continue until we have a maximal base.
Revised: We simply choose the index which gives the maximum gain in the objective function, while still satisfying the independence constraints, and continue until we have a maximal base.

Original: Note that when our matroid is the uniform matroid, thegreedy algorithm for choosing the maximum cost base reduces to the algorithm for the IHT projectionstep given in [7].
Revised: Note that when our matroid is the uniform matroid, the greedy algorithm for choosing the maximum cost base reduces to the algorithm for the IHT projection step given in [7].

Original: The same is true for the case of the partition matroid.
Revised: The same is true for the case of the partition matroid.

Original: C Convergence Analysis of IHT with Matroid constraints
Revised: C Convergence Analysis of IHT with Matroid constraints

Original: For the purposes of this analysis, we will not consider the use of any momentum terms, i.e., the stepsize µ is constant.
Revised: For the purposes of this analysis, we will not consider the use of any momentum terms, i.e., the step size µ is constant.

Original: The IHT algorithm in our setting, without any momentum terms is given below
Revised: The IHT algorithm in our setting, without any momentum terms is given below

Original: Algorithm 2: 
Revised: Algorithm 2: Vanilla IHT

Original: Input : Objective f : R n → R ; sparsity k , step size µ 1 Initialize w 2 repeat w ← Π C k ∩ R + ( w − µ ∇ f ( w )) ; 4 until stop criteria

Original: To begin the analysis we start with necessary deﬁnitions
Revised: To begin the analysis we start with necessary deﬁnitions

Original: Deﬁnition C.0.1 (Restricted Isometry Property) .
Revised: Deﬁnition C.0.1 (Restricted Isometry Property) .

Original: For each integer s ∈ N , the isometry constant δ s ofa matrix Φ is the smallest number such that
Revised: For each integer s ∈ N , the isometry constant δ s of a matrix Φ is the smallest number such that

Original: [Equation]
Revised: [Equation]

Original: for all s -sparse vectors x
Revised: for all s -sparse vectors x

Original: In other words, Φ is close to an isometric transformation for sparse vectors.
Revised: In other words, Φ is close to an isometric transformation for sparse vectors.

Original: Another useful observation is that for any set S ⊆
Revised: Another useful observation is that for any set S ⊆

Original: [ n ] , the projection operator Π S is a selectionmatrix, i.e., we can write
Revised: [ n ] , the projection operator Π S is a selection matrix, i.e., we can write

Original: [Equation]
Revised: [Equation]

Original: where δ i is an indicator function which equals 1 if i ∈ S , and 0 otherwise.
Revised: where δ i is an indicator function which equals 1 if i ∈ S , and 0 otherwise.

Original: This holds regardless ofthe speciﬁc constraints, which is useful since the constraint matroid represents a family of constraints.
Revised: This holds regardless of the speciﬁc constraints, which is useful since the constraint matroid represents a family of constraints.

Original: Lemma C.1.
Revised: Lemma C.1.

Original: If Φ satisﬁes the RIP assumption, then given a set S ⊆
Revised: If Φ satisﬁes the RIP assumption, then given a set S ⊆

Original: [ N ] with | S | ≤ k , then∀ w ∈ R n , the following holds
Revised: [ N ] with | S | ≤ k , then ∀ w ∈ R n , the following holds

Original: [Equation]
Revised: [Equation]

Original: Proof: Recall that Π S is a projection operator onto the index set S , so if | S | ≤ k , then Π S w hassparsity k for all w ∈ R n .
Revised: Proof: Recall that Π S is a projection operator onto the index set S , so if | S | ≤ k , then Π S w has sparsity k for all w ∈ R n .

Original: Given Φ ∈ R m × n satisﬁes the RIP assumption, we have
Revised: Given Φ ∈ R m × n satisﬁes the RIP assumption, we have

Original: [Equation]
Revised: [Equation]

Original: Denoting b = ΦΠ S w , X
Revised: Denoting b = ΦΠ S w , X

Original: = { x ∈ R n s.t (cid:107) x (cid:107) 2 = 1 } , and (cid:104) ., .
Revised: = { x ∈ R n s.t (cid:107) x (cid:107) 2 = 1 } , and (cid:104) ., .

Original: (cid:105) as the Euclidean inner product we canwrite
Revised: (cid:105) as the Euclidean inner product we can write

Original: [Equation]
Revised: [Equation]

Original: Where the second equality holds because Π S is a symmetric matrix, and the fact that , b (cid:105) = a T b .
Revised: Where the second equality holds because Π S is a symmetric matrix, and the fact that , b (cid:105) = a T b .

Original: Letx ∗ be the optimal value which maximizes dot product.
Revised: Let x ∗ be the optimal value which maximizes dot product.

Original: Using the Cauchy-Schwartz inequality, wecan show an upper bound
Revised: Using the Cauchy-Schwartz inequality, we can show an upper bound

Original: [Equation]
Revised: [Equation]

Original: We can also show a lower bound by computing the dot product for a speciﬁc value of x .
Revised: We can also show a lower bound by computing the dot product for a speciﬁc value of x .

Original: In particular,let x (cid:48)
Revised: In particular, let x (cid:48)

Original: =
Revised: =

Revised: ΦΠ S

Original:  ΦΠ S w (cid:107) ΦΠ S w (cid:107) 2 , the normalized form of the vector ΦΠ S w .
Revised: [Equation] ΦΠ S w [Equation] 2 , the normalized form of the vector ΦΠ S w .

Original: [Equation]
Revised: [Equation]

Original: Where the last equality is due to the fact that ΦΠ S w and ΦΠ S x (cid:48) are parallel.
Revised: Where the last equality is due to the fact that ΦΠ S w and ΦΠ S x (cid:48) are parallel.

Original: By the bounds assumedby RIP we get
Revised: By the bounds assumed by RIP we get

Original: [Equation]
Revised: [Equation]

Revised: 22 ≥ α k

Original: [Table]
Revised: [Table]

Original: w
Revised: Π

Original: [Table]
Revised: [Table]

Original: 22 × [Table]
Revised: w

Original: [Table]
Revised: [Table]

Original:  k
Revised: 22 × α k

Original: [Table]
Revised: [Table]

Original: Recall that x ∗ is a unit vector, and Π S modiﬁes vectors by setting speciﬁc indices to 0 , and leavingthe rest of the vector unchanged.
Revised: Recall that x ∗ is a unit vector, and Π S modiﬁes vectors by setting speciﬁc indices to 0 , and leaving the rest of the vector unchanged.

Original: Thus (cid:107) Π s x ∗ (cid:107) 22 ≤ (cid:107) x ∗ (cid:107) 22 ≤ 1 .
Revised: Thus (cid:107) Π s x ∗ (cid:107) 22 ≤ (cid:107) x ∗ (cid:107) 22 ≤ 1 .

Original: Also recall that x (cid:48)
Revised: Also recall that x (cid:48)

Original: =
Revised: =

Original: ΦΠ S w (cid:107) ΦΠ S w (cid:107) 2 , so x (cid:48)is already sparse.
Revised: ΦΠ S w (cid:107) ΦΠ S w (cid:107) 2 , so x (cid:48) is already sparse.

Original: Thus (cid:107) Π S x (cid:48)
Revised: Thus (cid:107) Π S x (cid:48)

Original: (cid:107) 

Revised: (cid:107)

Original: = (cid:107) x (cid:48) (cid:107) 22 = 1 .
Revised: = (cid:107) x (cid:48) (cid:107) 22 = 1 .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Revised: 22 ×

Revised: [Equation]

Original: ΦΠ S w
Revised: ΦΠ S x

Original: [Equation]
Revised: [Equation]

Original: 22 ×
Revised: 22 ≤

Original: [Equation]
Revised: [Equation]

Original: ΦΠ S x
Revised: ΦΠ S w

Original: [Equation]
Revised: [Equation]

Original: [-(cid:107) 22 ≤ (cid:107) ΦΠ S w (cid:107) 
-]
Revised: [-(cid:107) 22 ≤ (cid:107) ΦΠ S w (cid:107) 
-]

Original:  × (cid:107) ΦΠ S x ∗ (cid:107) 22 by the optimality of x ∗ .
Revised: 22 × (cid:107) ΦΠ S x ∗ (cid:107) 22 by the optimality of x ∗ .

Original: Putting theinequalities together, and taking the square root gives the desired bound
Revised: Putting the inequalities together, and taking the square root gives the desired bound

Original: [Equation]
Revised: [Equation]

Original: This gives us a bound on the eigenvalues of Π S Φ T ΦΠ S , which we will need need in the proof of theiterative invariant bound.
Revised: This gives us a bound on the eigenvalues of Π S Φ T ΦΠ S , which we will need need in the proof of the iterative invariant bound.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Given Φ which satisﬁes the RIP assumption, S 1 , S 2 ⊆ [ n ] such that | S 1 ∪ S 2 | ≤ k ,then ∀ w ∈ R n , the following inequality holds
Revised: Given Φ which satisﬁes the RIP assumption, S 1 , S 2 ⊆ [ n ] such that | S 1 ∪ S 2 | ≤ k , then ∀ w ∈ R n , the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: Similar to the proof of the previous lemma, we will begin by rewriting the norm as an inner product.
Revised: Similar to the proof of the previous lemma, we will begin by rewriting the norm as an inner product.

Original: Let X ={ x ∈ R n s.t (cid:107) x (cid:107) 2 = 1 }
Revised: Let X =

Original: [Equation]
Revised: [Equation]

Original: which is a valid equality since Π S 1 is symmetric.
Revised: which is a valid equality since Π S 1 is symmetric.

Original: Note that we are examining the dot product of 2vectors projected into the subspace of Φ .
Revised: Note that we are examining the dot product ofvectors projected into the subspace of Φ .

Original: Let us deﬁne 2 normalized vectors, which will denote thevectors before their projection by into the subspace deﬁned by Φ .
Revised: Let us deﬁne 2 normalized vectors, which will denote the vectors before their projection by into the subspace deﬁned by Φ .

Original: [Equation]
Revised: [Equation]

Original: Since Π S c 1 and Π S 1 are completely disjoint, we have , Y (cid:105) = 0 , which implies that (cid:107) X
Revised: Since Π S c 1 and Π S 1 are completely disjoint, we have , Y (cid:105) = 0 , which implies that (cid:107) X

Original: + Y (cid:107) =(cid:107) X (cid:107) + (cid:107) Y (cid:107) = 
Revised: + Y (cid:107) = (cid:107) X (cid:107) + (cid:107) Y (cid:107) = 2 .

Original: Since | S 1 ∪ S 2 |(cid:104) k , we know X + Y is a k -sparse vector.
Revised: Since | S 1 ∪ S 2 |(cid:104) k , we know X + Y is a k -sparse vector.

Original: Using the RIP assumptionyields
Revised: Using the RIP assumption yields

Original: [Equation]
Revised: [Equation]

Original: Symmetrically, we have (cid:107) X − Y (cid:107) 2 = 2 and X − Y is also k -sparse, which yields
Revised: Symmetrically, we have (cid:107) X − Y (cid:107) 2 = 2 and X − Y is also k -sparse, which yields

Original: [Equation]
Revised: [Equation]

Original: As a generalization of the fact that ab = ( a + b ) 2 − ( a − b ) 2we have
Revised: As a generalization of the fact that ab = ( a + b ) 2 − ( a − b ) 2we have

Original: [Equation]

Original: Φ X, Φ Y

Original: [Equation]
Revised: [Equation]

Original: Plugging in the previous inequalities we have
Revised: Plugging in the previous inequalities we have

Original: [Equation]
Revised: [Equation]

Original: Relating back to our original quantity
Revised: Relating back to our original quantity

Original: [Equation]
Revised: [Equation]

Original: Where the ﬁrst inequality is due to the fact that (cid:107) b (cid:107) 2 = 1 , and the projection by Π S 1 can only decrease2 norm value, and likewise for Π S c 1 .
Revised: Where the ﬁrst inequality is due to the fact that (cid:107) b (cid:107) 2 = 1 , and the projection by Π S 1 can only decrease 2 norm value, and likewise for Π S c 1 .

Original: Lemma C.3.
Revised: Lemma C.3.

Original: Using the Vanilla IHT algorithm, the following iterative invariant holds
Revised: Using the Vanilla IHT algorithm, the following iterative invariant holds

Original: [Equation]
Revised: [Equation]

Original: where w t is the iterate at time step t , w ∗ is the optimal value, ρ = (2 max { 2 µβ 2 k − 1 , 1 − 2 µα 2 k } +4 µ β 4 k − α 4 k) , β i is the isometry constant associated with i -sparse vectors
Revised: where w t is the iterate at time step t , w ∗ is the optimal value, ρ = (2 max { 2 µβ 2 k − 1 , 1 − 2 µα 2 k } + 4 µ β 4 k − α 4 k) , β i is the isometry constant associated with i -sparse vectors

Original: Proof: Let v = w t − µ ∇ f ( w t ) and S (cid:63) =
Revised: Proof: Let v = w t − µ ∇ f ( w t ) and S (cid:63) =

Original: supp ( w t +1 ) ∪ supp ( w ∗ ) .
Revised: supp ( w t +1 ) ∪ supp ( w ∗ ) .

Original: Applying the triangle inequalityyields
Revised: Applying the triangle inequality yields

Original: [Equation]
Revised: [Equation]

Original: Notice, by construction, t +1 , S c(cid:63) (cid:105) = ∗ , S c(cid:63) (cid:105) = 

Revised: Notice, by construction, t +1 , S c(cid:63) (cid:105) = ∗ , S c(cid:63) (cid:105) =

Original: Focusing on the ﬁrst term
Revised: Focusing on the ﬁrst term

Original: [Equation]
Revised: [Equation]

Original: Where the ﬁrst equality is due to the equivalence of subtracting values from some indices i ∈ S , andsubtracting all values and adding back indices i ∈ S c , and the inequality is due to the projectionw ∗ ∈ C k ∩ R
Revised: Where the ﬁrst equality is due to the equivalence of subtracting values from some indices i ∈ S , and subtracting all values and adding back indices i ∈ S c , and the inequality is due to the projection w ∗ ∈ C k ∩ R

Original: + and the fact that the projection w t
Revised: + and the fact that the projection w t

Original: +1 =
Revised: +1 =

Original: Π C k ∩ R + v is done optimally.
Revised: Π C k ∩ R + v is done optimally.

Original: Substitutingour the ﬁrst term with this inequality yields
Revised: Substituting our the ﬁrst term with this inequality yields

Original: [Equation]
Revised: [Equation]

Original: Expanding v and denoting the optimal error (cid:15)
Revised: Expanding v and denoting the optimal error (cid:15)

Original: =
Revised: =

Original: Φ w ∗ − y
Revised: Φ w ∗ − y

Original: [Equation]
Revised: [Equation]

Original: Where the second equality is due to the expansion of ∇ f ( w t ) , and the third equality is due to theequivalence between the total error in our current estimate and the error from our current estimate tothe optimal value w ∗ , added to the optimal error.
Revised: Where the second equality is due to the expansion of ∇ f ( w t ) , and the third equality is due to the equivalence between the total error in our current estimate and the error from our current estimate to the optimal value w ∗ , added to the optimal error.

Original: Substituting the expansion of v into the previousinequality yields
Revised: Substituting the expansion of v into the previous inequality yields

Original: [Equation]
Revised: [Equation]

Original: Where the ﬁrst equality is due to the rearrangement of terms and the fact that Π S ∗ w ∗ = w ∗ .
Revised: Where the ﬁrst equality is due to the rearrangement of terms and the fact that Π S ∗ w ∗ = w ∗ .

Original: Expanding the identity matrix as I =
Revised: Expanding the identity matrix as I =

Original: Π S ∗ + Π S c ∗ yields
Revised: Π S ∗ + Π S c ∗ yields

Original: [Equation]
Revised: [Equation]

Original: We will now use the previous lemmas to bound each of these terms.
Revised: We will now use the previous lemmas to bound each of these terms.

Original: Let us begin with the ﬁrst term2 (cid:107) ( I − 2 µ Π S ∗ Φ T ΦΠ S ∗ )
Revised: Let us begin with the ﬁrst term [Equation] I − [Equation]

Revised: µ Π

Original: Π S ∗ ( w t − w ∗ ) (cid:107) 2 .
Revised: [Equation] )Π [Equation] ( w t − w ∗ ) (cid:107) 2 .

Original: Let λ ( A ) denote the eigenvalues of a matrix A .
Revised: Let λ ( A ) denote the eigenvalues of a matrix A .

Original: We know
Revised: We know

Original: S ∗ ≤ 2 k , which allows us to apply Lemma 3 . 1 , which yields
Revised: S ∗ ≤ 2 k [Equation] which allows us to apply Lemma 3 . [Equation]

Revised: which yields

Original: [Equation]
Revised: [Equation]

Original: k which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: 2 k which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: To analyze the second term, let us denote S (cid:48)
Revised: To analyze the second term, let us denote S (cid:48)

Original: =
Revised: =

Original: supp ( w t ) ∪ supp ( w ∗ ) .
Revised: supp ( w t ) ∪ supp ( w ∗ ) .

Original: We can then write the secondterm as 4 µ (cid:107) Π S ∗ Φ T ΦΠ S c ∗ Π S (cid:48) ( w t − w ∗ ) (cid:107) 2 , and given the fact that | S (cid:48) ∪ S ∗ | ≤ 4 k , we can apply
Revised: We can then write the second term as 4 µ (cid:107) Π S ∗ Φ T ΦΠ S c ∗ Π S (cid:48) ( w t − w ∗ ) (cid:107) 2 , and given the fact that | S (cid:48) ∪ S ∗ | ≤ 4 k , we can apply

Original: Lemma 3 . 2 directly which yields
Revised: Lemma 3 . 2 directly which yields

Original: [Equation]
Revised: [Equation]

Original: Finally, we can bound the third term as follows, where X = { x∈ R [Equation] | |
Revised: Finally, we can bound the third term as follows, where X = { x

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: Combining the above inequalities yields the following iterative invariant
Revised: Combining the above inequalities yields the following iterative invariant

Original: [Equation]
Revised: [Equation]

Original: D Experiment Details
Revised: D Experiment Details

Original: For the ﬁrst experiment, construct a synthetic dataset with N = 1000 samples with M = 400features each, such that each feature value is sampled from a univariate Gaussian distribution.
Revised: For the ﬁrst experiment, construct a synthetic dataset with N = 1000 samples with M =features each, such that each feature value is sampled from a univariate Gaussian distribution.

Original: We construct the dataset such that 80% of the input vectors contain features sampled from someunivariate Gaussian distribution with mean µ 1 , and standard deviation σ 1 , and the remaining 20%of the data contains features sampled from a different univariate Gaussian distribution with meanµ 2 , and standard deviation σ 2 , such that cµ 2 = µ 1 and c 2 σ 2 = σ 1 .
Revised: We construct the dataset such that 80% of the input vectors contain features sampled from some univariate Gaussian distribution with mean µ 1 , and standard deviation σ 1 , and the remaining 20% of the data contains features sampled from a different univariate Gaussian distribution with mean µ 2 , and standard deviation σ 2 , such that cµ 2 = µ 1 and c 2 σ 2 = σ 1 .

Original: Essentially 80% of the data hasfeatures that are sampled from some Gaussian distribution, and the remaining 20% of the data hasfeatures sampled from a scaled down version of the same Gaussian [-distribution.
-]
Revised: Essentially 80% of the data has features that are sampled from some Gaussian distribution, and the remaining 20% of the data has features sampled from a scaled down version of the same Gaussian [-distribution.
-]distribution.

Original: In the second experiment, we observe the cost of the objective function using the hueristicsolution to the group sparsity problem across iterations, and compare this to the cost of the objectiveusing our proposed algorithm.
Revised: In the second experiment, we observe the cost of the objective function using the hueristic solution to the group sparsity problem across iterations, and compare this to the cost of the objective using our proposed algorithm.

Original: We take the average across 10 trials, and the mean objective cost ofapproach is represented by the solid lines, and the shaded regions represent the standard deviations.
Revised: We take the average across 10 trials, and the mean objective cost of approach is represented by the solid lines, and the shaded regions represent the standard deviations.

Original: Note that while the hueristic approach does produce a feasible solution to the group sparsity problem,the points selected to be in the sample differ from the points selected by our proposed algorithm.
Revised: Note that while the hueristic approach does produce a feasible solution to the group sparsity problem, the points selected to be in the sample differ from the points selected by our proposed algorithm.

Original: In the future, we hope to show empirically that the sample points selected by our algorithm arequalitatively superior to the sample points selected by the hueristic.
Revised: In the future, we hope to show empirically that the sample points selected by our algorithm are qualitatively superior to the sample points selected by the hueristic.

Original: This intuitively makes sense asour algorithm takes into account the full posterior, while the huerisitic only considers the group-wisepartioned posterior
Revised: This intuitively makes sense as our algorithm takes into account the full posterior, while the huerisitic only considers the group-wise