Coresets for Relational Data and The Applications
Abstract
A coreset is a small set that can approximately preserve the structure of the original input data set.
Therefore we can run our algorithm on a coreset so as to reduce the total computational complexity.
Conventional coreset techniques assume that the input data set is available to process explicitly.
However, this assumption may not hold in real-world scenarios.
In this paper, we consider the problem of coresets construction over relational data.
Namely, the data is decoupled into several relational tables, and it could be very expensive to directly materialize the data matrix by joining the tables.
We propose a novel approach called “aggregation tree with pseudo-cube” that can build a coreset from bottom to up.
Moreover, our approach can neatly circumvent several troublesome issues of relational learning problems [Khamis et al., PODS 2019].
Under some mild assumptions, we show that our coreset approach can be applied for the machine learning tasks, such as clustering, logistic regression and SVM.
Introduction
As the rapid development of information technologies, we are often confronted with large-scale data in many practical scenarios.
To reduce the computational complexity, a number of data summarization techniques have been proposed [46].
Coreset is a well-known sampling technique for compressing large-scale data sets [43; 27].
Roughly speaking, for a given large data set P , the coreset approach is to construct a new (weighted) set ˜ P that approximately preserves the structure of P and meanwhile the size | ˜ P | is much smaller than | P | .
Therefore, if we run an existing algorithm on ˜ P rather than
P , the runtime can be significantly reduced and the computational quality (e g., the optimization objectives for the machine learning tasks [53]) can be approximately guaranteed.
In the past decades, the coreset techniques have been successfully applied for solving many optimization problems, such as k -means clustering [27; 20], logistic regression [33; 44], Gaussian mixture models [39], and continual learning [11].
Usually we assume that the input data is stored in a matrix such that the coreset algorithm can easily access the whole data.
But this assumption may not hold in real-world scenarios.
For example, according to the Kaggle 2017 “State of Data Science Survey”, 65.5% of the data sets are relational data .
Namely, the data is decoupled into several relational tables, and we cannot obtain the data matrix (which is also called the “ design matrix ”) unless joining all the tables.
Relational database has a very large and fast growing market in this big data era.
It is expected to reach USD 122 .billion by 2027 [48].
Relational database has several favorable properties [54], e.g., the decoupled relational data can save a large amount of space, and it is friendly to several common statistical queries such as the aggregate functions COUNT , SUM and AVG in SQL.
In recent years, the study on “in-database learning” has gained a significant amount of attention in the areas of machine learning and data management [45; 35; 22; 37].
However, constructing a coreset for relational database is much more challenging.
Suppose there are s > 1 relational tables and the size of the largest table is N .
Let n be the number of the tuples obtained by joining all the tables, and then n can be as large as O ( N s ) [7] (we illustrate an instance of joining two tables in Table 1).
Obviously it will take extremely large time and space complexities to join all the tables and train a model on such scale of data.
To remedy this issue, a straightforward idea is trying to complete the computing task without explicitly constructing the design matrix.
Unfortunately, even for some very simple computational tasks, their implementations on relational data can be NP-hard.
To see why these implementations are so hard, we can consider the following simple clustering assignment task (which is often used as a building block for coresets construction [27; 20]): suppose the k cluster centers have already been given, and we want to determine the size of each cluster.
Note that if we have the design matrix, this task is trivial (just assign each point to its nearest center, and then calculate the cluster sizes based on the assignment).
However, if the data is decoupled into several relational tables, the problem of computing the assignment can be quite complicated.
Khamis et al.
[36] recently defined the problem of answering F unctional A ggregate Q ueries (FAQ) in which some of the input factors are defined by a collection of A dditive I nequalities 2 between variables (denote by FAQ - AI( m ) , where m is the number of additive inequalities).
The problem of computing the clustering assignment can be formulated as an instance of FAQ - AI( k − 1) (to determine the cluster membership for one point (tuple), we need to certify that its distance to its own cluster center is lower than the distances to the other k − 1 centers).
Recently, Khamis et al. [3] showed that evaluating even a FAQ - AI(1) instance is #P-hard, and approximating a FAQ - AI( m ) instance within any finite factor is NP-hard for any m > 1 .
Moseley et al.
[42] also showed that even approximating the clustering assignment to any factor is NP-hard for k ≥ 3 .
An intuitive understanding is that the query satisfying those inequalities require to access the information of each tuple on all the dimensions, where it is almost equivalent to constructing the entire design matrix.
1.1 Our Contributions
We consider Empirical Risk Minimization (ERM) problems in machine learning [56].
Let R d be the data space.
The training set P = { p 1 , p 2 , · · · , p n } ⊂ R d .
But we assume that this set P is not explicitly given, where it is decoupled into s relational tables (the formal definitions are shown in
Section 2).
The goal is to learn the hypothesis θ (from the hypothesis space H ) so as to minimize the empirical risk
[Equation]
where f ( · , · ) is the non-negative real-valued loss function .
Several coresets techniques on relational data have been studied very recently.
Samadian et al.
[49] showed that the simple uniform sampling yields a coreset for regularized loss minimization problems (note the uniform sampling can be efficiently implemented for relational database [59]).
Their sample size is Θ ( n κ · dim ) , where κ ∈ (0 , 1) (usually κ is set to be 1 / 2 in practice) and dim is the VC dimension of loss function (usually it is Θ( d ) ).
Thus the size can be Θ ( √ n · d ) , which is too large especially for high dimensional data.
Curtin et al.
[22] constructed a coreset for k -means clustering by building a weighted grid among the input s tables; the coreset yields a 9 -approximation for the clustering objective.
Independently, Ding et al.
[24] also applied the “grid” idea to achieve a (9 + ϵ ) -approximation for k -means clustering with distributed dimensions (attributes).
The major drawback of this “grid” idea is that the resulting coreset size can be as large as k s which is exponential in the number of tables s .
Moseley et al.
[42] recently proposed a coreset for k -means clustering on relational data by using the k -means++ initialization method [6], however, their approximation ratio is too high ( > 400 ) for real-world applications.
Moreover, the coresets techniques proposed in [22; 24; 42] can only handle k -means clustering, and it is unclear that whether they can be extended for other machine learning problems as the form of (1).
In this paper, we aim to develop an efficient and general coreset construction method for optimizing (1) on relational data.
First, we observe that real-world data sets often have low intrinsic dimensions (e.g, nearby a low-dimensional manifold) [10].
Our coreset approach is inspired by the well-known
Gonzalez’s algorithm for k -center clustering [29].
The algorithm greedily selects k points from the input point set, and the k balls centered at these selected k points with some appropriate radius can cover the whole input; if the input data has a low intrinsic dimension (e.g., the doubling dimension in Definition 3), the radius is no larger than an upper bound depending on k .
Therefore, the set of cluster centers (each center has the weight equal to the corresponding cluster size) can serve as a coreset for the ERM problem (1), where the error yielded from the coreset is determined by the radius.
Actually this k -center clustering based intuition has been used to construct the coresets for various applications before [34; 52; 21].
However, we have to address two challenging issues for realizing this approach on relational data.
First, the greedy selection procedure for the k cluster centers cannot be directly implemented for relational data.
Second, it is hard to obtain the size of each cluster especially when the balls have overlap (the k balls can partition the space into as many as 2 k parts).
Actually, both of these two issues can be regarded as the instances of FAQ - AI( k − 1) which are NP-hard to compute [3; 42].
Our approach relies on a novel and easy-to-implement structure called “ aggregation tree with pseudo-cube ”.
We build the tree from bottom to up, where each relational table represents a leaf.
Informally speaking, each node consists of a set of k cluster centers which is obtained by merging its two children; each center also associates with a “pseudo-cube” region in the space.
Eventually, the root of the tree yields a qualified coreset for the implicit design matrix.
The aggregation manner can help us to avoid building the high-complexity grid coreset as [22; 24].
Also, with the aid of “pseudo-cube”, we can efficiently estimate the size of each cluster without tackling the troublesome
FAQ - AI counting issue [3;
Comparing with the previous coresets methods, our approach enjoys several significant advantages.
For example, our approach can deal with more general applications.
In fact, for most ERM problems in the form of (1) under some mild assumptions, we can construct their coresets by applying our approach.
Also our coresets have much lower complexities.
It is worth to emphasize that our coreset size is independent of the dimensionality d ; instead, it only depends on the doubling dimension of the input data.
1.2 Other Related Works
The earliest research on relational data was proposed by Codd [18].
A fundamental topic that is closely related to machine learning on relational data is how to take uniform and independent samples from the full join results [13].
Recently, Zhao et al.
[59] proposed a random walk approach for this problem with acyclic join; Chen and Yi [15] generalized the result to some specific cyclic join scenarios.
Beside the aforementioned works [45; 35; 22; 24; 36; 3; 42], there also exist a number of results on other machine learning problems over relational data.
Khamis et al.
[4] and Yang et al.
[57] respectively considered training SVMs and SVMs with Gaussian kernel on relational data.
For the problem of linear regression on relational data, it is common to use the factorization techniques [50; 2].
The algorithms for training Gaussian Mixture Models and Neural Networks on relational data were also studied recently [16; 17].
We also refer the reader to the survey paper [51] for more details on learning over relational data.
Preliminaries
Suppose the training set for the problem (1) is a set P of n points in R d , and it is decoupled into s relational tables { T 1 , . .
, T s } with the feature (attribute) sets { D 1 , . .
, D s } .
Let D =
(cid:83) i D i and therefore the size | D | = d .
Actually, each table T i can be viewed as projection of P onto a subspace spanned by D i .
To materialize the design matrix of P , a straightforward way is to compute the join over these s tables.
With a slight abuse of notations, we still use “ P ” to denote the design matrix.
We also let [ s ] = { 1 , 2 , . . .
, s } for simplicity.
Tdd
Tdd
T 1 ▷◁ Tddd
Definition 1 (Join)
The join over the given s tables returns a n × d design matrix P = T 1 ▷◁ · · · ▷◁
T s , such that for any vector (point) p ∈ R d , p ∈ P if and only if ∀ i ∈
[ s ] , Proj D i ( p ) ∈
T i , where
Proj D i ( p ) is the projection of p on the subspace spanned by the features of D i .
Table 1 is a simple illustration for joining two relational tables.
To have a more intuitive understanding of join, we can generate a hypergraph G =
( V , E ) .
Each vertex of V corresponds to an individual feature of D ; each hyperedge of E corresponds to an individual relational table T i , and it connects all the vertices (features) of D i .
Then we can define the acyclic and cyclic join queries.
A join is acyclic if the hypergraph G =
( V , E ) can be ablated to be empty by performing the following operation iteratively: if there exists a vertex that connects with only one hyperedge, remove this vertex together with the hyperedge.
Otherwise, the join is cyclic.
A cyclic query usually is extremely difficult and has a much higher complexity than that of an acyclic query.
For example, for a cyclic join, it is even
NP-hard to determine that whether the join is empty or not [41].
Fortunately, most real-world joins are acyclic, which allow us to take full advantage of relational data.
Similar with most of the previous articles on relational learning [3; 4], we also assume that the join of the given tables is acyclic in this paper.
Counting is one of the most common aggregation queries on relational data.
It returns the number of tuples that satisfy some particular conditions.
The counting on an acyclic join (without additive inequalities) can be implemented effectively via dynamic programming [25].
But when the additive inequalities are included, the counting problem can be much more challenging and we may even need to materialize the whole design matrix [3].
For example, to count the tuples of { t ∈ T 1 ▷◁ T 2 & ∥ t ∥ 22 ≤ 5 } in Table 1, we need to check the whole design matrix T 1 ▷◁ T 2 for selecting the tuples that satisfy the constraint “ ∥ t ∥ 22 ≤ 5 ”.
For the machine learning problems studied in this paper, we have the following assumption for the loss function in (1).
Assumption 1 (Continuity)
There exist real constants α, z ≥ 0 , β ∈ [0 , 1) , such that for any p, q ∈ R d and any θ in the hypothesis space, we have
[Equation]
where ∥ · ∥ is the Euclidean norm in the space.
Remark 1 Different machine learning problems have different values for α, β and z .
For example, for the k -means clustering, we have z = 2 , β
= ϵ and α
= O ( 1 ϵ ) (where ϵ can be any small number in (0 , 1) ); for the k -median clustering, we have z = 1 , β =
and α = 1 .
Actually for a large number of problems, β is small or even 0 , e.g., the logistic regression and SVM with soft margin problems have the value β = 0 .
We define the following coreset with both multiplicative error and additive error.
We are aware that the standard coresets usually only have multiplicative errors [43; 27].
However, the deviation bounds for the ERM problems with finite training data set only yield additive error guarantees and the additive error is usually acceptable in practice [8; 55].
So the coresets with additive error have been also proposed recently [9].
Let P = { p 1 , p 2 , . . . , p n } be the training set (which is not explicitly given), and denote by ∆ the diameter of P (i.e., the largest pairwise distance of P ). Definition 2 ( ( ϵ 1 , ϵ 2 ) z -Coreset)
Suppose ϵ 1 ∈ (0 , 1) and ϵ 2 , z > 0 .
The ( ϵ 1 , ϵ 2 ) z -coreset, denoted as ˜ P , is a point set { c 1 , · · · , c | ˜ P | } with a weight vector W =
[ w 1 , w 2 , . .
, w | ˜ P | ] satisfying that
[Equation]
for any θ in the hypothesis space H , where ˜ F ( θ ) =
[Equation]
Usually we want the coreset size | ˜ P | to be much smaller than | P | .
So we can run our algorithm on ˜ P and save a large amount of running time.
As mentioned in Section 1.1, we also assume that the training set P has a low intrinsic dimension.
“Doubling dimension” is a widely used measure for indicating the intrinsic dimension of a given data set [40].
Roughly speaking, it measures the growth rate of the given data in the space.
Recently it has gained a lot of attention for studying its relation to coresets and machine learning problems [38; 32; 19].
For any c ∈ R d and r ≥ 0 , we use B ( c, r ) to denote the ball centered at c with radius r .
Definition 3 (Doubling Dimension)
The doubling dimension of a data set P is the smallest number ρ > 0 , such that for any p ∈ P and r ≥ 0 , P ∩ B ( p, 2 r ) is always covered by the union of at most 2 ρ balls with radius r .
It is easy to obtain the following proposition by recursively applying Definition 3 log ∆ r times.
Claim 1 For a given data set P and radius r > 0 , if P has the doubling dimension ρ , then it can be covered by ( ∆ r ) ρ balls of radius r .
Relational Coreset via Aggregation Tree
In this section, we present an efficient coreset construction method for relational data.
First, we introduce the technique of aggregation tree with pseudo-cube.
Actually there is an obstacle for obtaining the coreset from the aggregation tree.
The weight of each point of the coreset is determined by the size of its corresponding pseudo-cube; however, the pseudo-cubes may have overlap and it is challenging to separate them in the environment of relational data.
To explain our idea more clearly, we temporally ignore this “overlap” issue and show the “ideal” coreset construction result in
Section 3.1.
Then we show that the overlap issue can be efficiently solved by using random sampling and some novel geometric insights in Section 3.2.
Recall that our input is a set of s relational tables { T 1 , . .
, T s } with the feature (attribute) sets { D 1 , . .
, D s } .
Also D =
(cid:83) i D i and | D |
= d .
For ease of presentation, we generate a new feature set ˆ D i for each D
i as follows: initially, ˆ D
= D 1 ; starting from i = 2 to s , we let ˆ D i
= D i \ ( ∪ i − 1 l =
ˆ D l ) .
It is easy to see that (cid:83)
i ˆ D i
= D , and ∀ i, j ∈ [ s ] , i ̸
= j , ˆ D i ∩ ˆ D j = ∅ .
With a slight abuse of notations, we also use ˆ D i to represent the subspace spanned by the features of ˆ D i (so the Cartesian product ˆ D 1 × · · · × ˆ D s is the the whole space R d ).
For any point set Q ⊂ R d (resp., any point c ∈ R d ) and any subspace H of R d , we use Proj H ( Q ) (resp., Proj H ( c ) ) to denote the projection of Q (resp., c ) onto H .
3.1 Coreset Construction
Since our coreset construction algorithm is closely related to the Gonzalez’s algorithm for k -center clustering [29], we briefly introduce it first.
Initially, it arbitrarily selects a point from P , say c 1 , and sets C = { c 1 } ; then it iteratively selects a new point that is farthest to the set C , i.e., arg max p ∈ P min q ∈ C || p − q || , and adds it to C .
After k iterations, we obtain k points in C (suppose
C = { c 1 , · · · , c k } ).
The Gonzalez’s algorithm yields a 2 -approximation for k -center clustering.
If the optimal radius of k -center clustering on P is r opt , then P can be covered by the k balls B ( c 1 , 2 r opt ) , · · · , B ( c k , 2 r opt ) .
Together with Claim 1, for any given radius r > 0 , we know that if we run the Gonzalez’s algorithm on P with setting k = ( 2∆ r ) ρ , the set P can be covered by the balls B ( c 1 , r ) , · · · , B ( c k , r ) (since r opt ≤ r/ 2 ).
If we set r to be small enough, the obtained C can be a good approximation (informally a coreset) for P .
However, as mentioned in Section 1, such k -center clustering approach cannot be implemented on relational data since it is equivalent to an instance of FAQ - AI( k − 1) .
A straightfoward idea to remedy this issue is to run the Gonzalez’s algorithm in each subspace ˆ D i , and then compute the
Cartesian product to build a grid of size k s as the methods of [22; 24].
But this approach will result in an exponentially large coreset (e.g., there are s = 8 tables and k = 1000 ).
Below, we introduce our algorithm that can achieve a coreset with quasi-polynomial size.
Our high-level idea.
We construct a bottom-to-top tree T where each relational table T
i corresponds to a leaf.
For each node ν ∈ T , it is associated with a set of components: an index set I ν ⊆
[ s ] , the spanned subspace H ν
= Π i ∈
I ν
ˆ D i , and a collection of centers
C ν = { c ν, 1 , · · · , c ν,k } ⊂
H ν .
The center set C ν determines a set of pseudo-cubes (as Definition 4 below) in H ν .
Obviously, for a leaf node (table) ν of T i , the index set I ν
= { i } and the space H
ν
=
ˆ D i ; we directly run the Gonzalez’s algorithm on the projection of P on ˆ D i to obtain the set C ν .
Then, we grow the tree T from these leaves.
For each parent node ν p , we compute a grid C ν l × C ν r of size k 2 by taking the Cartesian product of the center sets from its two children ν l and ν r ; then we remove the “empty” grid points (we will explain the meaning of “empty” after Definition 4) and run the Gonzalez’s algorithm on the remaining grid points, which are denoted as (cid:94) C
ν l × C ν r , to achieve the set C ν p .
The index set
I ν p
= I
ν
l ∪ I ν r , and the subspace H ν p =
H ν l × H ν r .
Finally, the set C ν 0 of the root node ν 0 yields a coreset for the point set P .
See an illustration of the aggregation tree in Figure 1.
It is worth noting that our aggregation tree approach is fundamentally different to the hierarchical decomposition tree methods that were commonly studied in doubling metrics [30; 12] and other tree structures like kd -tree [23].
They build their trees from top to bottom, while our approach is from bottom to top.
Actually, our approach can be viewed as a “compressed” version of the grid coreset methods [22; 24], where we deal with a grid of size at most k 2 each time.
Also, the accumulated error can be also bounded since the height of T is O (log s ) .
Definition 4 (Pseudo-cube)
Given an index set I ⊆
[ s ] , a point c ∈ H = (cid:81)
i ∈ I
ˆ D i , and a number r ≥ 0 , we define the pseudo-cube
[Equation]
where Proj ˆ D i ( c ) is the projection of c onto the subspace ˆ D i .
A nice property of pseudo-cube is that it is a Cartesian product of a set of regions from those subspaces ˆ D i s. More importantly, it does not involve any cross-table constraints or additive inequalities.
Consequently, counting the size of a pseudo-cube is easy to implement over the relational data.
Another property is that the Cartesian product of two pseudo-cubes is still a pseudo-cube, but in a new subspace with higher dimension.
When we take the grid of the two center sets from two children nodes, we also obtain k 2 pseudo-cubes.
For each pseudo-cube, we count its size (i.e., the size of its intersection with P ); if the size is 0 , we call the corresponding grid point is “empty” and remove it before running the Gonzalez’s algorithm on the grid.
Let d ( A, B ) := max a ∈ A min b ∈ B || a − b || for any two sets A and B in R d (i.e., the directed Hausdorff
Distance).
First, for each h = 0 , 1 , · · · , ⌈ log s ⌉ , we define a key value
[Equation]
at the h -th level of T } .
[Equation]
By using these L h s, we present our coreset construction approach in Algorithm 1.
Theorem 1 Suppose the loss function f ( · , · ) satisfies Assumption 1.
If we set k =
[Equation]
Algorithm 1, the returned set C ν
[Equation]
node is constructed associated with corresponding subspace H set centers the run the
Gonzalez’s algorithm select k centers for T i on the subspace
ˆ D i
Also the
L Let Children = { ν 1 , . .
, ν s } and Parent = ∅ .
Initialize h = 0 to indicate the current level on T .
For h = 1 to ⌈ log s ⌉ : of remaining of k centers, let l h
= max( l h , d ( (cid:94) C
ν i × C ν j , C ν ))
, I ν =
I ν
i ∪ I ν j and H ν
= H
ν i × H ν j ; vi.
Parent =
Parent ∪ { ν } ; (c) L h = √ 2 h ( l h + √ 2 L h − 1 ) ; (d) Children =
Children ∪ Parent , Parent = ∅ ; 5.
Let ν 0 be the root node; for each c ν 0 ,i ∈ C ν 0 , we assign a weight w i = | P ∩ PC
I ν 0 ( c ν 0 ,i , L ⌈ log s ⌉ ) | (actually, this is not an accurate expression since the pseudo-cubes may have overlap with each other, and we will address this issue in Section 3.2).
return the set C ν 0 with the weights { w 1 , · · · , w k } .
with the weights { w 1 , · · · , w k } yields a ( β, ϵ 2 )
z -coreset.
The time complexity of constructing the tree T is O (cid:0) sk 2 Ψ( N, s, d ) + sk 3 (cid:1) , where Ψ( N, s, d ) is the complexity of performing the counting each time [1] (for counting an acyclic join, Ψ( N, s, d ) =
O (cid:0) sd 2 N log( N ) (cid:1) ).
Remark 2 Note that our coreset size is quasi-polynomial since it has a factor 2 O ((log s ) 2 ) .
But we would like to emphasize that this factor usually is small for most practical problems. For example, in the standard TPC-H benchmark [47], s is no larger than 8 , although the design matrix over joining the s tables can be quite large.
To prove the above theorem, we first introduce the following key lemma.
We let r 0 =
∆
k 1 ρ .
From
Claim 1 we know that the entire data set P can be covered by k balls with radius r 0 .
Lemma
For each h = 0 , 1 , · · · , ⌈ log s ⌉ , L h ≤ 3 h · 2 h 2+3 h +8r 0 .
Proof .
Since the Gonzalez’s algorithm yields an approximation factor 2 , from Claim 1 we know that
L 0 ≤ 2 r 0 .
Then we consider the case h ≥ 1 .
Suppose the algorithm is constructing the h -th level of
T .
In Step 4b we repeatedly select two nodes ν i , ν j from the ( h − 1) -th level that form a parent node ν at the h -th level.
Let C opt be the optimal solution of the k -center clustering on Proj H νi × H νj ( P ) .
Since P can be covered by k balls with radius r 0 , so can the projection Proj H νi × H νj ( P ) .
That is,
[Equation]
Also, since d (Proj H νi ( P ) , C ν i ) ≤ L h − 1 and d (Proj H νj ( P ) , C ν j ) ≤ L h − 1 , we have
[Equation]
Note that we use (cid:94) C
ν i × C ν j instead of
C ν i × C ν j in the above (7). The reason is that the Cartesian product C ν i × C ν j may contain some “empty” grid points (see Algorithm 1 step 4(b)(iii)), and the distance bound “ √ 2 L h − 1 ” does not hold for them.
Together with (6), we know
[Equation]
That is, if we run k -center clustering on (cid:94) C ν i × C ν j , the optimal radius should be no larger [-thanr 
-]than r 0 + √ 2 L h − 1 .
So if we run the 2 -approximation Gonzalez’s algorithm on (cid:94) C
ν i × C ν j and obtain the center set C ν , then
[Equation]
[Equation]
Further, we combine (7) and (9), and obtain
[Equation]
For each c ν,l ∈ C ν , we construct a pseudo-cube
[Equation]
Then we know that Proj H νi × H νj
( P ) is covered by the union of the obtained k pseudo-cubes.
Because | I ν | = | I ν
i ∪ I
ν j | ≤ 2 h , we have L h ≤ (2( r 0 + √ 2 L h − 1 )+
√ 2 L h − 1 ) ·√ 2 h .
Together with L 0 ≤ 2 r 0 , we can solve this recursion function and obtain L h ≤ 3 h · 2 h 2+3 h +8r 0 .
□
Lemma 1 indicates that the difference between C ν 0 and P is bounded in the space.
To prove it is a qualified coreset with respect to the ERM problem (1), we also need to assign a weight to each point of C ν 0 .
In Step 5 of Algorithm 1, we set the weight w i = | P ∩ PC I ν
( c ν 0 ,i , L ⌈ log s ⌉ ) | (for simplicity, we temporarily assume these pseudo-cubes are disjoint, and leave the discussion on the overlap issue to Section 3.2).
The detailed proof of Theorem 1 is placed to our full paper [14].
Remark
In a real implementation, if the coreset size is given (e.g., 5% of the input data size | P | ), we can directly set the value for k to run Algorithm 1.
Another scenario is that we are given a restricted variance between P and the output C ν 0 (i.e., the difference d ( P, C ν 0 ) is required to be no larger than a threshold).
Then we can try the value for k via doubling search.
For example, starting from a small constant k 0 , we can try k = k 0 , 2 k 0 , 2 2 k 0 , · · · , until the difference is lower than the threshold.
3.2 The Overlap Issue
We consider the k pseudo-cubes PC I ν
( c ν 0 ,i , L ⌈ log s ⌉ ) , 1 ≤ i ≤ k , obtained from Algorithm 1.
Since ν 0 is the root, it is easy to know I ν
=
[ s ] and H ν
= R d .
For the sake of simplicity, we use PC i to denote the pseudo-cube PC I ν {+
+} ( c ν 0 ,i , L ⌈ log s ⌉ ) and L to denote L ⌈ log s ⌉ below.
If these pseudo-cubes are disjoint, we can simply set the weight w i = | P ∩ PC i | as Step 5 of
Algorithm 1.
However, these pseudo-cubes may have overlaps and such an assignment for the weights cannot guarantee the correctness of our coreset.
For example, if a point q ∈ P is covered by two pseudo-cubes PC i 1 and PC i 2 , we should assign q to only one pseudo-cube.
Note that this overlap issue is trivial if the whole data matrix P is available.
But it can be troublesome for relational data, because it is quite inefficient to perform the counting on their union P ∩ (cid:0) PC i 1 ∪ PC i 2 (cid:1) .
Moreover, the k pseudo-cubes can partition the space into as large as 2 k different regions, and it is challenging to deal with so many overlaps.
Fortunately, we can solve this issue by using random sampling.
The key observation is that we can tolerate a small error on each weight w i .
Let δ ∈ (0 , 1) .
If we obtain a set of approximate weights
W ′ = { w ′ 1 , · · · , w ′ k } that satisfy w ′ i ∈ (1 ± δ ) w
i for 1 ≤ i ≤ k , then we have k (cid:88)
[Equation]
for any θ in the hypothesis space H .
We can use the following idea to obtain a qualified W ′ .
High-level idea.
Without loss of generality, we assume all the k pseudo-cubes are not empty (otherwise, we can directly remove the empty pseudo-cubes).
Then we consider the pseudo-cubes one by one.
For PC 1 , we directly set w ′ 1 = | P ∩ PC 1 | .
Suppose currently we have already obtained the values w ′ 1 , · · · , w ′ i 0 , and try to determine the value for w ′ i
+1 .
We take a uniform sample of m points from P ∩ PC i
+1 by using the sampling technique for relational data [59].
Each sampled point corresponds a binary random variable x : if it belongs to PC i 0 +1 \ ( ∪ i
i =1 PC i ) , x = 1 ; otherwise, x = 0 .
If m is sufficiently large, from the Chernoff bound, we can prove that the sum of these m random variables (denote by g ) over m can serve as a good estimation of
[Equation]
[Equation]
Thus we can set
[Equation]
A remaining issue of the above method is that the ratio τ
i
+1 can be extremely small, that is, we have to set m to be too large for guaranteeing the multiplicative “ 1 ± δ ” error bound.
Our idea for solving this issue is from the geometry.
If τ i 0 +1 is extremely small, we label PC i
+1 as a “light” pseudo-cube.
Meanwhile, it should has at least one “heavy” neighbor from { PC 1 , · · · , PC i 0 } .
The reason is that a small τ i
+1 implies that there must exist some 1 ≤ i 1 ≤ i 0 , such that the intersection
P ∩ PC
i 0 +1 ∩ PC i 1 takes a significant part of P ∩ PC i
+1 .
And we label PC i 1 as a “heavy” neighbor of PC i
+1 .
Moreover, the distance between PC i 1 and PC i
+1 should be bounded (since they have overlap in the space).
Therefore, we can just ignore PC i 0 +1 and use PC i 1 to represent their union PC i 1 ∪ PC i
+1 .
This idea can help us to avoid taking a too large sample for light pseudo-cube.
Overall, we have the following theorem, and the detailed proof is placed to our full paper [14].
Theorem 2 Suppose the loss function f ( · , · ) satisfies Assumption 1 and ϵ 1 > β .
If we set k in
Algorithm 1 as Theorem 1, and let m ≥ Θ(
[Equation]
1 log kλ ) where λ ∈ (0 , 1) , the returned set C νwith the weights W ′ yields a ( ϵ 1 , ϵ 2 )
z -coreset with probability 1 − λ .
The sampling procedure takes
O ( k 2 md ) time.
Experimental Evaluation
We evaluate the performance of our relational coreset on three popular machine learning problems, the SVM with soft margin ( α
= O ( ∥ θ ∥ 2 ) , β = 0 , and z = 1 ), the k c -means clustering 3 ( α = O ( 1 ϵ ) , β
= ϵ , and z = 2 , where ϵ can be any small number in (0 , 1) ), and the [-sampling procedure takes (cid:80) ki 
-]logistic regression ( α =
O ( ∥ θ ∥ 2 ) , β = 0 , and z = 1 ).
All the experimental results were obtained on a server equipped with 3.0GHz Intel CPUs and 384GB main memory. Our algorithms were implemented in Python with PostgreSQL 12.10.
We release our codes at Github [28]. Data sets and Queries. We design four different join queries ( Q1 - Q4 ) on three real relational data sets. Q1 and Q2 are designed on a labeled data set H OME C REDIT [31], and we use them to solve the SVM and logistic regression problems.
Q3 and Q4 are foreign key joins [5] designed on the unlabeled data sets Y ELP [58] and F AVORITA [26] respectively, and we use them to solve the k c -means clustering problem.
Baseline methods.
We consider five baseline methods for comparison.
(1) O RIGINAL : construct the complete design matrix P and run the training algorithm directly on it.
(2) O RI -G ON : construct
P as O RIGINAL , run the Gonzalez’s algorithm [29] on P to obtain the centers, and then run the training algorithm on the centers.
(3) U NIFORM : the relational uniform sampling algorithm [59].
R k - MEANS : the relational k c -means algorithm
(5) RC ORE : our relational coreset approach.
We consider both the running time and optimization quality.
We record the end-to-end runtime that includes the design matrix/coreset construction time and the training time.
For the optimization quality, we take the objective value F ( θ ∗ ) obtained by O RIGINAL as the optimal objective value; for each baseline method, we define “ Approx. ”
= F ( θ ) − F ( θ ∗ ) F ( θ ∗ ) , where F ( θ ) is the objective value obtained by the method. [Table] Due to the space limit, we only present part of the results (Table 2 and Figure 2) here. In general,
O RIGINAL and O RI -G ON are the most time-consuming ones because they need to construct the whole design matrix and perform the operations (like train the models or run the Gonzalez’s algorithm) on the data (even if the queries are foreign key joins). U NIFORM is always the fastest one, because it only takes the simple uniform sampling procedure; but its overall optimization performance is relatively poor.
Except O RIGINAL , our RC ORE has the best optimization quality for most cases with acceptable running time.
We leave the more detailed experimental results to our full paper [14].  Conclusion
In this paper, we propose a novel relational coreset method based on the aggregation tree with pseudo-cube technique.
Our method can be applied to several popular machine learning models and has provable quality guarantees.
In future, there are also several interesting problems deserved to study, e.g., the relational coresets construction for more complicated machine learning models, and the privacy-preserving issues for relational coresets when the input data contains sensitive attributes.  Acknowledgements
The research of this work was supported in part by National Key R&D program of China through grant 2021YFA1000900 and the Provincial NSF of Anhui through grant 2208085MF163.
We also want to thank the anonymous reviewers for their helpful comments.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Detailed experimental results are in our supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
In our supplement material.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]