Algorithms and Hardness for Learning Linear Thresholds from Label Proportions
Abstract
We study the learnability of linear threshold functions (LTFs) in the learning from label proportions (LLP) framework.
In this, the feature-vector classiﬁer is learnt from bags of feature-vectors and their corresponding observed label proportions which are satisﬁed by (i.e., consistent with) some unknown LTF.
This problem has been investigated in recent work ([27]) which gave an algorithm to produce an
LTF that satisﬁes at least ( 2 / 5 ) -fraction of a satisﬁable collection of bags, each of size  2 , by solving and rounding a natural SDP relaxation.
However, this SDP relaxation is speciﬁc to at most 2 -sized bags and does not apply to bags of larger size.
In this work we provide a fairly non-trivial SDP relaxation of a non-quadratic formulation for bags of size 3 .
We analyze its rounding procedure using novel matrix decomposition techniques to obtain an algorithm which outputs an LTF satisfying at least ( 1 / 12 ) -fraction of the bags of size  3 .
We also apply our techniques to bags of size q  4 to provide a ⌦ ( 1 / q ) -approximation guarantee for a weaker notion of satisﬁability.
We include comparative experiments on simulated data demonstrating the applicability of our algorithmic techniques.
From the complexity side we provide a hardness reduction to produce instances with bags of any constant size q .
Our reduction proves the NP-hardness of satisfying more than ( 1 / q ) + o (1) fraction of a satisﬁable collection of such bags using as hypothesis any function of constantly many LTFs, showing thereby that the problem is harder to approximate as the bag size q increases.
Using a strengthened analysis, for q = 2 we obtain a ( 4 / 9 ) + o (1) hardness factor for this problem, improving upon the ( 1 / 2 ) + o (1) factor shown by [27].
Introduction
Our work studies the computational learnability of linear threshold functions (LTFs) in the learning from label proportions (LLP) framework, which is a generalization of traditional supervised learning.
In this, a bag B is a set of some (say q ) feature vectors { x 1 , . . .
, x q } with a corresponding { 0 , 1 } -label proportion  B 2 [0 , 1] implying that exactly q  B out of the q feature-vectors have 1 as their true label.
Given a collection (or distribution) of ( B,  B ) consistent with an unknown classiﬁer, in LLP the goal is to ﬁt a feature-vector level classiﬁer hypothesis that matches the bag label proportions as closely as possible.
One way to formalize this is by deﬁning that a hypothesis classiﬁer satisﬁes a bag ( B,  B ) iff its predicted label proportion equals  B , with the goal being to maximize the number of bags satisﬁed by the hypothesis.
This notion of satisﬁability boils down to supervised learning when all bags are of size 1 , and is a reasonable measure of classiﬁer performance for small bags.
An LTF over d -dimensional feature-vectors x is given by pos ( g ( x )) for some linear function g ( x 1 , . . .
, x d ) =
P di =1 c i x i + c d +1 , where pos ( z ) :=
{ z> 0 } .
Recently, [27] studied the proper LLP learnability of LTFs i.e, given a collection of bags and their label proportions consistent with an unknown LTF, compute an LTF satisfying the maximum number of bags.
It is well known ([6]) that in supervised learning (all bags of size 1 ) LTFs are learnable by LTFs (i.e., all bags can be satisﬁed) using linear programming.
This however does not work for bags sizes > 1 , and neither are random
LTFs guaranteed to satisfy any signiﬁcant fraction of the bags.
The work of [27] studied this problem when all bags are of size  2 , giving an algorithm that satisﬁes at least ( 2 / 5 ) -fraction of all the bags, and
( 1 / 2 ) -fraction if all bags are non-monochromatic i.e.,  B 62 { 0 , 1 } for all bags B .
From the hardness side [27] showed that even on satisﬁable instances where all bags are non-monochromatic of size 2 , it is NP-hard to ﬁnd an LTF satisfying more that ( 1 / 2 ) + o (1) fraction of them.
The main algorithmic technique of [27] is based on the observation that the label proportion of a bag
B = { x 1 , x 2 } determines the sign of g ( x 1 ) g ( x 2 ) where pos ( g ) is a satisfying LTF with non-zero margin 1 i.e., g ( x 1 ) , g ( x 2 ) 6 = 0 .
Thus, one can write a collection of quadratic constraints over the coefﬁcients of g .
The corresponding semi-deﬁnite programming (SDP) relaxation can then be rounded using random hyperplanes to obtain the desired LTF.
However, the above approach is not directly applicable even for bags B = { x 1 , x 2 , x 3 } of size 3 since their label proportions no longer determine the products g ( x i ) g
( x j ) ( 1  i 6 = j  3 ).
Therefore, the following question remained: is there an efﬁcient algorithm which given a collection of ( B,  B ) s.t. | B |  3 consistent with some LTF, computes an LTF that satisﬁes at least ⌦ (1) -fraction of the bags .
Our work answers the above question in the afﬁrmative, using a fairly non-trivial SDP relaxation and new techniques to analyze the rounding algorithm.
In particular, we show that if allowed the presence of certain boolean variables the problem admits a non-quadratic formulation which nevertheless can be relaxed to an SDP.
For further analysis we prove a novel characterization of the condition A ⌫ B for two symmetric positive semi-deﬁnite (psd) matrices A and B in terms of their decomposition.
Our algorithm provides an LTF satisfying at least ( 1 / 12 ) -fraction of the bags of size  3 .
For bags of sizes  4 , we adapt this approach to provide a ⌦ ( 1 / q ) -approximation for a weaker notion of bag satisﬁability which is the same as satisﬁability for monochromatic bags, but only requires splitting the non-monochromatic bags.
We also show hardness reduction to this problem for bags of any constant size q  2 .
Unlike the reduction of [27], ours produces a mixture of non-monochromatic and monochromatic bags, and for general bag sizes q 2 Z + it yields a ( 1 / q ) + (1) hardness factor for any boolean function of constantly many LTFs as hypothesis, providing evidence that the problem becomes harder as the bag size q increases.
For the speciﬁc case of q = 2 we obtain a hardness factor of ( 4 / 9 ) + o (1) improving on the ( 1 / 2 ) + o (1) bound of [27].
An overview of our algorithms, hardness result and their analysis is provided later in this section.
1.1 Previous Related Work
The study of LLP is motivated by applications in which only the aggregated labels for sets (bags) of feature vectors are available due to privacy or legal [26, 28] constraints or inadequate or costly supervision [11, 10].
LLP has been applied to several weakly supervised tasks, for e.g. IVF prediction [19] and image classiﬁcation [7, 22].
Notably, small bag sizes – studied in this work – arise in real-world scenarios, e.g. [22] consider bags of size 50, and bag sizes 10 ⇠ 20 are relevant for IVF applications (see Sec 1.2 of [3]).
While there have been several applied works on LLP (e.g. [24, 29, 13]) the study of computational learning in the LLP framework has been – apart from the work of [27] – fairly sparse.
The LLP framework (as an analogue of PAC learning) was ﬁrst formalized in the work of [30].
They bounded the generalization error of a trained classiﬁer when taking the ( bag, label-proportion ) -pairs as instances sampled iid from some distribution.
Their loss function was different – a weaker notion than the strict bag satisfaction predicate that [27] and our work use.
As mentioned, LTFs [6] are well known to be properly learnable without any distributional assumptions.
In the presence of adversarial label noise however the problem is NP-hard even to approximate [1, 4, 9] with the optimal ( 1 / 2 + " ) -factor hardness shown by [14, 17], and generalized by [5] to hold even for constant degree polynomial thresholds as hypotheses.
1.2 Problem Deﬁnition
For an integer q , an instance of LLP-LTF [ q ] consists of ( X , B =
{ B ` } m ` =1 , {  ` } m ` =1 ) where X = { x 1 , . . .
, x n } ✓ R d is a set of feature-vectors, and B = { B 1 , . . .
, B m } ✓ 2 X s.t.
| B j |  q , is a collection of bags each of size at most q .
For each bag B ` there is a number  ` which is the sum of the { 0 , 1 } -labels of the vectors in the bag, satisfying  ` 2 { 0 , . . .
, | B ` |} , with the label proportion given by  ` :=  ` / | B ` | .
When  ` 2 { 0 , 1 } then B ` is said to be monochromatic i.e., bags which have same label (either 0 or 1 ) for all their feature-vectors.
The remaining bags B ` necessarily of size > 1 are called non-monochromatic .
A bag B ` 2 B is satisﬁed by some F : X ! { 0 , 1 } if
[Equation]
=
[Equation]
=
` | B ` | .
We say that a bag is split by F if
[Equation]
. , | B ` |  1 } , while it is unsplit by F if the latter assigns the same label to all the vectors in the bag.
We say that a bag B ` is weakly satisﬁed by F if (i) B is monochromatic and is satisﬁed by F , or (ii) B is non-monochromatic and is split by F .
Note that weak satisﬁability is implied by satisﬁability.
An instance of LLP-LTF [ q ] is said to be satisﬁable if there exists an LTF that satisﬁes all the bags.
It is said to be weakly satisﬁable if the LTF weakly satisﬁes all the bags.
The goal is to ﬁnd an LTF that (weakly) satisﬁes the most bags.
Choice of objective.
The satisﬁability condition is a natural generalization of the “classiﬁcation” objective in supervised learning in which a { 0 , 1 } -labeled example is either classiﬁed correctly or incorrectly.
For small-sized bags, it is also a reasonable approximation to objectives based on the deviation of
[Equation]  from  ` . More importantly, as we shall see later in this paper, the satisﬁability objective allows for a compact and tractable SDP relaxation in which any feasible solution can be rounded to an LTF with (in expectation) a non-trivial approximation guarantee. 1.3 Our Results
Our algorithmic result for satisﬁable LLP-LTF [3] is as follows.
Theorem 1.1.
Let I be a satisﬁable LLP-LTF [3] instance with m bags partitioned into m 0 monochromatic bags of size  2 , m 1 non-monochromatic bags of size 2 , m 2 monochromatic bags of bags of size 3 , and m 3 non-monochromatic bags of size 3 .
Then, there is a randomized polynomial time algorithm which on input I produces and LTF that satisﬁes in expectation at least (( m 0 / 2 + m 2 / 4 + m 3 / 6 ) /
+ m 1 / 2 ) bags.
In the worst case, (if m = m 3 ) the algorithm satisﬁes in expectation at least ( 1 / 12 ) -fraction of the bags.
The following theorem states our hardness result for satisﬁable LLP-LTF [ q ] and the improved hardness for satisﬁable LLP-LTF [2] .
Theorem 1.2.
For any ` 2 Z + and constant ⇣ > 0 it is NP-hard to ﬁnd any boolean valued function f of ` LTFs that satisﬁes more than ( 1 / q + ⇣ ) -fraction of the bags of a satisﬁable LLP-LTF [ q ] instance.
For q = 2 in particular, a strengthened result holds with a hardness factor of ( 4 / 9 + ⇣ ) .
We also provide the following algorithm for weakly-satisfying bags of a weakly-satisﬁable LLPLTF [ q ] instance for any q 2 Z + .
Theorem 1.3.
Let I be a weakly-satisﬁable LLP-LTF [ q ] instance with m bags.
Then, there is a randomized polynomial time algorithm which on input I produces an LTF that weakly-satisﬁes in expectation at least ( c 0 m / q ) bags for some absolute constant c 0 > 0 .
1.4 Overview of the Algorithm
First, observe that it is the non-monochromatic bags that make the LLP-LTF problem difﬁcult, as one can simply use linear programming to ﬁnd an LTF satisfying all the monochromatic bags.
This
LTF may however not satisfy even a single non-monochromatic bag.
Let us ﬁrst see how the algorithm of [27] for satisﬁable LLP-LTF [2] proceeds.
Since we can always append a coordinate with 1 to all feature vectors, assume that the satisfying LTF is given by pos ( h r , x i ) (where r is the normal vector of the separating hyperplane) with non-zero margin, the latter is possible by perturbing the LTF if necessary.
For a bag B = { x 1 , x 2 } , h r , x 1 ih r , x 2 i is either positive or negative depending on whether the bag is monochromatic or non-monochromatic.
There is a straightforward relaxation of this quadratic program to an SDP - substitute rr T with a symmetric psd matrix R and replace h r , x 1 ih r , x 2 i by x T 1 Rx 2 .
Solving this SDP and using the psd decomposition R =
L T L one obtains the same sign pattern for h Lx 1 , Lx 2 i .
Further, the non-zero margin property guarantees k Lx k
22 = h Lx , Lx i =
x T Rx > 0 for all the feature vectors x of the instance.
A standard hyperplane rounding of Lx and taking the best of the obtained LTF or its negation yields a random LTF that satisﬁes non-monochromatic bags with probability 1 / 2 and the monochromatic ones with probability 1 / 4 .
Note that the above algorithm crucially hinges on the fact that the label proportion of the 2 -sized bag determines the sign of h r , x 1 ih r , x 2 i .
This clearly is no longer true for a non-monochromatic B = { x 1 , x 2 , x 3 } of size 3 , and therefore it doesn’t seem possible to write an SDP relaxation with only terms of the form x T
i Rx j and solve for R as the relaxation of rr T .
Nevertheless, we observe that at least one of the two products h r , x 1 ih r , x j i ( j = 2 , 3 ) is negative.
Let us deﬁne boolean variables s { i,j } to be indicator of the event that h r , x i ih r , x j i < 0 .
Then, we have the following valid inequalities:
[Equation]
and
[Equation]
Of course, such constraints do not yield an SDP or a convex program due to the presence of the unknown variables s { i,j } in products with R .
The key step for obtaining an SDP is to relax s { i,j } R to a symmetric psd matrix
R { i,j } with the constraint R ⌫ R { i,j } which is valid since s { i,j } 2 { 0 , 1 } .
Now, the above two constraints can be rewritten as
[Equation]
and
[Equation]
From the last constraint above, we have x T 1 R { 1 , 2 } x 1 + x T 1 R { 1 , 3 } x 1  x T 1 Rx 1 , and assuming x T 1 R { 1 , 2 } x 1  x T 1 R { 1 , 3 } x 1 WLOG we have
[Equation]
along with,
[Equation]
The above suggests that the angle between Lx 1 and Lx 2 cannot be too small, where
R =
L T L .
Indeed, suppose for the moment that we could replace the LHS of the ﬁrst inequality above with h Lx 1 , z i and the LHS of the second inequality with h Lx 2 , z i with the guarantee that k z k 2  k Lx 1 k 2 .
A simple calculation shows that the angle between z and Lx 1 is at most ⇡ / 3 , while the angle between z and Lx 2 is at least ⇡ / 2 , implying a lower bound of ⇡ / 6 on the angle between Lx 1 and Lx 2 .
Thus, random hyperplane rounding will separate Lx 1 and Lx 2 with probability at least 1 / 6 , and the obtained LTF or its negation will satisfy the bag with probability at least 1 / 12 .
The only question that remains is whether such a z as assumed above exists.
We answer this in the afﬁrmative by proving the following: given psd A , 9 L s.t.
A =
L T L , and for any psd B these two conditions are equivalent: (i) A ⌫ B ; and (ii) , 9 C s.t B =
L T C and A ⌫ C T C .
Moreover, L is efﬁciently obtained by the spectral decomposition of A .
For our analysis, letting A = R and B =
R { 1 , 2 } , we can take z =
Cx 1 , and the last implication of (ii yields k Lx 1 k 2  k z k 2 .
This decomposition characterization of A ⌫ B for psd A , B seems novel to the best of the authors’ knowledge, and may prove useful in other geometric and SDP rounding techniques.
It is easy to see that (ii) )
(i).
The proof of the other direction is based on a speciﬁc choice of L which yields the decomposition of B =
L T C .
To show A ⌫ C T C we invoke a variant of Schur complement positive deﬁniteness condition.
For monochromatic 3 -sized bags we use a standard SDP relaxation and random hyperplane rounding analysis.
We extend the above algorithm to weakly satisfy bags of a weakly satisﬁable LLP-LTF [ q ] instance for q  4 .
Such instances also admit an analogous analysis for non-monochromatic bags as above and we obtain ( ⇤ ) and ( ⇤⇤ ) except with a factor of 1 / ( q  1) instead of 1 / 2 , yielding an ⌦ (1 /q ) probability for random hyperplane rounding splitting q -sized non-monochromatic bags.
Our techniques are also applicable to the related multiple instance learning (MIL) [7] of LTFs and we include an explanation in Appendix L.
Obtaining guarantees for satisfying non-monochromatic bags size of q  4 seems to require qualitatively stronger geometric techniques and in Appendix H of the supplementary we describe the technical issues in more detail.
We include an experimental validation of our algorithm for LLP-LTF [3] on simulated data, showing that our method outperforms random LTF classiﬁer, especially in the small margin scenarios.
In these scenarios, the LTF of our algorithm has high predictive accuracy on instance-level test data, demonstrating the practical applicability of our algorithmic methods.
1.5 Overview of Hardness for LLP-LTF [ q ]
The hardness reduction uses the template of a dictatorship test and combines it with a variant of the
Label Cover problem.
A dictatorship test over a domain [ M ] produces an instance I of the target problem, in our case LLP-LTF [ q ] , such that (i) (completeness) corresponding to each
i
[ M ] there is an LTF satisfying all bags of I , (ii) (soundness) an LTF that does not have any distinguished (relatively large) coefﬁcients does not satisfy more than some  < 1 fraction of the bags.
The crux is to construct dictatorship tests with large completeness vs soundness gap i.e., small  .
Fix any r 2 { 1 , . . .
, q } and consider the following distribution D r on bags of q feature vectors
X (1) , . . .
, X ( q ) 2 R M , each bag with label proportion r/q .
First, sample Z 2 R M ⇥ q so that each row of Z i is sampled iid uniformly from the set of vectors in { 0 , 1 , 2 } q which have exactly one coordinate with 2 , ( r  1) with 1 and rest 0 .
We derive the vectors X (1) , . . .
, X ( q ) from Z as follows for each j 2 [ q ] : if Z ij is 0 then set X ( j ) i = 0 , if Z ij is 1 then set X ( j ) i =  .
Independently for each i where Z ij = 2 , set X ( j ) i
=  w.p. (1  " ) , set X ( j ) i
= 1 w.p. " / 2 and set X ( j ) i = 2 w.p. " / 2 .
Here  is taken to be small depending on M and q , while " is a small constant depending on q but not on [ M ] .
Note that for any i , there exactly r of the q vectors X (1) , . . .
, X ( q ) have non-zero entries in the i th coordinates.
Thus, each coordinate yields an LTF pos ( X i ) which satisﬁes all the bags.
For the soundness analysis, consider any LTF given by pos ( h ( X )) , such that it has no large coefﬁcients.
Observe that { h ( X ( j ) ) } qj =1 are identically distributed but not necessarily independent, while conditioned on Z they are independent but not identical.
Using a fairly involved analysis we show is that there is a ﬁxed Gaussian distribution N ( µ, ⌃ ) (independent of the choice of Z , r) such that with high probability over the choice of Z each of { h ( X ( j ) ) } qj =1 are distributed close to N ( µ,  ) .
In effect, this implies that the probability that the bag is satisﬁed is at most,  r, ↵ + o (1) , where  r, ↵ :=
[Equation] and ↵ [Equation]
[Equation] , whre E is the expectation operator.
The above invariance is obtained through the randomness induced by the noise coordinates in X ( j ) for a given j i.e, those i for which Z ij is sampled to be 2 , on which X ( j ) i are independently sampled to be 1 or 2 w.p. " / 2 each.
Due to their small magnitude the  -valued coordinates in X ( j ) i can essentially be ignored.
After estimating bounds on the conditional (on Z ) expectation and variance of h ( X ( j ) )
we apply the Berry-Esseen theorem to obtain the desired invariance.
We can construct the D r on bags of q with label proportions r/q for all r 2 { 1 , . . .
, q } .
We note that the noise coordinates are identically distributed in each D r .
Thus, we are able to use the same µ and ⌃ for each r to obtain the  r, ↵ + o (1) bound for each r with the same ↵ .
If we weigh each of these distributions uniformly, using the easy derivation that
P qr =1  r, ↵  1 for ↵ 2 [0 , 1] , we obtain a (1 /q + o (1)) factor hardness.
For q = 2 , we obtain a better 4 / 9 + o (1) factor using explicit calculations.
Lastly, we use the folding trick to encode the Label Cover and combine the above dictatorship test only on the [ M ] labels of the Label Cover vertices.
This combination and the label decoding is along the lines as previous works e.g. by [18].
Like the reduction of [27], ours also works for functions of constantly many LTFs as hypotheses, requiring the application of the multi-dimensional version of
Berry-Esseen theorem.
The approach of decoupling by conditioning on Z is similar in spirit to that followed by [27] though their reduction has boolean coordinates which does not readily admit generalizations to larger bag sizes q .
The main contribution of our hardness result is the design and analysis a dictatorship test that works for all bag sizes q yielding bag-distributions of speciﬁc label proportions r/q ( r = 1 , . . .
, q ) with random-threshold like soundness  r, ↵ + o (1) .
Organization of the paper.
The next section provides some mathematical preliminaries and the proof of our novel characterization of A ⌫ B for psd matrices.
The latter is used in the proof of
Theorem 1.1 in Sec. 3 which provides and analyzes our algorithm A for LLP-LTF [3] .
Sec.
4 presents an experimental evaluation of our algorithm on simulated data.
In Sec. 5, Theorem 1.2 is derived from the statement of our hardness reduction whose proof is deferred to the Appendix C.
The proof of Theorem 1.3 is also omitted and appears in Appendix G.
Preliminaries
We state a few well known facts about matrices.
The pseudo-inverse of a diagonal matrix D =
Diag (  1 , . . .
,  r , 0 . . . , 0) with top r non-zero entries and the rest 0 is given by D † :=
Diag (   1 1 , . .
,   1 r , 0 . . . , 0) .
A symmetric matrix A has a decomposition A = UDU T for some diagonal matrix D and orthonormal matrix U i.e., satisfying
UU T =
U T U
= I .
The pseudo-inverse is A † =
UD † U T .
Deﬁnition 2.1 (see [21, 8]) .
For a real symmetric n ⇥ n matrix A , the following conditions are equivalent: (1) A ⌫ 0 , i.e A is positive semi-deﬁnite (psd), (2) UAU T ⌫ 0 for all orthonormal matrices U , (3) x T Ax  0 for all x 2 R n ,
(4) A = UDU T for some orthonormal U with D being a non-negative diagonal matrix(spectral decomposition), (5) all the principal minors of A have non-negative determinant.
For any two matrices, the Loewner order is given by A ⌫ B , A  B ⌫ 0 .
The square-root of a non-negative diagonal matrix D =
Diag (  1 , . . .
,  n ) is D 1 / 2 :=
Diag (  1 / 2 1 , . . .
,  1 / 2 n ) .
For a psd
A = UDU T , the square root is A 1 / 2 = UD 1 / 2 U T .
The following lemma, a variant of the the
Schur-complement deﬁniteness property, can be found on page 88 of [8], see also Thm.
4.3 of [15].
Lemma 2.2.
For any n ⇥ n matrices A , B and C where A and C are symmetric, let X =
A B B T C  .
Then, X ⌫
[Equation]
psd [Equation]
We prove the following lemmas which are used in our algorithmic results.
Lemma 2.3.
Given a real symmetric psd matrix A , 9 L s.t.
A =
L T L and the following are equivalent: (i) A ⌫ B , and (ii) 9 C s.t.
B =
L T C and A ⌫ C T C , for any real symmetric psd matrix B .
Further, L can be efﬁciently obtained from the spectral decomposition of A .
Proof.
It is easy to see that (ii) )
(i) as follows.
Considering any vector x we have,
[Equation]
where we use A ⌫ C T C and A =
L T L .
Thus, using (1)
[Equation]
Thus, (ii) )
(i).
The reverse is proved in Lemma 2.4 along with the explicit formula for L .
Lemma 2.4.
Let A and B be two real, symmetric, psd k ⇥ k matrices such that A ⌫ B ( ‡ ) .
Then, with the spectral decomposition A = UDU T =
L T L where U is orthonormal, D is non-negative diagonal and L = D 1 / 2 U T , there exists C
such that (i) B =
L T C , and (ii)
A ⌫ C T C .
Proof.
Let C :
= U T BU be symmetric psd (Defn. 2.1).
Condition ( ‡ ) of the lemma implies,
[Equation]
Suppose that D has top r diagonal elements positive and the rest zero.
Then C is zero outside of the top r ⇥ r submatrix.
Otherwise, D  C will have nonzero entries  C ir
=
C r 0 i in the ( i, r 0 ) and ( r 0 , i ) entries for some r 0 > r
and i .
On the other hand, the diagonal entry at ( r 0 , r 0 ) is  C r 0 ,r 0 =
N O S PLIT ( u 1 , u 2 , u 3 , Q ) :
[Equation]
[Equation]
PLIT
[Equation]
[Equation]
[Equation]
[Figure]
Given ( { ˜ x i } ni =1 , { B ` } m ` =1 , {  ` } m ` =1 ) .
Vars: real, symmetric psd R , R { i,j } 1  i < j  n , s.t.
8 i [Figure] [ n ] :
[Figure]
T
[Figure]
x i > 0 (13)
[Figure] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
since both ( D  C ) and C are psd and have non-negative diagonals, and thus the 2 ⇥ 2 principal minor of D  C given by the i th and r 0 th rows/columns has a negative determinant which contradicts
Defn.
Since C is zero outside of the top r ⇥ r submatrix, letting I r be diagonal matrix with ones in the top k entries and zero otherwise we have,
[Equation]
Letting C :
=
[Equation]
† CU T yields property (i) of the lemma. For the second property observe that,
[Equation]
using which A ⌫ C T C , U T AU ⌫ U T C T CU , D ⌫ CD † C ( X ⌫ 0 , where
X =
[Equation]
, and the last implication follows from Lemma 2.2.
It remains to show that
X ⌫ 0 .
For this let z = ( x 1 , . . .
, x k , y 1 , . .
. , y k ) , and x = ( x 1 , . .
, x k ) ,
y = ( y 1 , . . .
, y k ) .
Then,
[Equation]
Since C is symmetric psd we can write it as V T V so that
[Equation]
Substituting 2 x T Cy
[Equation]
T
[Equation]
+ y T
[Equation]
into the RHS of (4) we obtain,
[Equation]
[Equation]
by (2) which holds for any z .
Thus, X is psd which completes the proof.
Algorithm for LLP-LTF [3] 3.1 SDP Relaxation
We deﬁne two collections of constraints N O S PLIT and S PLIT for monochromatic and nonmonochromatic bags of size 3 respectively in Fig.
For a satisﬁable instance I =
( X = { x 1 , . . .
, x n } ✓ R d , B = { B ` } m ` =1 , {  ` } m ` =1 ) of LLP-LTF [3] let ˜ x
i 2 R d +1 be given by appending an extra 1 -valued coordinate to x i for i 2 [ n ] .
With this the corresponding SDP relaxation is given in
Fig. 2, and it enforces N O S PLIT constraints for monochromatic bags of size 3 and those given by
S PLIT for the non-monochromatic 3 -sized bags.
Constraints for margin and bags of size 2 are the same as in the algorithm of [27].
Feasibility of SDP-I. As discussed in Sec.
1.4, if pos ( h r , ˜ x i ) is the satisfying LTF, then we can set
R = rr T and R { i,j } =
R if h r , ˜ x i ih r , ˜ x j i < 0 and 0 otherwise.
The arguments for the margin and 2 -sized bag constraints are same as those in Sec 2.1 of [27], and those for the 3-sized bag constraints are informally presented in Sec.
We defer the formal proof to Appendix A.
3.2 SDP Algorithm and analysis
Fig.
3 provides the algorithm A for the satisﬁable LLP-LTF [3] instance I .
We have the following lemma for bags of size 3 .
Lemma 3.1.
Consider the linear form h obtained in Step 5 of A (Fig. 3).
Then, the probability of a non-monochromatic 3 -size bag being split by pos ( h ( . )) is at least 1 / 6 , and that of a 3 -sized monochromatic being unsplit by pos ( h ( . ))
is at least 1 / 4 .
Proof.
Let B be a bag of size 3 and by relabeling WLOG we can assume that B = { x 1 , x 2 , x 3 } .
Case: B non-monochromatic.
Using (10) we have
[Equation]
(18) where L is as deﬁned in Step 3 of A (Fig. 3).
By averaging and WLOG we can assume that ˜ x
[Equation]
and by applying Lemma 2.4 to the guarantee that R ⌫ R { 1 , 2 } (from (9)) we obtain that there exists a matrix C s.t.,
[Equation]
19) and
[Equation]
Further, using (8)
[Equation] [Equation]
Eqn. (13) implies k L ˜ x
b k 2 > 0 ( b = 1 , 2 ), and by (19) we also have k C ˜ x 1 k 2 > 0 .
Deﬁne the unit vectors:
[Equation]
From (19), (20) and (21) we obtain that h z 0 , z 1 i  1 / 2 , and h z 0 , z 2 i  0 .
For b = 0 , 1 we can write z b =
c b
z 0 + c b 1 z ?
b where k z ?
b k 2 = 1 and z ?
b ?
z
so that c 2 b 0 + c 2 b 1 = 1 .
Note that h z 0 , z 1 i  1 /implies that c 10  1 / 2 and therefore | c 11 |  p 3 / 2 .
Further, h z 0 , z 2 i  0 implies that c 20  0 .
Thus,
[Equation]
Thus, the angle between L ˜ x 1 and L ˜ x 2 is at least ⇡ / 6 .
From standard facts on random hyperplane rounding (see Appendix A of [27]) it is easy to see that pos ( h ( x 1 ))
6 = pos
( h ( x 2 )) with probability at least ( ⇡ / 6) / ⇡ = 1 / 6 .
Case: B monochromatic.
In this case, (13), (7) guarantee that { L ˜ x b | b = 1 , 2 , 3 } are non-zero vectors with pairwise non-negative inner products.
It is a well known fact (see [16]) that such vectors can be rotated to be contained in a three-dimensional orthant (cone subtended by three coordinate rays).
Thus, the probability that the bag is unsplit by pos ( h ( . )) is at least the probability that the inner products of three orthonormal vectors with g (as chosen in Step 4 of A ) all have the same sign.
Each of these three inner products is an independent standard Gaussian, so the latter probability is 1 / 4 .
Since our algorithm A when restricted bags of size  2 is the same as that given by [27], we can reuse the following lemma which summarizes the analysis in Sec.
2 of [27].
Lemma 3.2 (Sec. 2 of [27]) .
Any monochromatic bag of size  2 is unsplit by pos ( h ( . )) with probability at least 1 / 2 .
Any non-monochromatic bag 2 -sized bag is split by pos ( h ( . )) with probability at least 1 / 2 .
Further, h ( x i ) 6 = 0 ( 1  i  n ) w.p. 1.
Assuming that h does not vanish any x i (which happens w.p. 1) we obtain the following properties.
If a monochromatic bag is usplit by pos ( h ( . )) then it is satisﬁed by exactly one of pos ( h ( . )) and pos (  h ( . )) .
This also holds for any non-monochromatic bag of size 3 split by pos ( h ( . )) .
On the other hand a non-monochromatic bag of size 2 , if split by pos ( h ( . )) , is satisﬁed by both pos ( h ( . )) and pos (  h ( . )) .
This, along with Step 6 of A completes the proof of Theorem 1.1.
An analysis of the time complexity of A (which is asymptotically dominated by the time taken to solve the SDP) is provided in Appendix I.
Experimental Evaluation
[Table]
( A ) vs rand LTF ( R ). Bag size 3 . We compare our algorithm ( A ) to random LTF ( R ) evaluated on 25 instances for each row of Table 1 giving the avg.
% bags satisﬁed by each method, and the last two columns providing the accuracy on test dataset obtained by sampling a bag (same as the bag distribution) and sampling u.a.r.
one of the three featurevectors from the bag.
For each instance, m bags (of 3 d -dim.
vectors each) are sampled, where each is non-monochromatic w.p. 3 / 4 .
The small and large margin cases are analogous to the correlated and uncorrelated cases in the experiments of [27], and we similarly follow a best-of 5 -trials based rounding for A and best-of 5 u.a.r.
LTFs or their complements for R .
We see that (i) A satisﬁes on avg.
80-97% of the bags in the small margin cases, vastly outperforming R , the average feature-vector level test accuracy of the LTF produced by our algorithm is quite high: 96-98% for d = 10 and 85-90% for d = 40 , while that of random LTF is rather low at around 50-55%.
(ii)
A also betters R in most of the large margin cases.
Additional details are included in Appendix K which also provides similar experimental evaluation for weakly-satisfying LLP-LTF [4] . Remark. The SDP formulation in our experiments for 3 -sized bags differs slightly from the one in
Fig. 2 by using alternate valid constraints for non-monochromatic bags.
In particular, instead of x T i
{+R { i,j } x j  0 , i 
 = j 2 { 1 , 2 , 3 } (as described in Sec. 1.4) we add x T i R { i,j } x j + x T i R { i,k } x k <for each { i, j, k } = { 1 , 2 , 3 } .+}
It is easy to see that the new inequalities imply that there is i 2 { 1 , 2 , 3 } such that for each j 2 { 1 , 2 , 3 } \ { i } , x T i
R { i,j } x j < 0 .
Using this condition the rest of the analysis can be done as before yielding the same approximation guarantee, while it provided better observed experimental performance.
We defer a formal explanation to Appendix J.
Hardness Result
The following theorem, whose proof is provided in Appendix C, states our detailed hardness result .
Theorem 5.1.
For positive integers constants q > 1 , `  1 , and any constants ⇣ > 0 and { p r } qr =1 s.t.
P qr =1 p r = 1 , given an instance I of LLP-LTF [ q ] with p r fraction of bags of size q and label proportion r/q , for r 2 { 1 , . . . , q } , it is NP -hard to distinguish between the following cases:
YES Case.
There is an LTF that satisﬁes all the bags of I .
NO Case.
Any { 0 , 1 } -function f of at most ` LTFs satisﬁes at most  q,p 1 ,...,p q + ⇣ fraction of the bags in I where  q,p 1 ,...,p q := max ↵ 2 [0 , 1] (
P r =1 p r  q,r, ↵ ) and  q,r, ↵ :=
qr  ↵ r (1  ↵ ) q  r .
Proofs of Theorem 1.2.
We apply Theorem 5.1 with p r = 1 /q for r 2 [ q ] .
In the NO case, the total fraction of bags satisﬁed by f is  := max ↵ 2 [0 , 1] ⇣ 1 q
P qr =1  q,r, ↵ ⌘ + ⇣ for an arbitrarily small constant ⇣ > 0 .
Observing that
P qr =1  q,r, ↵ 
P qr =
q,r, ↵ =
( ↵ + (1  ↵ ))
q = 1 , we obtain that   1 /q + ⇣ .
This, along with the Yes case, proves Theorem 1.2 for LLP-LTF [ q ] .
For the case of q = 2 we show (in Appendix B) that min p 2 [0 , 1] max ↵ 2 [0 , 1] p ↵ 2 +2(1  p ) ↵ (1  ↵ )
= 4 / 9 to obtain a 4 / 9 + ⇣ hardness factor.
Conclusions
Our work develops novel linear algebraic techniques to design and analyze a non-trivial SDP relaxation based (1 / 12) -approximation for satisﬁable LLP-LTF [3] , for which no previous algorithm (other than trivial or random LTF) was known.
We also prove a 1 /q + o (1) factor hardness for
LLP-LTF
[ q ] for all constant q , and a strengthened 4 / 9 + o (1) factor for q = 2 , improving on the previous 1 / 2 + o (1) factor [27].
We extend our algorithm to bag sizes q  4 for for weaker notion of bag-satisﬁability, obtaining ⌦ (1 /q ) -approximate algorithm.
Experiments on simulated data of 3 -sized bags shows that our algorithm can provide substantially improved (over random LTFs) performance, both in terms of bag satisﬁability as well as on featurevector level test evaluation.
The main open question in this line of work is to develop algorithms for satisﬁable LLP-LTF [ q ] for q  4 .
Of course, learnability in the LLP setting can also be studied for other natural classiﬁers such as DNF formulas and decision trees.
Another interesting direction is to study variants of the bag satisﬁabliltiy objective such as those which minimize the average deviation (according to some distance e.g. ` 1 or ` 22 ) between the given bag label proportions and those induced by the solution classiﬁer.