Submodular Maximization in Clean Linear Time
Abstract
In this paper, we provide the first deterministic algorithm that achieves 1 / 2approximation for monotone submodular maximization subject to a knapsack constraint, while making a number of queries that scales only linearly with the size of the ground set n .
Moreover, our result automatically paves the way for developing a linear-time deterministic algorithm that achieves the tight 1 − 1 /e approximation guarantee for monotone submodular maximization under a cardinality (size) constraint.
To complement our positive results, we also show strong informationtheoretic lower bounds.
More specifically, we show that when the maximum cardinality allowed for a solution is constant, no deterministic or randomized algorithm making a sub-linear number of function evaluations can guarantee any constant approximation ratio.
Furthermore, when the constraint allows the selection of a constant fraction of the ground set, we show that any algorithm making fewer than Ω( n/ log( n )) function evaluations cannot perform better than an algorithm that simply outputs a uniformly random subset of the ground set of the right size.
We extend our results to the general case of maximizing a monotone submodular function subject to the intersection of a p -set system and multiple knapsack constraints.
Finally, we evaluate the performance of our algorithms on multiple real-life applications, including movie recommendation, location summarization,
Twitter text summarization, and video summarization.
Introduction
In the era of big data, one of the main challenges that we daily face is to make an informed decision while taking into account all the relevant information.
The study of how to make such near-optimal decisions from a massive pool of possibilities is at the heart of combinatorial optimization.
It is well-known that without further structures, finding a near-optimal solution for a combinatorial optimization problem is notoriously hard, similar to finding a global minimum of a general non-convex function.
Nevertheless, the problems of practical interest, such as the ones we encounter in machine learning applications, are often much more well-behaved which makes them amenable to (exact or approximate) optimization techniques.
One celebrated combinatorial structure is submodularity, also known as diminishing returns, stating that the added value of an element (e.g., image, sensor, etc.) to a context (data set of images, sensor network, etc.) decreases as the context in which it is considered grows.
Submodular functions have attracted significant interest from the machine learning community, and have led to the development of algorithms with near-optimal solutions for a wide range of applications, including outbreak detection [52], graph cuts in computer vision [40], image and video summarization [22, 32, 59, 70], active learning [19, 27, 28], compressed sensing and structured sparsity [3, 16], fairness in machine learning [13, 41], recommendation [60], human brain parcellation [64], model training [57, 58], and learning causal structures [66, 75].
For recent surveys on the applications of submodular functions in machine learning and artificial intelligence, we refer the interested reader to [7, 69].
In their pioneering work, Nemhauser et al.
[62] showed that a greedy algorithm achieves a tight 1 − 1 /e approximation for maximizing a monotone submodular function subject to cardinality constraint ( SMC ).
Running the greedy algorithm is too expensive as it requires O ( nk ) function evaluations, where n is the size of the ground set
and k is the maximum cardinality allowed by the constraint.
Since then, there has been a significant effort to develop faster algorithms [12, 55, 72] and extend the approximation guarantee to more complex constraints (such as knapsack and p -set system) [4, 32].
In this work, we propose the first deterministic linear time algorithm for maximizing a monotone submodular function subject to a knapsack constraints ( SMK ), achieving (1 / 2 − ε ) -approximation using O ε ( n ) function evaluations.
We note that some nearly-linear time algorithms have been previously proposed for this problem.
Ene and Nguyen [18] showed such an algorithm achieving (1 − 1 /e − ε ) -approximation, but the number of function evaluation used by their algorithm scales poorly with ε (namely, O ((1 /ε ) 1 /ε ) ), which makes it impractical even for moderate values of ε .
This has motivated Yaroslavtsev et al.
[73] to design an algorithm called Greedy+ that achieve ( 1 / 2 − ε ) - approximation and has a much more practical nearly-linear time complexity.
Nevertheless, our algorithm manages to combine the approximation guarantee of Greedy+ with a clean linear time.
For the special case of a cardinality constraint, i.e., when the elements of the ground set have the same cost, our algorithm automatically reduces to a deterministic linear-time algorithm that guarantees (1 − 1 /e − ε ) approximation via O ε ( n ) function evaluations.
This special result has also been independently obtained by [46, 47].
Previously, the state-of-the-art deterministic algorithm only managed to achieve this approximation ratio using O ε ( n log log n ) function evaluations [36] (albeit in their new arXiv version, [38] claimed a linear-time algorithm, but with a worse dependence on ε compared to our work).
Since the linear time complexity of our algorithm is the minimum required to read all the elements of the ground set, it is difficult to believe that any algorithm can improve over it while guaranteeing a reasonable approximation ratio.
However, note that a single function evaluation can provide information about many elements of the ground set, and therefore, it is not a priori clear that an algorithm making a sub-linear number of function evaluations may not exist (even if its total time complexity is linear).
Nevertheless, we are able to prove that this is indeed the case in at least two important regimes.
First, we show that when the maximum cardinality allowed for the solution is constant, no algorithm making a sub-linear number of function evaluations can guarantee any constant approximation ratio for SMC .
Second, when the constraint allows the selection of a constant fraction of the ground set, we show that any algorithm making fewer than Ω( n/ log( n )) function evaluations cannot perform better than an algorithm that simply outputs a uniformly random subset of the ground set of the right size.
Our technique also leads to a similar inapproximability result for unconstrained submodular maximization ( USM ).
These lower bounds, along with the corresponding linear-time algorithms, nearly complete our understanding of the minimum query complexity of
SMC , SMK , and USM problems.
It is important to note that our inapproximability results are based on information theoretic arguments rather than relying on any complexity assumptions.
We consider the more general setting where we maximize a monotone submodular function subject to a p -set system and d knapsack constraints ( SMKS ).
We study the trade-off between the time complexity and approximation ratio for this problem.
In particular, our results improve over the state-ofthe-art approximation for nearly-linear time algorithms [4].
Finally, we evaluate the performance of our algorithms using real-life and synthetic experiments, including movie recommendation, location summarization, vertex cover, twitter text summarization and video summarization.
1.1 Additional Related Work
The work of Badanidiyuru and Vondr´ak [4] is probably the first work whose main motivation was to find algorithms with guaranteed approximation ratios for submodular maximization problems that provably enjoy faster time complexities.
This work introduced the thresholding technique, which became a central component of basically every fast deterministic implementation of the greedy algorithm considered to date, including our algorithms.
In particular, using this technique, Badanidiyuru and Vondr´ak [4] described an O ε ( n log n ) time algorithm for maximizing a monotone submodular function subject to a cardinality constraint.
Sviridenko [67] was the first to obtain (1 − 1 /e ) -approximation for maximizing a monotone submodular function subject to a knapsack constraint.
Their original algorithm was very slow (having a time complexity of O ( n 5 ) ), but it can be improved to run in ˜ O ( n 4 ) time using the thresholding technique of [4].
The algorithm of Sviridenko [67] is based on making 3 guesses.
Recently, Feldman et al.
[24] showed that the optimal approximation ratio of 1 − 1 /e can be obtained using only 2 guesses, leading to ˜ O ( n 3 ) time.
They also showed that a single guess suffices to guarantee a slightly worse approximation ratio of 0 .
and requires only ˜ O ( n 2 ) time [24].
There exists also an impractical nearly-linear time algorithm for the problem due to [18].
Another fast algorithm was suggested by [4], but an error was later found in their analysis (see [18] for details).
We also note that multiple recent works aimed to bound the performance of
a natural (and historically important) greedy algorithm
A brief review on non-monotone submodular maximization subject to cardinality and knapsack constraints can be found in Appendix A.
When considering fast algorithms, it is important to consider also (semi-)streaming algorithms since these algorithms are usually naturally fast.
Among the first semi-streaming algorithms for submodular maximization was the work of Badanidiyuru et al.
[5], who obtained (1 / 2 − ε ) -approximation for SMC .
While this work was later improved by Kazemi et al.
[42] and extended to non-monotone functions [1], it was also proved that no semi-streaming algorithm can provide a better than 1 / 2approximation for this problem [23].
The more general case of a knapsack constraint was first considered in the semi-streaming setting by Yu et al.
Improved algorithms for this problem were later found by [37, 39] for single pass and by [38, 73] for multiple passes.
Maximizing a monotone submodular function subject to a p -set system constraint was first studied by Fisher et al.
[25], who showed that a natural greedy algorithm obtains ( p + 1) − 1 -approximation for this problem.
Many years later, Badanidiyuru and Vondr ´ ak [4] used their thresholding technique to get a nearly-linear time algorithm guaranteeing ( p + 2 d
+ 1 + ε ) − 1 -approximation even when there are d knapsack constraints (in addition to the p -set system constraint).
Recently, Badanidiyuru et al.
[6] presented a new technique that allows them to improve over the last approximation ratio.
They explicitly consider the implications of their ideas only for the regime where d ≤ p , and in this regime they achieve [2( p + 1 + ε )]
− 1 -approximation and ( p + 1 + ε ) − 1 -approximation in ˜ O
( nr/ε ) and ˜ O ( n 3 r 2 /ε ) time, respectively, where r ≤ n is the rank of p -set system.
We note, however, that some of the ideas of [6] can also be applied to the regime d > p , leading to ( p + d + 1) − 1approximation in ˜ O
( n 3 ) time.
1 One can observe that the improvement in the approximation ratio obtained by the new results of [6] comes at the cost of an increased time complexity.
Our results in Section 5 further study this trade-off between the time complexity and approximation ratio and improve over the state-of-the-art approximation for nearly-linear time algorithms.
Notations and Preliminaries
A set function f : 2 N → R over a ground set N is a function that assigns a numeric value to every subset of N .
Given such a function, an element u ∈ N and a set S ⊆ N , we denote by f ( u | S ) ≜
f ( S ∪ { u } ) − f ( S ) the marginal contribution of u to S with respect to f .
The set function f is submodular if f ( u | S ) ≥ f
( u | T ) for every two sets S ⊆ T ⊆ N and element u ∈ N \ T , and it is monotone if f ( u | S ) ≥ 0 for every element u ∈ N and set S ⊆ N .
All the set functions considered in this paper are also non-negative.
Additionally, given two sets S, T ∈ N , we define the shorthand f
( T | S ) ≜
f ( S ∪ T ) − f ( S ) .
Furthermore, given a set S and element u , we often use S + u as a shorthand for S ∪ { u } .
We assume throughout the paper that algorithms are able to access the objective function f only through a value oracle that given a set S ⊆
N returns f ( S ) .
This (standard) assumption allows us to prove inapproximability results that are based on information theoretic arguments, and therefore, are independent of any complexity assumptions.
A set system M is an ordered pair ( N , I ) , where N is a ground set
and I is a non-empty subset of 2 N that obeys the following property: if S ⊆ T ⊆ N and T ∈ I , then S also belongs to I .
It is customary to refer to the sets of I as independent (and to sets that do not belong to I as dependent).
A base of a set S ⊆
N is a subset of S that is independent and is not included in any other set having these properties.
The set system M = ( N , I ) is called p -set system for an integer p ≥ 1 if for every
S ⊆
N the ratio between the sizes of the largest base of S and the smallest base of S is at most p .
Knapsack Constraint
In this section we consider the Submodular Maximization subject to a Knapsack Constraint problem ( SMK ).
In this problem we are given a non-negative monotone submodular function f : 2 N → R ≥ 0 , a non-negative cost function c : N → R ≥ 0 and a budget B > 0 .
We say that a set S ⊆
N is feasible if it obeys c ( S ) ≤ B (where c ( S ) ≜ (cid:80)
u ∈ S c ( u ) ), and the objective of the problem is to find a set maximizing f
among all feasible sets S ⊆ N .
For simplicity, we assume below that c ( u ) ∈ (0 , B ] for every element u ∈ N and that B = 1 .
These assumptions are without loss of generality because (i) every element of N whose cost exceeds B can be simply removed, (ii) every element of N whose cost is 0 can be safely assumed to be part of the optimal solution ( f is monotone), and (iii) one can scale the costs to make the budget B equal to 1 .
When the knapsack constraint of SMK happens to be simply a cardinality constraint, we get a problem known as Submodular Maximization subject to a Cardinality Constraint ( SMC ).
Formally, in this problem we have an integer parameter 1 ≤ k ≤ n , and we are allowed to output any set whose size is at most k .
One can observe that SMC is a special case of SMK in which c ( u ) = 1 /k for every element u ∈ N .
The first step of our proposed solution to solve SMK and SMC problems is getting an estimate Γ of the value of an optimal solution (up to a constant factor).
We show how this can be done in Appendix B.
Then, in Section 3.1 we present a variant of the threshold-greedy algorithm of Badanidiyuru and
Vondr ´ ak [4] that, unlike the original algorithm of [4], uses the estimate Γ to reduce the number of thresholds that need to be considered.
In Section 3.2, by combining the threshold-greedy variant from Section 3.1 with a post-processing step, we show how to achieve the state-of-the-art algorithm for SMK .
The theoretical guarantee of this algorithm is given by Theorem 3.1.
Theorem 3.1.
For every ε > 0 , there exists a deterministic ( 1 / 2 − ε ) -approximation algorithm for Submodular Maximization subject to a Knapsack Constraint ( SMK ) that uses
O ( nε − 1 log ε − 1 ) time.
Complementary to our result for SMK , our proposed variant of the threshold-greedy algorithm leads to a deterministic linear algorithm for SMC , which is given by the next theorem.
Theorem 3.2.
For every ε > 0 , there exists a deterministic (1 − 1 /e − ε ) -approximation algorithm for Submodular Maximization subject to a Cardinality Constraint ( SMC ) using
O ( n/ε ) time.
3.1 Fast Threshold Greedy Algorithm
In this section, we present Algorithm 1 as a variant of the threshold-greedy algorithm from [4].
Algorithm 1 uses Γ , our estimate of the optimal value ( Γ is calculated by Algorithm 4 which is given in Appendix B).
Algorithm 4 enables us to reduce the number of thresholds that needed to be considered.
Specifically, Algorithm 1 considers an exponentially decreasing series of thresholds between 8 α · Γ and Γ /e , where α ≥ 1 is a parameter of the algorithm.
For every threshold considered, the algorithm adds to its solution every element such that (i) adding the element to the solution does not violate feasibility, and (ii) the density of the element with respect to the current solution exceeds that threshold.
In addition to the parameter α , Algorithm 1 gets a quality control parameter ε ∈ (0 , 1) .
The time complexity of Algorithm 1 is given by the next observation.
Observation 3.3.
The time complexity of Algorithm 1 is O ( nε − 1 log α ) .
Due to space constraints, the proof of this observation and the other proofs omitted from this section can be found in Appendix C.
Let ℓ be the value of h when Algorithm 1 terminated.
We analyze the approximation ratio of Algorithm 1 by describing two lower bounds on the performance of this algorithm.
The following lemma gives the simpler among these bounds.
Its proof is based on the observation that the elements of
T \ S ℓ were not added by Algorithm 1 to its solution even during the outer loop’s last iteration.
1 Let Γ (an estimate of the value of an optimal solution) be the output of Algorithm 4.
2 Let τ ← 8 α · Γ , h ← 0 and S 0 ← ∅ .
3 while τ > (1 − ε )Γ /e do for every element u ∈ N do if u ̸∈ S h , c ( u )
+ c ( S h ) ≤ 1 and f ( u | S h )
c ( u ) ≥ τ then
Let u h +1 ← u and S h +1 ← S h + u h
+1 .
Increase h by 1 .
Update τ ← (1 − ε ) τ .
9 return S h .
Lemma 3.4.
For every set T ⊆ N , if max u ∈ T c ( u ) ≤ 1 − c ( S ℓ ) , then f ( S ℓ ) ≥ f ( T ) − c ( T ) e · f ( OPT ) .
In particular, max u ∈ OPT c ( u ) ≤ 1 − c ( S ℓ ) implies f ( S ℓ ) ≥ (1 − 1 /e ) · f ( OPT ) .
The following lemma gives the other lower bound we need on the performance of Algorithm 1.
The proof of this lemma is based on the standard argument in greedy-like algorithms, namely, the observation that adding all the elements of T to a solution increases the value of the solution to at least T , and therefore, at least one element of T is a good choice that the algorithm can choose.
Lemma 3.5.
For every set ∅ ̸ = T ⊆ N and integer 0 ≤ h < ℓ , if max u ∈ T c ( u ) ≤ 1 − c ( S h ) , then f ( S h +1 )
− f ( S h ) c
( u h +1 ) ≥ min { (1 − ε ) · f ( T | S h ) c ( T ) , α · f ( OPT ) } .
Using the last two lemmata we can prove the following theorem, which implies Theorem 3.2.
In a nutshell, the proof of this theorem uses Lemma 3.4 when the output of Algorithm 1 has the maximum allowed cardinality, and Lemma 3.5 otherwise.
Theorem 3.6.
If we choose α = 1 , then, for every ε ∈ (0 , 1) , Algorithm 1 runs in O ( n/ε ) time and guarantees (1 − 1 /e − ε ) -approximation for SMC .
3.2 General Knapsack Constraint in Clean Linear Time
In this section we prove Theorem 3.1.
We should point out that Algorithm 1 by itself does not guarantee any constant approximation guarantee for general knapsack constraints.
However, this can be fixed by using the post-processing step described by Algorithm 2.
Recall that Algorithmmaintains a solution S k that grows as the algorithm progresses.
The post-processing step takes
O ( ε − 1 log ε − 1 ) snapshots of this solution at various times during the execution of Algorithm 1, and then tries to augment each one of these snapshots with a single element. Algorithm 1 Execute Algorithm 1 with α = ε − 1 .
Store all sets S 0 , S 1 , . . .
, S ℓ produced by this execution.
2 for i = 0 to ⌊ log 1+ ε ε − 1 ⌋ do
Let maximal value maximizing The time complexity of Algorithm 2 is given by the next observation.
Observation 3.7.
Algorithm 2 runs in O ( nε − 1 log ε − 1 ) time.
The rest of this section is devoted to analyzing the approximation ratio of Algorithm 2.
Consider the inequality
[Equation]
If this inequality does not hold for some integer i , then Algorithm 2 clearly obtains at least 1 / 2approximation.
Therefore, it remains to consider the case in which Inequality (1) applies for every integer 0 ≤ i ≤ log 1+ ε ε − 1 .
One can observe that this inequality implies that OPT is non-empty because f ( S (0) ) ≥ f ( ∅ ) by the monotonicity of f , and therefore, we can define r to be an element of OPT maximizing c ( r ) .
Additionally, we assume below that r is not the only element of OPT —if this assumption does not hold, then Algorithm 2 trivially returns an optimal solution.
Lemma 3.8.
If c ( r ) ≥ 1 − ε , then f ( S ℓ ) ≥ ( 1 / 2 − ε ) · f ( OPT ) .
If f ( { r } ) ≥ 1 / 2 · f ( OPT ) , then the last lemma follows immediately since { r } is one of the sets considered for the output of Algorithm 2 on Line 8.
Otherwise, we get that OPT \ { r } is a set with a lot of value taking a very small part of the budget allowed, and the proof of Lemma 3.8 uses
Lemmata 3.4 and 3.5 to argue that f ( S ℓ ) is always large when such a set exists.
So far we have proved that the approximation ratio of Algorithm 2 is at least 1 / 2 − ε when either
Inequality (1) is violated or c ( r ) ≥ 1 − ε .
The following lemma proves that the same approximation guarantee applies also when neither of these cases applies, which completes the proof of Theorem 3.1.
The proof of this lemma defines i r to be the maximal integer such that ε (1+ ε ) i r ≤ 1 − c ( r ) , and h r to be the index for which S h r = S ( i r ) .
If h r = ℓ , then the proof uses Lemma 3.4 to argue that f ( S h r ) is large.
Otherwise, the proof uses Lemma 3.5 to argue that f ( S h r +1 ) is large.
Lemma 3.9.
If c ( r ) ≤ 1 − ε , then Inequality (1) implies f ( S ℓ ) ≥ ( 1 / 2 − ε ) · f ( OPT ) .
Information-Theoretic Inapproximability Results
In this section we present our inapproximability results.
The first result that we show is inapproximability for SMC (see Section 3 for the definition of this problem) showing that, when the number k of elements allowed in the solution is constant, one cannot obtain a constant approximation ratio for this problem using o ( n ) value oracle queries.
Theorem 4.1.
Any (possibly randomized) algorithm guaranteeing α -approximation ( α ∈ (0 , 1] ) for Submodular Maximization subject to a Cardinality Constraint ( SMC ) must use Ω( αn/k ) value oracle queries.
In particular, this implies that the algorithm must make Ω( n ) value oracle queries when α and k are constants.
See Appendix D for the proof of Theorem 4.1.
Let us now consider the approximability of SMC in a different regime allowing larger values of k .
Specifically, we consider the regime in which the ratio between n and k is some constant β .
In this regime one can obtain β -approximation by simply outputting a uniformly random subset of N of size k (which requires no value oracle queries); thus, this regime cannot admit an extremely pessimistic inapproximability result like the one given by
Theorem 4.1.
Nevertheless, the following theorem shows that improving over the above-mentioned “easy” approximation guarantee requires a nearly-linear query complexity.
Theorem 4.2.
For any rational constant β ∈ (0 , 1) and constant ε > 0 , every (possibly randomized)
algorithm guaranteeing approximation ratio of β + ε for instances of Submodular Maximization subject to a Cardinality Constraint obeying k
= βn must use Ω( n log n ) value oracle queries.
Moreover, this is true even when the function f of is guaranteed to be a linear function.
We prove in Section E.1 a restricted version of Theorem 4.2 that applies only to deterministic algorithms.
Then, Section E.2 extends the proof to randomized algorithms, which implies Theorem 4.2.
Table 1 summarizes our knowledge about the number of value oracle queries necessary for obtaining various approximation guarantees for SMC .
Using the technique of Theorem 4.2, we can also prove another interesting result.
In the Unconstrained Submodular Maximization problem ( USM ), the input consists of a non-negative (not necessarily monotone) submodular function f : 2 N → R ≥ 0 .
The objective is to output a set S ⊆ N maximizing f .
Our result for USM is given by the next theorem (proved in Appendix F).
This theorem
[Table]
[Table]
) queries via brute force; [Table]
n ) [Table] [Table] [Table]
[Table]
[Table]
Greedy
[Table]
( nk ) queries
[Table]
[Table]
queries
[Table]
[Table]
– nearly completes our understanding of the minimum query complexity necessary to obtain various approximation ratios for USM .
See Table 2 for more detail.
Theorem 4.3.
For any constant ε > 0 , any algorithm for Unconstrained Submodular Maximization that guarantees an approximation ratio of 1 / 4 + ε must use Ω( n log n ) value oracle queries.
The necessity of the log n term in Theorems 4.2 and 4.3 is discussed in Appendix G. Specifically, we show in this appendix an algorithm that can be used to solve (exactly) the hard problem underlying these theorems using O ( n/ log n ) oracle queries.
This algorithm is not efficient in terms of its time complexity, but it shows that one cannot prove a lower bound stronger than Ω( n/ log n ) for this problem based on information theoretic arguments alone.
Set System and Multiple Knapsacks Constraints
In this section we consider the Knapsacks and a Set System Constraint problem ( SMKS ).
In this problem we are given a non-negative monotone submodular function f : 2 N → R ≥ 0 , a p -system
M = ( N , I ) , d ≥ 1 non-negative cost functions
c i : N → R ≥ 0 (one function for every integer 1 ≤ i ≤ d ) and d positive values B 1 , B 2 , . . .
, B d as Knapsack budgets.
We say that a set S ⊆
N is feasible if it is independent in M and also obeys
c
i ( S ) ≤ B i for every 1 ≤ i ≤ d (where c i ( S ) ≜ (cid:80)
u ∈ S
c i ( u ) ).
The objective of the problem is to find a set maximizing f among all feasible sets S ⊆ N .
Algorithm 3 is the basic version of our proposed solution for SMKS (in Appendix I.1 we explain how to make this basic algorithm nearly-linear).
Algorithm 3 gets two parameters.
The first of these parameters is an integer λ ≥ 1 .
Elements that have a value larger than λ − 1 with respect to at least one function c i are considered big elements, and are stored in the set B of the algorithm.
The other elements of N are considered small elements.
The algorithm never considers any solution that includes both big and small elements.
Instead, it creates one candidate solution S B from the big elements, and one candidate solution from the small elements, and then outputs the better among the two (technically, in some cases the algorithm outputs directly the candidate solution based on the small elements without comparing it to S B ).
The candidate solution S B is constructed using a procedure called BigElementsAlg that gets the set B as input and outputs a feasible set whose value is at least α · f ( OPT ∩ B ) , where α is a some value in (0 , 1] and OPT is an arbitrary optimal solution.
At this point we ignore the implementation of BigElementsAlg , and leave the value of α unspecified.
These gaps are filled in Section H.3.
Most of Algorithm 3 is devoted to constructing the candidate solution out of small elements, which we refer to as the “small elements solution”.
In the construction of this solution, Algorithm 3 uses its second parameter ρ ≥ 0 that intuitively should represent the density of the small elements of OPT .
The algorithm initializes the small elements solution to be empty, and then iteratively adds to it the element with the largest marginal contribution among the small elements that have two properties: (i) their addition to the solution does not make it dependent in M , and (ii) their density (the ratio between their marginal contribution and cost according to the linear constraints) is at least ρ .
This process of growing the small elements solution can end in two ways.
One option is that the process ends because no additional elements can be added to the solution (in other words, no element has the two properties stated above).
In this case the better among S B and the small elements solution obtained S k is returned.
The other way in which the process of growing the small elements solution can end is when it starts violating at least one linear constraint.
When this happens, Algorithmuses a procedure called SetExtract (see Algorithm 7 in Section H.1) to get a subset of the small elements solution that is feasible and also has a good value, and this subset is returned.
Algorithm 3 assumes access to an estimate ρ of the density of the small elements of an optimal solution for the problem.
In Section H.2 we explain how the dependence of the algorithm on ρ can be dropped without increasing the time complexity of the algorithm by too much.
Recall that
Section 1.1 demonstrated a trade-off between the time complexity and approximation guarantee of state-of-the-art algorithms for SMKS .
The next theorem further studies this trade-off, and in particular, improves over the state-of-the-art approximation for nearly-linear time algorithms.
The ˜ O notation in this theorem suppresses factors that are poly-logarithmic in n , d and ε − 1 .
Due to space constraints, the proof of Theorem 5.1 is differed to Appendix H.
Theorem 5.1.
For every ε > 0 , there exist algorithms that achieve [(1 + O ( ε ))
( p + 1 + 74 d )] − 1approximation and ( p + 1 . 5556 + 139 d + ε ) − 1 for Submodular Maximization subject to
Knapsacks and a Set System Constraint in ˜ O ( nd + n/ε ) and ˜ O ( n 2 + nd ) time, respectively. Algorithm
: Basic
// Build the set of big elements, and find a solution based on them.
1 Let B ← { u ∈ N | ∃ 1 ≤ i ≤ d c i ( u ) > λ − 1 } .
S B be
// Construct a solution from the small elements. if max 1 ≤ i ≤ d c i ( S k +1 ) ≤ 1 then Increase k by 1 . set of SetExtract ( λ, S k +1 ) .  Experimental Results
We have studied the submodular maximization problem subject to different constraints.
In this section, we compare our proposed algorithms with the state-of-the-art algorithms under the following constraints: (i) a carnality constraint (Section 6.1), (ii) a single knapsack constraint (Section 6.2), and (iii) combination of a p -system and d knapsack constraints (Appendix J.4).
6.1 Cardinally Constraint
Cardinality constraint is the most widely studied setup in the submodular maximization literature.
We compare our algorithm, F AST T HRESHOLD G REEDY (abbreviated as FTG) with L AZY G REEDY and B OOST R ATIO [46] under cardinality constraint.
L AZY G REEDY is an efficient implementation of the na¨ıve G REEDY algorithm in which the diminishing returns property of submodular functions is used to avoid oracle queries that are known to provide little gain [55].
It is well-known that
L AZY G REEDY leads to several orders of magnitude speedups over G REEDY in practice.
[-+
-]
In our first experiment, we implement a movie recommender system by finding a diverse set of movies for a user.
We adopt the approach of Lindgren et al.
[54] to extract features for each movie by using ratings from the MovieLens dataset [31].
For a given set of movies N ( |N| = n ) , let vector v
i represents the attributes of the i -th movie.
The similarity matrix M n × n between movies is defined by M ij =
e − λ · dist ( v i ,v j ) , where dist ( v i , v j ) is the euclidean distance between vectors v i , v j ∈ N .
For this application, we used the following monotone and submodular function to quantify the diversity of a given set of movies S : f
( S )
= log det( I + αM S ) , where I is the identity matrix and M S is a square sub-matrix of M consisting of the rows and columns corresponding to the movies in the set S .
Our objective is to maximize f under a cardinality constraint k .
In Figure 1a, we compare the utility of the algorithms on this instance.
We observe that F AST T HRESHOLD G REEDY with ε ∈ { 0 . 1 , 0 . 2 } performs as good as L AZY G REEDY , and B OOST R ATIO performs slightly worse.
In Figure 1b, we observe that the query complexity of F AST T HRESHOLD G REEDY is significantly less than that of L AZY G REEDY .
It is interesting to note that, as is guaranteed by our theoretical results, the number of oracle calls for F AST T HRESHOLD G REEDY is (almost) not increasing with k .
In our second experiment, which appears in Appendix J.1, we consider a location summarization application and show that similar conclusions apply also to this application.
S TOCHASTIC G REEDY is a fast but randomized approach for solving SMC [55].
In Appendix J. we present an experiment comparing F AST T HRESHOLD G REEDY with S TOCHASTIC G REEDY under the cardinality constraint in a vertex cover context.
In general, this experiment shows that the utility of F AST T HRESHOLD G REEDY is significantly better than that of S TOCHASTIC G REEDY , and that F AST T HRESHOLD G REEDY sometimes also outperforms S TOCHASTIC G REEDY in terms of the query complexity.
We also observe a high variability in the utility of solutions returned by
S TOCHASTIC G REEDY , which diminishes the benefit from this algorithm.
6.2 Single Knapsack Constraint
In this section, we evaluate the performance of Algorithm 2 (referred to as FTGP) with that of
D YNAMIC M RT [37] and D ENSITY G REEDY under a single knapsack constraint.
D ENSITY G REEDY greedily adds elements that maximize the ratio between their marginal gain and knapsack cost (but ignoring elements whose addition will result in a violation of the knapsack constraint).
In our first experiment for this constraint, we again consider the movie recommendation application from
Section 6.1.
The cost of each movie is defined to be proportional to the absolute difference between the rating of that movie and 10 (the maximum rating in iMDB).
In this application, the goal is to find a diverse set of movies while guaranteeing that the total rating of the picked movies is high enough [6].
From Figures 1c and 1d, we observe that Algorithm 2 significantly outperforms the other two algorithms with respect to both the utility and number of oracle calls metrics.
For our second experiment under the knapsack constraint, which appears in Appendix J.3, we consider Twitter text summarization.
In this experiment we again get that Algorithm 2 surpasses the baseline algorithms.
Conclusion and Societal Impact
An algorithm for maximizing a submodular function is conventionally measured in terms of two quantities: a) approximation guarantee, i.e., how well it performs with respect to the (exponential time) optimum algorithm, and b) query complexity, i.e., how many function evaluations it requires.
[Figure]
In their seminal work [Figure] Nemhauser and Wolsey [61] and Nemhauser et al.
[62] resolved part (a).
In this paper, after 44 years and building on a large body of prior work, we nearly resolved part (b) and portrayed a nearly complete picture of the landscape.
Specifically, we developed a clean lineartime algorithm for maximizing a monotone submodular function subject to a cardinality constraint (and more generally a knapsack constraint).
We also provided information-theoretic lower bounds on the query complexity of both constrained and unconstrained (non-monotone) submodular maximization.
Finally, we studied the trade-off between the time complexity and approximation ratio for maximizing a monotone submodular function subject to a p -system and d knapsack constraints.
This work does not have any negative societal impact.
Like most algorithms in the literature, our algorithms assume the ability to exactly evaluate the objective function.
In some applications exact evaluations is too costly, which has motivated works (such as [33]) that considered erroneous value oracles.
Extending our work in this direction is an interesting question for future research.
Funding Transparency Statement
Funding in direct support of this work: The work of Moran Feldman was supported in part by
Israel Science Foundation (ISF) grant no. 459/20.
The work of Amin Karbasi was supported in part by the NSF (IIS-1845032), ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enabled
Optimization at Scale (TILOS).
A Non-monotone Submodular Maximization: Literature Review
Maximization of non-monotone submodular functions subject to a cardinality constraint is a special case of the maximization of such functions subject to matroid constraints, and many works [17, 21, 26, 50, 71] on the problem considered this more general setting, culminating in 0 .
385 -approximation due to Buchbinder and Feldman [9].
As the above algorithms (especially the ones obtaining better approximation ratios) are quite involved and non-combinatorial, Buchbinder et al.
[10] proposed the random greedy algorithm, which is a combinatorial algorithm achieving an approximation ratio of 1 /e ≈ 0 .
367 using O ( nk ) queries (matching the state-of-the-art approximation ratio for general matroid constraints at the time).
Furthermore, by derandomizing random greedy, Buchbinder and
Feldman [8] obtained a 1 /e -approximation deterministic algorithm.
To improve the computational speed, Buchbinder et al.
[12] proposed an (1 /e − ε ) -approximation random sampling algorithm
that makes O ( nε 2 log 1 ε ) queries, and another algorithm that achieves the same approximation guarantee using O ( nε ln kε ) queries
when k
= O ( √ n ) .
Sakaue [63] showed that, with slight modification, a well-known algorithm called “stochastic greedy” [12, 55] can achieve almost 1 / 4 -approximation in expectation in linear time.
Additionally, based on the idea of interlacing two greedy procedures,
Kuhnle [45] developed a (1 / 4 − ε )
-approximation deterministic algorithm requiring O ( nε log kε ) queries.
The state-of-the-art algorithm for maximizing a non-monotone submodular function subject to a knapsack constraint is 0 .
385 -approximation that can be obtained by combining the algorithm of Buchbinder and Feldman [9] for optimization of the multilinear extension with the rounding scheme of Kulik et al.
Unfortunately, this state-of-the-art algorithm is randomized and quite slow, which has motivated works on deterministic and fast algorithms for the problem.
Mirzasoleiman et al.
[56] proposed a deterministic (1 / 10 − ε )
-approximation
algorithm using O ( n 2 log n ε ) queries; and the currently best deterministic algorithm is due to Han et al.
[30], who achieve an approximation ratio of (1 / 6 − ε ) via O ( nε log kε ) queries.
If one is willing to allow randomization, then a 1 / (3 + 2 √ 2) -approximation algorithm was propose by Amanatidis et al.
[2], which requires
O ( nε log nε ) queries.
This was later improved by Han et al.
[30], who showed that an approximation ratio of 1 / 4 is obtainable with O ( nε log kε ) queries.
B Estimating the Value of an Optimal Solution
In this section our objective is to design an algorithm (given as Algorithm 4) that given an instance of SMK produces a value Γ that obeys Γ ≤ f ( OPT ) ≤ s · Γ for some constant s ≥ 1 , where OPT is an arbitrary optimal solution.
Algorithm 4: E STIMATING f ( OPT )
Clearly Algorithm 4 can be implemented to work in O ( n ) time.
Therefore, we concentrate on bounding the quality of the estimate it produces.
The following observation shows that Algorithmdoes not underestimate OPT by more than a constant factor.
Let ˜ S denote the final value of the set
S in Algorithm 4.
Observation B.1.
f ( ˜ S ) ≥ f ( OPT ) / 2 .
Proof.
Consider an arbitrary element u ∈ OPT \ ˜ S .
The fact that u was never added to S by Algorithm 4 implies that at the time in which u was processed by Algorithm 4 the following inequality held f ( u | S ) c ( u ) ≤ f ( S ) .
Since elements are only ever added to S , by the monotonicity and submodularity of f , the last inequality implies
[Equation]
Using the last inequality, we now get
[Equation]
where the first inequality follows from the submodularity of f , and the last inequality holds because
OPT is a feasible solution and OPT \ ˜ S is a subset of it.
Plugging the definition of f ( OPT | ˜ S ) into the last inequality, and rearranging, yields
[Equation]
where the second inequality follows from the monotonicity of f .
To prove that Algorithm 4 also does not over estimate f ( OPT ) , we need to define some additional notation.
From this point on, it will be convenient to assume that the elements of N are ordered in the order in which Algorithm 4 considers them.
Given a set T ⊆ N , let T ≥ 1 be the minimal suffix of T (according to the above order) whose value according to c is at least 1 (unless c ( T ) < 1 , in which case T ≥ 1 = T ).
Clearly f ( ˜ S ≥ 1 ) ≤ f ( ˜ S ) by the monotonicity of f .
However, it turns out that f ( ˜ S ≥ 1 ) is never much smaller than f ( ˜ S ) .
Lemma B.2.
f ( ˜ S ≥ 1 ) ≥ f
( ˜ S ) / 2 .
Proof.
If c ( ˜ S ≥ 1 ) < 1 , then ˜ S ≥ 1 = ˜ S , which makes the lemma trivial.
Therefore, we may assume in the rest of the proof that c ( ˜ S ≥ 1 ) ≥ 1 .
The fact that an element u ∈ ˜ S ≥ 1 was added to S by Algorithm 4 implies that if, we denote by S u the set S immediately before u was processed, then we have the following inequality.
[Equation]
Since ˜ S ≥ 1 is a suffix of ˜ S , this inequality implies
[Equation]
\ ˜ S ≥ 1 )
[Equation]
where the penultimate inequality holds by the monotonicity of f , and the last inequality follows from our assumption that c ( ˜ S ≥ 1 ) ≥ 1 .
Plugging the definition of f ( ˜ S ≥ 1 |
˜ S \ ˜ S ≥ 1 ) into the last inequality gives
[Equation]
where the second inequality holds by the submodularity (and non-negativity) of f .
The lemma now follows by rearranging the last inequality.
Corollary B.3.
f ( ˜ S ) ≤ 4 · f ( OPT ) .
Proof.
Since ˜ S ≥ 1 is the minimal suffix of ˜ S whose size according to c is at least 1 , if we denote by u the first element of this suffix, then both { u } and ˜ S ≥ 1 − u are feasible solutions.
Hence,
[Equation] [Equation]
where the penultimate inequality follows from the submodularity of f , and the last inequality follows from Lemma B.2.
The corollary now follows by multiplying the last inequality by 4 .
Recall now that the output of Algorithm 4 is f ( ˜ S ) / 4 .
Therefore, Observation B.1 and Corollary B. imply together the following proposition.
Proposition B.4.
Algorithm 4 runs in O ( n ) time, and if we
C Omitted Proofs of Section
Observation 3.3.
The time complexity of Algorithm 1 is O ( nε − 1 log α ) .
Proof.
Recall that Algorithm 4 runs in O ( n ) time by Proposition B.4, and observe that every iteration of the outer loop of Algorithm 1 runs in O ( n ) time.
Therefore, to prove the lemma, it suffices to argue that the number of iterations made by this outer loop is O ( ε − 1 log α ) .
Formally, the number of iterations made by this loop is upper bounded by
[Equation]
Lemma 3.4.
For every set T ⊆ N , if max u ∈ T c ( u ) ≤ 1 − c ( S ℓ ) , then f ( S ℓ ) ≥ f ( T ) − c ( T ) e · f ( OPT ) .
In particular, max u ∈ OPT c ( u ) ≤ 1 − c ( S ℓ ) implies f ( S ℓ ) ≥ (1 − 1 /e ) · f ( OPT ) .
Proof.
The second part of the lemma follows from the first part by plugging T =
OPT since OPT , as a feasible set, obeys c ( OPT ) ≤ 1 .
Therefore, we concentrate below on proving the first part of the lemma.
Since elements are only ever added by Algorithm 1 to its solution set S k , the condition of the lemma implies that whenever an element u ∈ T \ S ℓ was considered on Line 5 of Algorithm 1, its density was too low compared to the value of the threshold τ at time.
In particular, this is true for the threshold τ ≤ Γ /e
in the last iteration of the outer loop of Algorithm 1, and therefore,
[Equation]
u
[Equation]
where S u represents here the set S k immediately before u is processed by Algorithm 1 in the last iteration of its outer loop.
By the submodularity of f , the last inequality holds also when S u is replaced with S ℓ , and therefore,
[Equation]
where the first inequality follows from the monotonicity of f , and the second inequality follows from the submodularity of f .
The lemma now follows from the last inequality by recalling that
Proposition B.4 guarantees Γ ≤ f ( OPT ) .
Lemma 3.5.
For every set ∅ ̸ = T ⊆ N and integer 0 ≤ h < ℓ , if max u ∈ T c ( u ) ≤ 1 − c ( S h ) , then f
[Equation]
S
[Equation]
Proof.
Let τ h
+1 be the value of τ when Algorithm 1 selects u
h +1 .
If τ h +1 = 8 α · Γ ≥ α · f ( OPT ) , then the lemma follows immediately from the condition used on Line 5 of the algorithm.
Therefore, we assume below that τ
h +1 < 8 α · Γ , which implies that no element of T \ S h was added to the solution of Algorithm 1 when the threshold τ was equal to (1 − ε ) − 1 τ
h
+1 .
Therefore, since elements are only ever added by Algorithm 1 to its solution, the submodularity of f guarantees
[Equation]
We now observe that the lemma follows immediately from the monotonicity of f when T ⊆ S h .
Therefore, we need to consider only the case of T ̸⊆ S h , and in this case the previous inequality implies
[Equation]
where the last inequality follows from the submodularity of f .
Theorem 3.6.
If we choose α = 1 , then, for every ε ∈ (0 , 1) , Algorithm 1 runs in O ( n/ε ) time and guarantees (1 − 1 /e − ε ) -approximation for SMC .
Proof.
The time complexity of Algorithm 1 was already proved in Observation 3.3.
Therefore, we concentrate here on analyzing the approximation guarantee of this algorithm.
Let k be the maximum cardinality allowed by the cardinality constraint ( i.e. , c ( u )
= 1 /k for every u ∈ N ).
We need to consider two cases depending on the value of ℓ .
If ℓ < k , i.e. , Algorithm 1 outputs a solution that is smaller than the maximum cardinality allowed, then the condition min u ∈ OPT c ( u ) ≤ 1 − c ( S ℓ ) of
Lemma 3.4 holds (because c ( S ℓ ) = ℓ/k ≤ 1 − 1 /k ), and therefore,
[Equation]
Assume now that ℓ = k .
In this case we need to use Lemma 3.5.
Since the condition of this lemma holds for every integer 0 ≤ h < ℓ and T =
OPT (we may assume that OPT ̸ = ∅ because otherwise Algorithm 1 is clearly optimal), the lemma implies, for every such k ,
[Equation]
where the second inequality holds because the feasibility of OPT implies c ( OPT ) ≤ 1 , and the submodularity and non-negativity of f imply f ( OPT | S h ) ≤ f ( OPT | ∅ )
≤ f ( OPT ) .
Rearranging the last inequality yields
[Equation]
1 ) · f
[Equation]
OPT
[Equation]
S
[Equation]
) − f ( S
[Equation]
+1 ) ,
which by the monotonicity of f , the observation that f ( S h ) ≤ f ( OPT ) because S h is a feasible solution, and the inequality 1 − x ≤ e − x (which holds for every x ∈ R ) implies
[Equation]
Finally, by combining the last inequality for all integers 0 ≤ h < ℓ , we get
[Equation]
where the equality holds since (cid:80) 0 ≤ h<ℓ c ( u h +1 )
=
c ( S ℓ ) = 1 by the definition of the case we consider, and the penultimate inequality follows from the non-negativity of f .
The theorem now follows by rearranging the last inequality.
Observation 3.7.
Algorithm 2 runs in O ( nε − 1 log ε − 1 ) time.
Proof.
Algorithm 1 runs in O ( nε − 1 log ε − 1 ) time by Observation 3.3.
Additionally, every iteration of the loop of Algorithm 2 runs in O ( n + ℓ ) =
O ( n ) time, where the equality holds because the fact that some element is added to the solution of Algorithm 1 whenever the index k increases guarantees that ℓ is at most n .
Therefore, to prove the observation it suffices to show that the loop of Algorithmiterates only O ( ε − 1 log ε − 1 ) times.
Formally, the number of iterations of this loop is upper bounded by
[Equation]
Lemma 3.8.
If c ( r ) ≥ 1 − ε , then f ( S ℓ ) ≥ ( 1 / 2 − ε ) · f ( OPT ) .
Proof.
If f ( { r } ) ≥ 1 / 2 · f ( OPT ) , then the lemma follows immediately since { r } is one of the sets considered for the output of Algorithm 2 on Line 8.
Therefore, we may assume that f ( { r } ) ≤ 1 / 2 · f ( OPT ) , which implies f ( OPT − r ) ≥ f ( OPT ) − f ( { r } ) ≥ 1 / 2 · f ( OPT ) by the submodularity of f .
Since c ( OPT ) ≤ 1 because OPT is a feasible set, we get that OPT − r is a set with a lot of value taking a very small part of the budget allowed (specifically, c ( OPT − r )
=
c ( OPT ) − c
( r ) ≤ 1 − (1 − ε )
= ε ).
Below we show that the existence of such a set implies that f ( S ℓ ) is large.
We also assume below f ( S ℓ ) ≤ f ( OPT − r ) because otherwise the lemma follows immediately since f ( OPT − r ) ≥ 1 / 2 · f ( OPT ) .
Consider first the case in which c ( S ℓ ) ≤ 1 − ε .
Since max u ∈ OPT − r c ( u ) ≤ c ( OPT − r ) ≤ ε ≤ 1 − c ( S ℓ ) in this case, Lemma 3.4 gives us immediately, for T =
OPT − r ,
[Equation]
It remains to consider the case of c ( S ℓ ) ≥ 1 − ε .
Let ¯ h be the minimal h such that c ( S h ) ≥ 1 − ε .
For every integer 0 ≤ h < ¯ h , we have max u ∈ OPT − r ≤ c ( OPT − r ) ≤ ε ≤ 1 − c ( S h ) , and therefore, applying Lemma 3.5 with T =
OPT − r yields
[Equation]
where the second inequality holds because c ( OPT − r ) ≤ ε , and the monotonicity, submodularity and non-negativity of f imply f ( OPT − r | S h ) ≤ f ( OPT | S h ) ≤ f ( OPT | ∅ ) ≤ f ( OPT ) .
Rearranging the last inequality yields
[Equation]
which by the monotonicity of f , our assumption that f ( S h ) ≤ f
[Equation]
S ℓ ) ≤ f ( OPT − r ) and the inequality 1 − x ≤ e − x (which holds for every x ∈ R ) implies
[Equation]
Finally, by combining the last inequality for all integers 0 ≤ h < ¯ h , we get
[Equation]
where the second inequality holds since (cid:80)
≤ h< ¯ h c ( u h +1 )
=
c ( S ¯ h ) ≥ 1 − ε by the definition of ¯ h and the penultimate inequality follows from the non-negativity of f .
Rearranging the last inequality, we now get, using the monotonicity of f ,
[Equation]
Lemma 3.9.
If c ( r ) ≤ 1 − ε , then Inequality (1) implies f ( S ℓ ) ≥ ( 1 / 2 − ε ) · f ( OPT ) .
Proof.
Let i r be the maximal integer such that ε (1 + ε ) i r ≤ 1 − c ( r ) .
Since we assume that c ( r ) ≤ 1 − ε , i r is non-negative.
On the other hand, we also have i r ≤ log 1+ ε ε − 1 because otherwise ε (1 + ε ) i r > 1 ≥ 1 − c ( r ) .
Therefore, the set S ( i r ) is well-defined, and one can observe that
[Equation]
where the first inequality follows from the way in which Algorithm 2 selects u ( i r ) because c ( S ( i r ) )
≤ ε (1 + ε ) i r ≤ 1 − c ( r ) , and the second inequality follows from Inequality (1).
Let h r be the index for which S h r
= S ( i r ) .
If h r = ℓ , then f ( S ℓ ) ≥ (1 − 1 /e ) · f ( OPT ) by Lemma 3.4 (for T = OPT ) because max u ∈ OPT c ( u ) =
c ( r ) ≤ 1 − c ( S ( i r ) )
= 1 − c ( S ℓ ) .
Therefore, we may assume h r < ℓ in the rest of this proof.
In particular, this implies that for every
0 ≤ h ≤ h r , by plugging T = OPT − r into Lemma 3.5, we get f
[Equation]
S h
[Equation]
− f
[Equation]
where the second inequality follows from the submoduarity of f ; the third inequality follows from f ’s monotonicity; and the last inequality holds since f ( S h r + r ) ≥ 0 , and max { c ( OPT − r ) , ε } ≤ 1 − c ( r ) by the condition of the lemma and the fact that OPT is a feasible solution.
Plugging now
Inequality (2) into the last inequality (and recalling that S h r = S ( i r ) ) yields f
[Equation]
S
[Equation]
−
[Equation]
Summing up the last inequality for all integers 0 ≤ h ≤ h r now gives f
[Equation]
S
h r +1 )
− f ( S 0 ) ≥
[Equation]
where the third inequality holds since the definition of S ( i r ) implies that h r is the maximal index for which c ( S h r ) ≤ ε (1 + ε ) i r , and the penultimate inequality follows from the definition of i r .
D Inapproximability of SMC for Constant k Values
In this section we prove Theorem 4.1, which we repeat here for convinience.
Theorem 4.1.
Any (possibly randomized) algorithm guaranteeing α -approximation ( α ∈ (0 , 1] ) for Submodular Maximization subject to a Cardinality Constraint ( SMC ) must use Ω( αn/k ) value oracle queries.
In particular, this implies that the algorithm must make Ω( n ) value oracle queries when α and k are constants.
Proof.
Let t = 2 k/α , and for every element u ∈ N , consider the set function f u : N → R ≥defined, for every set S ⊆ N , as
[Equation]
Below we consider a distribution D over instances of SMC obtained by choosing u uniformly at random out of N , and we show that this distribution is hard for every deterministic algorithm that uses o ( αn/k ) value oracle queries, which implies the theorem by Yao’s principle.
Let ALG be any deterministic algorithm using o ( αn/k ) value oracle queries, and let S 1 , S 2 , . . .
S ℓ be the sets on which ALG queries the value oracle of the objective function when this function is chosen to be min { t, | S |} .
We assume without loss of generality that S ℓ is the output set of
ALG .
Let E be the event that u does not belong to any of sets S 1 , S 2 , . . .
, S ℓ that are of size less than t .
Since ALG is deterministic, whenever the event E happens, ALG makes the same set of value oracle queries when given either min { t, | S |} or f u as the objective function, and outputs f u ( S ℓ ) =
min { t, | S |} ≤ | S ℓ | ≤ k in both cases.
Moreover, even when the event E does not happen,
ALG still outputs a set of value at most t since f u never returns larger value.
Finally, observe that the probability of the event E is at least 1 − ℓt/n , and therefore, when the input instance is drawn from the distribution D , the expected value of the output of ALG is at most
[Equation]
It remains to observe that the optimal solution for every instance in the support of the distribution D is { u } , whose value according to f u is t .
Therefore, the approximation ratio of ALG with respect to an instance of SMC drawn from D is no better than
[Equation]
where the last equality holds since ALG makes o ( αn/k ) queries, and ℓ is the number of queries that it makes given a particular input.
Hence, D is an hard distribution in the sense that ALG is not an α -approximation algorithm against an instance of SMC drawn from D .
E Inapproximability of SMC and Large k Values
We prove in Section E.1 a restricted version of Theorem 4.2 that applies only to deterministic algorithms.
Then, Section E.2 extends the proof to randomized algorithms, which implies Theorem 4.2.
E.1 Inapproximability of SMC for Deterministic Algorithms and Large k Values
We begin this section by considering the following problem, termed Set-Identification , which has a rational parameter β ∈ (0 , 1) .
An algorithm for this problem has access to a ground set N of size n and a non-empty collection C of subsets of N of size k
= βn (formally, C ⊆ { S ⊆ N | | S | = k } ) .
Each instance of Set-Identification also includes some hidden set C ∗ ∈ C , which the algorithm can access only via an oracle that answers the following type of queries: given a set
S ⊆ N , the oracle returns the size of S ∩ C ∗ .
The objective of the algorithm is to output a set S of size k whose intersection with C ∗ is as large as possible.
The majority of this section is devoted to proving the following proposition regarding deterministic algorithms for Set-Identification .
We then show that this proposition implies a version of
Theorem 4.2 for deterministic algorithms.
Proposition E.1.
For every α ∈ ( β, 1] and set C , any deterministic α -approximation algorithm for Set-Identification must make ( k +1) ·
[2( α − β ) 2 +ln β
+
( β − 1 − 1)ln(1 − β ) − k − 1 ln n ]+ln |C| ln( k +1) oracle queries.
Let ALG be an arbitrary deterministic algorithm for Set-Identification .
To prove Proposition E.1, we need to design an adversary that selects a hidden set C ∗ that is “bad” for ALG .
Our adversary does not immediately commit to a particular set C ∗ .
Instead, it maintains a list L of candidate sets that are consistent with all the oracle queries that ALG has performed so far.
In other words, the set L originally includes all the sets of C .
Then, whenever ALG queries the oracle on some set S , the adversary returns some answer a and eliminates from L every set C for which | S ∩ C | ̸
= a .
To fully define the adversary, we still need to explain how it chooses the answer a for the oracle query.
As the adversary wishes to keep the algorithm in the dark for as long as possible, it chooses the answer a to be the one that reduces the list L by the least amount.
Let L i be the list L after the adversary answers i queries.
Observation E.2.
For every integer i ≥ 0 , |L i | ≥ |C| / ( k + 1) i .
Proof.
Whenever the adversary has to answer a query regarding some set S , the answer corresponding to each set in the list L is some number in { 0 , 1 , . .
. , k } .
Therefore, at least one of these answers is guaranteed to reduce the list only by a factor of |{ 0 , 1 , . . .
, k }| =
k + 1 .
Thus, after i queries have been answered, the size of the list L reduces by at most a factor of ( k + 1) i .
To complete the proof of the observation, we recall that the original size of L is |C| .
Once ALG terminates and outputs some set S of size k , our adversary has to decide what set of L is the “real” hidden set C ∗ , which it does by simply choosing the set C ∗ ∈ L with the least intersection with S .
If the list L is still large at this point, then the adversary is guaranteed to be able to find in it a set with a low intersection with S .
The following lemma quantifies this observation.
Lemma E.3.
For every set S ⊆ N of size k , a non-empty subset L ⊆ C must include a set C ∈ L
such that | S ∩ C
[Equation]
<
[Equation]
Proof.
Let h = βk + k
[Equation]
n −
[Equation]
β − ( β − 1 − 1) ln(1 − β ) − 1 k +1 ln |L| ) / 2 .
We would like to upper bound the probability of a uniformly random subset R ⊆ N of size k to intersect at least h elements of S .
Since the expected size of the intersection of R and S is
[Equation]
and the distribution of R is hyper-geometric, we get by Inequality (10) in [65] (developed based on the works of [14, 35])
that
[Equation]
By Stirling’s approximation, the total number of subsets of N whose size is k is
[Equation]
Since Inequality (3) implies that at most a fraction of e − 2( h − βk ) 2 /k out of these sets has intersection with S of size at least h , we get that the number of subsets of N of size k whose intersection with S is of size at least h can be upper bounded by
[Equation]
The last bound implies that |L| must include at least one set whose intersection with S is less than h .
We are now ready to prove Proposition E.
[Equation]
Proof of Proposition E.1.
By Observation E.2 and Lemma E.3, if ALG makes i oracle queries, our adversary will be able to choose a hidden set C ∗ such that the intersection between the output set S of ALG and C ∗ is less than
[Equation]
Since the optimal solution for Set-Identification is the set C ∗ itself, whose intersection with itself is k
[Equation]
the above inequality implies that the approximation ratio of ALG is worse than
[Equation]
If the approximation ratio of ALG is at least α , then we get the inequality
[Equation]
which implies that the number i of oracle queries made by ALG obeys
[Equation]
Using Proposition E.1, we can now prove the promised version of Theorem 4.2 for deterministic algorithms.
Corollary E.4.
For any rational constant β ∈ (0 , 1) and (not necessarily constant) ε = ω ( (cid:113) log nn ) , every deterministic algorithm for SMC that guarantees an approximation ratio of β + ε for instances obeying k = βn must use Ω( ε 2 n log n ) value oracle queries.
Moreover, this is true even when the objective function f of SMC is guaranteed to be a linear function.
Proof.
Let C be the set of all subsets of N of size βn , and let k = βn .
By Stirling’s approximation, the size of the set C is at least
[Equation]
Therefore, the number of oracle queries made by any deterministic ( β + ε ) -approximation algorithm for Set-Identification with the above choice of parameters C and k is at least
[Equation]
ε 2 · Θ( n ) − Θ(log n ) Θ(log n ) =
Ω (cid:18) ε 2 n log n (cid:19) , where the last equality holds since we assume ε = ω ( (cid:113) log nn ) .
Assume now that we have an algorithm ALG for SMC with the guarantee stated in the corollary.
We construct an algorithm ALG ′ for Set-Identification that consists of the following three steps.
Construct an instance of SMC by setting k = βn and f ( S ) = | S ∩ C ∗ | .
Since f is a linear function, we can use ALG to find a set S ⊆ N of size at most βn such that f ( S ) ≥ ( β + ε ) · f ( C ∗ ) =
( β + ε )
| C ∗ | .
Return any subset S ′ ⊆
N of size βn that includes S .
Clearly, | S ′ ∩ C ∗ | ≥ | S ∩ C ∗ | = f ( S ) ≥ ( β + ε ) | C ∗ | .
The algorithm ALG ′ obtains ( β + ε ) -approximation for Set-Identification , and therefore, by the above discussion, it must use Ω( ε 2 n log n ) queries to the oracle of Set-Identification .
However, since ALG ′ queries the oracle of Set-Identification only once for every time that ALG queries the value oracle of f , we get that ALG must be using Ω( ε 2 n log n ) value oracle queries.
E.2 Inapproximability of SMC for Randomized Algorithms and Large k Values
In this section we extend the inapproximability result from Section E.2 to randomized algorithms.
To do that, we start by proving the following version of Proposition E.1 for randomized algorithms.
Proposition E.5.
For every α ∈ ( β, 1] and set C , any (possibly randomized) α -approximation algorithm for Set-Identification must make
[Equation]
oracle queries in the worst case.
Proof.
Let ALG be an algorithm of the kind described in the proposition, assume that the set C ∗ is chosen uniformly at random out of C and let S R be the output set of ALG .
Using a Markov like argument, we get that since ALG has an approximation guarantee of α and | S R ∩ C ∗ | is always at most | C ∗ | =
k ,
[Equation]
where the probability is over both the random choice of C ∗ and the randomness of ALG itself.
Recall now that ALG , as a randomized algorithm, can be viewed as a distribution over deterministic algorithms.
This means that there must be at least one (deterministic) algorithm ALG D in the support of this distribution such that its output set S D obeys
[Equation]
However, since ALG D is a deterministic algorithm, S D is a function of the set C ∗ alone, which implies that there exists a subset C ′ ⊆
C of size at least α − β 2 −
α − β · |C| such that the algorithm ALG D guarantees α + β 2 -approximation whenever C ∗ ∈ C ′ .
By Proposition E.1, the number of oracle queries used by ALG D must be at least
[Equation]
The proposition now follows since the number of oracle queries made by ALG D is a lower bound on the number of oracle queries made by ALG in the worst case.
Corollary E.6.
For any rational constant β ∈ (0 , 1) and (not necessarily constant) ε = ω ( (cid:113) log nn ) , every (possibly randomized) algorithm for SMC that guarantees an approximation ratio of β + ε for instances obeying k = βn must use Ω( ε 2 n log n ) value oracle queries.
Moreover, this is true even when the objective function f of SMC is guaranteed to be a linear function.
Proof.
Let C be the set of all subsets of N of size βn , and let k = βn .
As was proved in the proof of Corollary E.4,
[Equation]
and therefore, by Proposition E.5, the number of oracle queries made by any ( β + ε ) -approximation algorithm for Set-Identification with the above choice of parameters C and k is at least
[Equation]
where the last equality holds since we assume ε = ω ( (cid:113) log nn ) .
The rest of the proof is completely identical to the corresponding part in the proof of Corollary E.
(up to the need to add some expectations signs), and therefore, we omit it.
Theorem 4.2 is the special case of Corollary E.6 in which ε is a positive constant, and therefore, obeys ε = ω
( (cid:113) log nn ) .
F Inapproximability for USM
In this section we reuse the machinery developed in Section E.2 to get a query complexity lower bound for USM and prove Theorem 4.3.
Before getting to the proof, we recall that Table 2 gives the context of this theorem, and shows that this theorem nearly completes our understanding of the minimum query complexity necessary to obtain various approximation ratios for USM .
We assume throughout the section the following parameters for Set-Identification .
The parameter k is set to be n/ 2 , and C is the collection of all subsets of N of size k .
One can verify that the proof of Corollary E.6 also implies the following lemma (as a special case for β = 1 / 2 ).
Lemma F.1.
Given the above parameters, for any ε = ω ( (cid:113) log nn ) , every (possibly randomized) ( 1 / 2 + ε ) -approximation algorithm for Set-Identification must use Ω( ε 2 n log n ) oracle queries.
Our next objective is to show a reduction from Set-Identification to USM .
Let ALG be a ( 1 / 4 + ε ) -approximation algorithm for USM .
Using ALG , one can design the algorithm for SetIdentification (with the parameters we assume) that appears as Algorithm 5.
It is important to observe that the value oracle of the set function f defined on Line 1 of the algorithm can be implemented using one query to the oracle of Set-Identification .
Furthermore, f is the cut-function of a directed graph, and therefore, it is non-negative and submodular.
This allows Algorithm 5 to use
ALG to construct a set T of large expected value.
The set T is then replaced by it complement N \ T if this increases the value of f ( T ) .
Naturally, this replacement can only increase the expected value of the set T , and also gives it some deterministic properties that we need.
Algorithm 5 completes by converting the set T into an output set of size n/ 2 in one of two ways.
If T is too small, then a uniformly random subset of N \ T of the right size is added to it.
Otherwise, if T is too large, then a uniformly random subset of it of the right size is picked.
1 Define a function f : 2 N → R ≥ 0 by f ( S ) = | S ∩ C ∗ | · ( n 2 − | S \ C ∗ | ) .
2 Use ALG to find a set T that such that E [ f ( T )] ≥ ( 1 / 4 + ε ) · max S ⊆N f ( S )
= ( 1 / 4 + ε ) · n 2 / 4 .
3 if f ( T ) <
f ( N \ T ) then Update T ← N \ T .
4 if | T | ≤ n/ 2 then
Pick a uniformly at random subset R of N \ T of size n/ 2 − | T | .
return T ∪ R .
7 else
Pick a uniformly at random subset R of T of size n/ 2 .
return R .
Let T and ˜ T denote the values of the set T before and after Line 3 of Algorithm 5.
The following observation states the properties of the set ˜ T that we need below.
Observation F.2.
The set ˜ T obeys
• E [ f ( ˜ T )] ≥ ( 1 / 4 + ε ) · n 2 / 4 .
• | ˜ T ∩ C ∗ | ≥ | ˜
T \ C ∗ | .
Proof.
Line 3 of Algorithm 5 guarantees that f ( ˜ T ) ≥ f ( T ) , which implies
[Equation]
To see that the second part of the observation holds as well, note that Line 3 also guarantees f ( ˜ T ) ≥ f ( N \ ˜ T ) , which, by the definition of f , implies
[Equation]
Rearranging this inequality now gives
[Equation]
and dividing this inequality by n/ 2 yields the required inequality.
One consequence of the last lemma is given by the next corollary.
Corollary F.3.
E [ | ˜ T ∩ C ∗ | − | ˜
T \ C ∗ | ] ≥ nε/ 2 .
Proof.
Observe that
[Equation]
where the first inequality holds since Observation F.2 guarantees that | ˜ T ∩ C ∗
| − | ˜
T \ C ∗ | is always non-negative, and the last inequality follows also from Observation F.2.
The corollary now follows by dividing this inequality by n/ 2 .
We are now ready to prove the following lemma, which analyzes the performance guarantee of
Algorithm 5.
Lemma F.4.
If T ′ is the output set of Algorithm 5, then E [ | T ′ ∩ C ∗ | ] ≥ ( 1 / 2 + ε / 4 ) · n/ 2 , where the expectation is over the randomness of ALG and Algorithm 5.
Proof.
Let us begin this proof by fixing the set ˜ T .
In other words, until we unfix this set, all the expectations we use are assumed to be only over the randomness of Lines 5 and 8 of Algorithm 5.
Whenever Line 8 of the algorithm is used, we have
[Equation]
where the inequality holds since Observation F.2 guarantees
that | ˜ T ∩ C ∗ |−| ˜
T \ C ∗ | ≥ 0 .
Similarly, whenever Line 5 of Algorithm 5 is used, we get
[Equation]
Therefore, even if we unfix ˜ T , and take expectation also over the randomness of this set
[Equation]
we still get by the law of total expectation that
[Equation]
where the second inequality follows from Corollary F.3.
We can now prove Theorem 4.3, which we repeat here for convenience.
Theorem 4.3.
For any constant ε > 0 , any algorithm for Unconstrained Submodular Maximization that guarantees an approximation ratio of 1 / 4 + ε must use Ω( n log n ) value oracle queries.
Proof.
By Lemma F.1, Algorithm 5 must use at least Ω( ε 2 n log n ) queries to the oracle of SetIdentification .
However, aside from the queries used by ALG , Algorithm 5 queries this oracle only a constant number of times, which implies that ALG must be using Ω( ε 2 n log n ) such queries.
Recall now that ALG , as an algorithm for USM , queries the oracle of Set-Identification only by querying the value oracle of f , and every query to this value oracle results in a single query to the oracle of Set-Identification .
Therefore, ALG must be using Ω( ε 2 n log n ) value oracle queries.
This completes the proof of the theorem since ALG was chosen as an arbitrary ( 1 / 4 + ε ) -approximation algorithm for USM .
G Solving Set-Identification using O ( n/ log n ) Oracle Queries
Recall that the inapproximability results proved in Section 4 (namely, Theorems 4.2 and 4.3) are based on a reduction to a problem named Set-Identification (defined in Section E.1).
In Section E.2, we prove that any (possibly randomized) algorithm for Set-Identification must use Ω( n/ log n ) oracle queries to significantly improve over some “easy” approximation ratio.
In this section we show that this result is tight in the sense that O ( n/ log n ) oracle queries suffice to solve
Set-Identification exactly.
The algorithm we use for this purpose is given as Algorithm 6.
This algorithm is not efficient in terms of its time complexity.
Nevertheless, it shows that one cannot prove an inapproximability result requiring ω ( n/ log n ) oracle queries for Set-Identification based on information theoretic arguments only.
For simplicity, we assume in Algorithm 6 that the ground set N is simply the set [ n ] .
Given this assumption, we are able to base the algorithm on the following known lemma.
Lemma G.1 (Due to Lev and Yuster [53]) .
If q > (2 log 2 3 + o (1)) n log n , then there exists a binary matrix Q ∈ { 0 , 1 } q × n such that, for every vector s , the equation Q · x = s has at most one binary solution x ∈ { 0 , 1 } n .
Algorithm 6 begins by finding such a matrix Q (this can be done by brute-force enumeration since we do not care about the time complexity).
Then, the product Q · 1 C ∗ is calculated using q =
O ( n/ log n ) oracle queries, where 1 C ∗ is the characteristic vector of the set C ∗ ( i.e. , a vector that includes 1 in coordinate i if i ∈ C ∗ , and otherwise, includes 0 in this coordinate).
Once Algorithmhas the product Q · 1 C ∗ , Lemma G.1 guarantees that it is possible to recover the vector 1 C ∗ itself, which the algorithm can do, again, using brute-force.
1 Let q
=
O
( n/ log n ) be a large enough value so that it obeys the requirement of Lemma G.
2 Let Q ∈
q
×
n be a binary matrix
with
the properties stated
in
Lemma G.
for
do
Let
j -th line of Q
Let
j = | Q j ∩ C ∗
|
Can
be calculated
using
a single oracle query
th
coordinate q
is s
j
Note s
=
∈ {
n be a
for Q · x
=
s . /
/ By
Lemma G.1, x = 1 C ∗ is the sole solution for this equation in { 0 , 1 } n
H Proof of Theorem 5.
In this section we prove Theorem 5.1.
The rank of a set system M =
( N , I ) is defined as the maximum size of an independent set in it; and an element u ∈ N is a self-loop of this system if { u } is not independent, which implies that u does not appear in any independent set of M .
Below, we denote by r the rank of the set system M .
We also make a few simplifying assumptions.
• We assume that M does not include any self-loops or elements u ∈ N
such that c
i ( u ) >
B i for some integer 1 ≤ i ≤ d .
Any element violating these assumptions can be simply discarded since it cannot belong to any feasible set.
• We assume that the sum (cid:80) di =1 c i ( u ) is strictly positive for every element u ∈ N .
Elements violating this assumption can be added to any solution, and therefore, it suffices to solve the problem without such elements, and then add them to the obtained solution at the every end of the algorithm.
• We assume that B i = 1 for every integer 1 ≤ i ≤ d .
This can be guaranteed by scaling the cost functions
c i .
In Section H.1 we present and analyze a basic version of the algorithm that we use to prove Theorem 5.1 (in Appendix I.1 we explain how to make this basic algorithm nearly-linear).
The basic version of our algorithm assumes access to an estimate ρ of the density of the small elements of an optimal solution for the problem.
In Section H.2 we explain how the dependence of the algorithm on ρ can be dropped without increasing the time complexity of the algorithm by too much.
Finally, in Section H.3 we show how our algorithm can be used to derive the results stated in Theorem 5.1.
H.1 Basic Algorithm for SMKS
In this section we present and analyze a basic version algorithm of the algorithm that we use to prove Theorem 5.1.
This algorithm is given as Algorithm 3, and it gets two parameters.
The first of these parameters is an integer λ ≥ 1 .
Elements that have a value larger than λ − 1 with respect to at least one function c i are considered big elements, and are stored in the set B of the algorithm.
The other elements of N are considered small elements.
The algorithm never considers any solution that includes both big and small elements.
Instead, it creates one candidate solution S B from the big elements, and one candidate solution from the small elements, and then outputs the better among the two (technically, in some cases the algorithm outputs directly the candidate solution based on the small elements without comparing it to S B ).
The candidate solution S B is constructed using a procedure called BigElementsAlg that gets the set B as input and outputs a feasible set whose value is at least α · f ( OPT ∩ B ) , where α is a some value in (0 , 1] and OPT is an arbitrary optimal solution.
At this point we ignore the implementation of BigElementsAlg , and leave the value of α unspecified.
These gaps are filled in Section H.3.
Most of Algorithm 3 is devoted to constructing the candidate solution out of small elements, which we refer to below as the “small elements solution”.
In the construction of this solution, Algorithmuses its second parameter, which is a value ρ ≥ 0 that intuitively should represent the density of the small elements of OPT .
The algorithm initializes the small elements solution to be empty, and then iteratively adds to it the element with the largest marginal contribution among the small elements that have two properties: (i) their addition to the solution does not make it dependent in
M , and (ii) their density (the ratio between their marginal contribution and cost according to the linear constraints) is at least ρ .
This process of growing the small elements solution can end in one of two ways.
One option is that the process ends because no additional elements can be added to the solution (in other words, no element has the two properties stated above).
In this case the better among S B and the small elements solution obtained S k is returned.
The other way in which the process of growing the small elements solution can end is when it starts violating at least one linear constraint.
When this happens, Algorithm 3 uses a procedure called SetExtract to get a subset of the small elements solution that is feasible and also has a good value, and this subset is returned.
Let us now describe the procedure SetExtract , which appears as Algorithm 7.
As explained above, this procedure gets a set S of small elements that violates at least one of the linear constraints.
Its objective is to output a subset T of S that does not violate any linear constraint, but is not very small in terms of the linear constraints.
Since the set S passes by Algorithm 3 to SetExtract contains only elements of density at least ρ , this implies that the output set T of SetExtract has a significant value.
Algorithm 7 does its job by constructing λ + 1 subsets T 1 , T 2 , . .
, T λ +1 of S , and then outputting the subset with the maximum size with respect to the linear constraint.
The first subset T 1 is constructed by starting with the empty set, and then simply adding elements of S to
T 1 one by one, in an arbitrary order, until some element u 1 cannot be added because adding it will result in a set that violates some linear constraint.
The algorithm then constructs the second subset
T 2 in essentially the same way, but makes sure to include u 1 in it by starting with the set { u 1 } and then adding elements of S to T 2 one by one in an arbitrary order, until some element u 2 cannot be added because adding it will result in a set that violates some linear constraint.
The set T 3 is then constructed in the same way starting from the set { u 1 , u 2 } , and in general the set T j is constructed by starting from the set { u 1 , u 2 , . . .
, u j − 1 } and then adding to it elements of S in an arbitrary order until some element u j cannot be added because adding it will result in a set that violates some linear constraint.
Intuitively, this method of constructing the subsets T 1 , T 2 , . .
, T λ +1 guarantees that every element u j is rejected at most once from a subset due to the linear constraints.
Algorithm 7: SetExtract ( λ, S ) 1 for j = 1 to λ + 1 do
Let T j ← { u 1 , u 2 , . . .
, u j − 1 } .
for every element u ∈ S do if max 1 ≤ i ≤ d c i ( T j + u ) ≤ 1 then Add u to T j .
else
Denote the element u by u j from this point on, and exit the loop of Line 3.
6 return the set maximizing (cid:80) di =1 c i ( T ) among all sets T ∈ { T 1 , T 2 , . .
, T λ +1 } .
A formal statement of the guarantee of SetExtract is given by the next lemma.
Lemma H.1.
Assuming the input set S of SetExtract obeys
• c i ( S ) > 1 for some integer 1 ≤ i ≤ d and • max 1 ≤ i ≤ d c i ( u ) ≤ λ − 1 for every element u ∈ S ,
then the output set T is a subset of S such that max 1 ≤
i ≤ d c ( T ) ≤ 1 , but (cid:80) di
=1 c i ( T ) ≥ λλ +1 .
Proof.
For any 1 ≤ j ≤ λ + 1 , SetExtract initializes the set T j to contain j − 1 ≤ λ elements, which implies that immediately after this initialization the set T j obeyed max 1 ≤ i ≤ d c i ( T j ) ≤because of the second condition of the lemma.
After the initialization of T j , Algorithm 7 grows it by adding to it only elements whose addition does not make max 1 ≤ i ≤ d c i ( T j ) exceed 1 .
This method of growth guarantees that the set T j keeps obeying max 1 ≤ i ≤ d c i ( T j ) ≤ 1 throughout the execution of the algorithm.
Therefore, since T is chosen as the set T j for some integer j , it obeys max 1 ≤ i ≤ d c i ( T ) ≤ 1 .
Consider any iteration of the loop on Line 1 of SetExtract .
If this loop never reaches Line 5, then we are guaranteed that all the elements of S are added to T j , which contradicts the inequality max 1 ≤ i ≤ d
c i ( T ) ≤ 1 that we have proved above because we are guaranteed that c i ( S ) > 1 for some integer 1 ≤ i ≤ d .
Therefore, SetExtract reaches Line 5 in every iteration of the outter loop.
Specifically, for any integer 1 ≤ j ≤ λ , since the algorithm reached Line 5 in iteration number j of this loop, we must have max 1 ≤ i ≤ d
c i ( T j + u j ) > 1 .
Hence, d (cid:88) i =
[Equation]
d
[Equation]
c
[Equation]
u
[Equation]
d
[Equation]
i
[Equation]
c i ( u j ) .
Using this inequity and the observation that T λ
+1 includes all the elements u 1 , u 2 , . .
, u λ , we get d (cid:88)
[Equation]
i
[Equation]
We now get to the analysis of the full Algorithm 3.
Let ℓ be the final value of the variable k of
Algorithm 3.
Observation H.2.
Algorithm 3 outputs a feasible set.
Proof.
One can observe that the set S ℓ is feasible because the conditions of Lines 4 and 7 of Algorithm 3 guarantee that it assigns to S k only feasible sets for any 0 ≤ k ≤ ℓ .
Furthermore, S B is feasible by the properties we assume for the procedure BigElementsAlg .
Therefore, to prove the observation it only remains to show that the output set T of SetExtract ( λ, S ℓ +1 ) is feasible when
Line 8 is executed.
To see that this is indeed the case, we note that T is a subset of S ℓ +1 , which is an independent set of M , and therefore, T is also independent in M .
Additionally, Lemma H.1 implies that max 1 ≤ i ≤ d c i ( T ) ≤ 1 because (i) we are guaranteed by the condition of Line 7 that there exists an integer 1 ≤ d ≤ k
such that c
i ( S ℓ +1 )
=
c i ( S ℓ + v ℓ +1 ) > 1 when Line 8 is executed, and (ii) we are guaranteed that max u ∈ S ℓ +1 max 1 ≤ i ≤ d c i ( u ) ≤ λ − 1 since S ℓ +1 contains only small elements.
Next, we analyze the time complexity of Algorithm 3.
Lemma H.3.
Algorithm 3 has a time complexity of O ( λnd + nr + T B ) , where T B is the time complexity of BigElementsAlg .
Proof.
The construction of the set B requires O ( dn ) time, and therefore, the time complexity required for the entire Algorithm 3 except for the loop starting on Line 4 is O ( dn + T B ) .
In the rest of this proof we show that this loop requires O ( λnd + nr ) time, which implies the lemma.
The loop starting on Line 4 of Algorithm 3 runs at most r times because the size of the set S k grows by at least one after every such iteration (and this set always remains feasible in M ).
To understand the time complexity of the iterations of the last loop, we can observe that, assuming we maintain the values
c i ( S k ) , each such iteration takes O ( n + d ) time, with the exception of the following operations.
• In Line 4 we need to calculate the sum (cid:80) di =1 c i ( u ) for multiple elements u , which takes O ( d ) time per element.
However, we can pre-calculate this sum for all the elements of N in O ( nd ) time.
• Executing SetExtract ( λ, S k +1 ) requires O ( λ | S k +1 | d )
=
O ( λrd ) time.
However, this procedure is executed at most once by Algorithm 3.
Combining all the above, we get that the loop starting on Line 4 of Algorithm 3 requires only
O ( nd + λrd + r ( n + d ))
=
O ( λnd + nr ) time.
Our next objective is to analyze the approximation ratio of Algorithm 3.
Let E be the event that
Algorithm 3 returns through Line 8.
We begin the analysis of Algorithm 3 by looking separately at the case in which the event E happens and at the case in which it does not happen.
When the event
E happens, the output set of Algorithm 3 is the output set of SetExtract .
This set contains only high density elements and is large in terms of the linear constraints (by Lemma H.1), which provides a lower bound on its value.
The following lemma formalizes this argument.
Lemma H.4.
If the event E happens, then Algorithm 3 returns a solution of value at least λρλ +1 .
Proof.
Let T be the output set of SetExtract ( λ, S ℓ +1 ) .
We need to show that f ( T ) ≥ λρλ +1 .
Since
T is a subset of S ℓ +
= { v 1 , v 2 , . .
, v ℓ +1 } ,
[Equation]
where the first inequality follows from the submodularity of f , the second inequality follows from the definition of v k , and the last inequality follows from the guarantee of Lemma H.1.
Handling the case in which the event E does not happen is somewhat more involved.
Towards this goal, let us recursively define a set O k for every 0 ≤ k ≤ ℓ .
The base of the recursion is that for k = ℓ we define O ℓ = OPT \ ( B ∪{ u ∈ OPT | f
( u | S ℓ ) < ρ · (cid:80) di =1 c i ( u ) } ) .
Assuming O k +1 is already defined for some 0 ≤ k < ℓ , we define O k as follows.
Let D k
= { u ∈
O k
+1 \ S k |
S k + u ∈ I} .
If | D k | ≤ p , we define O k
=
O k +1 \ D k .
Otherwise, we let D ′
k be an arbitrary subset of D k of size p , and we define O k
=
O k +1 \ D ′ k .
Lemma H.5.
Assuming E does not happen, O 0 = ∅ .
Proof.
We prove by a downward induction the stronger claim that | O k | ≤ pk for every 0 ≤ k ≤ ℓ .
To prove this inequality for k = ℓ , we note that the fact that E did not happen and still Algorithmterminated after ℓ iterations implies that no element of O ℓ \ S ℓ can be added to S ℓ without violating independence in M .
Therefore, S ℓ is a base of S ℓ ∪ O ℓ in this p -system, which implies | O ℓ | ≤ p | S ℓ | = pℓ by the definition of a p -system since O ℓ ⊆
OPT is independent in M .
Assume now that the inequality | O k +1 | ≤ p ( k + 1) holds for some 0 ≤ k < ℓ , and let us prove | O k | ≤ pk .
There are two cases to consider.
If O k
=
O k +1 \ D
′ k , then |
O k | = | O k +1 | − | D ′ k
| ≤ p ( k + 1) − p = pk .
Otherwise, if O k
=
O k +1 \ D k , then, by the definition of D k , there are no elements of O k \ S k that can be added to S k without violating independence in M .
Therefore, S k is a base of S k ∪ O k in this p -system, which implies | O k | ≤ p | S k | = pk by the definition of a p -system since O k ⊆ O ℓ ⊆
OPT is independent in M .
Corollary H.6.
Assuming E does not happen, f ( S
[Equation]
Proof.
We prove by induction the stronger claim that, for every integer 0 ≤ k ≤ ℓ ,
[Equation]
For k = 0 this inequality follows from the non-negativity of f since S
= O 0 = ∅ by Lemma H.5.
Assume now that Inequality (4) holds for some value k − 1 obeying 0 ≤ k − 1 < ℓ , and let us prove it for k .
Consider the set ∆ k
= O k \ O k − 1 .
The construction of O k − 1 guarantees that every element of ∆ k can be added to S k − 1 without violating independence in M .
Furthermore, since ∆ k ⊆ O ℓ , every element u ∈ ∆ k also obeys f ( u | S k ) ≥ f ( u | S ℓ ) ≥ ρ · (cid:80) di
=1 c i ( u ) .
Therefore, every element of ∆ k obeys the condition of the loop on Line 4 of Algorithm 3 in the k -th iteration of the loop, and by the definition of v k , this implies
[Equation]
where the second inequality follows from the induction hypothesis and the submodularity of f , the penultimate inequality follows from the monotonicity of f and the observation that the construction of O k − 1 guarantees | ∆ k | ≤ p , and the last inequality follows again from the submodularity of f .
To use the last corollary, we need a lower bound on O ℓ ∪ S ℓ , which is given by the next lemma.
Lemma H.7.
f ( O ℓ ∪ S ℓ ) ≥ f ( OPT ) − f ( S B ) /α − ρ · (cid:104) d − | OPT
[Equation]
| λ (cid:105) .
Proof.
Observe that
[Equation]
where the first inequality follows from the submodularity and monotonicity of f , and the second inequality follows from the definition of S B .
To lower bound the rightmost side of the last inequality, we need to upper bound the last term in it.
[Equation]
where the last inequality holds since OPT is a feasible set and every element of B is big.
Plugging the last inequality into Inequality (5) completes the proof of the lemma.
We are now ready to lower bound the value of the solution produced by Algorithm 3.
While reading the following proposition, it useful to keep in mind that | OPT ∩ B | ≤ d ( λ − 1) because (i) the feasibility of OPT implies c i ( OPT ∩ B ) ≤ 1 for every integer 1 ≤ i ≤ d and (ii) every element u ∈ OPT ∩ B obeys
c
i ( u ) > λ − 1 for at least one such i .
Proposition H.8.
Let
[Equation]
If ρ ≥ ρ ∗ and the event E happened, or ρ ≤ ρ ∗ and the event E did not happen, then Algorithmoutputs a solution of value at least λρ ∗ / ( λ + 1) .
Furthermore, if ρ ∈ [(1 − δ ) ρ ∗ , ρ ∗ ] for some value δ ∈ (0 , 1] , then, regardless of the realization of the event E , Algorithm 3 outputs a solution of value at least (1 − δ ) λρ ∗ / ( λ + 1) .
Finally, Algorithm 3 runs in O ( λnd + nr + T B ) time, where T B is the time complexity of BigElementsAlg .
Proof.
If the event E happened, then Lemma H.4 guarantees that the output of Algorithm 3 is of value at least λρ/ ( λ + 1) .
Therefore, to complete the proof of the proposition, it suffices to show that when the event E does not happen and ρ ∗ ≥ ρ , the value of the output of the Algorithm 3 is at least λρ ∗ / ( λ + 1) ; and the rest of this proof is devoted to showing that this is indeed the case.
In the last case, Corollary H.6 and Lemma H.7 imply together
[Equation]
and therefore, the output set of Algorithm 3 is of value at least
[Equation]
In Appendix I.1 we show a modified version of Algorithm 3 that appears as Algorithm 9, accepts a quality control parameter ε ∈ (0 , 1 / 4) and employs the thresholding speedup technique due to Badanidiyuru and Vondr´ak [4].
Formally, the properties of Algorithm 9 are given by the following variant of Proposition H.8.
Proposition H.9.
Let
[Equation]
) / (1 + λ − 1 ) + d − | OPT ∩ B | /λ .
There exists an event ˜
E such that, if ρ ≥ ρ ∗ and the event ˜
E happened, or ρ ≤ ρ ∗ and the event ˜
E did not happen, then Algorithm 9 outputs a solution of value at least λρ ∗ / ( λ + 1) .
Furthermore, if ρ ∈ [(1 − δ ) ρ ∗ , ρ ∗ ] for some value δ ∈ (0 , 1] , then, regardless of the realization of the event ˜ E ,
Algorithm 9 outputs a solution of value at least (1 − δ ) λρ ∗ / ( λ + 1) .
Finally, Algorithm 9 runs in
O ( λnd + nε − 1 (log n +log ε − 1 )+ T B ) time, where T B is the time complexity of BigElementsAlg .
H.2 Guessing ρ
To use Algorithm 3 (or the faster Algorithm 9), one must supply a value for ρ .
Furthermore, according to Proposition H.8, it is best if the supplied value of ρ is close to ρ ∗ (throughout this section, unless stated explicitly otherwise, ρ ∗ refers to its value as defined in Proposition H.8).
In this section we present an algorithm that manages to supply such a value for ρ using binary search.
However, before presenting this algorithm, we first need to find a relatively small range which is guaranteed to include ρ ∗ .
Let α be a known lower bound on the value of α .
Observation H.10.
It always holds that
[Equation]
Proof.
According to the definition of ρ ∗ ,
[Equation]
where the first inequity follows from the submodularity and non-negativity of f , and the second inequality follows from the upper bound on | OPT ∩ B | given in the discussion before Proposition H. and the inequality λ ≥ 1 .
Similarly,
[Equation]
where the first inequality holds since every singleton is a feasible set by our assumptions, and the second inequality holds since 1 + λ − 1 ≥ 1 and | OPT ∩ B | /λ ≥ 0 .
We are now ready to present, as Algorithm 8, the algorithm that avoids the need to guess ρ .
This algorithm gets a quality control parameter δ ∈ (0 , 1) in addition to the parameter λ of Algorithm 3.
In Algorithm 8 we use the shorthand ρ ( i ) ≜ (1 + δ ) i · max u ∈N f ( { u } ) /
( p +
+ α − 1 + d ) .
Algorithm
: ρ Guessing Algorithm (
λ, δ )
Let i ← 0 , ¯ i ← (cid:108) log 1+ δ 2 np − log 1+ δ 1 p
+
+ α − 1 + d (cid:109) and k ← 0 .
2 while ¯ i
i
Algorithm
Proposition H.11.
Algorithm 8 has a time complexity of O (( λnd + nr + T B ) · (log δ − 1 +log(log n + log( α − 1 + d ))) and outputs a set of value at least (1 − δ ) λρ ∗ λ +≥ (1 − δ ) λ · f ( OPT ) λ ( p + 1 + α − 1 ) + ( λ + 1)( d − | OPT ∩ B | /λ ) .
Proof.
The number of iterations done by Algorithm 8 is at most
[Equation]
The time complexity guaranteed by the proposition now follows by multiplying the last expression with the time complexity of Algorithm 3 given by Proposition H.8.
It remains to prove the approximation guarantee stated in the proposition.
Assume towards a contradiction that this approximation guarantee does not hold, and let us show that this results in a contradiction.
Our first objective is to show that this assumption implies that the inequality
[Equation]
holds throughout the execution of Algorithm 8.
Immediately after the initialization of i and ¯ i , this inequality holds by Observation H.10.
Assume now that Inequality (6) held before some iteration of the loop of Algorithm 8, and let us explain why it holds also after the iteration.
By Proposition H.8, our assumption (that the approximation guarantee does not hold) implies that, in every iteration of the loop of Algorithm 8, the event E k happened if and only if ρ ( i k ) < ρ ∗ , and therefore, Algorithmupdates i in the iteration when ρ ( i k ) < ρ ∗ and updates ¯ i in the iteration when ρ ( i k ) ≥ ρ ∗ .
By Inequality (6) and the observation that ¯ i − i ≤ 1 when Algorithm 8 terminates, we get
[Equation]
However, the last inequality implies f ( A ′ ) ≥ (1 − δ ) λρ ∗ / ( λ + 1) by Proposition H.8, and hence, completes the proof.
In Appendix I.2 we give, as Algorithm 10, a modified version of Algorithm 8 that is based on
Algorithm 9 instead of Algorithm 3.
We prove for this algorithm the following proposition.
Proposition H.12.
Algorithm 10 has a time complexity of O (( λnd + nε − 1 (log n +log ε − 1 )
+ T B ) · (log δ − 1 + log(log n + log( α − 1 + d ))) and outputs a set of value at least
[Equation]
where ρ ∗ represents here the value stated in Proposition H.9.
H.3 Proof of Theorem 5.
In this section we use the machinery developed so far to prove Theorem 5.1.
To use Algorithmsor 10 one must choose the algorithm BigElementsAlg .
A very simple choice is to use an algorithm that just returns the best singleton subset of B .
This leads to the following theorem.
Theorem H.13.
For every value ε ∈ (0 , 1 / 4] , there is a polynomial time [(1+ O ( ε ))
( p +1+ 74 d )] − 1approximation algorithm for SMKS that runs in ˜ O ( nd + n/ε ) time.
Proof.
Given the above choice for BigElementsAlg , we can set α = | OPT ∩ B | − 1 because the submodularity of f implies
[Equation]
Thus, for λ ≥ 2 , it is valid to choose α − 1 =
d ( λ − 1) because of the upper bound on | OPT ∩ B | explained in the discussion before Proposition H.8.
If we now choose to use Algorithm 10 and set λ
= 2 and δ
= ε , we get by Proposition H.12 that the inverse of the approximation ratio of the algorithm we consider is at most
[Equation]
and its time complexity is
[Equation]
where the last equality holds because BigElementsAlg can be implemented by simply scanning all the n possible singleton sets, which requires a time complexity of T B
= O ( n ) .
A slightly more involved option for BigElementsAlg is an algorithm that enumerates over all sets of up to two big elements, and outputs the best such set that is feasible.
This leads to the following theorem.
Theorem H.14.
For every value ε ∈ (0 , 1 / 4] , there is a polynomial time ( p + 1 . 5556 + 139 d + ε ) - approximation algorithm for SMKS that runs in ˜ O
n 2 + nd ) time.
Proof.
In this proof we need the following known lemma.
Lemma H.15 (Lemma 2.2 of Feige et al.
Let f : 2 N → R be a submodular function, let A be an arbitrary subset of N , and let A ( p ) be a random subset of A containing every element of A with probability p (not necessarily independently).
Then,
[Equation]
Next, we show that one is allowed to choose α = min { 2 / | OPT ∩ B | , 1 } for the above described
BigElementsAlg .
If | OPT ∩ B | ≤ 2 then this is trivial.
Otherwise, by choosing R to be a uniformly random subset of OPT ∩ B of size 2 , we get that the value of the output of BigElementsAlg is at least
[Equation]
where the first inequality follows from Lemma H.15 and the observation that R includes every element of OPT ∩ B with probability exactly 2 / | OPT ∩ B | .
Hence, it is valid to set α − 1 = max { d ( λ − 1) / 2 , 1 } because of the upper bound on | OPT ∩ B | explained in the discussion before
Proposition H.8.
Consider now the algorithm obtained by plugging δ = ε 2 p +4 d +4 , λ
= 3 and the above choice for
BigElementsAlg into Algorithm 8.
We would like to show that this algorithm has all the properties guaranteed by the theorem.
By Proposition H.11, the time complexity of this algorithm is
[Equation]
where the last equality holds since a brute force implementation of BigElementsAlg runs in O ( n 2 ) time.
It remains to analyze the approximation ratio of our suggested algorithm.
Towards this goal, we need to consider a few cases.
If | OPT ∩ B | ≥ 2 , then, by Proposition H.11, the inverse of the approximation ratio of the above algorithm is no worse than
[Equation]
Otherwise, if | OPT ∩ B | ≤ 1 , then, by the same proposition, the inverse of the approximation ratio of the above algorithm is at most
[Equation]
The above inequalities prove an approximation ratio which is a bit weaker than the one guaranteed by the theorem.
To get exactly the approximation ratio guaranteed by the theorem, we need to be a bit more careful with the last case.
First, for | OPT ∩ B | = 1 , λ ( p + 1 + α − 1 )
+ ( λ +
[Equation]
( d − | OPT ∩ B | /λ
[Equation]
1 − δ
[Equation]
λ
[Equation] [Equation]
[Equation] [Equation]
Handling the case OPT ∩ B = ∅ is a bit more involved.
Since there are no big elements in this case in OPT , there is no need to take S B into account in the analysis of Algorithm 8.
It can be observed that by repeating the analysis, but ignoring this set, we can get that the value of the output set of this algorithm is also lower bounded by
[Equation]
which in our case implies that the inverse of the approximation ratio of our algorithm is at most
[Equation]
Theorem 5.1 now follows immediately from Theorems H.13 and H.14.
Remark: It is natural to consider also candidates for BigElementsAlg that enumerate over larger subsets of B .
However, this will require Ω( n 3 ) time, and is, therefore, of little interest as one can obtain a clean ( p + d + 1) -approximation for SMKS in ˜ O ( n 3 ) time (see Section 1.1).
I Fast Versions of Algorithms from Appendix H I.1 Nearly Linear Time Version of Algorithm
In this section we present a version of Algorithm 3 that runs in nearly linear time.
This version appears as Algorithm 9, and it gets a quality control parameter ε ∈ (0 , 1 / 4) (in addition to the parameters λ and ρ of Algorithm 3).
The speedup in this version of the algorithm is obtained using a technique due to Badanidiyuru and Vondr´ak [4] which employs a decreasing threshold τ .
In every iteration of the loop starting on Line 5 of Algorithm 9, the algorithm looks for elements whose marginal value exceeds this threshold.
This guarantees that the element selected by the algorithm
// Build the set of big elements, and find a candidate solution based on them.
1 Let B ← { u ∈ N | ∃ 1 ≤ i ≤ d c i ( u ) > λ − 1 } .
2 Let S B be the output set of BigElementsAlg ( B ) .
// Construct a solution from the small elements.
3 Let S 0 ← ∅ , k ← 0 , M ← max u ∈N f ( { u } ) and τ ← M .
4 while τ ≥ εM/ [(1 + ε ) n ] do for every element u ∈ N \ ( S k ∪ B ) do if S
+ u
I and u ) } then
Let v k +1 ← u , and let S k +1 ← S k
+ v k +1 .
if max by return has an almost maximal marginal among the elements that can be selected because the previous iteration of the same loop already selected every element that could be selected when τ was larger.
As explained in Section H.1, the properties we would like to prove for Algorithm 9 are summarized by Proposition H.9, which we repeat here for convenience.
Proposition H.9.
Let
[Equation]
There exists an event ˜
E such that, if ρ ≥ ρ ∗ and the event ˜
E happened, or ρ ≤ ρ ∗ and the event ˜
E did not happen, then Algorithm 9 outputs a solution of value at least λρ ∗ / ( λ + 1) .
Furthermore, if ρ ∈ [(1 − δ ) ρ ∗ , ρ ∗ ] for some value δ ∈ (0 , 1] , then, regardless of the realization of the event ˜ E ,
Algorithm 9 outputs a solution of value at least (1 − δ ) λρ ∗ / ( λ + 1) .
Finally, Algorithm 9 runs in
O ( λnd + nε − 1 (log n +log ε − 1 )+ T B ) time, where T B is the time complexity of BigElementsAlg .
We begin the proof of Proposition H.9 with the following lemma, which proves that Algorithmhas the time complexity stated in the proposition.
Lemma I.1.
Algorithm 9 has a time complexity of O ( λnd + nε − 1 (log n + log ε − 1 )
+ T B ) , where
T B is the time complexity of BigElementsAlg .
Proof.
The construction of the set B requires O ( dn ) time, and therefore, the time complexity required for the entire Algorithm 9 except for the loop starting on Line 4 is O ( dn + T B ) .
In the rest of this proof we show that this loop requires O ( λnd + nε − 1 (log n + log ε − 1 )) time, which implies the lemma.
The loop starting on Line 4 of Algorithm 9 runs at most the number of times that is required to decrease τ from M to εM/ [(1 + ε ) n ] , which is
[Equation]
Within each iteration of the loop on Line 4, the loop starting on Line 5 executes at most n times.
To understand the time complexity of the iterations of the last loop, we can observe that each such iteration takes a constant time with the exception of the following operations.
• In Line 6 we need to calculate the sum (cid:80) di
=1 c
i ( u ) , which takes O ( d ) time.
However, we can pre-calculate this sum for all the elements of N in O ( nd ) time.
• Checking the condition on Line 8 requires O ( d ) time (assuming we maintain the values
c i ( S k ) ).
However, this line is executed only when an element is added to the solution of the algorithm, which happens at most O
• Executing SetExtract ( λ, S k +1 ) requires O ( λ | S k +1 | d )
=
O ( λrd ) =
O ( λnd ) time.
However, this procedure is executed at most once by Algorithm 9.
Combining all the above, we get that the loop starting on Line 4 of Algorithm 9 requires only
O ( λnd + nε − 1 (log n + log ε − 1 )) time.
Like in Section H.1, we use ℓ below to denote the final value of the variable k .
Furthermore, one can observe that the proof of Observation H.2 applies to Algorithm 9 up to some natural modifications, and therefore, we are guaranteed that the output set of Algorithm 9 is feasible.
The rest of this section is devoted to bounding the approximation guarantee of this output set.
Let ˜ E be the event that Algorithm 9 returns through Line 9.
We consider separately the case in which the event ˜
E happens and at the case in which it does not happen.
When the event ˜
E happens, the output set of Algorithm 3 is the output set of SetExtract .
This set contains only high density elements and is large in terms of the linear constraints (by Lemma H.1), which provides a lower bound on its value.
This argument can be formalized, leading to the following lemma, whose proof is omitted since it is analogous to the proof of Lemma H.4.
Lemma I.2.
If the event ˜ E happens, then Algorithm 3 returns a solution of value at least λρλ +1 .
Handling the case in which the event ˜
E does not happen is somewhat more involved.
Towards this goal, we recursively define a set O k for every 0 ≤ k ≤ ℓ (note that the definition we give here is slightly different compared to the one given in Section H.1).
The base of the recursion is that for k = ℓ we define O ℓ = OPT \ ( B ∪ { u ∈ OPT | f
( u | S ℓ ) < max { ρ · (cid:80) di
=1 c i ( u )) , εM/n }} .
Assuming O k +1 is already defined for some 0 ≤ k < ℓ , we define O k as follows.
Let D k = { u ∈
O k +1 \ S k |
S k + u ∈ I} .
If | D k | ≤ p , we define O k
=
O k +1 \ D k .
Otherwise, we let D ′
k be an arbitrary subset of D k of size p , and we define O k
=
O k +1 \ D ′ k .
Lemma I.3.
Assuming ˜
E does not happen, O 0 = ∅ .
Proof Sketch.
The proof of this lemma is very similar to the proof of Lemma H.5.
The only difference is that arguing why O ℓ is a base of S ℓ ∪ O ℓ is a bit more involved now.
Specifically, consider the last iteration of the loop starting on Line 4 of Algorithm 9.
In this iteration the value of τ was at most εM/n , and therefore, every element of O ℓ would have been added to S ℓ during this iteration unless this addition violates independence in M .
Using the last lemma, we can now prove the following corollary, which corresponds to Corollary H. from Section H.1.
Corollary I.4.
Assuming ˜
E does not happen, f ( S ℓ ) ≥ f
( O ℓ ∪ S ℓ ) (1+ ε ) p +1 .
Proof.
We prove by induction the stronger claim that, for every integer 0 ≤ k ≤ ℓ ,
[Equation]
For k = 0 this inequality follows from the non-negativity of f since S
= O 0 = ∅ by Lemma I.3.
Assume now that Inequality (7) holds for some value k − 1 obeying 0 ≤ k − 1 < ℓ , and let us prove it for k .
Consider the set ∆ k
= O k \ O k − 1 .
Let u ′ be an element of ∆ k maximizing f ( u ′ | S k − 1 ) , and let τ ′ be the maximum value that τ takes in any iteration of the loop starting on Line 4 of Algorithmthat is not larger than f ( u ′ | S k ) .
Such a value exists because the inclusion ∆ k ⊆
O ℓ implies that every element u ∈ ∆ k obeys f ( u | S k − 1 ) ≥ f ( u | S ℓ ) ≥ max { ρ · (cid:80)
[Equation] [Equation]
i ( u ) , εM/n } .
Observe now that u ′ cannot belong to S ℓ because u ′ ∈
∆ k ⊆
O ℓ guarantees f ( u ′ | S ℓ ) > 0 , which implies that, during the iteration of of the loop starting on Line 4 that corresponds to τ ′ , the element u ′ was not added to the solution of Algorithm 9.
Let us study the reason that u ′ was not added.
By the definition of τ ′ , f ( u ′ | S k − 1 ) ≥ τ ′ .
Additionally, by the construction of O k − 1 , every element of ∆ k can be added to S k − 1 without violating independence in M .
These two facts imply that the reason that u ′ was not added must have been that, by the time Algorithm 9 considers the element u ′ in the iteration correspond to τ ′ , the solution of Algorithm 9 already contained at least k elements.
Since up to this time the algorithms adds to its solution only elements whose marginal contribution is at least τ ′ , this implies
[Equation]
Recall now that f ( S k ) − f
( S k − 1 )
=
f ( v k | S k − 1 ) .
Combining this equality with the previous one, we get
[Equation]
where the second inequality follows from the induction hypothesis and the submodularity of f , the penultimate inequality follows from the monotonicity of f and the observation that the construction of O k − 1 guarantees | ∆ k | ≤ p , and the last inequality follows again from the submodularity of f .
To use the last corollary, we need a lower bound on O ℓ ∪ S ℓ , which is given by the next lemma (and corresponds to Lemma H.7).
Lemma I.5.
[Equation]
Proof.
Observe that
[Equation]
where the first inequality follows from the submodularity and monotonicity of f , and the second inequality follows from the definition of S B .
To lower bound the rightmost side of the last inequality, we need to upper bound the two last terms in it.
[Equation]
where the last inequality holds since OPT is a feasible set and every element of B is big.
Additionally,
[Equation]
where the second inequality holds since | OPT ∩ B | is a subset of N , and therefore, is of size at most n ; and the last inequality holds since every element of N is a feasible solution by our assumption.
Plugging the last inequalities into Inequality (8) completes the proof of the lemma.
We are now ready to prove Proposition H.9.
Proof of Proposition H.9.
If the event ˜
E happened, then Lemma I.2 guarantees that the output of
Algorithm 9 is of value at least λρ/ ( λ + 1) .
Therefore, to complete the proof of the lemma, it suffices to show that when the event E does not happen and ρ ∗ ≥ ρ , the value of the output of the
Algorithm 9 is at least λρ ∗ / ( λ + 1) ; and the rest of this proof is devoted to showing that this is indeed the case.
In the last case, Corollary I.4 and Lemma I.5 imply together
[Equation]
and therefore, the output set of Algorithm 9 is of value at least
[Equation]
I.2 Nearly-Linear Time Version of Algorithm
In this section we describe a fast version of Algorithm 8.
This version appears as Algorithm 10.
There are only three differences between the two algorithms: (1) Algorithm 10 uses Algorithminstead of Algorithm 3, (2) the value of ¯ τ is slightly higher in Algorithm 9, and (3) the definition ρ ( i ) is slightly modified in Algorithm 9 to be ρ ( i ) ≜ (1 − 2 ε )(1+ δ ) i · max u ∈N f ( { u } ) /
( p +1+ α − 1 + d ) .
Algorithm δ )
←
[Table]
¯
The following observation corresponds to Observation H.10.
The proofs of the two observations are very similar.
Observation I.6.
For the value ρ ∗ stated in Proposition H.9,
[Equation]
Proof.
According to the definition of ρ ∗ ,
[Equation]
λ
[Equation]
d − |
[Equation]
∩ B |
[Equation]
λ
[Equation]
where the first inequity follows from the submodularity and non-negativity of f , and the second inequality follows from the upper bound on | OPT ∩ B | given in the discussion before Proposition H. and the inequality λ ≥ 1 .
Similarly,
[Equation]
where the first inequality holds since every singleton is a feasible set by our assumption, and the second inequality holds since 1 + λ − 1 ≥ 1 and | OPT ∩ B | /λ ≥ 0 .
Given the last observation, the proof of Proposition H.12 is identical to the proof of Proposition H. up to the following changes.
• One has to use Proposition H.9 and Observation I.6 instead of Proposition H.8 and Observation H.10.
• The calculation needed to bound the number of iterations performed by the algorithm becomes a bit more involved.
Specifically, we can upper bound it by
[Equation]
( p ) − ln(1 − 2 ε ) + ln( p + 1 + α − 1 + d )] − log 2 ln(1 + δ ) ≤ 4 + log 2 [ln n + ln 2 + ln(2 + α − 1 + d )]
+ log 2 δ −=
O (log δ − 1 + log(log n + log( α − 1 + d ))) .
J Additional Experiments J.1 Location Summarization Subject to a Cardinality Constraint
In this section we consider a location summarization application.
The goal is to find a representative summary of tens of thousands of Yelp business locations (from Charlotte, Edinburgh, Las Vegas,
Madison, Phoenix and Pittsburgh) by using their related attributes [6].
The representativeness of a business at location i for another business at a location j is quantified by a similarity measure
M i,j = e − λ · dist ( v i ,v j ) , where v i and v j are vectors of attributes of the facilities at locations i and j .
To select a good representation of all the locations N = { 1 , . .
. , n } , we use the monotone and submodular facility location function [44] given by
[Equation]
In Figure 2a, we observe that L AZY G REEDY performs slightly better than F AST T HRESHOLDG REEDY .
On the other hand, Figure 2b shows that the query complexity of F AST T HRESHOLDG REEDY is much lower.
J.2 Comparing F AST T HRESHOLD G REEDY with S TOCHASTIC G REEDY
Recall that S TOCHASTIC G REEDY is a fast but randomized approach for maximizing submodular functions subject to a cardinality constraint [55].
In the next experiment, we compare F ASTT
HRESHOLD G REEDY with S TOCHASTIC G REEDY under the cardinality constraint.
We consider
[Figure]
the monotone and submodular vertex cover function.
This function is defined over vertices of a (directed) graph G =
( V, E ) .
For a given vertex set S ⊆ V , let N ( S ) be the set of vertices which are pointed to by S , i.e., N ( S ) ≜ { v ∈ V | ∃ u ∈ S s.t.
( u, v ) ∈ E } .
The vertex cover function f : 2 V → R ≥ 0 is then defined as f ( S ) =
| N ( S ) ∪ S | .
We use two different graphs: i) a random graph of n = 10 6 nodes with an average degree of 2 , where 20 additional nodes with degrees 50 are added (the neighbors of these high degree nodes are chosen randomly); ii) Slashdot social network [51].
In Appendix J.2, we observe that the utility of F AST T HRESHOLD G REEDY is significantly better than that of S TOCHASTIC G REEDY .
We also observe a high variability in the utility of solutions returned by S TOCHASTIC G REEDY .
Furthermore, F AST T HRESHOLD G REEDY ( ε = 0 .
8) outperforms S TOCHASTIC G REEDY ( ε ∈ { 0 . 1 , 0 . 2 } ) in terms of both utility and query complexity.
Note that, while S TOCHASTIC G REEDY is performing quite well in many practical scenarios, the theoretical guarantee of this algorithm holds only in expectation, and there are cases resulting in high variance.
In these high variance cases, one has to run S TOCHASTIC G REEDY multiple times, which diminishes the benefit from the algorithm.
J.3 Twitter Text Summarization
In this section, we consider Twitter text summarization with the goal of producing a representative summary of Tweets around the first day of January 2019 [29].
In this application, the objective is to find a diverse summary from the Twitter feeds of news agencies.
The monotone and submodular function f used is defined over a ground set N of tweets as follows: f ( S )
= (cid:80) w ∈W (cid:112)(cid:80) e ∈ S score ( w, e ) , where W is the set of all the English words [42].
If word w is in
Tweet e , then we have score ( w, e ) =
{ number of retweets e has received } .
Otherwise, we define score ( w, e ) = 0 .
The cost of each tweet is proportional to the time difference (in months) between the date of that Tweet and the first day of January 2019 [29].
Like in Section 6.2, it is evident from
Figures 4a and 4b that Algorithm 2 surpasses the baseline algorithms.
J.4 p -Set System and d Knapsack Constraints
In this section we describe a set of experiments designed to compare the performance of Algorithm 10 with several other algorithms under the combination of a p -system and d knapsack constraints.
We consider B ARRIER G REEDY [6], F AST [4], G REEDY and D ENSITY G REEDY as our baselines.
G REEDY keeps adding elements one by one according to their marginal gain as long as the p -system and knapsack constraints allow it.
D ENSITY G REEDY is very similar to G REEDY with the only difference being that it picks elements according to the ratio between their marginal gain and their total knapsack cost.
Note that G REEDY and D ENSITY G REEDY are heuristic algorithms without any theoretical guarantees for the setup of this experiment.
In the first experiment of this section, we again consider the location summarization application from Section J.1.
The goal is to maximize (9) subject to the following constraints: (i) the maximum number of locations from each city is 5 , (ii) the total size of the summary is at most 20 , and (iii)
[Figure]
i
[Figure]
v
[Figure] [Figure]
two knapsacks constraints c 1 and c 2 , where c i ( j ) = distance ( j, POI i ) is the normalized distance of location j from a given location in its city (for c 1 , POI 1 is the down-town; and for c 2 , POI 2 is the international airport in that city).
For ease of understanding the results of the experiments, the distances are normalized to make one unit of knapsack budget equivalent to 100km.
Figures 5a and 5b compare algorithms for varying values of knapsack budget.
In terms of utility, we observe that B ARRIER G REEDY is the best performing algorithm followed by our algorithm (Algorithm 10)
[Figure]
Despite the very competitive performance of Algorithm
[Figure]
its query complexity is almost an order of magnitude faster than that of B ARRIER G REEDY .
The second application is a video summarization task over a collation of videos from the VSUMM dataset [15].
The features for each frame of a video are generated by a pre-trained ResNetmodel [34, 43].
The similarity between two frames i and j is defined by e − λ · dist ( x i ,x j ) , where dist ( x i , x j ) is the Euclidean distance of the corresponding features of the frames.
Similarly to the movie recommendation applications of Sections 6.1 and 6.2, the goal of the summarization task is to maximize the monotone and submodular function f ( S )
= log det( I + αM S ) subject to the combination of the following constraints: (i) the maximum allowed cardinality of the final summary is k (in Figures 5c and 5d, we compare algorithms by varying this value), (ii) the maximum number of allowed frames from each video is 5, and (iii) a single knapsack constraint.
The knapsack cost for each frame u is defined as H( u ) / 20 , where H( u ) is the entropy of u .
This extra knapsack constraint allows us to bound the required bits to encode the produced summary by using the entropy of each frame as a proxy for its encoding size.
We again observe that B ARRIER G REEDY and our algorithm (Algorithm 10) produce the summaries with the highest utilities (see Figure 5c).
Furthermore, in
Figure 5d, we observe that Algorithm 10 is the fastest algorithm.
Particularly, Algorithm 10 is several orders of magnitudes faster than B ARRIER G REEDY and F AST .
[Figure]