Nonnegative Tensor Completion via Integer Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
Unlike matrix completion, tensor completion does not have an algorithm that isknown to achieve the information-theoretic sample complexity rate.
This paperdevelops a new algorithm for the special case of completion for nonnegative tensors.
We prove that our algorithm converges in a linear (in numerical tolerance) numberof oracle steps, while achieving the information-theoretic rate.
Our approach isto define a new norm for nonnegative tensors using the gauge of a particular 0-1polytope; integer linear programming can, in turn, be used to solve linear separationproblems over this polytope.
We combine this insight with a variant of the FrankWolfe algorithm to construct our numerical algorithm, and we demonstrate itseffectiveness and scalability through computational experiments using a laptop ontensors with up to one-hundred million entries.
Introduction
Tensors generalize matrices.
A tensor ψ of order p is ψ ∈ R r 1 ×···× r p , where r i is the dimension ofthe tensor in the i -th index, for i = 1 , . . .
, p .
Though related, many problems that are polynomial-timesolvable for matrices are NP-hard for tensors.
For instance, it is NP-hard to compute the rank of atensor (Hillar and Lim, 2013).
Tensor versions of the spectral norm, nuclear norm, and singular valuedecomposition are also NP-hard to compute (Hillar and Lim, 2013; Friedland and Lim, 2014).
1.1 Past Approaches to Tensor Completion
Tensor completion is the problem of observing (possibly with noise) a subset of entries of a tensor andthen estimating the remaining entries based on an assumption of low-rankness.
The tensor completionproblem is encountered in a number of important applications, including computer vision (Liu et al.
,2012; Zhang et al., 2019), regression with only categorical variables (Aswani, 2016),
healthcare(Gandy et al., 2011; Dauwels et al., 2011), and many other application domains (Song et al., 2019).
Although the special case of matrix completion is now well understood, the tensor version of thisproblem has an unresolved tension.
To date, no tensor completion algorithm has been shown to achievethe information-theoretic sample complexity rate.
Namely, for a tensor completion problem on arank k tensor with sample size n , the information theoretic rate for estimation error is (cid:112) k · (cid:80) i r i /n(Gandy
et al., 2011).
In fact, evidence suggests a computational barrier in which no polynomial-timealgorithm can achieve this rate (Barak and Moitra, 2016).
One set of approaches has polynomial-timecomputation but requires exponentially more samples than the information-theoretic rate (Gandyet al., 2011; Mu et al., 2014; Barak and Moitra, 2016; Montanari and Sun, 2018), whereas another setof approaches achieves the information-theoretic rate but (in order to attain global minima) requiressolving NP-hard problems that lack numerical algorithms (Chandrasekaran et al., 2012; Yuan and Zhang, 2016, 2017; Rauhut and Stojanac, 2021).
However, algorithms that achieve the information-theoretic rate have been developed for a few specialcases of tensors.
Completion of nonnegative rank-1 tensors can be written as a convex optimizationproblem (Aswani, 2016).
For symmetric orthogonal tensors, a variant of the Frank-Wolfe algorithmhas been proposed (Rao et al., 2015).
Though this latter paper does not prove their algorithm achievesthe information-theoretic rate, this can be shown using standard techniques (Gandy et al., 2011).
Thislatter paper is closely related to the approach we take in this paper, with one of the key differencesbeing the design of a different separation oracle in order to support a different class of tensors.
1.2 Contributions
This paper proposes a numerical algorithm for completion of nonnegative tensors that provablyconverges to a global minimum in a linear (in numerical tolerance) number of oracle steps, whileachieving the information-theoretic rate.
Nonnegative tensors are encountered in many applications.
For instance, image and video data usually consists of nonnegative tensors (Liu et al., 2012; Zhanget al., 2019).
Notably, the image demosaicing problem that must be solved by nearly every digitalcamera (Li et al., 2008) is an instance of a nonnegative tensor completion problem, though it has notpreviously been interpreted as such.
Nonnegative tensor completion is also encountered in specificinstances of recommender systems (Song et al., 2019), healthcare applications (Gandy et al., 2011;
Dauwels et al., 2011), and statistical regression contexts (Aswani, 2016).
Our approach is to define a new norm for nonnegative tensors using the gauge of a specific 0-1polytope that we construct.
We prove this norm acts as a convex surrogate for rank and has lowstatistical complexity (as measured by the Rademacher average for the tensor, viewed as a functionfrom a set of indices to the corresponding entry), but is NP-hard to approximate to an arbitraryaccuracy.
Importantly, we prove that the tensor completion problem using this norm achieves theinformation-theoretic rate in terms of sample complexity.
However, the resulting tensor completionproblem is NP-hard to solve.
Nonetheless, because our new norm is defined using a 0-1 polytope,we can use integer linear optimization as a practical means to solve linear separation problems overthe polytope.
We combine this insight with a variant of the Frank-Wolfe algorithm to construct ournumerical algorithm, and we demonstrate its effectiveness and scalability through experiments.
Norm for Nonnegative Tensors
Consider a tensor ψ ∈ R r 1 ×···× r p .
To refer to a specific entry in the tensor we use the notationψ x := ψ x 1 ,...,x p , where x = ( x 1 , . .
, x p ) , and x i ∈
[ r i ] denotes the value of the i -th index, with[ s ] := { 1 , . . .
, s } .
Let ρ = (cid:80) i r i , π = (cid:81)
i r i , and R =
[ r 1 ] × · · · × [ r p ] .
A nonnegative rank-1 tensor is ψ x = (cid:81) pk =
1 θ ( k ) x k , where θ ( k ) ∈
R r k + are nonnegative vectors indexedby the different values of x k ∈
[ r k ] .
To simplify notation, we drop the superscript in θ ( k ) x k and writethis as θ x k when clear from the context.
For a nonnegative tensor ψ , its nonnegative rank is
[Equation]
where we define the ball of nonnegative rank-1 tensors whose maximum entry is λ ∈ R + to be
[Equation]
and B ∞ =
lim λ →∞ B λ .
A nonnegative CP decomposition is given by the summation ψ = 70 (cid:80)
k ψ k , where ψ k ∈ B ∞ for k ∈ [rank + ( ψ )] .
2.1 Convex Hull of Nonnegative Rank-1 Tensors
Now for λ ∈ R + consider the finite set of points:
[Equation]
Using standard techniques from integer optimization (Hansen, 1979; Padberg, 1989), the above setof points can be rewritten as a set of linear constraints on binary variables: S λ
= { ψ : λ · (1 − p )
+λ · (cid:80) pk
=1 θ x k ≤ ψ x , 0 ≤ ψ x ≤ λ · θ x k , θ x k ∈ { 0 , 1 } , for k ∈ [ p ] , x ∈ R} .
Our first result is that theconvex hulls of the set of points S λ and of the ball B λ are equivalent.
Proposition 2.1.
We have the relation that C λ := conv( B λ )
= conv( S λ ) .
Proof.
We prove this by showing the two set inclusions conv( S λ ) ⊆ conv( B λ ) and conv( B λ )
⊆conv( S λ ) .
The first inclusion is immediate since by definition we have S λ ⊂ B λ , and so we focuson proving the second inclusion.
We prove this by contradiction: Suppose conv( B λ ) ̸⊂ conv( S λ ) .
Then there exists a tensor ψ ′ ∈ B λ with ψ ′ ̸∈ conv( S λ ) .
By the hyperplane separation theorem,there exists φ ∈ R r 1 ×···× r p and δ > 0 such that ⟨ φ, ψ ′ ⟩ ≥ ⟨ φ, ψ ⟩ + δ for all ψ ∈ conv( S λ ) , where⟨· , ·⟩ is the usual inner product that is defined as the summation of elementwise multiplication.
Nowconsider the multilinear optimization problem
[Equation]
Proposition 2.1 of (Drenick, 1992) shows there exists a global optimum ψ ′′ of (4) with ψ ′′ ∈ S λ .
By construction, we must have ⟨ φ, ψ ′′ ⟩ ≥ ⟨ φ, ψ ′ ⟩ , which implies ⟨ φ, ψ ′′ ⟩ ≥ ⟨ φ, ψ ⟩ + δ for allψ ∈ conv( S λ ) .
But this last statement is a contradiction since ψ ′′ ∈ S λ ⊆ conv( S λ ) .
Remark 2.2 .
This result has two useful implications.
First, C λ is a polytope since it is the convex hullof a finite number of bounded points.
Second, the elements of S λ are the vertices of C λ , since anyindividual element cannot be written as a convex combination of the other elements.
We will call the set C λ the nonnegative tensor polytope.
A useful observation is that the followingrelationships hold: B λ = λ B 1 , S λ = λ S 1 , and C λ = λ C 1 .
2.2 Constructing a Norm for Nonnegative Tensors
Since the set of nonnegative tensors forms a cone (Qi et al., 2014), we need to use a modifieddefinition of a norm.
A norm on a cone K is a function p : K → R + such that for all x, y ∈ K thefunction has the following three properties: p ( x ) =
if and only if x = 0 ; p ( γ · x ) = γ · p ( x ) forγ ∈ R + ; and p ( x + y ) ≤ p ( x ) + p ( y ) .
The difference with the usual norm definition is subtle: Thesecond property here is required to hold for γ ∈ R + , whereas in the usual norm definition we requirep ( γ · x ) =
| γ | · p ( x ) for all γ ∈ R .
We next use C λ to construct a new norm for nonnegative tensors using a gauge (or Minkowskifunctional) construction.
Though constructing norms using a gauge is common in machine learning(Chandrasekaran et al., 2012), the convex sets used in these constructions are symmetric about theorigin.
Symmetry guarantees that scaling the ball eventually includes the entire space (Rockafellarand Wets, 2009).
However, in our case C λ is not symmetric about the origin, and so without proof wedo not a priori know whether scaling C 1 eventually includes the entire space of nonnegative tensors.
Thus we have to explicitly prove the gauge is a norm.
Proposition 2.3.
The function defined as
[Equation]
is a norm for nonnegative tensors ψ ∈ R r 1 ×···× r p + .
Proof.
We first prove the above function is finite.
Consider any nonnegative tensor ψ ∈ R r 1 ×··· r p + ,and note there exists a decomposition ψ = (cid:80) πi
=1 ψ k with ψ k ∈ ∥ ψ ∥ max · B 1 (Qi et al., 2016).
(Thisholds because we can choose each ψ k to have a single non-zero value that corresponds to a differententry of ψ .)
Hence by Proposition 2.1, ψ k ∈ ∥ ψ ∥ max · C 1 .
Recalling the decomposition of ψ , thismeans ψ ∈ π
∥ ψ ∥ max · C 1 which by definition means ∥ ψ ∥ + ≤ π ∥ ψ ∥ max .
Thus ∥ ψ ∥ + must be finite.
Next we verify that the three properties of a norm on a cone are satisfied.
To do so, we first observethat by definition: C 1 is convex; 0 ∈ C 1 ; and C 1 is closed and bounded.
Thus by Example 3.50of (Rockafellar and Wets, 2009) we have { ψ : ∥ ψ ∥ + = 0 } = { 0 } , which means the first normproperty holds.
Also by Example 3.50 of (Rockafellar and Wets, 2009)
we have λ C 1 ⊆ λ ′
C 1 for all0 ≤ λ ≤ λ ′ , which means the second norm property holds.
Last, note Example 3.50 of (Rockafellarand Wets, 2009) establishes sublinearity of the gauge, and so the third norm property holds.
2.3 Convex Surrogate for Nonnegative Tensor Rank
An important property of our norm ∥ ψ ∥ + is that it can be used as a convex surrogate for tensor rank,whereas the max and Frobenius norms cannot.
Proposition 2.4.
If ψ is a nonnegative tensor, then we have ∥ ψ ∥ max ≤ ∥ ψ ∥ + ≤ rank + ( ψ ) ·
∥ ψ ∥ max .
If rank + ( ψ ) = 1 , then ∥ ψ ∥ + = ∥ ψ ∥ max .
Proof.
Consider any λ ≥ 0 .
If ψ ∈ S λ , then by definition ∥ ψ ∥ max = λ .
By the convexity of normswe have that: if ψ ∈ C λ , then ∥ ψ
∥ max ≤ λ .
This means that for all λ ≥
we have C λ ⊆
U λ := { ψ :∥ ψ ∥ max ≤ λ } .
Thus we have the relation inf { λ ≥ 0 | ψ ∈ λ U 1 } ≤ inf { λ ≥
| ψ ∈ λ C 1 } .
But theleft side is ∥ ψ ∥ max and the right side is ∥ ψ ∥ + .
This proves the left side of the inequality.
Next consider the case where rank + ( ψ ) = 1 .
By definition we have ψ ∈ ∥
ψ ∥ max · B 1 .
Thusψ ∈
∥ ψ ∥ max · C 1 , and so ∥ ψ ∥ + ≤ ∥ ψ ∥ max .
This proves the rank-1 case.
Last we prove the right side of the desired inequality.
Consider any nonnegative tensor ψ .
Recall itsnonnegative CP decomposition ψ = (cid:80) rank + ( ψ ) k
= ψ k with ψ k ∈ B ∞ .
The triangle inequality gives
[Equation]
where the equality follows from the above-proved rank-1 case since rank + ( ψ k ) = 1 by definition ofthe CP decomposition.
But ∥ ψ k ∥ max ≤ ∥ ψ ∥ max since the tensors are nonnegative.
Remark 2.5 .
These bounds are tight.
The lower and upper bounds are achieved by all nonnegativerank-1 tensors.
The identity matrix with k columns achieves the upper bound with rank + ( ψ ) = k .
Measures of Norm Complexity
3.1 Computational Complexity
We next characterize the computational complexity of our new norm on nonnegative tensors.
Proposition 3.1.
It is NP -hard to approximate the nonnegative tensor norm ∥ · ∥ + to arbitraryaccuracy.
Proof.
The dual norm is ∥ φ ∥ ◦ = sup {⟨ φ, ψ ⟩ | ∥ ψ ∥ + ≤ 1 } for all tensors φ ∈ R r 1 ×···× r p .
Theorems3 and 10 of (Friedland and Lim, 2016) show that approximation of the dual norm ∥ · ∥ ◦ is polynomialtime reducible to approximation of the norm ∥ · ∥ + .
We proceed by showing a polynomial-timereduction to approximation of ∥ · ∥ ◦ .
Without loss of generality assume p = 2 and d := r 1 =r 2 .
The appendix of (Witsenhausen, 1986) proves that MAX CUT is polynomial-time reducible tosup {⟨ φ, ψ ⟩ | ψ ∈ S 1 } .
However, since the objective function is linear and the set S 1 consists of thevertices of C 1 , this means that optimization problem is equivalent to sup {⟨ φ, ψ ⟩ | ψ ∈ C 1 } .
This isthe desired polynomial-time reduction of MAX CUT to approximation of the dual norm ∥ · ∥ ◦ because
C 1 = { ψ : ∥ ψ ∥ + ≤ 1 } .
The result now follows by recalling that approximately solving MAX CUT toarbitrary accuracy is NP-hard (Papadimitriou and Yannakakis, 1991; Arora et al., 1998).
Corollary 3.2.
Given K ∈ R + and ψ ∈ R r 1 ×···× r p + , it is NP -complete to determine if ∥ ψ ∥ + ≤ K .
Proof.
The problem is NP-hard by reduction from the problem of Proposition 3.1.
In particular, abinary search can approximate the norm using this decision problem, ( ∥ ψ ∥ + ≤ K )? , as an oracle.
By Proposition 2.4, the search can be initialized over the interval [0 , π ∥ ψ ∥ max ] .
Furthermore, thedecision problem is in NP because we can use the (polynomial-sized) θ from (2) as a certificate.
3.2 Stochastic Complexity of Norm
We next show that our norm ∥ ψ ∥ + has low stochastic complexity.
Let X = { x ⟨ 1 ⟩ , . . .
, x ⟨ n ⟩} , andsuppose σ i are independent and identically distributed (i.i.d.)
Rademacher random variables (i.e.,σ i = ± 1 with probability 12 ) (Bartlett and Mendelson, 2002; Srebro et al., 2010).
The Rademachercomplexity for a set of functions H is R ( H ) =
E (sup h ∈H 1 n | (cid:80) n i =1 σ i · h ( x ⟨ i ⟩ ) | ) , and the worst case
Rademacher complexity of H is W ( H ) =
sup X E σ (sup h ∈H 1 n | (cid:80) ni
=1 σ i · h ( x ⟨ i ⟩ ) | ) .
These notionscan be used to measure the complexity of sets of matrices (Srebro and Shraibman, 2005) or tensors(Aswani, 2016).
The idea is to interpret each nonnegative tensor as a function ψ : R → R + from aset of indices x ∈ R to the corresponding entry of the tensor ψ x .
This complexity notion is useful forthe completion problem because it can be directly translated into generalization bounds.
Proposition 3.3.
We have R ( C λ ) ≤ W ( C λ ) ≤ 2 λ (cid:112) ρ/n .
Proof.
First note that from their definitions we get R ( C λ ) ≤ W ( C λ ) .
Next define the set P λ
= {± ψ :ψ ∈ S λ } , and recall that C λ = conv( S λ ) by Proposition 2.1.
This means W ( C λ ) =
W ( S λ ) (Ledouxand Talagrand, 1991; Bartlett and Mendelson, 2002).
Next observe that
[Equation]
where in the second line we replaced the supremum with a maximum, since the set P λ is finite, andused the Finite Class Lemma (Massart, 2000) with r = max ψ ∈P λ (cid:113)(cid:80) ni
=1 ( ψ x ⟨ i ⟩ ) 2 ≤ λ √ n .
Thisinequality on r is due to the fact that P λ consists of tensors whose entries are from {− λ, 0 , λ } .
Thus
W ( C λ ) ≤ λ (cid:112) (2 log 2) · ( ρ + 1) /n .
The result follows by noting that log 2 · ( ρ + 1) ≤ 2 ρ .
Remark 3.4 .
Because ψ has π = O ( r p ) entries, the Rademacher complexity in a typical tensornorm (e.g., max and Frobenius norms) will be O ( (cid:112) π/n ) =
O ( (cid:112) r p /n ) .
However, the Rademachercomplexity in our norm ∥ ψ
∥
+ is O ( (cid:112) ρ/n ) =
O ( (cid:112) rp/n ) , which is exponentially smaller.
Tensor Completion
Suppose we have data ( x ⟨ i ⟩ , y ⟨ i ⟩
) ∈ R × R for i = 1 , . . . , n .
Let I =
{ i 1 , . .
, i u } ⊆
[ n ] be any setof points that specify all the unique x ⟨ i ⟩ for i = 1 , . . . , n , meaning the set U
= { x ⟨ i 1 ⟩ , . . .
, x ⟨ i u ⟩}does not have any repeated points and x ⟨ i ⟩ ∈ U for all i = 1 , . . . , n .
The nonnegative tensorcompletion problem using our norm ∥ ψ ∥ + is given by
[Equation]
We will study the statistical properties of the above estimate and describe the elements of algorithmto solve the above optimization problem.
The purpose of defining the set U is that it is used inconstructing the algorithm used to solve the above problem.
4.1 Statistical Guarantees
Though in the previous section we calculated a Rademacher complexity for nonnegative tensors in
C λ viewed as functions, here we use an alternative approach to derive generalization bounds.
Thereason is that generalization bounds using the Rademacher complexity are not tight here.
Our approach is based on the observation that the nonnegative tensor completion problem (8)using our norm ∥ ψ ∥ + is equivalent to a convex aggregation problem (Nemirovski, 2000; Tsybakov,2003; Lecué et al., 2013) for a finite set of functions.
In particular, by Proposition 2.1 we have{ ψ : ∥ ψ ∥ + ≤ λ } =
C λ = conv
( S λ ) .
The implication is we can directly apply existing results forconvex aggregation to provide a tight generalization bound for the solution of (8).
Proposition 4.1 ((Lecué et al., 2013)) .
Suppose | y | ≤ b almost surely.
Given any δ > 0 , withprobability at least 1 − 4 δ we have that
[Equation]
where c 0 is an absolute constant and
[Equation]
[Equation]
Remark 4.2 .
We make two comments.
First, note that ζ n = o ( (cid:112) ρ/n ) .
Second, in some regimes ζ ncan be considerably faster than the (cid:112) ρ/n rate.
Generalization bounds under specific noise models, such as an additive noise model, follow as acorollary to the above proposition combined with Proposition 2.4.
Corollary 4.3.
Suppose φ is a nonnegative tensor with rank + ( φ )
=
k and ∥ φ ∥ max ≤ µ .
If( x ⟨ i ⟩ , y ⟨ i ⟩ ) are independent and identically distributed with | y ⟨ i ⟩ − φ x ⟨ i ⟩ | ≤ e almost surely and 204 E y ⟨ i ⟩ = φ x ⟨ i ⟩ .
Then given any δ > 0 , with probability at least 1 − 4 δ we have
[Equation]
where ζ n is as in (10) and c 0 is an absolute constant.
4.2 Computational Complexity
Though (8) is a convex optimization problem, our next result shows that solving it is NP-hard.
Proposition 4.4.
It is NP-hard to solve the nonnegative tensor completion problem (8) to an arbitraryaccuracy.
Proof.
Define the ball of radius δ > 0 centered at a nonnegative tensor ψ to be B ( ψ, δ )
= { φ : ∥ φ −ψ ∥ F ≤ δ } .
Next define W ( C 1 , δ ) =
(cid:83) ψ ∈C 1 B ( ψ, δ ) and W ( C 1 , − δ )
= { ψ ∈ C 1 : B ( ψ, δ ) ⊆
C 1 } .
The weak membership problem for C 1 is that given a nonnegative tensor ψ and a δ > 0 decidewhether ψ ∈ W ( C 1 , δ ) or ψ / ∈ W ( C 1 , − δ ) .
Theorem 10 of (Friedland and Lim, 2016) shows thatapproximation of the norm ∥ ·
∥ + is polynomial-time reducible to the weak membership problem for
C 1 .
Since Proposition 3.1 shows that approximation of the norm ∥ ·
∥ + is NP-hard, the result followsif we can reduce the weak membership problem to (8).
Suppose we are given inputs ψ and δ for the weak membership problem.
Choose x ⟨ i ⟩ for i = 1 , . . .
, πsuch that each element in R is enumerated exactly once.
Next choose y ⟨ i ⟩ = ψ x ⟨ i ⟩ .
Finally, note ifwe solve (8) and the minimum objective value is less than or equal to δ , then we have ψ ∈ W ( C 1 , δ ) ;otherwise we have ψ / ∈ W ( C 1 , − δ ) .
The result now follows since this was the desired reduction.
We note that the decision version of (8), that is ascertaining whether a given tensor ψ attains anobjective less than a given value while also satisfying ∥ ψ ∥ + ≤ λ , is NP-complete.
This followsdirectly from Corollary 3.2 and Proposition 4.4.
4.3 Numerical Computation
Although it is NP-hard, there is substantial structure that enables efficient numerical computationof global minima of (8).
The key observation is that C 1 is a 0-1 polytope, which implies thelinear separation problem on this polytope can be solved using integer linear optimization.
Integeroptimization has well-established global algorithms that are guaranteed to solve the separationproblem for this polytope.
This is a critical feature that enables the use of the Frank-Wolfe algorithmor one of its variants to solve (8) to a desired numerical tolerance.
In fact, the Frank-Wolfe variantknown as Blended Conditional Gradients (BCG) (Braun et al., 2019) is particularly well-suited forcalculating a solution to (8) for the following reasons:
The first reason is that our problem has structure such that the BCG algorithm will terminate(within numerical tolerance) in a linear number of oracle steps.
A sufficient condition for this linearconvergence is if the feasible set is a polytope and the objective function is strictly convex over thefeasible set.
For our problem (8), the objective function can be made strictly convex over the feasibleset by an equivalent reformulation.
Specifically, we use the equivalent reformulation in which wechange the feasible set from { ψ :
∥ ψ ∥ + ≤ λ }
= C λ to Proj
U
( C λ ) where the projection is done overthe unique indices specified by the set U .
In fact, this projection is trivial to implement because itsimply consists of discarding the entries of ψ that are not observed by the x ⟨ i ⟩ indices.
The second is that the BCG algorithm uses weak linear separation, which accommodates earlytermination of the associated integer linear optimization.
Integer optimization software tends to findoptimal or near-optimal solutions considerably faster than certifying the optimality of such solutions.
Furthermore, a variety of tuning parameters (Berthold et al., 2018) can be used to accelerate the searchfor good primal solutions when exact solutions are not needed.
We also deploy a fast alternatingmaximization heuristic in order to avoid integer optimization oracle calls where possible.
Thus,early-termination allows us to deploy a globally convergent algorithm with practical solution times.
Hence we use the BCG algorithm to compute global minima of (8) to arbitrary numerical tolerance.
For brevity we omit a full description of the algorithm, and instead focus on the separation oracle, which is the main component specifically adapted to our application.
The oracle is described in
Algorithm 1, where ⟨· , ·⟩ is the dot product for tensors viewed as vectors; for notational simplicitywe state the oracle in terms of the original space, ignoring projection onto U .
Output condition (1)provides separation with some vertex φ , while (2) requires certification that no such vertex exists.
In implementing the weak separation oracle, we first attempt separation with our alternating maximization procedure.
Described as Algorithm 2, it involves the following objective function:
[Equation]
The separation problem is thus treated as a multilinear binary optimization problem, and the algorithmsuccessively minimizes in each dimension.
We apply this heuristic a fixed number of times, randomlycomplementing entries in the incumbent each time.
The procedure runs in polynomial-time, andis not guaranteed to separate nor can it certify that such separation is impossible.
However, in ourexperiments it succeeds often, offering substantial practical speedups.
If alternating maximization fails to separate, then we solve the integer programming problem:
[Equation]
Note that early termination is deployed: we stop whenever an incumbent solution is found such thatthe objective is greater than Φ /K .
If no such solution exists, then the integer programming solver isguaranteed to (eventually) establish a dual bound z U such that ⟨ c, ψ − φ ⟩ ≤ z U ≤ Φ .
Numerical Experiments
Here we present results that show the efficacy and scalability of our algorithm for nonnegative tensorcompletion.
Our experiments were conducted on a laptop computer with 8GB of RAM and an Intel
Core i5 2.3Ghz processor with 2-cores/4-threads.
The algorithms were coded in Python 3.
We used
[Figure] [Figure]
[Figure] [Figure]
Gurobi v9.1 (Gurobi Optimization, LLC, 2021) to solve the integer programs (13).
As a benchmark,we use alternating least squares (ALS) – which is often called a “workhorse” for numerical tensorproblems (Kolda and Bader, 2009) – as well as two state-of-the-art methods implemented by the
PyTen package (Song et al., 2019), namely the simple low rank tensor completion (SiLRTC) algorithm(Liu
et al., 2012) and the trace norm regularized CP decomposition (TNCP) algorithm.
The PyTenpackage is available from https://github.com/datamllab/pyten under a GPL 2 license.
To minimize the impact of hyperparameter selection in our numerical results, we provided the groundtruth values when possible.
For instance, in our nonnegative tensor completion formulation (8) wechose λ to be the smallest value for which we could certify that ∥ ψ ∥ + ≤ λ for the true tensor ψ .
Thiswas accomplished by construction of the true tensor ψ .
For ALS and TNCP, we used a nonnegativerank k that was the smallest value for which we could certify that rank + ( ψ ) ≤ k .
This was alsoaccomplished by construction of the true tensor ψ .
A last note is that ALS often works better whenused with L2 regularization (Navasca et al., 2008).
The hyperparameter for the L2 regularization for
ALS was chosen in a way favorable to ALS in order to maximize its accuracy.
5.1 Experiments with Third-Order Tensors
Our first set of results concerns tensors of order p = 3 with increasing dimensions.
In each experiment,the true tensor ψ was constructed by randomly choosing 10 points from S 1 and then taking a randomconvex combination.
This construction ensures ∥ ψ ∥ + ≤ 1 and rank + ( ψ ) ≤ 10 .
We used n = 500samples (with indices sampled with replacement).
Each experiment was repeated 100 times, and theresults are shown in Figure 1.
A table of these values and their standard error is found in the Appendix.
We measured accuracy using normalized mean squared error (NMSE) ∥ (cid:98) ψ − ψ ∥ 2 F / ∥ ψ ∥ 2 F , which is amore stringent measure than used in Corollary 4.3 because the statistical theoretical result does notinclude normalization.
The results show that our approach provides modestly higher accuracy butrequires more computation time.
However, computation time remains on the order of seconds for thevarious tensor sizes.
We note that the NMSE value of approximately 0 .
48 that TNCP converges to isthe value achieved by a tensor that is identically the average of y ⟨ i ⟩ in all its entries.
5.2 Experiments with Increasing Tensor Order
Our second set of results concerns tensors with increasing order p , where each dimension takes thevalue r i = 10 for i = 1 , . .
p .
In each experiment, the true tensor was constructed by the method in
Sec. 5.1.
We used n = 10 , 000 samples (with indices sampled with replacement).
Each experimentwas repeated 100 times, and the results are shown in Figure 2 with the full values and standard errorgiven in the Appendix.
SiLRTC and TNCP were unable to run for tensors with 10 8 entries.
Theresults show that our approach provides much higher accuracy but requires more computation time.
However, computation remains on the order of minutes even for the largest tensor with 10 8 entries.
5.3 Experiments with Increasing Sample Size
Our third and fourth set of results concerns tensors of size 10 × 6 and 10 × 7 , respectively, whereexperiments are conducted with increasing sample size, given in the table as percentage of totalentries.
The true tensor ψ was constructed as in Sec. 5.1.
We begin with sample percentage 0.01%(with indices sampled with replacement), and extend each set of experiments’ sample size by oneorder of magnitude.
Each experiment was repeated 100 times, and the results are shown in Figures 3and 4 with the full values and standard error given in the Appendix.
Again, the numerical resultsshow that our approach provides substantially higher accuracy but requires more computation time.
The computation time remains on the order of minutes for most of the sampling schemes, exceptingthe draw of 10 6 entries for a tensor with 10 7 entries (i.e., about 1.5 hours).
Conclusion
We defined a new norm for nonnegative tensors and used it to develop an algorithm for nonnegative tensor completion that provably converges in a linear number of oracle steps and meets theinformation-theoretic sample complexity rate.
Its efficacy and scalability were demonstrated usingexperiments.
The next step is to generalize this approach to all tensors.
In fact, our norm definitions,optimization formulations, algorithm design, and theoretical analysis all extend to general tensors.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This isfoundational research and not tied to particular applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Supplemental Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec. 5.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec.
5 for acitation to the PyTen package.
(b) Did you mention the license of the assets?
[Yes] See Sec.
5 for the license of the PyTen
package.(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
There is new code in the Supplemental Material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
[Table] [Table]
[Table] [Table]