Optimal Query Complexities for Dynamic Trace Estimation
Abstract
We consider the problem of minimizing the number of matrix-vector queries needed for accurate trace estimation in the dynamic setting where our underlying matrix is changing slowly, such as during an optimization process.
Specifically, for any m matrices A 1 , ..., A m with consecutive differences bounded in
Schatten- 1 norm by ↵ , we provide a novel binary tree summation procedure that simultaneously estimates all m traces up to " error with  failure probability with an optimal query complexity of e
O ( m ↵ p log(1 /  ) / " + m log(1 /  ))
, improving the dependence on both ↵ and  from Dharangutte and Musco (NeurIPS, 2021).
Our procedure works without additional norm bounds on A i and can be generalized to a bound for the p -th Schatten norm for p 2 [1 , 2] , giving a complexity of e
O ( m ↵ p log(1 /  ) / " ) p + m log(1 /  )) .
By using novel reductions to communication complexity and information-theoretic analyses of Gaussian matrices, we provide matching lower bounds for static and dynamic trace estimation in all relevant parameters, including the failure probability.
Our lower bounds (1) give the ﬁrst tight bounds for Hutchinson’s estimator in the matrix-vector product model with Frobenius norm error even in the static setting , and (2) are the ﬁrst unconditional lower bounds for dynamic trace estimation, resolving open questions of prior work.
Introduction
Implicit matrix trace estimation is ubiquitous in numerical linear algebra and arises naturally in a wide range of applications, see, e.g., [25].
In this problem, we are given an oracle which gives us matrix-vector products A x 1 , A x 2 , · · · , A x m for an unknown n ⇥ n square matrix A and queries x 1 , . . .
, x m of our choice, that may be chosen adaptively.
In typical applications, one cannot afford to compute the diagonal entries of A explicitly, due to A being implicitly represented and computational constraints.
The goal is to efﬁciently estimate Tr A using only matrix-vector products.
In machine learning and data science, applications of trace estimation include training Gaussian
Processes [8, 11], triangle counting [1], computing the Estrada Index [10, 9], and studying optimization landscapes of deep neural networks from Hessian matrices [12, 26].
In these applications, it is common that A is represented implicitly due to its large memory footprint.
For example, while it is possible to compute Hessian-vector products via Pearlmutter’s trick [20], it is prohibitive to compute or store the Hessian matrix H , see, e.g., [12].
Moreover, A may be a matrix function f of another matrix B in some applications.
Since computing f ( B ) is expensive, it is desirable to apply implicit trace estimation.
For example, during the training of Gaussian Processes, the marginal log-likelihood contains a heavy-computation term, i.e., the log of the determinant of the covariance matrix, log(det( K )) , where K 2 R n ⇥ n and n is the number of data points.
The canonical way of computing log(det( K )) is via a Cholesky factorization on
K , which takes O ( n 3 ) time.
Instead, implicit trace estimation methods provide fast algorithms for 36th on Neural Information Processing Systems (NeurIPS 2022). approximating log(det( K ))
=
[Equation]
ni =1 log(  i ) =
tr(log( K )) on large-scale data.
Therefore, it is important to understand the fundamental limits of implicit trace estimation as the query complexity , i.e., the minimum number of matrix-vector multiplications required to achieve a desired accuracy and success rate.
Static trace estimation and Hutchinson’s method.
On the algorithmic side, Hutchinson’s method [14] is a simple and widely used method for trace estimation.
Let Q =
[ q 1 , . . .
, q ` ] 2 R n ⇥ ` be ` vectors with i.i.d.
standard Gaussian or Rademacher random variables.
Given matrix-vector multiplication access to A , Hutchinson’s method estimates tr( A ) by t = 1 q
[Equation]
q tr( Q T AQ ) .
It is known [2] that the estimator satisﬁes that for any " ,
[Equation]
[Equation]
[Equation]
provided the number ` of queries satisﬁes `  C log(1 /  ) / " 2 for some ﬁxed constant C .
For Hutchinson’s method, there is also previous work which showed for queries of the form x > A x , ⌦ (1 / " 2 ) queries are required [21]; however, this does not imply even a lower bound for non-adaptive algorithms that use matrix-vector queries.
Though stronger algorithmic results and matching lower bounds are known for the important case of PSD matrices in the non-adaptive setting [18, 16], the optimality of Hutchinson’s estimator as an trace estimator for general square matrices in the matrixvector product model still remains an open problem.
Notably, Hutchinson’s method chooses the query vectors non-adaptively and it is furthermore unclear whether adaptivity could help.
More generally, there has been a ﬂurry of recent work that gives trace estimators with o (1 / " 2 ) query complexity but with a different error guarantee.
Speciﬁcally, let us consider a Schatten- p norm error guarantee, where the goal is to provide an estimate t such that
[Equation]
where k A k p denotes the Schatten- p norm.
For p = 1 , a previous work [18] proposes a variance-reduced version of Hutchinson’s method that uses only O (1 / " ) matrix-vector product queries to achieve a nuclear norm error of " k A k
⇤ , in contrast to the O (1 / " 2 ) queries used when the error is in the Frobenius norm.
When the matrix is positive semideﬁnite (PSD), the nuclear norm error is equivalent to a (1 + " ) multiplicative approximation to the trace.
Their work, along with a subsequent work [16], shows that ⌦ (1 / " ) queries are therefore sufﬁcient and necessary to achieve a (1 + " ) multiplicative trace approximation in this setting.
While this line of work mainly focuses on PSD matrices and nuclear norm error, we consider trace estimation on general square matrices with Schatten- p norm error for any p 2 [1 , 2] .
Furthermore, we note that the variance-reduced Hutchinson’s method splits the queries between approximating the top O (1 / " ) eigenvalues, i.e., by computing a rank- O (1 / " ) approximation to
A , and performing Hutchinsons’s method on the remainder.
Due to the low rank approximation subroutine, the query complexity’s dependence on the failure probability is more concretely
O ( p log(1 /  ) / "
+ log(1 /  )) for additive " k A k ⇤ error.
The additive log(1 /  ) rate is shown to be necessary when non-adaptive queries are used, but it is an open problem whether adaptive queries can remove the additive log(1 /  ) term for trace estimation with Schatten- p norm error [16].
This motivates the natural question:
Question 1:
Is Hutchinson’s method optimal in terms of " and  for static trace estimation of general square matrices, even when adaptivity is allowed?
How do we generalize Hutchinson’s method for error in general Schatten-
p norms?
Dynamic trace estimation.
In various applications the input matrix is not ﬁxed.
For example, during model training, we need to estimate the trace of a dynamically changing Hessian matrix with respect to some loss function.
One may assume that the change at each step is not very large.
Motivated by such a scenario, a recent work by Dharangutte and Musco [6] studies dynamic trace estimation.
Formally, let p 2 [1 , 2] and A 1 , A 2 , · · · , A m be n ⇥ n matrices in a stream such that (1)
k
A i k p  1 for all i > 1 , where k · k p denotes the Schatten- p norm, and (2) k
A i +
A
i k p  ↵ < 1 for all i  m  1 .
The goal is to output a sequence of estimates t 1 , · · · , t m such that for each
i 2 [ m ] ,
[Equation]
probability
[Equation]
[Table]
Query Complexity
Matrix Type
Failure
[Table]
Algorithm Type [2, 22]
O (log(1 /  ) / " 2 ) general square  non-adaptive, p =[18]
O ( p log(1 /  ) / "
+ log(1 /  ))
PSD  adaptive, p = 1 [18]
O (log(1 /  ) / " )
PSD  non-adaptive, p =[16]
O ( p log(1 /  ) / "
+ log(1 /  ))
PSD  non-adaptive, p =
This work
O (( p log(1 /  ) / " ) p + log(1 /  ))
[Table]
[Table] [Table]
p
Lower Bounds
[Table]
Adaptive) [18] ⌦ (1 / ( b + " log(1 / " ))) general square, bit constant adaptive, p =
This work⌦ ⇣ 1 " p ( b +log(1 / " ))
+ log(1 /  ) ( b +loglog(1 /  ))
⌘
[Table]
adaptive, general p
This work⌦ ⇣
( p log(1 /  ) / " ) p ⌘
[Table]
[Table] [Table]
) [18] ⌦ (1 / " )
[Table] [Table]
=
[Table]
[16] ⌦ ( p log(1 /  ) / "
+ log(1 /  )
loglog(1 /  ) )
PSD, ram  non-adaptive, p =
This work 4 ⌦ ⇣ log p/ 2 (1 /  ) / ( " p ( b + log(1 / " )))
⌘ general square, bit  non-adaptive, general p
This work⌦ ⇣
( p log(1 /  ) / " ) p + log(1 /  )
loglog(1 /  ) ⌘ general square, ram  non-adaptive, general p
and lower bounds on the query complexity for static trace estimation.
In the bit complexity model, each entry of the query vector is speciﬁed by b bits, and the dependence on b is necessary.
1 : A static upper bound generalizing Hutch++ [18] to Schatten-
p norm error (Theorem C.1).
: An adaptive lower bound via communication complexity of the Gap Equality and Approximate Orthogonality problem (Theorem 4.1), which combines Theorem D.2 and Theorem D.3, resolving an open problem that log(1 /  ) queries are required in the adaptive setting.:
An adaptive lower bound via information-theoretic analysis of Gaussian Wigner matrices (Theorem 4.2), showing optimal dependence on log(1 /  ) .
: A non-adaptive lower bound via communication complexity of Augmented Indexing (Theorem F.1), optimal in all parameters up to the bit complexity term.:
A non-adaptive lower bound combining our Theorem 4.2 and the prior result from [16].
via matrix-vector multiplication query access to the ﬁrst i matrices ( A j ) ij =1 .
Naïvely, one could estimate each Tr A i independently using Hutchinson’s method.
This, however, does not exploit that the changes are bounded at each step.
Alternatively, one can rewrite Tr A i as Tr A 1 +
P ii =
Tr(  i ) , where  i =
A i  A i  1 , by linearity of the trace, and apply Hutchinson’s method on each term.
Unfortunately, this scheme suffers from an accumulation of errors over the steps.
The prior work [6] is focused on p = { 1 , 2 } and improves upon the naïve ideas above.
For p = 1 , the authors give a method that uses O ⇣ m p ↵ /  / " + p 1 /  / " ⌘ queries.
For p = 2 , they provide an algorithm with query complexity O ( m ↵ log(1 /  ) / " 2 +log(1 /  ) / " 2 ) and a conditional lower bound showing that this is tight.
This leaves open the question:
Question 2: Can we design improved algorithms for dynamic trace estimation under a general Schatten norm assumption?
Can we prove an unconditionally optimal lower bound?
1.1 Our Results
Our work resolves the proposed questions (nearly) optimally, and we next discuss our main results.
Static trace estimation: For Question 1, we prove query complexity lower bounds for implicit trace estimation in both bit complexity and real RAM models of computation, resolving the open problem of establishing unconditional lower bounds for the optimality of Hutchinson’s method even in the adaptive setting.
To do so, we provide new reductions from classic communication complexity problems, including G AP -E QUALITY and A PPROXIMATE -O RTHOGONALITY , to matrix trace estimation.
Our main lower bounds demonstrate that log(1 /  ) queries are always needed even with adaptivity and for general p , there is an additional 1 / " p dependence.
A key idea is a communication protocol simulation using the product of two matrices rather than the sum, as was used in prior work on PSD lower bounds [18].
Theorem 1.1 (Informal; see Theorem 4.1) .
In the bit complexity model, where each entry of each query vector is speciﬁed using b bits,
[Equation]
number of adaptive queries is necessary to achieve " k A k p error with probability at least 1   .
When adaptivity is not allowed, we give a stronger lower bound (Theorem F.1) of ⌦ (log p/ 2 (1 /  ) / " p ) .
This matches the guarantee of Hutchinson’s non-adaptive estimator up to a constant factor, for which random sign vectors sufﬁce and so one can take b
= O (1) .
We also provide a query complexity lower bound in the real RAM model for general Schatten pnorms with p 2 [1 , 2] by using Gaussian ensembles and controlling the remaining entropy of the distribution conditioned on prior queries.
In the special case of p = 2 (i.e., Frobenius norm error guarantee), our bound again matches the classic Hutchinson’s method up to a constant factor for p = 2 , and an additive log(1 /  ) factor for p < 2 .
Note that in the non-adaptive setting, our lower bound in the RAM model can also be improved for p < 2 to include a log(1 /  ) / log log(1 /  ) factor.
Therefore, this lower bound emphasizes that our dependence on log(1 /  ) in the " -dependent term is tight, even in the adaptive setting.
Theorem 1.2 (Informal; see Theorem 4.2) .
In the real RAM model, where the queries are realvalued, for sufﬁciently small " and any p 2 [1 , 2] , ⌦ [Equation] of adaptive queries is necessary to achieve " k A k p error with probability at least 1   .
On the algorithmic front, we give a matching upper bound for static trace estimation for general
Schatten-
p norm error for p 2 [1 , 2] .
The argument requires a careful balancing of the " and  parameters in the low rank approximation of the Hutch++ procedure from [18].
See Theorem C. for a full statement.
Dynamic trace estimation: To answer Question 2, we ﬁrst give an improved algorithm for dynamic trace estimation that uses a binary tree-based decomposition to estimate all matrix traces with only a small logarithmic overhead.
The algorithm improves upon the previous work [6] and gets an optimal dependence on 0 < ↵ ,  < 1 , up to logarithmic factors.
Speciﬁcally, for p = 1 , the prior work gives a method that uses O [Equation] queries for small " , while our algorithm gives an improved O ( m ↵ p log(1 /  ) / " ) bound with a linear dependence on ↵ and square root dependence on log(1 /  ) .
For p = 2 , our algorithm matches the query complexity of O ( m ↵ log(1 /  ) / " 2 ) given by previous work.
Furthermore, our algorithm works under a general Schatten p -norm assumption for any p 2 [1 , 2] :
Theorem 1.3 (Informal; see Theorem 3.1 and Theorem C.2) .
For any p 2 [1 , 2] , there is a dynamic trace estimation algorithm that achieves error " and failure rate  at each step.
The algorithm uses a total of
[Equation]
matrix-vector product queries.
Furthermore, for p = 1 , it can be improved to
[Equation]
Furthermore, since our algorithm avoids the variance reduction technique from [6], we may relax the assumptions of dynamic trace estimation and require only the ﬁrst matrix to have norm k A 1 k  1 , instead of asking the entire sequence A i to be bounded in such a way.
While the norm bound on all A i is crucial for the algorithm in [6] (rerunning the analysis naïvely would give a worse query complexity of O ( m 3 ↵ 3 / " ) ), our tree-based algorithm achieves a nearly optimal query complexity even when the norm of A i grows, and we suffer only a log m overhead in that case.
Moreover, in our experiments, we ﬁnd that our algorithm signiﬁcantly outperforms previous algorithms on real and synthetic datasets.
See Section 6 for our experimental results.
To complement our algorithms, we give unconditional lower bounds showing that our algorithm is nearly optimal.
Our lower bounds rely on a reduction from dynamic trace estimation to static matrix trace estimation from [6] and make use of our new lower bounds in the static setting.
In particular, the reduction shows that if for a ﬁxed set of parameters " ,  , p , a static trace estimation scheme requires ⌦ ( r ) queries, then ⌦ ( m ↵ r ) queries are necessary for any dynamic algorithm.
Combining this observation with our static trace estimation lower bounds, we get:
Theorem 1.4 (Informal; see Theorem 5.2 and Theorem 5.3) .
For any p = [1 , 2) , our algorithm attains the optimal query complexity, up to bit complexity and logarithmic terms.
More speciﬁcally, we prove lower bounds that match the ﬁrst term in our upper bound (4) for all p 2 [1 , 2] .
For p = 1 , we give a lower bound (Theorem 5.4) matching the the second term in (5) as well, showing that the m (log(1 /  )) additive dependence is necessary.
For p = 2 , the prior work [6] gives a upper bound of O ( m ↵ log(1 /  ) / " 2 + log(1 /  ) / " 2 ) .
Our lower bounds are unconditional and show that the ﬁrst term is tight.
Moreover, the second term is necessary due to the static lower bound when m = 1 .
This result is not contradicted by the claim of
Theorem 5.4.
In particular, when ↵  " 2 , Theorem 5.4 is weaker than the ⌦ ( m ↵ log(1 /  ) / " 2 ) lower bound; and when ↵ < " 2 , the construction by itself requires " / ↵ update steps to change the trace by " , which leads to a lower bound of ⌦ ( m ↵ log(1 /  ) / " ) , again weaker than ⌦ ( m ↵ log(1 /  ) / " 2 ) .
Preliminaries
A matrix A 2 R n ⇥ n is symmetric positive semi-deﬁnite (PSD) if it is real, symmetric and has nonnegative eigenvalues.
Hence, x > Ax  0 for all x 2 R n .
Let tr( A ) =
P ni =
A ii denote the trace of
A .
Let k A k F = (
P ni =
P nj =
A 2 ij ) 1 / 2 denote the Frobenius norm and k A k op =
sup k v k 2 =1 k
A v kdenote the operator norm of A .
We let k A k p =
P i  pi ) 1 /p be the Schatten- p norm, where  i are the singular values of A .
Two special cases are the Frobenius norm, which equals the Schattennorm ( k A k F =
k A k 2 ) and the nuclear norm, equals the Schatten- 1 norm ( k A k ?
=
k
A k 1 ) .
Algorithm for Dynamic Trace Estimation
We give an algorithm for dynamic trace estimation under a general Schatten- p norm assumption, for p 2 [1 , 2] .
For p = 1 , our algorithm provides an improved guarantee upon the DeltaShift++ procedure from [6].
In a later section we complement the result by showing that it is indeed nearoptimal.
Speciﬁcally, we give an algorithm that achieves the following guarantees:
Theorem 3.1 (Improved dynamic trace estimation) .
Let A 1 , A 2 , · · · , A m be n ⇥ n matrices such that (1) k
A i k ?
 1 for all i , and (2) k
A i +1  A
i k ?
 ↵ for all i  m  1 .
Given matrix-vector multiplication access to the matrices, a failure rate  > 0 and error bound " , there is an algorithm that outputs a sequence of estimates t 1 , · · · , t m such that for each
i 2 [ m ] , |
[Equation]
i
[Equation]
i |  " , with probability at least 1   .
The algorithm uses a total of O
[Equation]
matrix-vector multiplication queries to A 1 , A 2 , · · · , A m .
Compared with DeltaShift++ in [6], this guarantee provides an exponential improvement in  and a polynomial improvement in ↵ for p 6 = 2 , while maintaining the optimal dependence on m and " .
3.1 Algorithm
We now describe our algorithm.
The ﬁrst idea is to partition the m updates into groups of size s =
d 1 / (2 ↵ ) e .
Each group will be treated independently, and we will use
[Equation]
queries on each group.
This leads to our claimed query complexity, as there are O ( m ↵ ) groups.
Note that if ↵ < " , since | Tr( A j  A j  1 ) |  k A j  A j  1 k ⇤  ↵ , the trace can change by at most an additive ↵ , so we can simply ignore every subsequence of length " / ↵ .
Therefore, we only need to apply our estimators to m ↵ / " matrices.
Without loss of generality, consider a group of matrices A 1 , · · · , A 1 / 2 ↵ .
As the ﬁrst step, we estimate Tr( A j  A j  1 ) for each j  2 by using the Hutch++ static trace estimator [18] as a black box.
Then, for each even integer j = 2 k (for an integer 2  k  s/ 2 ), we also estimate
Tr( A 2 k  A 2( k  1) ) in the same way.
More generally, for each integer j = 2 ` k , for 0  ` < log 2 s , we use Hutch++ to approximate Tr
[Equation]
.
We view this scheme as a binary tree: the bottom level consists of leaves corresponding to the trace difference of neighboring matrices, and nodes at level ` correspond to the trace difference of matrices that are 2 ` apart in their indices.
To output an estimate of Tr A i , we will write i in its binary representation and approximate it by
Tr( A 1 ) plus a sequence of O (log(1 / ↵ )) differences, at most one for each level in the binary tree.
By setting the success rates and errors bounds at each level carefully, we can achieve the desired error guarantee of Equation (6).
To formalize the construction, we ﬁrst cite the following guarantee of the Hutch++ algorithm:
Lemma 3.2 (Hutch++, nuclear norm, Theorem 5 of [18]) .
The Hutch++ estimator uses
[Equation]
⌘ matrix-vector multiplication queries such that given any square matrix A and parameters " 0 ,  0 , with probability at least 1   0 , the algorithm’s output t satisﬁes
[Equation]
Let Hutch++ ( A , " 0 ,  0 ) denote the output of Hutch++ on matrix A with parameters " 0 ,  0 .
It will be invoked with different parameters at different levels of the binary tree construction.
A description of the algorithm is given by the pseudocode Algorithm 1, with a helper function Algorithm 2.
For simplicity of analysis, note that since we can add dummy matrices (say, extra copies of A 1 ), we assume that each group has size exactly s = d 1 / (2 ↵ ) e and s is a power of two.
This blows up the total number of matrices by at most a constant factor.
,
3.2 Analysis
The analysis of the algorithm is rather lengthy and is delayed to Appendix C.1.
In addition, we give a general analysis of the algorithm under Schatten-
p norm assumption and the speciﬁc improved bounds for p = 1 in Appendix C.2 and show how to relax the bounded norm assumption in
Appendix C.3.
Lower Bounds for Adaptive Trace Estimation
In this section, we provide (nearly) optimal lower bounds for trace estimation with adaptive matrixvector multiplication queries, under general square matrices and Schatten- p norm error.
4.1 Adaptive Lower Bound, Bit Complexity
First, we show two separate lower bounds under bit complexity model, both proven via reductions from communication complexity problems.
One shows an ⌦ (1 / " p ) lower bound (Theorem D.2) and the other ⌦ (log(1 /  ))
(Theorem D.3), up to bit complexity terms.
Combined together, they yield:
Theorem 4.1 (Adaptive query lower bound, bit complexity) .
Any algorithm that accesses a square matrix A via matrix-vector multiplication queries requires at least
[Equation]
queries to output an estimate t such that with probability at least 1   , | t
Tr
A |  " k A k p , for any p 2 [1 , 2] , where the query vectors may be adaptively chosen with entries speciﬁed by k bits.
The proofs of the theorems can be found in Appendix D.1.
4.2 Adaptive Lower Bound, RAM
Next, we prove a tight lower bound under the real RAM model (Theorem 4.2).
The bounds hold for any Schatten- p norm error.
Our proof is via information-theoretic analysis of random Gaussian matrices and is delayed to Appendix D.2.
Theorem 4.2 (Lower Bound for Any Schatten Norm) .
For all p 2 [1 , 2] ,  > 0 and 0 < " <
(log(1 /  )) 1 / 2  1 /p , any algorithm that takes in any input matrix A and succeeds with probability at least 1   in outputting an estimate t such that | t  tr( A ) |  " k A k p requires m = ⌦ p log(1 /  ) " !
p !
matrix-vector multiplication queries.
Lower Bounds for Dynamic Trace Estimation
Using the query complexity lower bounds for adaptive trace estimation, we can now prove tight lower bounds for dynamic trace estimation.
The recent work of Dharangutte and Musco [6] only provides a conditional lower bound, assuming that Hutchinson’s scheme is optimal.
We remove this assumption and make the lower bound unconditional.
We additionally prove a lower bound by constructing an explicit hard instance in the dynamic setting.
Our lower bounds hold under a general
Shatten norm assumption and nearly matches the guarantee of our algorithm.
5.1 Lower Bounds via Static-to-Dynamic Reduction
We ﬁrst show a lower bound for dynamic trace estimation under a Frobenius norm assumption.
This immediately implies that the DeltShift algorithm due to [6] is optimal for p = 2 .
First, we cite a static-to-dynamic reduction from [6] and its implication.
The reduction shows how to solve a static instance using a dynamic trace estimation scheme, and therefore any hardness on the static problem translates to the dynamic setting as well.
It holds generally for an error bound in any Schatten norm.
For completeness, we give a proof in Appendix E.1.
Lemma 5.1 (Conditional lower bound for dynamic trace estimation [6]) .
Suppose that any algorithm that achieves Equation (2) for static trace estimation must use ⌦ ( r ) matrix-vector product queries.
Then any dynamic trace estimation algorithm requires ⌦ ( r ↵ m ) matrix-vector product queries under a general Schatten- p norm assumption, when ↵ = 1 / ( m  1) .
It follows immediately from this lemma and our adaptive query lower bound (Theorem 4.1):
Theorem 5.2 (Unconditional lower bound for dynamic trace estimation, bit) .
For all p 2 [1 , 2] and " ,  2 (0 , 1) , any algorithm for dynamic trace estimation under a Schatten- p norm assumption must use at least
[Equation]
matrix-vector multiplication queries, where each entry of the query vectors is speciﬁed by k bits.
Combining the same reduction (Theorem D.6) with our previous real RAM lower bound (Theorem 4.2) in the static setting gives:
Theorem 5.3 (Unconditional lower bound for dynamic trace estimation, RAM) .
For all p 2 [1 , 2] ,  > 0 and 0 < " <
(log(1 /  )) 1 / 2  1 /p , any algorithm for dynamic trace estimation under a
Schatten-
p norm assumption must use at least ⌦ [Equation] ⌘ p ⌘ matrix-vector multiplication queries.
5.2 Lower Bound via Explicit Hard Instance
Using the hard instance based on G AP -E QUALITY in the static setting (from the proof of Theorem D.3), we give an explicit hardness construction against any dynamic trace estimation scheme.
This yields the following lower bound, and its proof is in Appendix E.2.
Theorem 5.4.
For all p 2 [1 , 2] and " ,  2 (0 , 1 / 4) , any algorithm for dynamic trace estimation under Schatten- p norm assumption must use at least
[Equation]
matrix-vector multiplication queries, where each entry of the query vectors is speciﬁed by k bits.
Experiments
We experimentally validate our algorithmic results.
We compare Algorithm 1, with the following procedures on both synthetic and real datasets.
More experimental details are in Appendix G.
• Hutchinson’s:
Apply the classic Hutchinson’s scheme for each Tr( A i ) independently.
• DiffSum: Approximate t 1 ⇡
Tr( A 1 ) and each neighboring difference d i ⇡
Tr( A i )  Tr( A i  1 ) using Hutchinson’s independently.
Then output t i =
t 1 +
[Equation]
2 d [Equation]
• DeltaShift: The main algorithm of [6].
The experiments from [6] demonstrate that
DeltaShift outperforms DiffSum and other Hutchinson-based schemes on various datasets.
Synthetic data.
We simulate a dynamic trace estimation instance by ﬁrst generating a (symmetric) random matrix A n ⇥ n and then adding random perturbations over T = 100 time steps.
The details and results are found in Appendix G.1.
[Figure] [Figure]
Counting triangles.
Our ﬁrst experiment on a real-world dataset is on counting triangles in dynamic undirected (simple) graphs.
Note that the number of triangles in a graph equals 1 6 Tr A 3 , where A is the adjacency matrix of the graph.
Thus, triangle counting reduces to trace estimation.
We use two arXiv collaboration networks with 5 , 242 and 9 , 877 nodes [17].
The nodes represent authors, and edges indicate co-authorships.
To simulate a real-world scenario, we add a random clique of size at most 6 to the graph in each step, indicating a group of researchers jointly publishing a paper.
We note that our algorithm signiﬁcantly outperforms other methods (Figure 1).
Neural network weight matrix.
We evaluate the performance of the algorithms on a sequence of weight matrices of a neural network, generated during the training process.
In particular, we choose a three-layer neural network with a hidden layer of 100 ⇥ 100 .
We train the network on the MNIST dataset via mini-batch SGD and consider the ﬁrst 1 , 000 steps, when the weights are changing most rapidly.
Our algorithm achieves much smaller error than DiffSum and DeltaShift (Figure 2). Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]