Paper: Towards Establishing Guaranteed Error for Learned Database Operations (ICLR-2024)

Reviewer comment (Evaluation):
Concerns about the problem setting -- the studied problem does not quite correspond to the learned DB operators. For the learned index, it is fine for the model to make errors as we can maintain some delta to help correct the error. After all, the training and testing data are the same. For the learned cardinality estimator, the absolute error is not what people are using for evaluation -- q-error is. This is because for a true cardinality of 10K, an error of 100 is more acceptable than an error of 10 for a true cardinality of 10.

Author response:
Regarding the error metric for learned indexing, we are unsure what the reviewer means by “maintaining delta”. The typical practice in learned indexing is to do binary (or exponential) search (Kraska et al., 2018; Ferragina & Vinciguerra, 2020; Ding et al., 2020) from the final estimate on the sorted array to find the exact answer, in which case, as R3Q3 mentions one may be interested in the $\log_2$ error of the model (which is the runtime of this binary search step). We have added a discussion to the newly added Appendix A showing how our bounds can be translated to log2 error, as discussed next. Ideally, to ensure $\log_2$ error is at most an error parameter $\tau$, setting $\epsilon=2^\tau$ in our bounds would provide the required model size given the $\log_2$ error $\tau$. This is true when considering worst-case error (i.e., Theorem 1) and in our upper-bounds on the required model size when considering average case error (i.e., Theorem 4, Lemma 2 and part (ii) of Theorems 2,3,5). That is, our bounds on the mentioned results simply translate to bounds for $\log_2$ error by replacing $\epsilon$ with $2^\tau$. However, this is not true in the case of lower bounds for average error, (i.e., Corollary 1 and part (i) of Theorems 2,3,5) and the required model size can be smaller than what the results would suggest if we set $\epsilon=2^\tau$. We leave study of the log2 error in these cases to future work, although we expect similar proof techniques that were used in this paper to be applicable to that case as well.


Regarding the error for cardinality estimation, although q-error is a useful metric for the evaluation of cardinality estimators, worst-case guarantees on absolute error can provide more meaningful guarantees on the performance of a system. For example, Negi et al. 2021 show that a cardinality estimator that achieves a worst-case q-error of 4100 can lead to a similar runtime as another method with a worst-case q-error of 100. Indeed a worst-case bound on q-error is a worse indicator of the system’s performance compared with a worst-case bound on absolute error. For instance, the q-error of 4100 can correspond to a query with true cardinality 4100 and estimated cardinality 1 (in which case one needs to read an extra 4100 items from disk, which will take little resources, both computation and memory wise) while the error can also correspond to an estimate of 1,000,000 for a query whose true cardinality is 4,100,000,000 (in which case one needs to read an extra 4 billion records, a huge computation and memory cost). On the other hand, the absolute error in those two cases will respectively be 4,399 and  4,099,000,000 a much better indicator of the extra computation caused by the error of the cardinality estimator. Overall, a worst-case bound on absolute error can readily translate to performance metrics (e.g., worst-case memory consumption and number of disk accesses), while worse-case q-error in itself is not informative. 

Finally, note that cardinality estimation is not just used for query optimization, and has use-cases in approximate query processing (similar to range-sum estimation), where the goal is to return to the user an estimated query answer (e.g., for the query of how many people are in an area), which is common in analytical applications. In such settings, the absolute error is useful as it provides a bound on what the true query answer can be, and is often directly shown to the user as error bars.

Reviewer comment (Writing):
It would be great to better articulate the problem settings (See W1).

Author response:
Regarding the error metric for learned indexing, we are unsure what the reviewer means by “maintaining delta”. The typical practice in learned indexing is to do binary (or exponential) search (Kraska et al., 2018; Ferragina & Vinciguerra, 2020; Ding et al., 2020) from the final estimate on the sorted array to find the exact answer, in which case, as R3Q3 mentions one may be interested in the $\log_2$ error of the model (which is the runtime of this binary search step). We have added a discussion to the newly added Appendix A showing how our bounds can be translated to log2 error, as discussed next. Ideally, to ensure $\log_2$ error is at most an error parameter $\tau$, setting $\epsilon=2^\tau$ in our bounds would provide the required model size given the $\log_2$ error $\tau$. This is true when considering worst-case error (i.e., Theorem 1) and in our upper-bounds on the required model size when considering average case error (i.e., Theorem 4, Lemma 2 and part (ii) of Theorems 2,3,5). That is, our bounds on the mentioned results simply translate to bounds for $\log_2$ error by replacing $\epsilon$ with $2^\tau$. However, this is not true in the case of lower bounds for average error, (i.e., Corollary 1 and part (i) of Theorems 2,3,5) and the required model size can be smaller than what the results would suggest if we set $\epsilon=2^\tau$. We leave study of the log2 error in these cases to future work, although we expect similar proof techniques that were used in this paper to be applicable to that case as well.


Regarding the error for cardinality estimation, although q-error is a useful metric for the evaluation of cardinality estimators, worst-case guarantees on absolute error can provide more meaningful guarantees on the performance of a system. For example, Negi et al. 2021 show that a cardinality estimator that achieves a worst-case q-error of 4100 can lead to a similar runtime as another method with a worst-case q-error of 100. Indeed a worst-case bound on q-error is a worse indicator of the system’s performance compared with a worst-case bound on absolute error. For instance, the q-error of 4100 can correspond to a query with true cardinality 4100 and estimated cardinality 1 (in which case one needs to read an extra 4100 items from disk, which will take little resources, both computation and memory wise) while the error can also correspond to an estimate of 1,000,000 for a query whose true cardinality is 4,100,000,000 (in which case one needs to read an extra 4 billion records, a huge computation and memory cost). On the other hand, the absolute error in those two cases will respectively be 4,399 and  4,099,000,000 a much better indicator of the extra computation caused by the error of the cardinality estimator. Overall, a worst-case bound on absolute error can readily translate to performance metrics (e.g., worst-case memory consumption and number of disk accesses), while worse-case q-error in itself is not informative. 

Finally, note that cardinality estimation is not just used for query optimization, and has use-cases in approximate query processing (similar to range-sum estimation), where the goal is to return to the user an estimated query answer (e.g., for the query of how many people are in an area), which is common in analytical applications. In such settings, the absolute error is useful as it provides a bound on what the true query answer can be, and is often directly shown to the user as error bars.

Reviewer comment (Theory):
It would be great to justify the usefulness of worst-case/average-case guarantees instead of instance-level guarantees (See W2).

Author response:
We would like to clarify that in the context of the instance-optimal framework of learned models, our bounds study the worst-case instances for such instance-optimal methods. That is, we study how high the error of a model can be on an instance, even if we optimize it for that instance. Such worst-case guarantees are useful when one needs to reason about the performance of a system without having seen the specific data instances it will be used on. Two examples of when this is useful are presented in our response to R4Q2. We present further details here.

Worst-case guarantees serve a different purpose than instance-level guarantees. Instance-level guarantees will have to depend on the properties of the instance (e.g., bounds in Zeighami & Shahabi, 2023 depend on the p.d.f of data distribution while bounds in Ferragina et al., 2020 depend on the distribution of gaps in an array). Such bounds can only be calculated after the instance is observed, and will need to be recomputed if an instance changes. This does not allow a system to provide apriori guarantees on the performance of the system, which worst-case guarantees provide. For instance, a cloud service provider needs to know how much resources it needs to allocate to learned models apriori (e.g., to calculate costs and ensure availability). Instance level guarantees do not help, since instances will change and so will the amount of resources needed. However, a worst-case guarantee allows the service provider to calculate the maximum amount of resources it needs, and therefore guarantee the reliability of the system. A similar example holds in terms of setting model size, where one may not be able to afford to redo hyperparameter tuning when the data instance changes, and thus setting model size to what is suitable across instances will be beneficial, which can be done through worst-case bounds.

Reviewer comment (Theory):
It would be great to show more implications of the theoretical results (see W3).

Author response:
Please see the newly added paragraph to the introduction and our response R2Q2 for a discussion on the implications of our theoretical results. Here, we describe in more detail two practical use cases enabled by our theoretical results. 

The first practical use case is to ensure that the model size used is at least what the bounds suggest. This ensures that, as data instances change or new data instances are observed, the model has enough capacity to answer queries and reduces the need for hyperparameter tuning.  For instance, while a linear model (which has a very small size) may be sufficient to answer queries for data from a uniform distribution, if the data changes one will need a more complex model (of larger size). By choosing a model size that guarantees a desired accuracy level across data instances (i.e., using our worst-case guarantees), one can avoid redoing hyperparameter tuning as the data instance changes.

Another practical use case of our results is in resource management for database systems, especially significant for cloud service providers that store and manage the learned models for a large number of database instances. Our bounds can tell such service providers how much resources they need to be able to support learned models in their systems with a desired accuracy level. This can be used in cost calculation, and to make sure enough resources are available to support a learned model across all instances (i.e., to provide reliability guarantees).

Reviewer comment (Theory):
What is the unique property of learned DB operators that are used during the proof? Putting it in another way, is this method applicable to arbitrary functions and not limited to learned DB operators?

Author response:
Our results only apply to database operations and not arbitrary functions. We study the properties of database functions to provide bounds on the required model size to represent database operations. Our proofs specifically study the properties of the three database operations in question (i.e., indexing, cardinality estimation and range-sum estimation). The main property of database operations used in our proofs is: if two databases differ in a set of points S, how different the answer to queries will be on those two databases? Our proofs utilize properties of the set S to bound the difference between query answers on different databases. Such bounds are query-dependent, and our proofs study this for the three different database operations (presented in technical Lemmas 3-8 in our appendix, which are one of our main technical contributions). The consequence of such a bound is that if a model (with a specific parameter setting, e.g., after training) achieves a small error on some dataset, then it will also achieve a small error on other datasets that don’t differ from it too much, but will achieve large error on datasets that do differ from it too much (the bound is used to rigorously quantify this statement). Thus, we can count how many different model parameter settings are needed to achieve small errors on all possible datasets, which directly gives a bound on the required model size (as we need to use enough bits to realize all the required different model parameter settings). Please also see the newly added section Appendix B1 and our response to R3Q2 for more intuition on the proofs.

Reviewer comment (Writing):
Q1) The main text of the paper is both overly formal and gives very little intuition about the actual proof. If the main text is just going to go over the results and implications, it might be better to drop the formal notation and give intuitive / illustrated examples of each problem and bound.

Author response:
R3Q1-2. We have added a new subsection to Appendix B, further discussing the intuition behind our proofs which we present here. As, the reviewer observed, given that there are $2^\sigma$ possible bitstrings (for model size $\sigma$), multiple datasets must be mapped to the same model parameter values (that is, after training, multiple datasets will have the exact same model parameter values). To obtain a bound on the error, the proof shows that some datasets that are mapped to the same parameter values will be too different from each other (i.e., queries on the datasets have answers that are too different), so that the exact same parameter setting cannot lead to answering queries accurately for both datasets. The proofs do this by constructing $2^\sigma+1$ different datasets such that query answers differ by more than $2\epsilon$ on all the $2^\sigma+1$ datasets. Thus two of those datasets must be mapped to the same model parameter values, and the model must have an error more than $\epsilon$ on at least one of them, which completes the proof. The majority of the proofs are on how this set of $2^\sigma+1$ datasets is constructed. Specifically, the proofs construct a set of datasets, where each pair differs in some set of points $S$. The main technical challenge is constructing/showing that for any of the two datasets that differ in a set of points $S$, the maximum or average difference between the query answers is at least $2\epsilon$. This last statement is query dependent, and our technical Lemmas 3-8 in the appendix are proven to quantify the difference between query answers between the datasets based on the properties of the set $S$ and the query type. This is especially difficult in the average-case scenario, as one needs to study how the set $S$ affects all possible queries.

Reviewer comment (Theory):
Q2) The authors did not convey enough intuition behind the proof in the main text. The appendices are too advanced for a reader without the required background. As far as can be understood, the authors assume that a learned structure can, at most, represent 2^n different functions of bitstrings, show an isomorphism between bitstring functions and learned indexing, and then assert that since only 2^n bitstring functions can be covered, any database admitting more than 2^n values must have at least imperfect prediction. From there, they work through to the error bound. More explanation of this would be appreciated.

Author response:
R3Q1-2. We have added a new subsection to Appendix B, further discussing the intuition behind our proofs which we present here. As, the reviewer observed, given that there are $2^\sigma$ possible bitstrings (for model size $\sigma$), multiple datasets must be mapped to the same model parameter values (that is, after training, multiple datasets will have the exact same model parameter values). To obtain a bound on the error, the proof shows that some datasets that are mapped to the same parameter values will be too different from each other (i.e., queries on the datasets have answers that are too different), so that the exact same parameter setting cannot lead to answering queries accurately for both datasets. The proofs do this by constructing $2^\sigma+1$ different datasets such that query answers differ by more than $2\epsilon$ on all the $2^\sigma+1$ datasets. Thus two of those datasets must be mapped to the same model parameter values, and the model must have an error more than $\epsilon$ on at least one of them, which completes the proof. The majority of the proofs are on how this set of $2^\sigma+1$ datasets is constructed. Specifically, the proofs construct a set of datasets, where each pair differs in some set of points $S$. The main technical challenge is constructing/showing that for any of the two datasets that differ in a set of points $S$, the maximum or average difference between the query answers is at least $2\epsilon$. This last statement is query dependent, and our technical Lemmas 3-8 in the appendix are proven to quantify the difference between query answers between the datasets based on the properties of the set $S$ and the query type. This is especially difficult in the average-case scenario, as one needs to study how the set $S$ affects all possible queries.

Reviewer comment (Evaluation):
Q3) Learned index structures today generally care about log-loss (i.e., the average log2 of the absolute differences), since this is the number of binary search steps one will need for the "last mile" search.

Author response:
R3Q3. We have added a discussion to the newly added Appendix A showing how our bounds can be translated to $\log_2$ error, as discussed next. Ideally, to ensure $\log_2$ error is at most an error parameter $\tau$, setting $\epsilon=2^\tau$ in our bounds would provide the required model size given the $\log_2$ error $\tau$. This is true when considering worst-case error (i.e., Theorem 1) and in our upper bounds on the required model size when considering average case error (i.e., Theorem 4, Lemma 2 and part (ii) of Theorems 2,3,5). That is, our bounds on the mentioned results simply translate to bounds for $\log_2$ error by replacing $\epsilon$ with $2^\tau$. However, this is not true in the case of lower bounds for average error, (i.e., Corollary 1 and part (i) of Theorems 2,3,5) and the required model size can be smaller than what the results would suggest if we set $\epsilon=2^\tau$. We leave the study of the $\log_2$ error in these cases to future work, although we expect similar proof techniques that were used in this paper to be applicable to that case as well.

Reviewer comment (Presentation):
Presentation a bit misleading: The paper gives the impression as if the results apply to general database operators over all sorts of tabular data (e.g., SQL queries over mix of categorial/numerical data) while the results are limited to orthogonal/axis-aligned range queries (intersection of range selections along each dimensions). In general the limitations of this work are not outlined clearly.

Author response:
R2W1/Q1: The reviewer has correctly pointed out that our results on cardinality and range-sum estimation only apply to axis-parallel range queries for numerical data (which is a very common setting, e.g., Kipf et al., 2018; Ma & Triantafillou, 2019; Zeighami et al. 2023). We have modified the introduction to specify this earlier on. We have also added Sec. A to the appendix to discuss various other considerations not addressed in the paper, such as cardinality estimation for joins, other aggregation functions and other error metrics.

Reviewer comment (Experiments):
Significance unclear: The empirical study is too limited to give a clear idea how much predictive power is gained via the derived formulas and the general discussion does not clearly explain the implications for learned indexing and related topics. Non-learned baselines such as random sampling are missing.

Author response:
R2W2/Q2: We have added a new paragraph to the introduction further clarifying the implication of our results, as discussed next. There are two important predictive statements made by our theorems, following two interpretations of our theorems. First, our results provide a lower bound on the worst-case error given a model size (this view is presented in our experiments). This shows what error is achievable by a model of a certain size. Our experiments illustrate that our bound on error is meaningful, showing that models achieve error values close to what the bound suggests. Second, our results provide a lower bound on the required model size to achieve the desired accuracy level across datasets. This has important implications for resource management in database systems. For instance, this helps a cloud service provider decide how much resources it needs to allocate (and calculate the cost) for the learned model to be able to guarantee an accuracy level. Moreover, our bound provides the first theoretical guideline on how the model size should be set in practice to be able to guarantee a desired accuracy level.  

Moreover, based on the reviewer’s suggestions, we have added random sampling as a baseline in Figure 1. The figure shows that sampling performs worse than learned models for uniform distribution while it performs similarly for GMMs (Sample should be compared with Linear as they both have the same size). We note that although presenting results for sampling in our experiments provides more context to our empirical study, the goal of our experiments is not a detailed empirical evaluation of different modeling choices, and we refer the reviewer to the related work (e.g., Kipf et al., 2018; Zeighami et al. 2023) for such a comparison. The goal of our experiments is to show our theoretical bounds are meaningful, showing that existing modeling choices achieve similar error values as our bound suggests. We also note that our theoretical bounds suggest (discussed in Sec. 5 and also in our response to R2W3) that the gap between learned models and sampling will grow as dimensionality increases (we use one-dimensional datasets in our experiments).

Reviewer comment (Novelty):
Literature: The related work discussion is too limited in scope. It does not discuss VC dimensionality of orthogonal range queries (which is well-known and discussed in the referenced works), epsilon approximations in the computational geometry literature and data summaries in the database literature that pose questions that are similar in spirit to the model size question just with non-learned models such as samples and histogram-based data structures. Examples of related work in the aforementioned topics:

- Mustafa, N. H., & Varadarajan, K. R. (2017). Epsilon-approximations and epsilon-nets. arXiv preprint arXiv:1702.03676.

- Suri, S., Tóth, C. D., & Zhou, Y. (2004, June). Range counting over multidimensional data streams. In Proceedings of the twentieth annual symposium on Computational geometry (pp. 160-169).

- Shekelyan, M., Dignös, A., & Gamper, J. (2017). Digithist: a histogram-based data summary with tight error bounds. Proceedings of the VLDB Endowment, 10(11), 1514-1525.

- Cormode, G., Garofalakis, M., Haas, P. J., & Jermaine, C. (2011). Synopses for massive data: Samples, histograms, wavelets, sketches. Foundations and Trends® in Databases, 4(1–3), 1-294.

- Wei, Z., & Yi, K. (2018). Tight space bounds for two-dimensional approximate range counting. ACM Transactions on Algorithms (TALG), 14(2), 1-17.

Author response:
R2W3/Q3. We thank the reviewer for pointing out the related work and the interesting connection between our results and non-learned data models. We have updated the related work section based on the reviewer’s suggestion. Please see the newly added paragraph in the related work section for a detailed answer to the reviewer’s question. To summarize, the newly added paragraph includes a discussion of theoretical results in the related non-learned literature (such as $\epsilon$-approximation, sampling and use of VC dimensionality, non-learned data summaries). We specifically compare our bounds with bounds on $\epsilon$-approximations, showing that $\epsilon$-approximations (which include random sampling) have a worse lower bound compared with our theorems in high dimensions. This shows that random sampling (and other $\epsilon$-approximations) can be a much worse data model in terms of space efficiency compared with other modeling choices in high dimensions.

Reviewer comment (Experiments):
The range-sum estimation queries considered in the paper are also restrictive. In particular, including other forms of aggregation would be very useful for practical database applications.

Author response:
We thank the reviewer for the comments. We have updated the paper based on the reviewers’ comments, and major changes are marked in blue in the updated file. 

R1W2. We have added a paragraph in the newly added Appendix A that discusses providing bounds for other aggregation functions as follows.  Although we expect similar proof techniques as what was used to apply to other aggregations such as min/max/avg, we note that studying worst-case bounds for min/max/avg may not be very informative. Intuitively, this is because, for such aggregations, one can create arbitrary difficult queries that require the model to memorize all the data points. For instance, consider datasets for which the $(d+1)$-th dimension, where the aggregation is applied, only takes 0 or 1 values, while the other dimensions can take any values in a domain of size $u$. Now avg/min/max for any range query will have an answer between 0 and 1 (for min/max the answer be exactly 0 or 1). However, unless the model memorizes all the points exactly (which requires a model size close to the data size), its worst-case error can be up to 0.5. This is because queries with a very small range can be constructed that match exactly one point in the database, and unless the model knows the exact value of all the points, it will not be able to provide a correct answer for all such queries. Note that the error of 0.5 is very large for avg/min/max queries, and a model that always predicts 0.5 also obtains a worst-case error 0.5. This is not the case for sum/count aggregations whose query answers range between 0 to $n$, and a model with an absolute error of 0.5 can be considered a very good model for most queries when answering sum/count queries. 

To summarize, worst-case errors of min/avg/max queries are disproportionately affected by smaller ranges, while smaller ranges have a limited impact on the worst-case error of count/sum queries.  As such we expect that for the case of min/max/avg queries, one needs to study the error for a special set of queries (e.g., queries with a lower bound on their range, as done in  Zeighami et al. (2023), which also makes similar observations) to be able to obtain meaningful bounds. We leave such a study to future work.

Reviewer comment (Theory):
Could you please comment on the two points mentioned in Weaknesses. In particular, could you please comment on the possibility of using the ideas of the paper to provide error guarantees for aggregate operators min, max, and average.

Author response:
We thank the reviewer for the comments. We have updated the paper based on the reviewers’ comments, and major changes are marked in blue in the updated file. 

R1W2. We have added a paragraph in the newly added Appendix A that discusses providing bounds for other aggregation functions as follows.  Although we expect similar proof techniques as what was used to apply to other aggregations such as min/max/avg, we note that studying worst-case bounds for min/max/avg may not be very informative. Intuitively, this is because, for such aggregations, one can create arbitrary difficult queries that require the model to memorize all the data points. For instance, consider datasets for which the $(d+1)$-th dimension, where the aggregation is applied, only takes 0 or 1 values, while the other dimensions can take any values in a domain of size $u$. Now avg/min/max for any range query will have an answer between 0 and 1 (for min/max the answer be exactly 0 or 1). However, unless the model memorizes all the points exactly (which requires a model size close to the data size), its worst-case error can be up to 0.5. This is because queries with a very small range can be constructed that match exactly one point in the database, and unless the model knows the exact value of all the points, it will not be able to provide a correct answer for all such queries. Note that the error of 0.5 is very large for avg/min/max queries, and a model that always predicts 0.5 also obtains a worst-case error 0.5. This is not the case for sum/count aggregations whose query answers range between 0 to $n$, and a model with an absolute error of 0.5 can be considered a very good model for most queries when answering sum/count queries. 

To summarize, worst-case errors of min/avg/max queries are disproportionately affected by smaller ranges, while smaller ranges have a limited impact on the worst-case error of count/sum queries.  As such we expect that for the case of min/max/avg queries, one needs to study the error for a special set of queries (e.g., queries with a lower bound on their range, as done in  Zeighami et al. (2023), which also makes similar observations) to be able to obtain meaningful bounds. We leave such a study to future work.