PAC Synthesis of Machine Learning Programs
Abstract
We study the problem of synthesizing programs that include machine learning components such as deep neural networks (DNNs).
We focus on statistical properties, which are properties expected to hold with high probability—e.g., that an image classiﬁcation model correctly identiﬁes people in images with high probability.
We propose novel algorithms for sketching and synthesizing such programs by leveraging ideas from statistical learning theory to provide statistical soundness guarantees.
We evaluate our approach on synthesizing list processing programs that include DNN components used to process image inputs, as well as case studies on image classiﬁcation and on precision medicine.
Our results demonstrate that our approach can be used to synthesize programs with probabilistic guarantees.
Introduction
Machine learning has recently become a powerful tool for solving challenging problems in artiﬁcial intelligence.
As a consequence, there has been a great deal of interest in incorporating machine learning components such as deep neural networks (DNNs) into real-world systems, ranging from healthcare decision-making [1, 2, 3], to robotics perception and control [4, 5].
In these domains, there is often a need to ensure correctness properties of the overall system.
To reason about such properties, we need to reason about properties of the incorporated machine learning components, focusing on statistical properties that should hold with high probability with respect to the distribution of inputs—e.g., we may want to ensure that the DNN detects 95% of pedestrians.
We propose a framework for synthesizing programs that incorporate machine learning components while satisfying statistical correctness properties.
Our framework consists of two components.
First, it includes a novel statistical sketching algorithm, which builds on the concept of sketching [6] to provide statistical guarantees.
At a high level, it takes as input a sketch (i.e., a program with certain parts left unspeciﬁed), annotated with speciﬁcations encoding statistical properties that are expected to hold, as well as holes corresponding to real-valued thresholds for making decisions (e.g., the conﬁdence level at which to label an image as containing a pedestrian or to diagnose a patient with a disease).
Since statistical properties depend on the data distribution, it additionally takes as input a labeled dataset of training examples (separate from those used to train the DNNs).
Then, our algorithm selects values to ﬁll the holes in the sketch so all the given speciﬁcations are satisﬁed.
Second, our framework uses this sketching algorithm in conjunction with a syntax-guided synthesizer [7] to synthesize programs in a speciﬁc domain that provably satisfy statistical guarantees.
Our strategy is to ﬁrst synthesize a sketch whose speciﬁcations encode overall correctness, and then apply our sketching algorithm to ﬁll the holes in the sketch so these speciﬁcations are satisﬁed.
We have implemented our approach as S TAT C ODER .
We evaluate its ability to synthesize list processing programs satisfying statistical properties, where the program inputs are images, and DNN components are used to classify objects in these images.
We also perform three case studies of our sketching algorithm: one on ImageNet classiﬁcation and another on a medical prediction task.
Overview
Here, we provide an overview of our approach; we give details on our sketch language in Section A, our sketching algorithm in Section C, and our synthesis algorithm in Section D.
Statistical sketching.
We assume given a DNN component f : X → [0 , 1] that, given an image x ∈ X , predicts whether x contains a person.
In particular, f ( x ) is a score indicating its conﬁdence that x contains a person; higher score means more likely to contain a person.
We do not assume the the scores are reliable—e.g., they may be overconﬁdent.
We assume that the ground truth label y ∗ ∈ Y = { 0 , 1 } indicates whether x contains a person.
For example, f ( x ) may be the probability that an image contains a person according to a pretrained DNN such as ResNet [8]; then, the goal is to tailor this DNN to the current task in a way that provides correctness guarantees.
In particular, our goal is to choose a threshold c ∈ [0 , 1] such that the program returns that the given image x contains a person if f ( x ) has conﬁdence at least 1 − c —i.e., f ( x ) ≥ 1 − c , or equivalently, 1 − f ( x ) ≤ c .
Furthermore, we want c to be correct in the following sense:
[Equation]
That is, if the image contains a person (i.e., y ∗ = 1 ), then the classiﬁer should say so (i.e., 1 − f ( x ) ≤ c ).
We do not require the converse—i.e., the program may incorrectly conclude that an image contains a person even if it does not.
That is, we want soundness (i.e., no false negatives) but not necessarily completeness (i.e., no false positives).
However, we cannot guarantee soundness for every x ; instead, we guarantee it with high probability—i.e., we say c is (cid:15) -approximately correct if
[Equation]
In adddition, our algorithm relies on training examples (cid:126)z = { ( x 1 , y ∗ 1 ) , ..., ( x n , y ∗ n ) } to choose c , where ( x i , y ∗ i ) ∼ p are i.i.d.
samples from p ( x, y ∗ ) .
Thus, as with probably approximately correct (PAC) bounds from statistical learning theory [9, 10], we need to additional allow a possibility that our algorithm fails altogether due to the randomness in our training examples (cid:126)z .
In particular, consider an algorithm A that chooses c
= A ( (cid:126)z ) ; then, we say A is ( (cid:15), δ ) -PAC if
P p ( (cid:126)z ) (cid:0) A ( (cid:126)z ) is (cid:15) -approximately correct (cid:1) ≥ 1 − δ where p ( (cid:126)z ) is the distribution over the training examples (cid:126)z , and δ ∈ R > 0 is another user-provided conﬁdence level.
Then, given the sketch, a value δ ∈ R > 0 , and a dataset (cid:126)z , our algorithm synthesizes a value of c to ﬁll ??
1 such that c is (cid:15) -approximately correct with probability at least 1 − δ .
Our sketching algorithm formulates the problem of synthesizing c as a binary classiﬁcation problem.
In particular, for each training example ( x, y ∗ ) ∈
X × Y , it constructs the value
z = (cid:26) 1 − f ( x ) if y ∗ =
−∞ otherwise .
Note that for a given choice of c , the value w = 1 ( z ≤ c ) indicates whether c is correct for ( x, y ∗ ) — either y ∗ = 0 , in which case since z =
−∞ so the antecedent y ∗ = 1 in (1) trivially holds, or y ∗ = 1 , in which case z = 1 − f ( x ) and w indicates whether the consequent of (1) holds.
Thus, thinking of z as a random function of ( x, y ∗ ) , our goal is to choose c such that P p ( z ) ( z ≤ c ) ≥ 1 − (cid:15) .
In other words, c can be thought of as a threshold in a binary classiﬁcation problem, where the input is z and the desired label is always w = 1 .
The difference from typical binary classiﬁcation is that we do not want to minimize the number of errors; instead, our goal is to minimize c subject to a constraint on the error rate—i.e., we want the most optimistic value of c that satisﬁes the PAC guarantee.
In Section C, we provide an algorithm for solving this problem with PAC guarantees.
Synthesis algorithm.
Next, suppose we want to synthesize a program that counts the number of people in a list of images (cid:96) =
( x 1 , ..., x n ) .
Intuitively, we can do so by writing a simple list processing program around our DNN for detecting people.
In particular, letting
[Equation]
be our DNN component, where the detection threshold has been left as a hole, then the sketch
[Equation]
counts the number of people in (cid:96) .
Given a few input-output examples along with the ground truth labels for each image, we can use a standard enumerative synthesizer to compute the sketch ˜
P ex , assuming predict person returns the ground truth label.
In particular, this sketch has a single hole in the DNN component predict person that remains to be ﬁlled.
Note that ˜ P ex evaluates correctly if predict person returns the ground truth label, but in general, it may make mistakes.
Thus, the correctness property for the synthesized program P ex needs to account for the possibility that predict person may return incorrectly.
Mirroring the correctness property for a single prediction, suppose we want a program P ex that conservatively overestimates the number of people in (cid:96) .
1 In particular, given conﬁdence levels (cid:15), δ ∈ R > 0 , we say a completion P ex of ˜ P ex is (cid:15) -approximately correct if
[Equation]
where α =
( (cid:96), y ∗ ) is an example, and (cid:74) P (cid:75) (cid:96) denotes the output of running program P on input (cid:96) .
Then, we say our synthesis algorithm is ( (cid:15), δ ) -probably approximately correct (PAC) if
P
[Equation]
is (cid:15) -approximately correct ) ≥ 1 − δ, where P ex = A
( ˜ P ex , (cid:126)α ) is the program synthesized using our algorithm and training examples (cid:126)α .
Using our statistical sketching algorithm, we can provide ( (cid:15) (cid:48) , δ (cid:48) )
-PAC guarantees on predict person for any (cid:15) (cid:48) , δ (cid:48) ∈ R > 0 ; thus, the question is how to choose (i) the appropriate speciﬁcation, (ii) the parameters of this speciﬁcation, and (iii) the conﬁdence levels (cid:15) (cid:48) , δ (cid:48) .
These choices depend on the speciﬁcation that we want to ensure for the synthesized program P ex .
In our example, we can use the speciﬁcation above—i.e., that predict person returns 1 with high probability if there is a person:
[Equation]
In general, the speciﬁcation on predict person may have additional parameters (in particular, for real-valued predictions, an error tolerance e ).
Next, we need to choose (cid:15) (cid:48) , δ (cid:48) .
While there is only one hole, predict person is executed multiple times (assuming length ( (cid:96) ) > 1 ).
We need to choose (cid:15) (cid:48) and δ (cid:48) so that with high probability, predict person is correct for all applications.
For simplicity, we assume given an upper bound N ∈ N on the maximum possible length of (cid:96) (we discuss how we might remove this assumption in Section E).
Given N , we take (cid:15) (cid:48)
=
(cid:15)/N and δ (cid:48) =
δ/N ; then, we use our sketching algorithm to synthesize c to ﬁll the hole in predict person .
By a union bound, for a given list (cid:96) , all applications of predict person are correct with probability at least 1 − (cid:15) , and this property holds with probability at least 1 − δ .
Under this event, P ex returns correctly—i.e., P ex satisﬁes the desired ( (cid:15), δ ) -PAC guarantee.
Evaluation
We provide partial experimental results here; see Section F for details and additional results.
Experimental setup.
We consider synthesizing programs that operate over the predictions made by a state-of-the-art DNN for object detection.
We assume given a DNN component ˆ f that given an image x , is designed to detect people and cars in x .
We use a pretrained state-of-the-art object detector called Faster R-CNN [4] available in PyTorch [11], tailored to the COCO dataset [12], which is a dataset of real-world images containing people, cars, and other objects.
There are multiple variants of
Faster R-CNN; we use the most accurate one, X101-FPN with 3 × learning rate schedule.
We represent this DNN as a component ˆ f : X → Y = D ∗ × R , where ˆ f ( x ) =
(ˆ y ( x ) , ˆ p ( x )) consists of a list of detections d ∈ ˆ y ( x ) along with a correctness score ˆ p ( x ) that the prediction is correct.
Each detection d ∈ D
=
R 2 ×
Z is itself a tuple d = ( b, z ) including the position b and predicted category of the object.
The ground truth label y ∗ for an image x is a list of detections d ∈ y ∗ .
In general, we cannot expect to get a perfect match between the predicted bounding boxes and the ground truth ones.
[Table]
Typically, two detections d, d ∗ match , denoted (cid:107) d − d ∗ (cid:107) ≤ e , where e is a speciﬁed error tolerance, if the distance between their centers satisﬁes (cid:107) b − b ∗ (cid:107) ∞ ≤ e .
Furthermore, we write (cid:107) ˆ y ( x ) − y ∗ (cid:107) ≤ e if | ˆ y ( x ) | = | y ∗ | and there exists a one-to-one correspondence between d ∈ ˆ f ( x ) and d ∗ ∈
y ∗ such that (cid:107) d − d ∗ (cid:107) ≤ e .
Then, we deﬁne predict : X → ( Y ∪ ∅ ) by
[Equation]
In other words, the speciﬁcation says that a correct prediction is if the error tolerance is below a level ??
e to be speciﬁed.
Thus, given e and (cid:15) to ﬁll ??
e and ??
(cid:15) , respectively, our sketching algorithm synthesizes a threshold c to ﬁll ??
c in a way that guarantees that this speciﬁcation holds.
Then, predict returns ˆ y ( x ) if the DNN is sufﬁciently conﬁdent in its prediction, and ∅ otherwise.
For example, if a robot acting in the world is using results from the program to navigate, then it can act conservatively (e.g., safely come to a stop) when the program returns ∅ to ensure safety; alternatively, it might fall back on a more accurate but computationally expensive model to make predictions.
We use our synthesis algorithm in conjunction with this prediction component and a standard domainspeciﬁc language (DSL) of list processing programs, and PAC parameters (cid:15) =
δ = 0 .
We use n = 1000 COCO validation set images for synthesis, and the remaining 1503 for evaluation.
We compare to a baseline that uses a naïve statistical estimator to synthesize c instead of statistical sketching.
At a high level, it uses the more traditional method of chooses c to minimize the empirical error rate, whereas our algorithm minimizes c subject to a constraint on the error rate.
We evaluate our approach on synthesizing ﬁve programs.
For the program synthesized using each approach, we report the following metrics: (i) the ∅ rate—i.e., The rate at which ¯ P returns ∅ , and (ii) the failure rate—i.e., the rate at which ¯ P makes mistakes.
Results.
Results are shown in Table 1.
Both S TAT C ODER and the baseline always achieve the desired failure rate bound of (cid:15) = 0 .
However, S TAT C ODER outperforms the baseline by a large margin in terms of ∅ rate, since it uses a learning algorithm tailored to our setting.
These results demonstrate the effectiveness of S TAT C ODER at synthesizing accurate programs while satisfying a PAC guarantee.
Conclusion
We have proposed algorithms for synthesizing machine learning programs that come with PAC guarantees.
Our technique leverages novel statistical learning bounds to achieve these guarantees.
We have empirically demonstrated how our approach can be used to synthesize list processing programs that manipulate images using DNN components while satisfying PAC guarantees
A Sketch Language
In this section, we describe the syntax and semantics of our sketch language, as well as the desired correctness properties we expect that synthesized programs should satisfy.
Syntax.
Our sketch language is shown in Figure 1.
Intuitively, in the expression φ ( P, c ) { Q } ω(cid:15) , Q is a speciﬁcation that we want to ensure holds, P is a score (intuitively, it should indicate the likelihood that Q holds, but we make no assumptions about it), c is a threshold below which we consider Q to be satisﬁed, (cid:15) is the allowed failure probability, and ω indicates whether we want a conditional guarantee (i.e., ω
= | , the guarantee (1)) or implication guarantee (i.e., ω = ⇒ , the guarantee ( ?? )).
We assume that P evaluates to a value in R , c ∈ R , and Q evaluates to a value in { 0 , 1 } .
Note that Q is itself a program; unlike programs P , it can use ground truth inputs y .
Finally, either c and (cid:15) in this expression can be left as a hole ??
(but not both simultaneously).
We say P is complete if it contains no holes and partial otherwise.
We use P to denote the space of programs, ¯ P ⊆ P to denote the space of complete programs, and ¯ P ∈ ¯ P to denote a complete program.
For P ∈ P , we use Φ( P ) to denote the expressions φ ( P (cid:48) , c ) { Q } ω(cid:15) in P (including cases where c or (cid:15) is a hole), Φ c ??
( P ) ⊆
Φ( P ) to denote the expressions φ ( P (cid:48) , ??)
{ Q } ω(cid:15) in P , Φ (cid:15) ??
( P ) ⊆
Φ( P ) to denote the expressions φ ( P (cid:48) , c ) { Q } ω ??
in P , and Φ ??
( P ) =
Φ c ??
( P ) ∪ Φ (cid:15) ??
( P ) .
Semantics.
We deﬁne two semantics for programs P , shown in Figure 1:
• Train semantics: Given a training valuation α ∈ A , where α : X ∪ Y → C maps both inputs and ground truth inputs y to values, the train semantics (cid:74) · (cid:75) ∗ α evaluate Q instead of φ ( P, c ) .
Since they ignore φ , they can be applied to both partial and complete programs.
• Test semantics: Given a test valuation β ∈ B , where β : X → C maps inputs to values, the test semantics (cid:74) · (cid:75) β evaluate φ ( P, c ) instead of Q .
They only apply to complete programs.
Correctness properties.
We deﬁne what it means for a complete program to be correct—i.e., satisﬁes its speciﬁcations.
We begin with correctness of a single speciﬁcation.
Deﬁnition A.1.
Given a distribution p ( α ) over test valuations α ∈ A , φ ( ¯ P, c ) { Q } | (cid:15) is approximately sound if it satisﬁes the conditional guarantee[-
-]
[Equation]
and { Q } ⇒ (cid:15) is approximately sound if it satisﬁes the implication guarantee
[Equation]
This property can be thought of as probabilistic soundness; it says that we should have φ ( ¯ P, c ) ⇒ Q with high probability, which means that φ ( ¯ P, c ) is a sound overapproximation of Q .
Deﬁnition A.2.
A complete program ¯ P is approximately correct (denoted ¯ P ∈ ¯ P ∗ ) if every expression φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) in ¯ P is approximately sound.
B Statistical Veriﬁcation
We describe our algorithm for verifying a complete program.
Our algorithm (Algorithm 1) takes as input a complete program ¯ P , training valuations (cid:126)α =
( α 1 , ..., α n ) , where α 1 , ..., α n ∼ p are i.i.d. samples, and a conﬁdence level δ ∈ R > 0 , and outputs a value
A ( ¯ P, (cid:126)α ) ∈ { 0 , 1 } indicating whether ¯ P is approximately complete, which is correct with probability at least 1 − δ with respect to p ( (cid:126)α ) .
Our algorithm is based on statistical veriﬁcation [13, 14, 15].
These algorithms leverage concentration inequalities from probability theory to provide high-probability correctness guarantees.
Concentration inequalities are theorems that provide rigorous bounds on the rate of convergence of statistical estimators.
For instance, consider a Bernoulli distribution p =
Bernoulli ( µ ) with unknown mean µ .
Given samples z 1 , ..., z n ∼ p , Hoeffding’s inequality [16] says that the empirical mean ˆ µ ( (cid:126)z ) = n − 1 (cid:80) ni
=
z i converges to µ :
[Equation]
(cid:1) ≥ 1 − δ where δ = 2 e − 2 n(cid:15) 2 , (2) i.e., ˆ µ ( (cid:126)z ) is a good approximation of µ with high probability.
In our setting, given training valuation α and a speciﬁcation E = φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) in ¯ P , we let z α = (cid:74) E (cid:75)
α and z ∗ α = (cid:74) E (cid:75) ∗ α .
Then, (cid:15) -approximate soundness of E is equivalent to µ =
P p ( α ) ( z α | z ∗ α ) ≥ 1 − (cid:15) if ω = | , or µ = P p ( α ) ( z ∗ α ⇒ z α ) ≥ 1 − (cid:15) if ω = ⇒ .
That is, (cid:15) -approximate soundness is equivalent to µ ≥ 1 − (cid:15) , where µ is the mean of a Bernoulli random variable z
α that is a function of a random variable α with distribution p ( α | z ∗ α ) (if ω = | ) or the mean of a Bernoulli random variable z ∗ α ⇒ z
α that is a function of α with distribution p ( α ) (if ω = ⇒ ).
However, z α is potentially a complicated function of α and p ( α ) is unknown, so µ is hard to compute directly.
Instead, given i.i.d. samples α 1 , ..., α n ∼ p ( α ) , we can construct the samples z α 1 , ..., z α n and z ∗ α 1 , ..., z ∗ α n and use them estimate µ :
[Equation]
if ω = [Equation] Then, we can use (2) to bound the error of ˆ µ ( (cid:126)α ) —e.g., if ˆ µ ( (cid:126)α ) ≥ 1 − (cid:15) 2 and | ˆ µ ( (cid:126)α ) − µ | ≤ (cid:15) 2 with probability at least 1 − δ , then µ ≥ 1 − (cid:15) with probability at least 1 − δ .
However, this approach is inefﬁcient since Hoeffding’s inequality is not tight for our setting.
Instead, our veriﬁcation algorithm (Section B.2) leverages a concentration inequality tailored to our setting (Section B.1).
Finally, we disucss how our approach can be used in the context of runtime monitoring (Section B.3).
B.1 A Concentration Bound
Problem formulation.
Consider a Bernoulli distribution p =
Bernoulli ( µ ) with unknown mean µ ∈ [0 , 1] .
Given (cid:15) ∈ R > 0 , our goal is to determine whether µ ≥ 1 − (cid:15) .
For instance, a sample z ∼ p may indicate a desired outcome (e.g., a correctly classiﬁed input), in which case µ is the correctness rate and (cid:15) is a desired bound on the error rate; then, our goal is to check whether the µ meets the desired error bound.
More precisely, we want to compute ψ ∈ { 0 , 1 } such that
[Equation]
[Equation]
That is, ψ is a sound overapproximation of the property µ ≥ 1 − (cid:15) (i.e., ψ = 1 implies µ ≥ 1 − (cid:15) ).
To compute such a ψ , we are given a training set of examples (cid:126)z = ( z 1 , ..., z n ) ∈ { 0 , 1 } n , where z 1 , ..., z n ∼ p are n i.i.d.
samples from p .
An estimator is a mapping ˆ ψ : R n → R .
We say such an estimator is approximately correct if it satisﬁes the condition (3)—i.e.
, ˆ ψ ( (cid:126)z ) ⇒ ( µ ≥ 1 − (cid:15) ) .
In general, we cannot guarantee ˆ ψ ( (cid:126)z ) is approximately correct due to the randomness in the training examples (cid:126)z .
Thus, we allow a probability δ ∈ R > 0 that ˆ ψ ( (cid:126)z ) is not approximately correct.
Deﬁnition B.1.
Given (cid:15), δ ∈ R > 0 , ˆ ψ is ( (cid:15), δ ) -PAC if P p ( (cid:126)z ) (cid:0) ˆ ψ ( (cid:126)z ) ⇒ ( µ ≥ 1 − (cid:15) ) (cid:1) ≥ 1 − δ .
In other words, ˆ ψ ( (cid:126)z ) is approximately correct with probability at least 1 − δ according to the randomness in p ( (cid:126)z ) .
Our goal is to construct an ( (cid:15), δ ) -PAC estimator ˆ ψ ( (cid:126)z ) .
Algorithm procedure V ERIFY ( ¯ P, (cid:126)α, δ ) m ← | Φ( P ) | for φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) ∈
Φ( ¯ P ) do
Compute (cid:126)z (cid:126)α according to (5)
Compute ˆ ψ ( (cid:126)z (cid:126)α ) according to (4) with ( (cid:15), δ/m ) if ¬ ˆ ψ ( (cid:126)z (cid:126)α ) then return false end if end for return true end procedure
Estimator.
Given (cid:15), δ ∈ R > 0 , consider the estimator ˆ ψ ( (cid:126)z ) =
1 ( L ( (cid:126)z ) ≤ k ) where k = max (cid:40) h ∈ N (cid:12)(cid:12)(cid:12) (cid:12)(cid:12) h (cid:88) i
= (cid:18) n i (cid:19) (cid:15) i (1 − (cid:15) )
n − i ≤ δ (cid:41) (4) and where L ( (cid:126)z ) = (cid:80) nz ∈ (cid:126)z (1 − z ) .
Intuitively, L ( (cid:126)z ) counts the number of errors, so we conclude the desired property holds as long as L ( (cid:126)z ) is below a threshold k .
This threshold is chosen so ˆ ψ is ( (cid:15), δ ) -PAC—in particular, δ upper bounds the CDF of the binomial distribution evaluated at k .
To compute the solution k in (4), we start with h = 0 and increment it until it no longer satisﬁes the condition.
To ensure numerical stability, this computation is performed using logarithms.
Note that k does not exist if the set inside the maximum in (4) is empty; in this case, we choose ˆ ψ ( (cid:126)z ) = 0 , which trivially satisﬁes the PAC property.
We have the following; see Section H.1 for a proof:
Theorem B.2.
The estimator ˆ ψ is ( (cid:15), δ ) -PAC.
B.2 Veriﬁcation Algorithm
Problem formulation.
A veriﬁcation algorithm A : ¯ P ×
A n → { 0 , 1 } takes as input a complete program ¯ P ∈ ¯ P , and a set of test valuations (cid:126)α =
( α 1 , ..., α n ) ∈
A n , where α 1 , ..., α n ∼ p are i.i.d. samples from a distribution p ( α ) .
For example, p ( α ) may be the distribution of input images to an image classiﬁer encountered while running in production, that have been manually labeled using crowdsourcing.
Then, A ( ¯ P, (cid:126)α ) ∈ { 0 , 1 } should indicate whether ¯ P is approximately sound—i.e., whether every expression φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) ∈ Φ( ¯ P ) is approximately sound.
We say that A is sound if
A ( ¯ P, (cid:126)α ) ⇒ ¯ P ∈ ¯ P ∗ .
As before, we cannot guarantee that A is sound; instead, given δ ∈ R > 0 , we want this property to hold with probability at least 1 − δ according to p ( (cid:126)α ) .
Deﬁnition B.3.
A veriﬁcation algorithm A : ¯ P ×
A n → { 0 , 1 } is δ -probably approximately sound if for all ¯ P ∈ ¯ P , P p ( (cid:126)α ) (cid:0)
A ( ¯ P, (cid:126)α ) ⇒ ¯ P ∈ ¯ P ∗ (cid:1) ≥ 1 − δ .
Algorithm.
Our veriﬁcation algorithm is shown in Algorithm 1.
It check approximate correctness of ¯ P by checking approximate soundness of each φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) ∈
Φ( ¯ P ) .
It does so by allocating a δ/m probability of failure for each expression, where m =
| Φ( ¯ P ) | is the number of such expressions.
Next, we describe how our algorithm checks approximate soundness for a single expression φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) .
Given a single test valuation α ∼ p , consider the indicators
[Equation]
and
[Equation]
That is, z α indicates whether φ ( ¯ P (cid:48) , c ) holds, and z ∗ α indicates whether Q holds.
Then, φ ( ¯ P (cid:48) , c ) { Q } is approximately sound if and only if
[Equation]
Next, note that z α ∈ { 0 , 1 } is a Bernoulli random variable with mean µ
= P p ( α ) ( z α | z ∗ α ) (if ω = | ) or µ = P p ( α ) ( z ∗ α ⇒ z α ) (if ω = ⇒ ).
Thus, given (cid:126)α = ( α 1 , ..., α n ) , where α 1 , ..., α n ∼ p are i.i.d.
samples,
[Equation] [Equation]
is a vector of i.i.d.
samples from Bernoulli ( µ ) .
Then, the estimator ˆ ψ ( (cid:126)z (cid:126)α ) in (4) with parameters ( (cid:15), δ/m ) indicates whether µ ≥ 1 − (cid:15) with high probability—i.e., if ˆ ψ ( (cid:126)z (cid:126)α ) = 1 , then µ ≥ 1 − (cid:15) holds with probability at least 1 − δ/m according to p ( (cid:126)α ) .
The following guarantee follows from
Theorem B.2 by a union bound over expressions in Φ( ¯ P ) :
Theorem B.4.
Algorithm 1 is δ -probably approximately sound.
B.3 Runtime Monitoring
One challenge is that the speciﬁcations considered by our framework depend on the distribution of the data.
As a consequence, if this distribution changes, then our correctness guarantees may no longer hold.
This potential failure mode, called distribution shift [17, 18], is a major challenge for machine learning components.
A key feature of our framework is that it can be used not only to sketch or verify the program before it is deployed, but also to continuously re-sketch the program based on feedback obtained in production to account for potential distribution shift.
The primary requirement for using this approach is the need for feedback—i.e., continuing to collect labeled examples in production.
In some settings, this kind of feedback is naturally available; otherwise, a solution is to manually label a small fraction of examples—e.g., using crowdsourcing [19].
Given ground truth labels for the input examples encountered in production, our veriﬁcation algorithm can be straightforwardly adapted to the runtime setting.
In particular, our system collects examples during execution; once it collects at least N examples, it re-runs veriﬁcation or sketching.
It can do so after every subsequent example, or every K examples.
Finally, we may want to discard an examples after T steps, both for computational efﬁciency and to account for the fact that the data distribution may be shifting over time so older examples are less representative.
Here, K, N, T ∈ N are hyperparameters.
Finally, we note that our statistical sketching algorithm can similarly be adapted to the runtime setting.
C Statistical Sketching
Next, we describe our algorithm for synthesizing values c and (cid:15) to ﬁll holes in a given sketch.
Our algorithm, shown in Figure 2, takes as input a sketch P , training valuations (cid:126)α =
( α 1 , ..., α n ) , where α 1 , ..., α n ∼ p are i.i.d.
samples, and a conﬁdence level δ ∈ R > 0 , and outputs a complete program
A ( ¯ P, (cid:126)α ) ∈
¯ P ∗ that is approximately correct with probability at least 1 − δ with respect to p ( (cid:126)α ) .
Our algorithm synthesizes c and (cid:15) in a bottom-up fashion, so that all subtrees of the current expression are complete.
Our sketching algorithm uses probabilistic bounds in conjunction with the given samples (cid:126)α to provide guarantees.
Intuitively, since we are estimating parameters from data, our problem is a statistical learning problem [9], so we can leverage techniques from statistical learning theory to provide guarantees on the synthesized sketch.
For synthesizing c —i.e., an expression E = φ ( P, c ) { Q } ω(cid:15) .
Letting z α = (cid:74) P (cid:75) α ∈ R and z ∗ α = (cid:74) E (cid:75) , then c is (cid:15) -approximately correct if z α ≤ c conditioned on z ∗ α =
(if ω = | ) or whenever z ∗ α =(if ω = ⇒ ) with probability at least 1 − (cid:15) with respect to p ( α ) .
In either case, synthesizing c is equivalent to a binary classiﬁcation problem with labels z ∗ α , with a one-dimensional hypothesis space c ∈ R and a one-dimensional feature space z α ∈ R .
Furthermore, this problem is simple— c is a linear classiﬁer.
Thus, we could use standard learning theory results to provide guarantees.
However, we can obtain sharper guarantees using a learning theory bound specialized to our setting.
We build on a bound based on [10] (Section C.1) tailored to the realizable setting, where there exists a classiﬁer that makes zero mistakes.
Our setting is realizable, since c = ∞ always makes zero mistakes.
The main difference is that their bound always chooses a classiﬁer that makes zero mistakes, which can be overly conservative.
We prove a novel generalization bound that allows for some number k of mistakes that is a function of (cid:15) , δ , and n .
Synthesizing a value (cid:15) is a bit different, since we are not classifying examples that depend on a single α , but examples that depend on (cid:126)α .
Thus, we can formulate it as a learning problem where the examples are (cid:126)α ; however, this approach is complicated due to the need to ﬁgure out how to divide our given samples (cid:126)α into multiple sub-examples (cid:126)α 1 , ... (cid:126)α n .
Instead, we use an approach based on
Hoeffding’s inequality [16] (Section C.2) to infer (cid:15) .
In particular, Hoeffding’s inequality gives us a lower bound on the correctness rate P p ( α ) ( z α | z ∗ α ) ≥ 1 − (cid:15) (if ω = | ) or P p ( α ) ( z ∗ α ⇒ z α ) ≥ 1 − (cid:15) (if ω = ⇒ ), and we can simply use this (cid:15) .
Finally, our sketching algorithm uses the above two approaches to synthesize c and (cid:15) (Section C.3).
C.1 A Learning Theory Bound
Problem formulation.
We consider a unary classiﬁcation problem with one-dimensional feature and hypothesis spaces.
In particular, given a probability distribution p ( z ) over z ∈ R (the feature), the goal is to select the smallest possible threshold t ∈ R (the hypothesis) such that
[Equation]
for a given (cid:15) ∈ R > 0 .
That is, we want the smallest possible t such that z ∈ ( −∞ , t ] with probability at least 1 − (cid:15) according to p ( z ) .
We denote the subset of t that satisﬁes (6) by
[Equation]
To compute such a t , we are given a training set of examples (cid:126)z = ( z 1 , ..., z n ) ∈ R n , where z 1 , ..., z n ∼ p are n i.i.d.
samples from p .
An estimator ˆ t is a mapping ˆ t : R n → R .
Then, the constraint (6) is [Equation] ) ∈ T (cid:15) ; we say such a ˆ t is (cid:15) -approximately correct —i.e., it is correct for “most” samples z ∼ p .
In general, we are unable to guarantee that ˆ t is approximately correct due to the randomness in the training examples (cid:126)z .
Thus, we additionally allow for a small probability δ ∈ R > 0 that ˆ t is not approximately correct.
Deﬁnition C.1.
Given (cid:15), δ ∈ R > 0 , ˆ t is ( (cid:15), δ ) -PAC if P p ( (cid:126)z ) (ˆ t ( (cid:126)z ) ∈ T (cid:15) ) ≥ 1 − δ .
That is, ˆ t ( (cid:126)z ) is approximately correct with probability at least 1 − δ according to p ( (cid:126)z ) .
Our goal is to construct an ( (cid:15), δ ) -PAC estimator ˆ t ( (cid:126)z ) that tries to minimize ˆ t ( (cid:126)z ) .
Estimator.
Given (cid:15), δ ∈ R > 0 , consider the estimator ˆ t ( (cid:126)z ) =
inf t ∈ R (cid:8) t ∈ R (cid:12)(cid:12) L ( t ; (cid:126)z ) ≤ k (cid:9) + γ ( (cid:126)z ) where k = max (cid:40)
[Equation]
(cid:41) (7) where the empirical loss is L ( t ; (cid:126)z ) =
(cid:80) z ∈ (cid:126)z 1 ( z > t ) , and where γ ( (cid:126)z ) > 0 is an arbitrary positive function.
Intuitively, the empirical loss counts the number of mistakes that t makes on the training data—i.e., z ∈ (cid:126)z such that z (cid:54)∈ ( −∞ , t ] .
To compute the solution k in (7), we start with h = 0 and increment it until it no longer satisﬁes the condition.
To ensure numerical stability, this computation is performed using logarithms.
Note that k does not exist if the set inside the maximum in (7) is empty; in this case, we choose ˆ ψ ( (cid:126)z ) = 0 , which trivially satisﬁes the PAC property.
To compute ˆ t ( (cid:126)z ) , we sort the training examples z 1 , ..., z n by magnitude, so z 1 ≥ z 2 ≥ ... ≥ z n .
Finally, z k + solves the minimization problem in (7), so ˆ t ( (cid:126)z ) =
z k +1 + γ
( (cid:126)z ) .
If k does not exist, then we choose ˆ t ( (cid:126)z ) = ∞ , which trivially satisﬁes the PAC property.
We have the following; see Section H.2 for a proof:
Theorem C.2.
The estimator ˆ t ( (cid:126)z ) in (7) is ( (cid:15), δ ) -PAC.
C.2 A Concentration Bound
Problem formulation.
Consider a Bernoulli distribution p =
Bernoulli ( µ ) with unknown mean µ ∈ [0 , 1] .
Our goal is to compute a lower bound ν ∈ [0 , 1] of µ —i.e., µ ≥ ν .
For example, if µ is the error rate of a classiﬁer, then ν is a lower bound on this rate.
To compute ν , we are given a training set (cid:126)z = ( z 1 , ..., z n ) ∈ { 0 , 1 } n , where z 1 , ..., z n ∼ p are n i.i.d.
samples from p .
An estimator is a mapping ˆ ν : R n → R .
We say ˆ ν is correct if it satisﬁes µ ≥ ˆ ν ( (cid:126)z ) .
We are unable to guarantee that ˆ ν ( (cid:126)z ) is correct due to the randomness in the training examples (cid:126)z .
Thus, we additionally allow for a small probability δ ∈ R > 0 that ˆ ψ ( (cid:126)z ) is not correct—i.e., it is probably correct (PC) .
Algorithm 2 Use learning theory to sketch ¯ P that is approximately correct.
procedure S KETCH ( P, (cid:126)α, δ ) m ← | Φ ??
( P ) | for E ∈ BottomUp (Φ ??
( P )) do if E = φ ( ¯ P (cid:48) , ??)
{ Q } ω(cid:15) then
Compute (cid:126)z (cid:126)α according to (9)
Compute ˆ t ( (cid:126)z (cid:126)α ) according to (7) with ( (cid:15), δ/m )
Fill the hole ??
with ˆ t ( (cid:126)z α ) else if E = φ ( ¯ P (cid:48) , c ) { Q } ω ??
then
Compute (cid:126)z (cid:126)α according to (9)
Compute ˆ ν ( (cid:126)z (cid:126)α ) according to (8) with δ/m
Fill the hole ??
with 1 − ˆ ν
( (cid:126)z (cid:126)α ) end if end for return true end procedure
Deﬁnition C.3.
Given δ ∈ R > 0 , ˆ ν is δ -PC if P p ( (cid:126)z ) (cid:0) µ ≥ ˆ ν ( (cid:126)z ) (cid:1) ≥ 1 − δ .
In other words, ˆ ν ( (cid:126)z ) is correct with probability at least 1 − δ according to the randomness in p ( (cid:126)z ) .
Our goal is to construct an δ -PC estimator ˆ ν ( (cid:126)z ) .
Estimator.
Given δ ∈ R > 0 , consider the estimator
[Equation]
(8) where ˆ µ ( (cid:126)z ) = n − 1 (cid:80) z ∈ (cid:126)z z is an estimate of µ based on the samples (cid:126)z ; we take ˆ ν ( (cid:126)z ) =[-
-]
if (8) is negative.
Intuitively, the second term in ˆ ν ( (cid:126)z ) is a correction to ˆ µ ( (cid:126)z ) to ensure it is ( (cid:15), δ ) -PC, based on Hoeffding’s inequality [16].
We have the following; see Section H.3 for a proof:
Theorem C.4.
The estimator ˆ ν is δ -PC.
C.3 Sketching Algorithm
Problem formulation.
A sketching algorithm A : P × A n → ¯ P takes as input a partial program
P ∈ P , together with a set of test valuations (cid:126)α = ( α 1 , ..., α n ) ∈
A n , where α 1 , ..., α n ∼ p are i.i.d. samples from an underlying distribution p ( α ) .
Then, ¯ P =
A ( P, (cid:126)α ) should be a complete program that is approximately correct by ﬁlling each hole in expressions
φ ( P (cid:48) , ??)
{ Q } ω(cid:15) ∈
Φ c ??
( P ) with a value c ∈ R and each hole in expressions φ ( P (cid:48) , c ) { Q } ω ??
∈ φ (cid:15) ??
( P ) with a value (cid:15) ∈ R > 0 .
We assume that every expression in Φ( P ) has a hole—i.e., Φ( P ) =
Φ ??
( P ) ; otherwise, we cannot guarantee that the existing thresholds in these expressions are approximately sound.
Deﬁnition C.5.
A partial program P ∈ P is a full sketch , denoted P ∈ P 0 , if Φ ??
( P ) =
Φ( P ) .
Then, we say A is correct if A ( P, (cid:126)α ) ∈
¯ P ∗ .
We cannot guarantee this property; instead, given δ ∈ R > 0 , we want it to hold with probability at least 1 − δ according to p ( (cid:126)α ) .
Deﬁnition C.6.
A sketching algorithm A : P 0 × A n → ¯ P is δ -probably approximately correct (PAC) if for all P ∈ P 0 , we have P p ( (cid:126)α
[Table]
Note that this deﬁnition does not include (cid:15) since these values are provide in the given sketch.
Algorithm.
Our sketching algorithm is shown in Algorithm 2.
At a high level, it ﬁlls each hole so that the resulting expressions φ ( ¯ P (cid:48) , c ) { Q } ω(cid:15) are all approximately sound.
The order in which these expressions are processed is important; a expression cannot be processed until all its descendants have been processed.
This order ensures that ¯ P (cid:48) is complete, so it can be evaluated.
In Algorithm 2, the function BottomUp ensures that the expressions in Φ ??
( P ) is processed in such an order.
The algorithm allocates a δ/m probability of failure for each expression, where m = | Φ ??
( P ) | .
Synthesizing c .
We describe how our algorithm synthesizes a threshold c for an expression E = φ ( ¯ P (cid:48) , ??)
{ Q } ω(cid:15) .
Given a single test valuation α ∼ p , consider the values
[Equation]
and
[Equation]
Given c ∈ R , it follows by deﬁnition of (cid:74) · (cid:75) α that
[Equation]
Thus, E is approximately sound for some c ∈ R
if and only if
[Equation]
Given (cid:126)α =
[Equation]
1 , ..., α n ) , where α 1 , ..., [Equation] ∼ p i.i.d.,
[Equation]
is a vector of i.i.d.
samples.
The estimator ˆ t ( (cid:126)z (cid:126)α ) in (7) with parameters ( (cid:15), δ/m ) ensures approximate soundness with high probability—i.e.,
[Equation]
holds with probability at least 1 − δ/m according to p ( (cid:126)α ) .
Synthesizing (cid:15) .
We describe how our algorithm synthesizes a conﬁdence level (cid:15) for an expression
E = φ ( ¯ P (cid:48) , c ) { Q } ω ?? .
Given a single test valuation α ∼ p , consider the values z
[Equation]
and
[Equation]
Note that we compute these values even though the (cid:15) is a hole, since (cid:74) · (cid:75) α and (cid:74) · (cid:75) ∗ α do not depend on (cid:15) .
Also, note that unlike the case of synthesizing c , where z α ∈ R is a score, in this case, z α ∈ { 0 , 1 } is a binary value.
Given (cid:15) ∈ R > 0 , E is (cid:15) -approximately sound for (cid:15)
if and only if
P p
[Equation]
α ) ( z α | z
[Equation]
α ) ≥ 1 − (cid:15) if ω = | or
P p
[Equation]
α ) ( z
[Equation]
⇒ z α ) ≥ 1 − (cid:15) if ω = ⇒ .
Given (cid:126)α = ( α 1 , ..., α n ) , where α 1 , ..., α n ∼ p are i.i.d.
samples, (cid:126)z (cid:126)α deﬁned in (9) is a vector of i.i.d. samples from Bernoulli ( µ ) .
Then, the estimator ˆ ν ( (cid:126)z (cid:126)α ) in (8) with parameter δ/m is a lower bound on µ with high probability—i.e.,
P p
[Equation]
z
[Equation]
≥
[Equation]
ν ( (cid:126)z
[Equation]
) if ω = | or
P p ( α ) ( z
[Equation]
⇒ z α ) [Equation] ˆ ν ( (cid:126)z (cid:126)α ) if ω = ⇒ .
holds with probability at least 1 − δ/m according to p ( (cid:126)α ) .
Thus, it sufﬁces to choose 1 − (cid:15) =
ˆ ν ( (cid:126)z (cid:126)α ) .
The following guarantee follows from Theorems C.2 & C.4 by a union bound over Φ( ¯ P ) :
Theorem C.7.
Algorithm 2 is δ -PAC.
D Synthesis Algorithm
We now describe a syntax-guided synthesizer that uses our sketching algorithm to identify programs with machine learning components while satisfying a desired error guarantee.
In general, to design such a synthesizer, we need to design a space of speciﬁcations along with a domain-speciﬁc language (DSL) of programs.
For clarity, we focus on a speciﬁc set of design choices; as we discuss in
Section E, our approach straightforwardly generalizes in several ways.
We consider the following choices:
• Speciﬁcations: We consider speciﬁcations ˜ ψ
= ( ψ, (cid:15), e ) , consisting of both a traditional part ψ indicating the logical property that the train semantics of the program should satisfy (provided either as a logical formula or input-output examples), and a statistical part ( (cid:15), e ) indicating that the program should have error at most e with probability at least 1 − (cid:15) with respect to p ( α ) , or else return ∅ .
• DSL:
We consider a DSL (shown in Figure 2) of list processing programs where the inputs are images of integers.
Our DSL includes components designed to predict the integer represented by a given image.
These components return the predicted value if its conﬁdence is above a certain threshold, and return ∅ otherwise.
Values ∅ are propagated as ∅ by all components in our DSL—i.e., if any input to a function is ∅ , then its output is also ∅ .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
Algorithm 3 Use learning theory to synthesize ¯ P that is approximately correct.
procedure S YNTHESIZE ( (cid:126)α, ψ, (cid:15), e, N, δ ) ˜
P ← SynthesizePartialSketch ( ψ ) (cid:126)α synth , (cid:126)α sketch ← Split ( (cid:126)α )
P ← arg max P (cid:48) ∈ FillAll ( ˜ P,(cid:15),e ) Score ( Sketch ( P (cid:48) , (cid:126)α synth , δ )) return Sketch ( P, (cid:126)α sketch , δ ) end procedure
For clarity, we refer to speciﬁcations ˜ ψ as task speciﬁcations and speciﬁcations on DSL components as component speciﬁcations .
As a running example, consider the program in Figure 3.
This program predicts the value x of the image input 1 (as an integer) and values (cid:96) of the images in the list input(as real values), and then sums the values in (cid:96) that are greater than equal to x .
It contains three components that have component speciﬁcations: the two machine learning components predict int and predict ﬂoat , along with the inequality cond- ≤ .
The ﬁrst two component speciﬁcations ensure that the corresponding machine learning model returns correctly (or ∅ ) with high probability.
For the last one, note that in the expression y 1 ≤ y 2 , the inputs y 1 and y 2 may have a small amount of prediction error, so if they are to close together (i.e., | y 1 − y 2 | ≤ c for some c ∈ R ≤ 0 ), then y 1 ≤ ymight be incorrect.
Thus, to ensure ≤ returns correctly, cond- ≤ returns ∅ if | y 1 − y 2 | ≤ c .
Finally, note that we use ω = ⇒ , indicating that our goal is to synthesize ¯ P such that the the overall success rate is bounded—i.e., P p ( α ) (cid:0) (cid:74) ¯ P (cid:75)
α
= ∅ ∨ | (cid:74) ¯ P (cid:75)
α − (cid:74) ¯ P (cid:75) ∗ α | > e (cid:1) ≥ 1 − (cid:15) .
We could use ω = | here if we instead wanted to bound the probability of failure conditioned on (cid:74) ¯ P (cid:75) α (cid:54)
= ∅ .
Given labeled training examples (cid:126)α , a task speciﬁcation ˜ ψ , a maximum list length N , and a conﬁdence level δ , our algorithm shown in Algorithm 3 synthesizes a complete program ¯ P that satisﬁes ˜ ψ with probability at least 1 − δ .
At a high level, this algorithm proceeds in three steps:
• Step 1: First, our algorithm uses the logical speciﬁcation ψ to identify a sketch ˜ P whose train semantics is consistent with ψ .
Note that the train semantics for sketches in our DSL in
Figure
are
well-deﬁned
even when the holes left unﬁlled
We refer to ˜ P as a partial sketch , since it has additional holes that cannot be ﬁlled by our sketching algorithm.
• Step 2: While our algorithm uses our sketching algorithm described in Algorithm 2 to ﬁll holes ??
c in ˜ P , it must ﬁrst ﬁll the holes ??
(cid:15) and ??
e (described below), which cannot be handled by this algorithm.
To this end, it analyzes the program to identify constraints on
[Figure]
the values of (cid:15) and e that can be assigned to each hole ??
(cid:15) and ??
e , respectively and satisfy the desired task
speciﬁcation ( (cid:15), e ) .
Given candidate values (cid:126)e and (cid:126)(cid:15) , it constructs the sketch P = Fill ( ˜ P,(cid:126)(cid:15),(cid:126)e ) , and evaluates the success rate Score ( P ) (i
e., how often (cid:74) P (cid:75) α (cid:54) = ∅ ).
It chooses the sketch P that maximizes this objective over a ﬁnite set of choices of (cid:126)(cid:15) and (cid:126)e .
• Step 3: Finally, it uses a held-out set of labeled examples (cid:126)α sketch in conjunction with our sketching algorithm in Algorithm 2 to synthesize We use a held-out set since Theorem C. only holds if the examples (cid:126)α sketch are not used to construct the sketch P .
In Figure 3, we show the partial sketch ˜
P ex along with two analyses which are used to help compute the search space over (cid:126)(cid:15) and (cid:126)e .
Below, we describe our DSL and synthesis algorithm in more detail.
D.1 Domain-Speciﬁc Language
Our DSL is summarized in Figure 2.
To be precise, this ﬁgure shows sketches in our language; ﬁlling holes in these sketches produces a program in our language.
At a high level, the language consists of standard list processing operators such as map, ﬁlter, and fold, along with a set of functions that can be applied to individual integers, real numbers, or images.
Machine learning components.
Our DSL has three machine learning components: predict int , predict ﬂoat , and cond-ﬂip .
The ﬁrst two predict the value in a given image.
They are identical except for their component speciﬁcation; whereas the integer predictions must be exactly correct, the real-valued predictions are allowed to have bounded error.
We describe these speciﬁcations below.
This difference gives the user ﬂexibility in terms of what kind of guarantees they want to provide.
The third machine learning component checks if the input image is ﬂipped along the vertical axis.
We include it to demonstrate how our approach can combine multiple machine learning components.
It only returns an image if it is conﬁdent about its prediction; otherwise, it returns ∅ .
Component speciﬁcations.
Intuitively, there are two kinds of component speciﬁcations in our language: (i) require that the output is exactly correct, and (ii) require that the error of the output is bounded.
There are four components in (i): predict int , cond-ﬂip , cond- ≤ , and cond- ≥ .
The ﬁrst two are straightforward—they consist of a machine learning component, and return the predicted value if the prediction conﬁdence is a threshold to be synthesized, and return ∅ otherwise.
The latter two are result from challenges handling inequalities on real-valued predictions.
In particular, real-valued predictions (i.e., by predict ﬂoat ) can be wrong by a bounded amount, yet the return value of ≤ and ≥ is a Boolean value that must be exactly correct.
Thus, these components include a component speciﬁcation indicating that their output must be correct with high probability.
Note that the scoring function used in the condition is | y 1 − y 2 | ; intuitively, if the inputs y 1 and y 2 are far apart (i.e., | y 1 − y 2 | is large), then the predicted result is less likely to be an error.
The predict ﬂoat component is the only one in (ii).
The only difference from predict int is that it only requires that the prediction is correct to within some bounded amount of error—i.e., | ˆ f ( x ) − y ∗ | ≤ e , for some e ∈ R ≥ 0 .
Note that e is left as a hole to be ﬁlled.
Holes.
Our language has three kinds of holes.
The ﬁrst two are holes ??
c and ??
(cid:15) ; these are in our sketch DSL in Figure 2.
Note that in that DSL, each component speciﬁcation could only have either c or (cid:15) as a hole, but here we allow both to be left as holes; our algorithm searches over choices of (cid:15) to ﬁll holes ??
(cid:15) , and uses our sketching algorithm in Algorithm 2 to ﬁll holes ??
c .
The third kind of hole is the hole ??
e in the component speciﬁcation | ˆ f ( x ) − y ∗ | ≤ ??
e for predict ﬂoat , which indicates the magnitude of error allowed by the prediction of that component.
As with ??
(cid:15) holes, the ??
e holes are ﬁlled by our algorithm before our sketching algorithm is applied.
Intuitively, ??
(cid:15) (resp., ??
e ) holes must be ﬁlled in a way that satisﬁes the overall (cid:15) failure probability guarantee (resp., e error guarantee) in the user-provided task speciﬁcation ˜ ψ .
D.2 Synthesis Algorithm
Our algorithm (Algorithm 3) takes as input labeled training examples (cid:126)α , a task speciﬁcation ˜ ψ = ( ψ, (cid:15), e ) , and δ ∈ R > 0 , and returns a program ¯ P that satisﬁes ˜ ψ with probability ≥ 1 − δ .
Step 1: Syntax-guided synthesis.
Our algorithm ﬁrst synthesizes a partial sketch ˜ P in our DSL whose train semantics satisﬁes ψ
—i.e., UNSAT α,y (cid:0) y = (cid:74) ˜
P (cid:75) ∗ α ∧ ¬ ψ ( α, y ) (cid:1) .
Importantly, note that (cid:74) ˜ P (cid:75) ∗ α is well-deﬁned even though there are holes in ˜ P .
We can compute ˜ P using any standard synthesizer.
Step 2: Sketching (cid:15) and e .
Next, our algorithm ﬁlls the holes ??
(cid:15) in ˜ P with values (cid:126)(cid:15) and holes ??
e with values (cid:126)e to obtain a sketch P = F ILL ( ˜ P,(cid:126)(cid:15),(cid:126)e ) .
Since P only has holes ??
[Equation] , we can use
Algorithm 2 to ﬁll these holes in a way that guarantees correctness for the given values (cid:126)(cid:15) and (cid:126)e —i.e.,
[Equation]
where ¯ P is a completion of P where the holes ??
c in P have been ﬁlled with values (cid:126)c .
We need to use ¯ P since the test semantics are not well-deﬁned for sketches P .
In particular, we need to choose values (cid:126)(cid:15) and (cid:126)e that ensure that (10) holds for all possible completions ¯ P of P .
Furthermore, we not only want to choose (cid:126)(cid:15) and (cid:126)e to ensure correctness, but also to maximize a quantitative property of ¯ P .
In particular, we want to choose it in a way that maximizes the probability that P does not return ∅ —i.e., maximize the score
[Equation]
Note that the score depends critically on the choice of thresholds (cid:126)c used to ﬁll holes ??
c in P .
Thus, given a set of candidate choices (cid:126)(cid:15) and (cid:126)e , our algorithm constructs the corresponding sketch P (cid:48) =
Fill ( ˜ P,(cid:126)(cid:15),(cid:126)e ) , uses our sketching algorithm to ﬁll the holes ??
c in P (cid:48) to obtain ¯ P (cid:48) =
Sketch ( P (cid:48) , (cid:126)α, δ ) , and ﬁnally scores ¯ P (cid:48) .
Then, our algorithm chooses P (cid:48) with the highest score.
In Algorithm 3, we let
FillAll ( ˜ P, (cid:15), e ) denote the set of all sketches P (cid:48) constructed from candidates (cid:126)(cid:15) and (cid:126)e .
One important detail is that Algorithm 2 requires that P is a straight-line program—i.e., it cannot handle loops.
For now, we assume that we are given a bound N ∈ N on the maximum length of any input list.
Then, we can unroll list operations such as map, ﬁlter, and fold into straight-line code.
Algorithm 3 uses this strategy to apply Algorithm 2 to sketches P .
We describe how we can remove the assumption that we have an upper bound N in Section E.
Step 3: Sketching c .
Finally, we use Algorithm 2 to choose values (cid:126)c to ﬁll holes ??
c in the highest scoring sketch P from the previous step, and return the result ¯ P =
Sketch ( P, (cid:126)α sketch , δ ) .
Importantly, in the previous step, P is chosen based on a subset (cid:126)α synth of the training examples (cid:126)α , whereas in this step, ¯ P is constructed based on a disjoint subset (cid:126)α sketch .
We choose these two subsets to be of equal size since Algorithm 2 is sensitive to the number of examples in (cid:126)α .
This strategy ensures that P does not depend on the random variable (cid:126)α sketch , thereby ensuring that Theorem C.7 holds.
D.3 Search Space Over (cid:126)(cid:15) and (cid:126)e
Here, we describe how we choose candidates (cid:126)(cid:15) and (cid:126)e in Step 2 so that the candidate sketches
P (cid:48)
=
Fill ( ˜ P,(cid:126)(cid:15),(cid:126)e ) satisfy (10).
At a high level, for (cid:126)(cid:15) , for each component f of ˜ P with an ??
(cid:15) hole, we compute (cid:74)
˜ P (cid:75) # f , which is the number of times f occurs in the unrolled version of ˜ P ; then, we consider (cid:126)(cid:15) =
( (cid:15) f 1 , ..., (cid:15) f d ) such that
(cid:80) f (cid:15) f ≤ (cid:15) .
For (cid:126)e , for each component f of ˜ P with an ??
e hole, we compute (cid:74) ˜
P (cid:75) err : (cid:126)e (cid:55)→ e (cid:48) , which is a linear function mapping (cid:126)e to an upper bound e (cid:48) on the error of the output; then, we consider (cid:126)e such that (cid:74)
˜
P (cid:75) err ( (cid:126)e ) ≤ e .
We provide details below.
[Table]
[Figure]
Search space over (cid:126)(cid:15) .
First, we describe our search space over parameter values (cid:126)(cid:15) used to ﬁll holes ??
(cid:15) so that the overall failure rate is at most (cid:15) .
Note that here, (cid:126)(cid:15) =
( (cid:15) f 1 , ..., (cid:15) f k ) , where F ˜ P = { f 1 , ..., f k } are subexpressions of ˜ P of the form predict int , predict ﬂoat , cond-ﬂip , cond- ≤ , or cond- ≥ , since each of these subexpressions contains exactly one hole of the form ??
(cid:15) .
Intuitively, we can ensure correctness via a union bound—i.e., if the sum of the (cid:15) f is bounded by (cid:15) , then the overall failure probability is also bounded by (cid:15) .
The key caveat is that to apply Algorithm 2, we need to unroll the sketch P = Fill ( ˜ P,(cid:126)(cid:15),(cid:126)e ) .
Thus, we need to count a value (cid:15) f multiple times if the corresponding subexpression f occurs multiple times in the unrolled version of P .
In particular, the rules (cid:74) P (cid:75)
# f (cid:48) shown in Figure 4 are designed to count the number of occurrences of the subexpression f (cid:48) in the unrolled version of P .
Note that in these rules, f (cid:48) refers to a speciﬁc subexpression, and 1 ( f = f (cid:48) ) refers to whether f is that speciﬁc subexpression; multiple uses of the same construct (e.g., a program with two uses of predict int ) are counted separately.
These rules are straightforward; for instance, when unrolling the fold operator, the expressions for the list L and the initial value B are included exactly once, whereas the function expression F occurs N times.
Then, to ensure that the failure probability is at most (cid:15) , it sufﬁces for (cid:126)(cid:15) to satisfy
[Equation]
Now, let ∆ F ˜ P
= { (cid:126)x ∈ R |F
˜ P | | ∀ f . 0 ≤ x f ≤ 1 ∧ (cid:80) f ∈F ˜ P x f = 1 } be the regular simplex in R |F
˜ P | .
Now, given any (cid:126)x ∈ ∆ F ˜ P , letting (cid:15) f
= x f · (cid:15)/ (cid:74) ˜ P (cid:75) # f , then (11) is satisﬁed.
In our algorithm, we search over a ﬁnite set of points from ∆ F ˜ p , and construct the corresponding set of values (cid:126)(cid:15) .
In Figure 6, the rule for ﬁlter applies f 1 = cond- ≤ and f 2 = predict int each N = 3 times (where N is the given bound on the list length), so we have (cid:74) ˜
P ex (cid:75) # f 1 = (cid:74) ˜
P ex (cid:75) # f 2 = 3 .
Similarly, map applies f 3 = predict ﬂoat a total of N = 3 times, so (cid:74) ˜
P ex (cid:75) # f 1 = 3 .
As an example of a point in our search space, taking (cid:126)x = (1 / 3 , 1 / 3 , 1 / 3) yields (cid:126)(cid:15) =
Search space over (cid:126)e .
Next, we describe our search space over parameter values (cid:126)e used to ﬁll holes ??
e so the overall error is at most e .
Similar to before, (cid:126)e = ( e f 1 , ..., e f h ) , but this time G ˜ P = { f 1 , ..., f h } are subexpressions of ˜ P of the form predict ﬂoat , which each contain exactly one hole of the form ??
e .
In this case, we deﬁne an analysis that bounds the overall error of the output of ¯ P = Fill ( P,(cid:126)(cid:15),(cid:126)e ) for any (cid:126)(cid:15) as a function of (cid:126)e .
More precisely, (cid:74) P (cid:75) err satisﬁes the following property:
[Equation]
[Equation]
for all (cid:126)(cid:15) and (cid:126)e , and for all α such that all component speciﬁcations in Fill ( P,(cid:126)(cid:15),(cid:126)e hold for (cid:126)α .
In other words, (12) bounds the error of the output for examples α such that predictions fall within the desired error bounds (failures happen with probability at most (cid:15) according to our choices of (cid:126)(cid:15) ).
Note that (12) uses the L ∞ norm.
For scalar outputs, we have (cid:107) x − x (cid:48) (cid:107) ∞ =
| x − x (cid:48) | .
For list outputs, for the L ∞ norm to be well-deﬁned, we need to ensure that x = (cid:74) Fill ( P,(cid:126)(cid:15),(cid:126)e ) (cid:75) α and x (cid:48)
= (cid:74) ¯ P (cid:75) ∗ α are of the same length (at least, when all component speciﬁcations are satisﬁed).
In particular, the only potential case where x and x (cid:48) have unequal lengths is if ¯ P contains a ﬁlter operator.
We focus on ﬁltering real-valued lists; ﬁltering integer-valued lists is similar (and there are no operations to ﬁlter list-valued lists or image-valued lists).
In the real-valued case, the ﬁlter function must be either cond- ≤ and cond- ≥ .
Assuming the component speciﬁcations on cond- ≤ and cond- ≥ are satisﬁed, then their (Boolean) outputs are guaranteed to be equal, so the outputs of the ﬁlter operator have equal length under train and test semantics.
Thus, (cid:107) x − x (cid:48) (cid:107) ∞ is well-deﬁned.
Given (cid:74) P (cid:75) err , our goal is to compute (cid:126)e satisfying
[Equation]
As with (cid:126)(cid:15) , we can construct a candidate (cid:126)e for any point in x ∈ ∆ G ˜ P by taking e f = x f · e/a f , where (cid:74) ˜ P (cid:75) err =
(cid:80) f ∈G ˜ P a f · e f .
In Figure 3, we have (cid:74) ˜ P (cid:75) err = 3 · e f 3 , so
there is a single candidate e f 3 = e/ 3 .
Next, we describe the rules (cid:74) P (cid:75) err , which are shown in Figure 4 (right).
They compute an symbolic expression of the form η = (cid:80) f ∈G ˜ P a f · e f ∈ E ˜ P , where a f ∈ R ≥ 0 and e f is a symbol.
Given (cid:126)e , an expression η can be evaluated by substituting (cid:126)e for the symbols e f in η .
Now, the rule for function application assumes given a function abstraction (cid:74) F (cid:75) err :
E ˜ P
→ E ˜ P .
In particular, (cid:74) F (cid:75) err is the identity function except for predict ﬂoat , + , and − .
The case predict ﬂoat follows since we have assumed that the component speciﬁcation holes, and the component speciﬁcation for f = predict ﬂoat says exactly that | (cid:74)
¯ f (cid:75)
α − (cid:74) f
(cid:75) ∗ α | ≤ e f for any completion
¯ f of f .
For + and − , letting η = (cid:80) [Equation]
e f and η (cid:48) = [Equation]
, we deﬁne
[Equation]
The rule for map follows since we are using the L ∞ norm, so the bound is applied elementwise.
The remaining rules are straightforward.
In Figure 3, the rule for predict ﬂoat returns e f 3 , so the rule for map returns 3 · e f 3 (since the given bound on the list length is N = 3 ).
The remaining rules propagate this value, so (cid:74) ˜
P ex (cid:75) err = 3 · e f 3 .
Finally, the fact that (cid:74) · (cid:75) err is a linear function follows by structural induction.
Additional components (e.g., multiplication) can result in nonlinear expressions, but a similar approach applies.
Overall search space.
Our overall search space consists of pairs (cid:126)(cid:15) and (cid:126)e such that (cid:126)(cid:15) satisﬁes (11) and (cid:126)e satisﬁes (13); given such a pair, FillAll ( ˜ P, (cid:15), e ) includes the program P =
Fill ( ˜ P,(cid:126)(cid:15),(cid:126)e ) .
Together, (11) and (13) ensure the desired property (10).
In particular, for any completion ¯ P of P , (13) ensures that | (cid:74) ¯ P (cid:75) α − (cid:74) P (cid:75) ∗ α | ≤ e as long as α satisﬁes all the component speciﬁcations, and (11) ensures that α satisﬁes the component speciﬁcations with probability at least 1 − (cid:15) over p ( α ) .
E Discussion
Generality.
In Section D, we described a synthesizer tailored to the language in Figure 2.
Our approach generalizes straightforwardly in several ways.
First, we note that the predict int and predict ﬂoat machine learning components are not speciﬁc to images of integers, and represent general classiﬁcation and regression problems, respectively.
Furthermore, we can also include additional list processing components as long as we provide the abstract semantics (cid:74) · (cid:75) # and (cid:74) · (cid:75) err .
Thus, our algorithm can be viewed as a general algorithm for synthesizing list processing programs with DNNs for classiﬁcation and regression, where the speciﬁcation is that with high probability, the program should return the either the correct answer (within some given error tolerance) or ∅ .
We can also modify the speciﬁcation in certain ways; for instance, we can ignore certain kinds of errors by modifying the annotations on predict int and predict ﬂoat .
For instance, to allow for one-sided errors in regression problems (e.g., it is ﬁne to say “person” when there isn’t one but not vice versa), we can simply drop the absolute values from the task speciﬁcation ψ and from the annotations on predict ﬂoat .
For this case, the algorithm for allocating errors e works as is, but in general, it may need to be modiﬁed to ensure the annotations imply the speciﬁcation.
Bound on examples.
In Section D, we assumed given a bound N on the maximum length of any list observed during program execution.
Intuitively, we can circumvent this assumption by computing a high probability bound N ; the error probability can be included in the user-provided allowable error rate (cid:15) .
In particular, let (cid:74) [Table] [Table] P p ( α ) ( (cid:74) ¯ P (cid:75) α = ∅ ) , and the “Failure Rate” P p
[Equation]
α [Equation] P (cid:75) α (cid:54) = ∅ ∧ | (cid:74)
[Equation]
P
[Equation] P [Equation] ∅ | ) > e .
α .
Then, suppose we can obtain N such that
[Equation]
Now, if we synthesize a completion ¯ P of ˜ P with overall error rate ≤ (cid:15)/ 2 , then by a union bound, the total error rate is ≤ (cid:15) .
Finally, to obtain such an N , we can use the speciﬁcation
[Equation]
Letting c be the synthesized value used to ﬁll the hole, the speciﬁcation says that (cid:74) ˜ P (cid:75) len α ≤ c with probability at least (cid:15)/ 2 according to p ( α ) , which is exactly the desired condition on N ; thus, we can take N = c .
Note that since the speciﬁcation is true , we can use either | or ⇒ .
F Evaluation
We describe our evaluation on synthesizing list processing programs, as well as on three case studies: i ) a state-of-the-art image classiﬁer, (ii) a random forest trained to predict Warfarin drug dosage, and (iii) object
detection.
F.1 Synthesizing List Processing Programs with Image Classiﬁcation
Experimental setup.
We evaluate our synthesis algorithm on our list processing domain in Section D.
Inputs are lists of MNIST digits [20].
We use a convolutional DNN (two convolutional layers followed by two fully connected layers, with ReLU activations) [21] to predict the integer in an image, trained on the MNIST training set; it achieves 99.2% accuracy.
We also train a single layer DNN, which is 4.04 × faster but only 98.5% accurate.
Finally, for inputs with the ﬂip component, with consider input images ﬂipped along their horizontal axis.
We train a DNN to predict whether a given image is ﬂipped; it achieves 99.6% accuracy.
[Figure]
For the synthesizer, we use a standard enumerative synthesizer that returns the smallest program in terms of depth (but chooses arbitrarily among equal depth programs).
We give it 5 labeled inputoutput examples as a speciﬁcation ψ , along with the type of the function to be synthesized [22, 23].
For the search space over each (cid:126)(cid:15) and (cid:126)e , we consider values (cid:126)x 0 ∈ { 1 , 3 , 5 } d , where d = |F ˜ P | or d = |G ˜ P | , and then take (cid:126)x = (cid:126)x 0 / (cid:107) (cid:126)x 0 (cid:107) 1 to normalize it to ∆ d .
We also compare to (i) a baseline “No Search”, which only considers a single (cid:126)x 0 = (1 , ..., 1) , and (ii) a baseline “ k = 0 ”, which uses a variant of our generalization bound that uses either k = 0 (or k = ∅ , if there are insufﬁcient samples); this strategy captures the guarantees provided by traditional generalization bounds from statistical learning theory [10, 24, 25].
We use our algorithm with parameters (cid:15) =
δ = 0 .
05 , e = 6 , and N = 3 .
We use 2500 MNIST test set images for each α synth and α sketch , and the remainingfor evaluation.
Next, we consider four variants of our DSL:
Int: Restrict to components with integer type and omit the cond-ﬂip component • Float:
Same as “int”, but include components with real types
• Flip:
Same as “int”, but include the ﬂip component • Fast:
Same as “int”, but use the fast neural network.
For each variant, we consider ﬁve list processing tasks, which are designed to exercise different kinds of components.
These programs all take as input a list (cid:96) of images x ∈ (cid:96) ; in addition, several of them take as input a second image k that encodes some information relevant to task.
Then, they output an integer or real value (as speciﬁed by ψ ).
The tasks are shared across the different DSL variants, but speciﬁc programs change based on the available components.
Results.
We show results in Table 2.
For the program ¯ P synthesized using each our approach
S TAT C ODER and our baseline that does not search over (cid:126)(cid:15) and (cid:126)e , we show the following metrics:
• ∅ Rate: The rate at which ¯ P returns ∅ —i.e., P p ( α ) ( (cid:74) ¯ P (cid:75) α = ∅ ) .
• Failure Rate: The rate at which ¯ P makes mistakes—i.e., P p ( α ) (cid:0) (cid:74) ¯ P (cid:75) α (cid:54) = ∅ ∧ | (cid:74) ¯ P (cid:75) α − (cid:74) ¯ P (cid:75) ∗ α
= ∅ | (cid:1) >
e.
As can be seen, both S TAT C ODER and the baseline always achieve the desired failure rate bound of (cid:15)
= 0 .
Furthermore, by searching over candidates (cid:126)(cid:15) and (cid:126)e , S TAT C ODER substantially outperforms the baseline, achieving an 8 × reduction in ∅ rate on average.
For simpler programs (i.e., sum and max), the two perform similarly since there is only a single hole, so the search space only contains one candidate.
However, for larger programs, the search improves performance by up to an order of magnitude.
There is a single case where the baseline performs better (the fourth program in the “ﬂip”
DSL), due to random chance since the dataset α sketch used to synthesize the ﬁnal program ¯ P from ˜ P differs from the dataset α synth used to choose (cid:126)(cid:15) and (cid:126)e .
S TAT C ODER outperforms the “ k = 0 ” baseline by an even larger margin, due to the fact that the generalization bound is overly conservative; these results demonstrate the importance of using a generalization bound specialized to our setting rather than a more traditional generalization bound that minimizes the empirical risk.
Next, in Figure 5, we show how these results vary as a function of the speciﬁcation parameters (cid:15) , δ , and e .
As can be seen, (cid:15) has the largest effect on ∅ and failure rates, followed by e ; as expected, δ has almost no effect since the dependence of our bound on δ is logarithmic.
Finally, we note that the failure rates for the “fast” DSL are very low.
Thus, we could use our technique to chain together the fast program with the slow one, along the same lines as discussed in our case study in Section F.3; we estimate that doing so results in a 3 × speedup on average. [Table]
[Table]
F.2 Synthesizing List Processing Programs with Object Detection
Experimental setup.
Next, we consider synthesizing programs that operate over the predictions made by a state-of-the-art DNN for object detection.
We assume given a DNN component ˆ f that given an image x , is designed to detect people and cars in x .
We use a pretrained state-of-the-art object detector called Faster R-CNN [4] available in PyTorch [11], tailored to the COCO dataset [12], which is a dataset of real-world images containing people, cars, and other objects.
There are multiple variants of Faster R-CNN; we use the most accurate one, X101-FPN with 3 × learning rate schedule.
We represent this DNN as a component ˆ f : X → Y = D ∗ × R , where ˆ f ( x ) =
(ˆ y ( x ) , ˆ p ( x )) consists of a list of detections d ∈ ˆ y ( x ) along with a correctness score ˆ p ( x ) that the prediction is correct.
Each detection d ∈ D
=
R 2 ×
Z is itself a tuple d = ( b, z ) including the position b and predicted category of the object.
The ground truth label y ∗ for an image x is a list of detections d ∈ y ∗ .
In general, we cannot expect to get a perfect match between the predicted bounding boxes and the ground truth ones.
Typically, two detections d, d ∗ match , denoted (cid:107) d − d ∗ (cid:107) ≤ e , where e is a speciﬁed error tolerance, if the distance between their centers satisﬁes (cid:107) b − b ∗ (cid:107) ∞ ≤ e .
Furthermore, we write (cid:107) ˆ y ( x ) − y ∗ (cid:107) ≤ e if | ˆ y ( x ) | = | y ∗ | and there exists a one-to-one correspondence between d ∈ ˆ f ( x ) and d ∗ ∈
y ∗ such that (cid:107) d − d ∗ (cid:107) ≤ e .
Then, we deﬁne predict : X → ( Y ∪ ∅ ) by (
[Equation]
then ˆ y [Equation] else ∅ ) .
In other words, the speciﬁcation says that a correct prediction is if the error tolerance is below a level ??
e to be speciﬁed.
Thus, given e and (cid:15) to ﬁll ??
e and ??
(cid:15) , respectively, our sketching algorithm synthesizes a threshold c to ﬁll ??
c in a way that guarantees that this speciﬁcation holds.
Then, predict returns ˆ y ( x ) if the DNN is sufﬁciently conﬁdent in its prediction, and ∅ otherwise.
We can use this component in conjunction with our synthesis algorithm in the same way that it uses predict ﬂoat .
In particular, we deﬁne the abstract semantics
[Equation]
These semantics enable it to select the error tolerance e to ﬁll ??
e .
The remainder of the synthesis algorithm proceeds as in Section F.1.
We use parameters (cid:15) =
δ = 0 .
2 , e = 20 . 0 , and N = 3 , and use n = 1000 COCO validation set images for each α synth and α sketch and the remaining 1503 for evaluation.
We use larger (cid:15) and δ since the accuracy of the object detector is signiﬁcantly lower than that of the image classiﬁer, so the ∅ rates are very high for smaller choices.
We evaluate our approach on synthesizing ﬁve programs, which include additional list processing components: (i) ( product L L (cid:48) ) , which returns the list of all pairs ( x, x (cid:48) ) such that x ∈ (cid:74) L (cid:75) and x (cid:48) ∈ (cid:74) L (cid:48) (cid:75) , (ii) ( compose f f (cid:48) ) , which returns the composition λx.f
( f (cid:48) ( x )) , (iii) ( is z (cid:48) D ) , which returns 1 ( z = z (cid:48) ) , where (cid:74) D (cid:75)
=
( b, z ) is a detection and z
(cid:48) ∈
Z is an object category, and (iv) ( distance D D (cid:48) ) , which returns the distance (cid:107) b − b (cid:48) (cid:107) ∞ between two detections (cid:74) D (cid:75) =
( b, z ) and (cid:74) D (cid:48) (cid:75)
=
( b (cid:48) , z (cid:48) ) .
Their abstract semantics are straightforward: for (cid:74) ·
(cid:75) # , they each evaluate each of their arguments once, and for (cid:74) · (cid:75) err , the only one that propagates errors is distance , for which
[Equation]
Results.
We provide results in Table 3.
The trends are similar to Section F.1; the main difference is that search does not help in this case, likely because there is only a single machine learning component
[Figure] [Figure]
omit ResNet alone since its running time (82.6 minutes) is signiﬁcantly above the scale.
so optimizing the allocation does not signiﬁcantly affect performance.
Finally, we can chain these programs with a faster object detector to reduce running time; see Section F.5.
F.3 Case Study 1: ImageNet Image Classiﬁcation
Correctness.
Consider program shown in Figure 6, which classiﬁes images as “person” (returns true ) or “not person” (returns false ).
The function is_person takes as input an image x , and optionally the ground truth label y ∗ (which is only used during sketching).
The speciﬁcation in is_person says that the program should return true with high probability if the image is of a person (i.e., y ∗ = 1 ).
The predicate 1 (1 − f ( x ) ≤ c ) is shown in blue, where the value of c has been left as a hole ??
1 , the speciﬁcation y ∗ = 1 is shown in green in the curly braces, and the value (cid:15) = 0 .
05 is shown in green in the square braces.
We perform a case study in the context of this program (though for labels other than “person”).
We consider the ImageNet dataset [19], a large image classiﬁcation benchmark with over one million images in 1000 categories, including various different animals and inanimate objects.
We consider the ResNet-152 DNN architecture [8], a state-of-the-art image classiﬁcation model trained on ImageNet that achieves about 88% accuracy overall.
For both architectures, we use the implementation in PyTorch [11].
To use our system, we split the ImageNet validation set consisting of 50,000 held-out images into (at most) 25,000 for synthesis (i.e., the synthesis set ) and 25,000 for validation.
Because ImageNet has so many labels, each object category has very few examples in the validation dataset (50 on average).
Thus, we group the labels into larger, coarse-grained categories, focusing on ones that correspond to many ﬁne-grained ImageNet labels.
We consider “dog” (130 labels, 6,500 images) “bird” ( labels, 2,950 images), “insect” (27 labels, 1,350 images), “car” (21 labels, 1050 images), “snake” (17 labels, 850 images), and “cat” (13 labels, 650 images).
The default one we use is “car”; this category contains vehicles such as passenger cars, bikes, busses, trolleys, etc.
For the scoring function, given a coarse-grained category Y ⊆
Y , we use the sum of the ﬁne-grained label probabilities—i.e., f ( x ) =
(cid:80)
y ∈
Y p ( x, y ) , where p ( x, y ) is the predicted probability of label y according to ResNet-152.
Then, we use our sketching algorithm to synthesize c to ﬁll ??
We show results in the ﬁrst and fourth rows of Figure 7.
Note that the red curves ideally equal the blue curves, but are slightly conservative to account for synthesis being based on ﬁnitely many samples.
The value of (cid:15) has the biggest effect on performance, since it directly governs recall; as (cid:15) grows, recall drops (as desired) and precision substantially improves.
In contrast, the performance does not vary signiﬁcantly with δ .
These trends match sample complexity guarantees from learning theory relevant to our setting of n =
O (log(1 /δ ) /(cid:15) )
Next, as n grows larger, recall can more closely match the desired maximum, allowing precision to improve dramatically (the non-monotone effect is most likely due to random chance).
Finally, the dependence on the target label is also governed by the number of synthesis images in each category.
Improving speed.
Next, we describe how our framework can be used to compose f with a second
DNN f fast , which is much faster than f but has lower accuracy.
Intuitively, we want to use f fast when we can guarantee its prediction is correct with high probability, and use f otherwise.
This approach has been used to reduce running time [26, 27]; our framework can be used to do so while providing rigorous accuracy guarantees.
The code for this approach is shown in is_person_fast in Figure 6.
As before, the idea is to compute a threshold c (cid:48) such that the prediction f fast ( x ) ≥ 1 − c (cid:48) is correct with high probability.
There are two differences.
First, if we conclude that there might be a person in the image according to f fast , then we return the prediction according to f (instead of true ).
While f fast is guaranteed to detect 95% images with people with high probability, it may have more false positives than f ; calling f after f fast reduces these false positives.
Second, the correctness guarantee is with respect to the prediction ˆ y = {+
+}
( f ( x ) ≥ 1 − c ) rather than y ∗ .
We could use y ∗ , but there is no need—if ˆ y is incorrect, then it is not helpful for f fast to predict correctly since it falls back on ˆ y .
For f fast , we use AlexNet, which achieves about 57% accuracy overall; in particular, we use f fast ( x ) = (cid:80)
y ∈ Y
p fast ( x, y ) , where p fast ( x, y ) is the predicted probability of label y according to AlexNet.
Then, we conclude that x (may) have label y if f fast ( x ) ≥ 1 − c (cid:48) , where c (cid:48) is synthesized by our algorithm.
We obtain results on an Nvidia GeForce RTX 2080 Ti GPU.
We show results on the second and third rows of Figure 7.
All results shown are for the combined predictions (i.e., using both AlexNet and
ResNet), and are estimated on the validation set.
For running time, we omit results for ResNet since its running time is 82 .
6 minutes, which is more than 4 × the running time of our combined model.
For the “dog” category, our approach reduces running time 6 × from 82.6 minutes to 13.8 minutes without any sacriﬁce in precision or recall.
Thus, our approach signiﬁcantly reduces running time while achieving the desired error rate.
Furthermore, comparing to Figure 7 (d), the precision does not signiﬁcantly decrease across most labels.
It does suffer for the labels “car” and “snake”.
Intuitively, for these labels, there are relatively few examples in the synthesis set, so the synthesis algorithm needs to choose more conservative thresholds.
Since the fast program has two thresholds whereas the original program only has one, it is more conservative in the latter case.
This difference is reﬂected in the fact that Figure 7 (e) has higher recall than (d), especially for “car” and “snake”.
Importantly, these results rely on the fact that we are tailoring our predictions to a single category— i.e., our system enables the user to tailor the predictions of pretrained DNN models such as ResNet and AlexNet to their desired task.
For instance, it can focus on predicting cars rather than achieving good performance on all 1000 ImageNet categories.
Runtime monitoring.
As described in Section B.3, our framework can monitor the synthesized program at runtime, which is useful since PAC guarantees are speciﬁc to the data distribution p ( x, y ∗ ) .
Thus, if the program is executed on data from a different distribution, called distribution shift [17, 18], then our guarantees may not hold.
Monitoring requires us to obtain ground truth labels y ∗ for inputs x encountered at run time; then, we use these ground truth labels to estimate the failure rate of the model and ensure it is below the desired value (cid:15) .
We show how we can monitor the correctness of is_person_fast .
In this case, we can easily obtain ground truth labels since the speciﬁcation for ??
2 can be obtained by evaluating f ( x ) .
We want to avoid running f on every input since this would defeat the purpose of using a fast DNN; instead, we might run it once every N iterations for some large N .
The function monitor_correctness implements this check, generating a ground truth label once every N = 100 iterations on average.
Note that we formulate the check as a probabilistic assertion [28]—i.e.,
[Equation]
[Figure] [Figure]
which has the semantics
[Figure]
[Equation]
, which is the speciﬁcation in is_person_fast .
When our framework synthesizes a value c (cid:48) to ﬁll ?? 2 in is_person_fast , it uses the same value to ﬁll ??
2 in monitor_correctness .
Then, at run time, it accumulates pairs ( x, ˆ y ) , where ˆ y =
( f ( x ) ≤ c ) , in calls to monitor_correctness and uses them to check whether the probabilistic assertion in that function is true.
To evaluate whether monitoring can detect shifts, we select two subsets of the “car” category: (i) bikes, including motor bikes, and (ii) passenger cars, excluding busses, trucks, etc., with 6 ﬁne-grained labels each.
Then, we consider a shift from the car category to the bike category—i.e., if we imagine that bikes were instead labeled as cars, would the recall of our program continue to be above the desired threshold.
First, we check whether it proves correctness when the data distribution does not shift—i.e., using the test images labeled “passenger car”.
We run our veriﬁcation algorithm on this property using the test set images labeled As expected, our veriﬁcation algorithm correctly concludes that both the recall and the running time are within the expected bounds.
Then, we check whether it proves correctness when the data distribution shifts—i.e., using the test images labeled “bike”.
In this case, our veriﬁcation algorithm concludes that recall is incorrect, but running time is correct.
Indeed, the average running time is now lower—intuitively, f fast is incorrectly rejecting many “car” images, which reduces recall (undesired) as well as running time (desired).
As a side note, our framework can also be used to monitor quantitative properties.
For instance, we can keep monitor how frequently the branch f fast ( x ) >
c is taken—i.e., avoiding the need to evaluate f ( x ) .
In Figure 6, monitor_running_time includes a probabilistic assertion [Equation]
to perform this check.
This assertion says that 1 − f fast ( x ) > c (cid:48) with probability at least 1 − (cid:15) —i.e., the faster branch in is_person_fast should be taken at least 1 − (cid:15) fraction of the time according to p ( x, y ∗ ) .
We might not know what is a reasonable value of (cid:15) —i.e., the rate at which f fast predicts there is a person in the image.
Thus, we leave it as a hole ??
Given training examples (cid:126)z , our framework can be used to synthesize a value of (cid:15) to ﬁll this hole.
F.4 Case Study 2: Precision Medicine
Warfarin dosing task.
Next, we consider a task from precision medicine.
In particular, we consider a random forest trained to predict dosing level for the Warfarin drug based on individual covariates such as genetic biomarkers [29].
Personalized dosing can improve patient outcomes, but signiﬁcant errors can lead to adverse events if not quickly corrected.
The ideal dosage is a real-valued label.
The goal is to train a model to predict this dosage as a decision support tool for physicians.
For simplicity, we build on an approach that converts the problem into a classiﬁcation problem by discretizing this value into labels Y
= { high , medium , low } dose [30].
Then, the goal is to maximize accuracy while ensuring that very few patients for whom a high dose is predicted but should have been assigned a low dose, and vice versa.
Experimental setup.
We split the dataset (5,528 examples) into training (1,658 examples), synthesis (2,764 examples), and test (1,106 examples) sets.
Then, we use scikit-learn [31] to train a random forest f : X × Y
→ R with 100 trees on the training set, where f ( x, y ) ∈
R is the probability assigned to label y ∈ Y , and use f in conjunction with the program shown in Figure 9.
This program includes two thresholds c low and c high , and only assigns a low dose to a patient with covariates x if f ( x ) ≥ 1 − c low , and similarly for a high dose—i.e., it only assigns the riskier outcomes when f is sufﬁciently conﬁdent in its prediction.
Importantly, the speciﬁcation on c low refers not to the error rate on predictions for patients for whom
y = low, but for whom y = high—i.e., we want to choose c low to ensure precision speciﬁcally on patients for whom
y = high, and conversely for c high .
We use our synthesis algorithm to synthesize values of c low and c high that satisfy these speciﬁcations.
Correctness.
Figure 9 shows the results of our approach (black) compared to directly predicting the highest probability label according to the random forest f (red), always predicting “medium” (green), and the desired error rate (blue), as a function of the maximum error rate (cid:15) , the maximum failure probability δ , and the number of synthesis examples n .
The top plots show the error rate, which is the maximum of the rate at which patients with y = low are assigned a high dose, and the rate at which patients y = high are assigned a low dose; this value should be below the blue line.
The bottom plot shows the overall accuracy of the program—i.e., how often its predicted dose equals the ground truth dose.
All values are estimated on the held-out test set.
As before, (cid:15) has the largest impact on performance since it directly controls the error rate; however, once it hits (cid:15) = 0 .
06 , performance levels off since its accuracy now equals that of f , and the program never assigns a dose not predicted by f .
Performance is ﬂat as a function of δ .
Finally, performance increases quickly as n goes from 500 to 1000 , but plateaus thereafter, again once accuracy equals that of f .
Runtime monitoring.
In the case of Warfarin dosing, the doctor administers an initial dose to the patient (possibly the predicted dose, depending on the doctor’s judgement), and gradually adjusts it based on the patient response.
Thus, we eventually observe the ground truth dose that should have been recommended, which we can use to monitor our program.
This process is achieved by the monitor_correctness subroutine; here, obtain_result returns the true dose eventually observed for a patient with covariates x .
We evaluate whether our runtime monitoring can detect shifts in the data distribution that lead to a reduction in performance.
We consider a shift in terms of the ethnicity of the patients, which has recently been identiﬁed as an important challenge in algorithmic healthcare [32].
In particular, we consider a model trained using non-Hispanic White patients (2, examples), which we refer to as the “majority patients”, and test it on Black, Hispanic, and Asian patients (2,559 examples), which we refer to as the “minority patients”.
First, we check if it proves correctness when the data distribution does not shift—i.e., we train the random forest, and synthesize and verify the program on majority patients.
As expected, it successfully veriﬁes correctness.
Next, we check if it proves correctness when there is a shift—i.e., we train the random forest and synthesize the program on majority patients, but verify the program on minority patients.
As expected, it rejects the program as incorrect.
Finally, recall that whether veriﬁcation is successful depends on how many test examples are provided; thus, we also evaluate how many test examples are needed in this setting.
To make sure we have enough examples, we use all examples in this case.
Then, we ﬁnd that for 2,000 test examples, our veriﬁcation algorithm successfully proves correctness, but for 500, 1,000, or 1,500 test examples, it fails.
Intuitively, the number of test examples needed to verify correctness needs to be more than the number use to synthesize the parameters, or else the synthesized thresholds will be more precise (i.e., closer to their “optimal” value) and the veriﬁcation algorithm will not have enough data to validate them.
In this case, we use 1,000 synthesis examples, so about 2 × as many test examples are needed to verify correctness.
F.5 Case Study 3: Object Detection
Object detection.
We assume given a DNN component f that given an image x , is designed to detect people in x .
Our formulation of object detection in this section is slightly different than the
[Figure] [Figure]
previous setup.
In particular, d ∈ f ( x ) is a list of detections , which is a pair d = ( b, p ) including a bounding box b ∈ R 4 that encodes the center, width, and height of a rectangular region of x , and a value p ∈ [0 , 1] that is the predicted probability that b exists.
In addition, the ground truth label y ∗ for an image x is a list of bounding boxes b ∈ y ∗ .
In general, we cannot expect to get a perfect match between the predicted bounding boxes and the ground truth ones.
Typically, two bounding boxes b, (cid:48) match if have signiﬁcant overlap—in particular, their intersection-over-union satisﬁes IOU ( b, b (cid:48) ) ≥ ρ for some threshold ρ ∈ [0 , 1] ; we use a standard choice of ρ = 0 .
We denote that b and b (cid:48) match in this sense by b ∼ = b (cid:48) .
Finally, b approximately matches a bounding box in y ∗ it b ∼ = b (cid:48) for some b (cid:48) ∈ y ∗ , which we denote by b ˜ ∈
y ∗ .
Experimental setup.
We use a pretrained state-of-the-art object detector called Faster R-CNN [4] available in PyTorch [11], tailored to the COCO dataset [12].
There are multiple variants of Faster
R-CNN; we use the most accurate one, termed X101-FPN with 3 × learning rate schedule.
For each predicted bounding box, this model additionally outputs a predicted object category (e.g., “person”), as well as the size of the bounding box (“small”, “medium”, and “large”).
For most of our evaluation, we use “person” and “large”.
We specify alternative choices when we used them; in particular, we additionally consider 6 of the 91 object categories: “person” (10777 bounding boxes), “car” ( bounding boxes), “truck” (414 bounding boxes), “motorcycle” (367 bounding boxes), “bike” ( bounding boxes), and “bus” (283 bounding boxes).
We split the COCO validation set intosynthesis images and 3000 test images.
Correctness.
Our goal is to detect a majority of people.
In particular, we consider synthesizing a threshold c and selecting all bounding boxes with probability above c —i.e.,
[Equation]
This task is more challenging to specify than our examples so far since f ( x ) is a structured output.
In particular, we are not reasoning about whether f ( x ) is correct with high probability with respect to p ( x, y ∗ ) , but whether bounding boxes ( b, p ) ∈ f
( x ) are correct.
Thus, we need a distribution p ( b | x ) over bounding boxes b in an image x .
Given such a distribution, our goal is to choose c so
[Equation]
where p ( x, y ∗ ) is the data distribution.
Intuitively, this property says that f ( x, c ) contains at least a 1 − (cid:15) fraction of ground truth bounding boxes.
A reasonable choice for p ( b | x ) is the uniform distribution over f ( x, 0) —i.e., the set of all bounding boxes predicted by f .
One issue is when a ground truth bounding box b ∈ y ∗ is completely missing from f ( x, 0) ; in this case, b would not occur in p ( b | x ) , so (14) would not count it as an error even though it is missing from f ( x, c ) for any c .
To handle this case, we simply add ( b, 0) to f ( x ) during synthesis for such bounding boxes b —i.e., f predicts b occurs with probability zero.
The program for achieving this goal is shown in the subroutine detect_ppl in Figure 10.
We use our algorithm in conjunction with the synthesis examples to synthesize the parameter ??
1 for this program, using the default values (cid:15) =
δ = 0 .
and the object category “person”.
In Figure 11, we show the recall (red), desired lower bound on recall (blue), and precision (black) as a function of (a) (cid:15) , (b) δ , and (c) the object category y .
The trends are largely similar to before—e.g., performance varies signiﬁcantly with (cid:15) and the object category, but not very much with δ .
For (c), we use (cid:15) = 0 .to
facilitate comparison to our fast program described below.
Improving speed.
We use a similar approach to improve speed as before—i.e., given a fast object detector f fast , we want to use it to check the image, and only send it to the slow object detector f if necessary.
A challenge compared to image classiﬁcation is that the object detection model does not operate at the level of bounding boxes, which is the level at which we deﬁned correctness, but at the level of images.
Thus, we cannot decide whether we want to run the slow model independently for each detection d ∈ f fast ( x ) ; instead, we have to make such a decision for an image x as a whole.
Intuitively, we check whether the fast model returns any detections in the given image x .
To this end, we compute the maximum score p across all detections ( b, p ) ∈
f fast ( x ) —i.e.,
[Equation]
Then, we want to guarantee that y ∗ = ∅ if this score is below some threshold that ensures that y ∗ = ∅ ; this property is equivalent to its contrapositive
[Equation]
(15) where the right-hand side of the implication is equivalent to f fast ( x ) ≥ 1 − c —i.e., the score is above the threshold 1 − c .
As before, we cannot ensure this property holds with probability one, so instead we use the high-probability variant
[Equation]
This approach is shown in the detect_ppl_fast subroutine in Figure 10.
We note that this approach does not provide guarantees as strong as the ones for image classiﬁcation—in particular, there is a chance that the false negative images x of f fast (i.e., x does not satisfy (15)) will contain larger numbers of ground truth bounding boxes compared to true positive images.
Then, the recall at the level of bounding boxes may be less than 1 − 2 (cid:15) .
However, we ﬁnd that it works well in practice; intuitively, f fast is more likely to have false negative images that contain fewer ground truth bounding boxes.
For f fast , we use a variant of Faster R-CNN termed R50-FPN with 3 × learning rate schedule, which is the fastest variant available.
Then, we synthesize the parameters of ??
1 and ??
2 in Figureusing the synthesis examples.
As before, all results are run on an Nvidia GeForce RTX 2080 Ti
GPU.
In Figure 11, we show the recall (red), desired lower bound on recall (blue), and precision (black) of our approach as a function of (d) (cid:15) , (e) δ , and (f) the object category y .
Similarly, we show the running time (on the entire test set) of the combined program slow + fast (black), fast alone (green), and slow alone (red).
As can be seen, our approach reduces running time by more than 2 × except in the case of “person” (28% reduction) and “truck” (45% reduction).
The person speedup is relatively small because so many of the images in the COCO dataset contain people.
Compared to the image classiﬁcation setting, we obtain a smaller speedup since the gap between the fast and slow models is not as large, and also because we can only avoid using the slow model for images that contain zero detections.
Furthermore, comparing Figure 11 (c) and (f) (i.e., slow alone vs. slow + fast, respectively), for categories “car” and “truck”, we suffer no loss in precision, though we suffer a small loss in precision for the others.
Finally, we note that in Figure 11 (e), for δ = 0 .
15 and δ = 0 .
2 , the estimated recall falls slightly below the desired lower bound on recall.
This result is most likely due to random chance, either because of randomness in the synthesis set or because these values are estimates based on a random test set.
In particular, 0 . 15 is a fairly high failure probability (note that the results across δ are correlated, since we are using the same synthesis and test sets across all δ ).
Runtime monitoring.
We use runtime monitors to check that our program meets the desired bounds both in terms of error rate (the subroutine monitor_correctness in Figure 10) and running time (the subroutine monitor_speed in Figure 10).
These approaches are the same as for image classiﬁcation—the correctness monitor checks that the error rate (i.e., f fast ( x ) concludes there are no detections but f ( x ) (cid:54) = ∅ ) is below the desired rate (cid:15) , and the running time monitor checks that f is not called too often (i.e., f fast ( x ) concludes there are no detections sufﬁciently frequently).
To evaluate these monitors, we consider a shift from the default “large” bounding boxes we use to “small” and “medium”.
Intuitively, the smaller bounding boxes correspond to objects farther in the background, which are harder to detect but also tend to be less important (e.g., an autonomous car may not care as much about detecting far-away pedestrians).
The trends are as before.
First, we ﬁnd that the monitors correctly prove correctness when there is no shift.
Second, we ﬁnd that the running time does not increase due to the shift, so the running time monitor continues to prove correctness.
Finally, our correctness monitor rejects correctness for the shift to “small” bounding boxes; interestingly, it proves correctness for “medium” bounding boxes, which suggests that our synthesized program generalizes to this case.
G Related Work
Synthesizing machine learning programs.
There has been work on synthesizing programs that include DNN components [33, 34, 35, 36, 37] and on synthesizing probabilistic programs [38, 39]; however, they do not provide guarantees on the synthesized program.
There has been work on synthesizing control policies that satisfy provable guarantees [40, 41, 42, 43]; however, they focus on the setting where the learner can interact with the environment, and are not applicable to our supervised learning setting.
Finally, there has been work on synthesizing programs with probabilistic constraints [44], but requires that the search space of programs has ﬁnite VC dimension.
Veriﬁed machine learning.
There has been recent interest in verifying machine learning programs— e.g., verifying robustness [45, 46, 47, 48, 49], fairness [50, 51], and safety [46, 52].
More broadly, there has been work verifying systems such as approximate computing [53, 54, 55, 56] and probabilistic programming [57, 28].
The most closely related work is [58, 59, 60, 61], which verify semantic properties of machine learning models by sampling synthetic inputs from a user-speciﬁed space.
In contrast, our focus is on synthesizing machine learning programs.
Statistical veriﬁcation.
There has been work leveraging statistical bounds to verify stochastic systems [13, 14, 15], probabilistic programs [57, 28], and machine learning programs [51].
Our veriﬁcation algorithm in Section B relies on bounds similar to the ones used in these approaches [13].
To the best of our knowledge, we are the ﬁrst to focus on synthesis; in contrast to veriﬁcation, our approach relies on bounds from learning theory to provide correctness guarantees.
Conformal prediction.
There has been work on conformal prediction [62, 63, 64, 65], including applications of these ideas to deep learning [66, 67, 68, 69], which aim to use statistical techniques to provide guarantees on the predictions of machine learning models.
In particular, they provide conﬁdence sets of outputs that contain the true label with high probability.
Our techniques are inspired by these approaches, extending them to a general framework of synthesizing machine learning programs that satisfy provable guarantees.
H Proofs H.1 Proof of Theorem B.
It sufﬁces to show that if µ < 1 − (cid:15) , then P p ( (cid:126)z ) ( ˆ ψ ( (cid:126)z )) < δ .
First, note that since z 1 , ..., z n are i.i.d.
Bernoulli random variables with mean µ , then 1 − z 1 , ..., 1 − z n are i.i.d.
Bernoulli random variables with mean ν = 1 − µ .
Their sum L ( (cid:126)z ) is a binomial random variable—i.e., L ( (cid:126)z ) ∼ Binomial ( n, ν ) .
Also, note that the condition µ < 1 − (cid:15) is equivalent to ν > (cid:15) .
Thus, we have
[Equation]
≤
[Equation]
where the ﬁrst inequality follows by standard properties of the CDF of the Binomial distribution.
The claim follows.
H.2 Proof of Theorem C.
First, deﬁne
[Equation]
Intuitively, t 0 (cid:15) ∈ R is the threshold that determines whether t is (cid:15) -approximately correct.
In particular, it is clear that t ∈ T (cid:15) for all t > t 0 (cid:15) and t (cid:54)∈ T (cid:15) for all t < t 0 (cid:15) ; in general, t 0 (cid:15) ∈ T (cid:15) may or may not hold.
Thus, it sufﬁces to show
[Equation] ) ≤ t 0 (cid:15) ) <
δ.
To this end, note that the constraint L ( t ; (cid:126)z ) ≤ k in (7) implies (cid:88)
[Equation]
≤ k.
Thus, on event ˆ t ( (cid:126)z ) ≤ t 0 (cid:15) , we have ˆ t ( (cid:126)z ) − γ ( (cid:126)z ) ≤ t 0 (cid:15) − γ ( (cid:126)z ) , so k ≤ (cid:88) z ∈ (cid:126)z 1 ( z > ˆ t ( (cid:126)z ) − γ ( (cid:126)z
[Equation]
≤ (cid:88) z ∈ (cid:126)z
[Equation]
( z > t 0 (cid:15) − γ ( (cid:126)z )) .
As a consequence, we have
[Equation] γ ( (cid:126)z )
[Equation]
≥ k (cid:33) .
Next, since t 0 (cid:15) − γ ( (cid:126)z ) < t 0 (cid:15) , we have t 0 (cid:15) − γ ( (cid:126)z ) (cid:54)∈ T (cid:15) —i.e., (cid:15) <
P p ( z ) ( z > t 0 (cid:15) − γ ( (cid:126)z ))
=
E p ( z ) ( 1 ( z > t 0 (cid:15) − γ ( (cid:126)z ))) .
In other words, the random variables 1 ( z > t 0 (cid:15) − γ ( (cid:126)z )) for z ∈ (cid:126)z are i.i.d.
Bernoulli random variables with mean ν > (cid:15) .
Thus, we have
[Equation]
where the ﬁrst inequality follows by standard properties of the CDF of the Binomial distribution.
The claim follows.
H.3 Proof of Theorem C.
First, we have the following classical inequality [16]:
Theorem H.1.
(Hoeffding’s inequality)
We have
[Equation]
Now, letting t = (cid:113) log(1 /δ )
2 n , we have
[Equation]
where the second-to-last inequality follows from Theorem H.1.
The claim follows.