H OW D ID THE M ODEL C HANGE ?
E FFICIENTLY
A S
SESSING M ACHINE L EARNING API S HIFTS
Anonymous authors
Paper under double-blind review
A BSTRACT
ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications.
A challenge for users is that such APIs continuously change over time as the providers update models, and changes can happen silently without users knowing.
It is thus important to monitor when and how much the ML
APIs’ performance shifts.
To provide detailed change assessment, we model ML
API shifts as confusion matrix differences, and propose a principled algorithmic framework, MASA, to provably assess these shifts efﬁciently given a sample budget constraint.
MASA employs an upper-conﬁdence bound based approach to adaptively determine on which data point to query the ML API to estimate shifts.
Empirically, we observe signiﬁcant ML API shifts from 2020 to 2021 among 12 out of 36 applications using commercial APIs from Google, Microsoft, Amazon, and other providers.
These real-world shifts include both improvements and reductions in accuracy.
Extensive experiments show that MASA can estimate such API shifts more accurately than standard approaches given the same budget.
I NTRODUCTION
Machine learning (ML) prediction APIs have made it dramatically easier to build ML applications.
For example, one can use Microsoft text API (Mic, a) to determine the polarity of a text review written by a customer, or Google speech API (Goo, b) to recognize users’ spoken commands received by a smart home device.
These APIs have been gaining in popularity (MLa; Chen et al., 2020), as they eliminate the need for ML users to train their own models.
Monitoring and assessing the performance of these third-party ML APIs over time, however, is under-explored.
ML API providers continuously collect new data or change their model architectures (Qi et al., 2020) to update their services, which could silently help or harm downstream applications’ performance.
For example, as shown in Figure 1 (a) and (b), we observe a 7% overall accuracy drop of IBM speech API on the AUDIOMNST dataset in March 2021 compared March 2020.
In our systematic study of 36 API and dataset combinations, there are 12 cases where the API’s performance changed by more than 1% on the same dataset from 2020 to 2021 (sometimes for the worse).
Such performance shifts are of serious concern not only because of potential disruptions to downstream tasks, but also because consistency is often required for audits and oversight.
Therefore, it is important to precisely assess shifts in an API’s predictions over time.
Moreover, in this assessment, it is often more informative to quantify how the entire confusion matrix of the API has changed rather than just the overall accuracy.
In the IBM case in Figure 1, it is interesting that a major culprit of the drop in performance is the 2021 model mistaking “four” for “ﬁve”.
In other settings, changes in the confusion matrix could still cause issues even if the overall accuracy stays the same.
In this paper, we formalize the problem of assessing API shifts as estimating changes in the confusion matrix on the same dataset in a sample-efﬁcient manner (i.e., with few calls to the API itself to minimize dollar cost).
The straightforward approach is to compare the API’s prediction on randomly sampled data.
However, this can require a large number of API calls to estimate the confusion matrix, which is expensive given that each call costs money.
To help address this challenge, we propose
MASA, a principled algorithm for ML API shift assessments.
MASA efﬁciently estimates shifts in the API’s confusion matrix by clustering the dataset and adaptively sampling data from different clusters to query the API.
MASA automates its sampling rate from different data clusters based on the uncertainty in the confusion matrix estimation.
For example, MASA may query the ML API on
[Figure]
more samples with the true label “four” than “one”, if it is less sure about the estimated performance change on the former.
Employing an upper-conﬁdence-bound approach to estimate the uncertainties,
MASA enjoys a low computation and space cost as well as a fast estimation error rate guarantee.
MASA’s adaptive sampling substantially improves the quality of estimation for API shifts.
In extensive experiments on real world ML APIs, MASA’s assessment error is often an order of magnitude smaller than that of standard uniform sampling with same sample size (e.g., Figure 1 (c)).
To reach the same tolerable target estimation error, MASA can reduce the required sample size by more than 50%, sometimes up to 90%.
Contributions.
In short, our main contributions include:
We demonstrate that commercial Ml APIs can experience signiﬁcant performance shifts over time, and formulate ML API shift assessments via confusion matrix difference estimation as an important practical problem.
We will release the dataset consisting of 1,224,278 samples annotated by ML APIs in different date to stimulate more research on ML API shifts.
We propose MASA, an algorithm to assess the ML API performance shifts efﬁciently.
MASA adaptively determines querying the ML API on which data points to minimize the shift estimation error under a sample size constraint.
We show that MASA enjoys a low computation cost and performance guarantees.
We evaluate MASA on real world APIs from Google, Microsoft, Amazon and other providers for tasks including speech recognition, sentiment analysis, and facial emotion recognition.
MASA leads to estimation errors an order
of magnitude smaller than standard uniform sampling
using the same sample size, or over 90% fewer samples to reach the same tolerable estimation error.
Related Work.
Distribution shifts in ML deployments: Performance shifts in ML systems have been observed in applications like disease diagnosis (Lipton et al., 2018), facial recognition (Wang et al., 2020), and Inference of molecular structure (Koh et al., 2020).
Most of them are attributed to distribution shifts, i.e., the distribution of the test and training datasets are different.
Distribution shifts are usually modeled as covariate shifts (Shimodaira, 2000; Sugiyama et al., 2007;
QuiñoneroCandela et al., 2009) or label shifts (Lipton et al., 2018;
Saerens et al., 2002;
Azizzadenesheli et al., 2019; Zhao et al., 2021).
API shifts are orthogonal to distribution shifts: instead of attributing the performance shifts to data distribution changes, API shifts concern with ML APIs changes which changes its predictions on the same dataset.
The methods for detecting distribution drifts typically rely on changes in data feature statistics and can not detect changes in the API on the same data.
To the best of our knowledge, MASA is the ﬁrst work to systematically investigate ML API shifts.
Deploying and monitoring ML APIs: Several issues in deployed ML APIs have been studied.
For example, Buolamwini & Gebru (2018) showed that strong biases toward minority can exist in
[Figure]
commercial APIs and Ribeiro et al.
(2020) revealed that several bugs in commercial APIs can be detected using checklists.
Kang et al.
(2020) extend program assertions to monitor and improve deployed ML models.
Chen et al.
(2020) consider the trade-offs between accuracy performance and cost via exploiting multiple APIs.
On the other hand, the proposed MASA focuses on estimating (silent) API performance changes cheaply and accurately, which has not been studied before.
Stratiﬁed sampling and multi-arm bandits: Stratiﬁed sampling has proved to be useful in various domains, such as approximate query processing (Chaudhuri et al., 2007), population mean estimation (Carpentier & Munos, 2011; Carpentier et al., 2015), and complex integration problems (Leprêtre et al., 2017).
A common approach is to model stratiﬁed sampling as a multi-arm bandit (MAB) problem: view each data partition as an arm, and set the regret as the variance of the obtained estimator.
Plenty of algorithms exist for solving standard MAB problems, such as the epsilon greedy algorithm
(Slivkins, 2019), upper-conﬁdence-bound approach (Auer et al., 2002), and Thompson sampling method (Russo et al., 2018).
While those algorithms aim at playing the best arm as often as possible, stratiﬁed sampling’s goal is often to estimate the average of all arms, and thus needs to allocate the number of calls for all arms aptly based on their variance.
In contrast to estimating the population average in standard stratiﬁed sampling, our goal is to simultaneously estimate a matrix whose entries can be correlated.
Thus, we have an objective (uncertainty score) that is different from theirs (variance), which also leads to an optimal allocation different from that of standard stratiﬁed sampling.
This difference requires additional statistical bounds to prove convergence, which we provide in our paper.
It also results in a different upper-conﬁdence-bound term and convergence rate compared to standard bandit algorithms.
T HE API S HIFT P ROBLEM
Empirical assessment of ML API shifts.
We start by making an interesting observation: Commercial ML APIs’ performance can change substantial over time on the same datasets .
We investigated twelve standard datasets across three different tasks, namely, YELP (Dat, c), IMDB (Maas et al.),
WAIMAI (Dat, b), SHOP (Dat, a) for sentiment analysis, FER+ (Goodfellow et al., 2015), RAFDB (Li et al.), EXPW (Zhang et al.), AFNET (Mollahosseini et al., 2019) for facial emotion recognition, and DIGIT (Dat, d), AMNIST (Becker et al., 2018), CMD (Warden, 2018), FLUENT (Lugosch et al.), for speech recognition.
For each dataset, we evaluated three commercial ML APIs’ accuracy in April 2020 and April 2021.
Figure 2 summarizes the overall accuracy changes.
There are several interesting empirical ﬁndings.
First, API performance changes are quite common.
In fact, as shown in Figure 2, API performance changes exceeding 1% occurred in about 33% of all (36) considered ML API-dataset combinations.
Since the data distribution remains ﬁxed, such a change is due to ML APIs’ updates.
Second, the API updates can either help or hurt the accuracy performance depending on the datasets.
For example, as shown in Figure 2 (a), the Amazon sentiment analysis
API’s accuracy increases on YELP, WAIMAI, and SHOP, but decreases on IMDB.
In addition, the update of Microsoft facial emotion recognition API only affects performance on the FER+ dataset, as
[Figure]
shown in Figure 2 (b). Another interesting ﬁnding is that the magnitude of the performance change can be quite different.
In fact, most of the accuracy differences are between 1–3%, but on DIGIT dataset, Google’s accuracy change is more than 20%.
Fine-grained assessment of API shift as changes in the confusion matrix.
Based on feedback from practitioners, accuracy change alone is insufﬁcient, and attribution to per class change is often much more informative (Tsipras et al., 2020;
Hou et al., 2019; Luque et al., 2019).
Thus, a natural idea is to quantify an ML API’s performance by its confusion matrix.
We assess the change of the confusion matrix over time as a measure of API shift.
Formally, consider an ML service for a classiﬁcation task with L labels.
For a data point x from some domain X , let ˆ y ( x ) ∈
[ L ] denote its predicted label on x , and y ( x ) be the true label.
For example, for sentiment analysis, x is a text paragraph, and the task is to predict if the polarity of x is positive or negative.
Here L = 2 , and ˆ y ( x ) = 1 implies positive predicted label while y ( x ) = 2 indicates negative true label.
The confusion matrix is denoted by CCC ∈
R L × L where
CCC i,j (cid:44) Pr[ y ( x ) =
i, ˆ y ( x ) =
j ] .
Given a confusion matrix of the ML API measured previously (say, a few months ago), CCC o , the ML API shift is deﬁned as ∆ CCC (cid:44) CCC − CCC O .
Using confusion matrix difference to quantify the ML API shift is informative.
E.g, the overall accuracy change is simply the trace of ∆ CCC .
It also explains which label gets harder or easier for the updated API.
Still consider, e.g., sentiment analysis.
Given a 2% overall accuracy change, ∆ CCC 1 , 2 = 1% and ∆ CCC 2 , 1 = − 3% implies that the change is due to predicting less (-3%) negative texts as positive, by sacriﬁcing the accuracy on positive texts slightly (1%).
This suggests that the
API could have been updated with more negative training texts.
MASA: ML API S HIFT A SSESSMENT
Now we present MASA, an algorithmic framework efﬁciently to assess ML API shifts.
Suppose the old confusion matrix CCC o and a large labeled dataset D are available.
Given a query budget N , our goal is to generate ∆ˆ CCC , an estimation of the API shifts as accurately as possible by querying the ML
API ˆ y ( · ) on N samples drawn from D .
MASA achieves its goal via an adaptive sampling approach (Figure 3).
It ﬁrst divides the given dataset D into several partitions (clusters).
Then it adaptively decides which sample to query the ML
API in an iterative manner: at each iteration, it selects one data partition based on some uncertainty measure (deﬁned below), and queries the ML API on one sample randomly drawn from this partition.
The API’s prediction is obtained to update the uncertainty measure as well as the estimated shift ∆ˆ CCC .
This process is repeated until the ML API has been queried N times or if a stopping rule is reached.
We explain each step in detail next.
3. D ATA P ARTITIONING
A key intuition in MASA is that not all samples are equally informative for estimating API shifts.
Consider, for example, a vision API makes perfect predictions on “dog” images, and guesses randomly on ”cat“ pictures.
The “dog” images are less informative, as even a small sample of “dog” queries would tell that there is essentially no confusion for this class.
Intuitively, within a sample budget, an estimator with more samples from “cat” pictures should be more accurate overall than that from “dog”.
Generally, harder images tend to be more informative.
Thus, it is a natural idea to partition all data points based on factors that may correlate with their informativeness, and sample from those partitions separately.
In MASA, we use partitions D i,k that each contain the points with true label i and difﬁculty level k .
The difﬁculty level is an integer indicating how hard it is to predict the data point’s label.
It needs not be perfect, and can be simply the discretized prediction conﬁdence generated by some simple ML models.
A total of L labels and
K distinct difﬁculty labels lead to a total of LK partitions.
If the uncertainty or variability of the ML
API’s prediction on each partition is different, then drawing a different number of samples from each partition may improve the shift assessment performance compared to standard uniform sampling.
We verify this empirically in our evaluation (Section 4).
3. B UDGET A LLOCATION P ROBLEM
Given the data partition, two questions arise: (i) how many samples should be drawn from each partition, and (ii) how to estimate the ML API shifts given available samples.
The second question is relatively straightforward.
Note that the API shifts satisfy
[Equation]
where the ﬁrst equation is by deﬁnition, the second is due to total probability rule, the third uses the fact that x ∈ D
i,k implies y ( x ) =
i , and the last equation applies conditional probability.
Here,
Pr[ x ∈ D
i,k ] is simply ratio of size of partition D
i,k and entire dataset D , known a prior.
To assess ∆ CCC i,j , we only need to estimate Pr[ˆ y ( x ) =
j | x ∈ D i,k ] , the predicted label distribution on partition
D i,k .
It can be estimated simply via the frequency of predicting label j among all available samples drawn from D i,k .
Now we consider the sample allocation problem.
For ease of notation, we denote Pr[ x i ∈
D
i,j ] by ppp i,k , Pr[ˆ y ( x ) =
j | x ∈ D i,k ] and its estimation by µµµ i,k,j and ˆ µµµ i,k,j , respectively.
Then for deterministic sample allocations, the squared Frobenius norm error can be written as
[Equation]
Thus we use the loss L ( A , N ) (cid:44)
[Equation] (cid:1) 2 to measure the performance of any sample budget allocation algorithm
A using N samples.
For any ﬁxed N , our goal is to ﬁnd a sample budget allocation algorithm
A to minimize the loss L ( A, N ) .
Notably, we can generalize it for other scenarios by replacing (∆ CCC − ∆ˆ CCC ) with WWW (cid:12) (∆ CCC − ∆ˆ CCC ) , where (cid:12) is element-wise multiplication and WWW is an L × L weight matrix.
Different choices of WWW can penalize the error of each entry in ∆ˆ CCC differently and serve for different purposes.
For example, if misclassifying label 1 as label 2 is the only focus, then we can simply set WWW 1 , 2 = 1 and WWW i,j = 0 ,
∀
( i, j ) (cid:54) =
If we use the identity matrix as the weight WWW , then it becomes equivalent to estimating the overall accuracy by minimizing the variance.
To minimize the loss for the general scenarios, Algorithm 1 (which is explained in the rest of this section) is still applicable by simply multiplying the API selection formula (line 3 in
Algorithm 1) with its corresponding weights.
3. U NCERTAINTY S CORE AND O PTIMAL A LLOCATION
The optimal sample allocation is directly connected to how informative each data partition is.
To see this, let us ﬁrst introduce the notation of uncertainty score for each data partition.
Input : ML API ˆ y ( · ) , query budget N , partitions D i,k ,
ppp ∈ R L × K , CCC o ∈ R L × L , and a >
Output : Estimated ML API Shift ∆ˆ CCC ∈ R L ×
ˆ L 1 Set NNN =
L × K , ˆ µµµ =
L × K × L ,
ˆ σσσ =
L × K ,HHH =
L × K × L (cid:46) Initialization 2 for n ← 1 to N do ( i ∗ , k ∗ ) ←  
( i, k
[Equation]
, if
[Equation]
(cid:19) , o/w (cid:46) Determine data partition
Sample x n from D i ∗ ,k ∗ and query the ML API to obtain ˆ y ( x n )
[Table]
[Equation]
(cid:46) Update sample size
[Table]
[Equation]
, ∀ j ∈ [ L ] (cid:46) Update predicted label distribution
[Equation]
←  
[Equation]
if NNN
[Equation]
NNN
[Equation]
, o
[Equation]
w (cid:46) Update uncertainty score
[Table]
[Equation]
) +(cid:46) Update label frequency 9 end 10 Return ∆ˆ
CCC ∈ R L × L where ∆ˆ CCC i,j = (cid:80)
Kk =1 ppp i,k ˆ µµµ i,k,j − CCC oi,j , ∀ i, j (cid:46) Confusion estimation
Deﬁnition 1.
σσσ 2 i,k (cid:44) (1 − (cid:80)
Lj =1 Pr 2 [ˆ y ( x ) =
j | x ∈ D i,k ]) denotes the uncertainty score of D i,k .
The uncertainty score quantiﬁes how informative each D i,k is by subtracting from 1 the sum of the square of each label’s probability mass.
The uncertainty score is related to collision entropy (discussed in Appendix A), and determines the optimal allocation as follows.
Lemma 1.
Let A ∗ be the sample allocation algorithm that achieves the smallest expected squared
Frobenius norm error.
Then the number of samples drawn from D i,k by A ∗ is
[Equation]
Lemma 1 shows that the optimal budget allocation depends on the uncertainty score, but in practice, we do not know the uncertainty score before drawing samples and querying the ML API.
Thus, a natural question is how to estimate the uncertainty score σσσ 2 i,k .
Suppose n samples, x 1 , x 2 , · · · , x n , are drawn from partition D
i,k .
Then we can estimate σσσ 2 i,k by ˆ σσσ 2 i,k (cid:44) 1 − n ( n − 1) n (cid:88) s = n (cid:88) t : t =1 ,t (cid:54) =
s 1 ˆ y ( x s )
=
ˆ y ( x t ) (3.1)
3. A N U NCERTAINTY - AWARE A DAPTIVE S AMPLING A LGORITHM
Now we have a chicken-and-egg problem: estimating the uncertainty scores is needed to ﬁnd the optimal sample allocation, but sampling from all partitions is needed to estimate their uncertainty scores.
To overcome this issue, we adopt an iterative sampling approach, as shown in Algorithm 1.
At each iteration, it alternates between (i) uncertainty score-based new sample selection (line- 4) and (ii) uncertainty score and predicted label distribution update using the new sample (line- 8).
After querying the ML API N times, the API shifts are obtained by (iii) fusing the estimated predicted label distribution on each partition (line 10).
We give the details as follows.
Uncertainty score and predicted label distribution update.
After obtaining the predicted label for a sample from partition D i ∗ ,k ∗ , we need to update (i) the number of samples already drawn from this partition, denoted by N N N i ∗ ,k ∗ , (ii) the estimated predicted label distribution, denoted by ˆ µµµ i ∗ ,k ∗ ,j , ∀ j , and (iii) the estimated uncertainty score, ˆ σσσ 2 i ∗ ,k ∗ .
For NNN i ∗ ,k ∗ and ˆ µµµ i ∗ ,k ∗ ,j (line 5-6), we use standard incremental update approach (Cotton, 1975), which requires constant space and computational cost per iteration.
For ˆ σσσ 2 i ∗ ,k ∗ , we additionally maintain the number of label j being predicted among all samples drawn from D i ∗ ,k ∗ , denoted by HHH i ∗ ,k ∗ ,j (line 8).
This enables a fast incremental update of ˆ σσσ 2 i ∗ ,k ∗ (line 7).
Uncertainty score-based new sample selection.
To determine on which partition to select a new sample, we use an upper-conﬁdence-bound approach on the weighted uncertainty score (second case in line 3), after ensuring two samples have been drawn from each partition (ﬁrst case in line 3).
Two samples are needed for an initial estimation of each partition’s uncertainty score.
Here, we use a parameter a > 0 to balance between exploiting knowledge of uncertainty score ( ˆ σσσ 2 i,k ) and exploring more partitions ( 4 (cid:113) 1 NNN i,k ).
We quantify the performance of MASA v.s.
the optimal allocation algorithm
A ∗ as follows.
Theorem 2.
If a > 2 log L + log K + 94 log N and N > 4 LK , then we have
[Equation]
Roughly speaking, Theorem 2 shows that the loss gap between the API shift estimated by MASA and the (unreachable) optimal allocation algorithm ceases in the rate of N − 5 / 4 .
Note that the loss of the optimal allocation decays in the rate of N − 1 .
Thus, as N gets larger and larger, the relative gap becomes more and more negligible.
E XPERIMENTS
We apply MASA to estimate the shifts of several real world ML services for various tasks.
Our goal is three-fold: (i) understand if and why MASA assess the API shifts efﬁciently, (ii) examine how much sample cost MASA can reduce compared to standard sampling, and (iii) exploit the trade-offs between estimation accuracy and query cost achieved by MASA.
We also study how the hyperparameters affect MASA’s performance, left to Appendix C.
Tasks, ML APIs, and datasets.
As shown in Section 2, we have observed 12 of 36 cases where there is a >1% overall accuracy performance change of an ML API.
Thus, we focus on evaluating
MASA’s performance on those 12 cases.
Except for case study, all experiments were averaged over 1500 runs.
In all tasks, we created partitions using difﬁculty levels induced by a cheap open source model from GitHub.
More details are in Appendix C.
Sentiment analysis: a case study on Amazon API.
We start by a case study on Amazon API on a sentiment analysis dataset, YELP to understand MASA’s performance.
We adopt MASA with sample budget 2000.
The dataset is divided into 4 partitions D + ,l , D + ,h , D − ,l , D − ,h , depending on whether the true label is positive (+) or negative (-), and quality score produced by the 2020 version is lower (l) or higher (h) than the median.
We ﬁrst note that the API shift gives an interesting explanation to Amazon API’s accuracy change.
In fact, as shown in Figure 4 (a-c), the accuracy increase is mostly because more texts (2.7%) with negative altitudes are correctly classiﬁed.
One possible explanation is that the API has been retrained on a dataset with more negative texts.
Next, we observe that MASA produces accurate estimation of the API shift by comparing Figure 4 (c) and (d).
This is primarily due to (i) that the data partitioning separates more uncertain data from less uncertain ones, and (ii) that adaptive sampling learns the uncertainty level effectively.
Figure 4 (e-f) shows that while the partitions’ size is similar, their uncertainty scores are different.
For example, higher quality score implies a much smaller uncertainty for positive texts ( D + ,h and D + ,l in Figure 4(f)).
As shown in Figure 4 (g), MASA indeed learns to utilize such imbalanced uncertainty: its sampling allocation for each partition is quite close to the optimal allocation (dark star point).
Finally, it is worth noting that MASA outperforms standard uniform sampling notably, as shown in Figure 4(h).
This is because uniform sampling does not exploit the uncertainty of each partition.
Budget savings achieved by MASA.
In many applications, it sufﬁces to obtain an estimated API shift close to the true shift, e.g., within a 1% Frobenius norm error.
Thus, a natural question arises: to reach the same tolerable estimation error, how much sampling cost can MASA reduce compared to standard sampling approaches?
To answer this question, we compare MASA with two natural approaches: (i) uniform sampling and (ii) stratiﬁed sampling by drawing same number of samples for each true label.
For each approach,
[Figure]
i t i
[Figure]
t i ve
[Figure] [Figure]
i t i ve
[Figure]
t i ve
[Figure]
[Figure] [Figure]
) CM
[Figure]
i t i ve
[Figure]
t i ve
[Figure] [Figure]
i t i
[Figure]
ve
[Figure] [Figure]
[Figure] [Figure]
[Table]
we measure the number of samples needed to reach 1% Frobenius norm error with probability 95%, via an upper bound on the estimated Frobenius error.
The details are left to Appendix C. As shown in
Table 1, MASA usually requires more than 70% fewer samples to reach such tolerable Frobenius norm error than the uniform and stratiﬁed sampling, primarily due to its shift estimation is more accurate.
Uniform and stratiﬁed sampling required the similar number of samples because the upper bounds on their estimated Frobenius error are similar.
Trade-offs between estimation error and query budget .
Next we examine the trade-offs between API shift estimation error and sample size achieved by MASA, shown in Figure 5.
We ﬁrst note that, across all 12 observed API shifts, MASA consistently outperforms standard uniform sampling for any ﬁxed sample size.
In fact, the achieved estimation error of MASA is usually an order of magnitude smaller than that of uniform sampling.
This veriﬁes that MASA can provide
[Figure]
more accurate assessments of API shifts in diverse applications.
Second, some API shifts are easier to estimate than others.
For example, for Google API shift on AMNIST, using 1000 samples already gives an expected squared Frobenius norm error lower than 10 − 4 , while it usually requiressamples for other shifts.
This is probably because the skew in its uncertainties among different partitions is more severe than other shifts
[Figure]
C ONCLUSION
In this paper
[Figure]
we identify and formulate the problem of characterizing ML API shifts.
Our systematic empirical study has shown that API model updates are frequent, and that some updates can reduce performance substantially.
Quantifying such shifts is an important but understudied problem that can greatly affect the reliability of applications using ML-as-a-service.
To capture ﬁne-grained change assessment, we model the ML API shifts as confusion matrix differences.
Adaptive sampling methods are typically designed for estimating a single scalar (e.g., adaptive population mean) or training a model (e.g., active learning), and thus not directly applicable for API shift estimation.
Uniform sampling is natural but requires a large number of samples.
Thus, we propose an algorithmic framework, MASA, to adaptively assess the API shifts using as few samples as possible.
Our work focuses on estimating changes in the confusion matrix because the confusion matrix is often what is used by practitioners to assess API performance.
We acknowledge that confusion matrices are most applicable for classiﬁcation tasks, and other measures need to be used for more complex APIs (e.g.
OCR, NLP).
While this is a limitation, classiﬁcation with a small to moderate number of classes of interest is a common use case for ML APIs, and this is an important starting point since it has not been studied before.
We will release our dataset of 1,224,278 samples annotated by commercial APIs in different dates as the ﬁrst dataset and resource for studying ML API performance shifts.