Outsourcing Training without Uploading Data via Efﬁcient Collaborative Open-Source Sampling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
As deep learning blooms with growing demand for computation and data resources,outsourcing model training to a powerful cloud server becomes an attractive alternative to training at a low-power and cost-effective end device.
Traditional outsourcing requires uploading device data to the cloud server, which can be infeasible inmany real-world applications due to the often sensitive nature of the collected dataand the limited communication bandwidth.
To tackle these challenges, we proposeto leverage widely available open-source data , which is a massive dataset collectedfrom public and heterogeneous sources (e.g., Internet images).
We develop a novelstrategy called Efﬁcient Collaborative Open-source Sampling (ECOS) to constructa proximal proxy dataset from open-source data for cloud training, in lieu of clientdata.
ECOS probes open-source data on the cloud server to sense the distributionof client data via a communication- and computation-efﬁcient sampling process,which only communicates a few compressed public features and client scalar responses.
Extensive empirical studies show that the proposed ECOS improves thequality of automated client labeling, model compression, and label outsourcingwhen applied in various learning scenarios.
Source codes will be released.
Introduction
Nowadays, powerful machine learning services are essential in many devices that supports our dailyroutines.
Delivering such services is typically done through client devices that are power-efﬁcientand thus very restricted in computing capacity.
The client devices can collect data through built-insensors and make predictions by machine learning models.
However, their stringent computingpower often makes the local training prohibitive, especially for high-capacity deep models.
Onewidely adopted solution is to outsource the cumbersome training to cloud servers equipped with massive computational power, using machine-learning-as-a-service (MLaaS).
Amazon Sagemaker [22],
Google ML Engine [6] and Microsoft Azure ML Studio [4] are among the most successful industrialadoptions, where users upload training data to designated cloud storage, and the optimized machinelearning engines then handle the training.
One major challenge of the outsourcing solution in manyapplications is that the local data collected are sensitive and protected by regulations, thereforeprohibiting data sharing.
Notable examples include General Data Protection Regulation (GDPR)
[1]and Health Insurance Portability and Accountability Act (HIPPA)
On the other hand, recent years witnessed a surging amount of general-purpose and massive datasetsauthorized for public use, such as ImageNet [11], CelebA [24], and MIMIC [19].
Moreover, manytask-speciﬁc datasets used by local clients can be well considered as biased subsets of these large public datasets [32, 21].
Therefore, the availability of these datasets allows us to use them to modelconﬁdential local data, facilitating training outsourcing without directly sharing the local data.
Oneapproach is to use the private client dataset to craft pseudo labels for a public dataset in a conﬁdentialmanner [44, 31], assuming that the public and local data are identically-and-independently-distributed( iid ).
In addition, Alon et al. showed that an iid public data can strongly supplement client learning,which greatly reduces the private sample complexity [3].
However, the iid assumption can often be toostrong for general-purpose open-source datasets, since they are usually collected from heterogeneoussources with distributional biases from varying environments.
For example, a search of ‘digits’ onlineyields digits images from handwriting scans, photos, to artwork of digits.
In this paper, we relax the iid assumption in training outsourcing and instead consider the availabilityof an open-source dataset.
We ﬁrst study the gap between the iid data and the heterogeneous opensource data in training outsourcing, and show the low sample efﬁciency of open-source data.
We showthat in order to effectively train a model from open-source data that is transferable to the client data,the open-source data needs to communicate more samples than those of iid data.
The main reasonbehind such low sample efﬁciency is that we accidentally included out-of-distribution (OoD) samples,which poison the training and signiﬁcantly degrade accuracy at the target (client) data distribution [5].
We propose a novel framework called Efﬁcient Collaborative Open-source Sampling (ECOS) totackle this challenge, which ﬁlters the open-source dataset through an efﬁcient collaboration betweenthe client and server and does not require client data to be shared.
During the collaboration, the serversends compressed representative features (centroids) of the open-source dataset to the client.
Theclient then identiﬁes and excludes OoD centroids and returns their privately computed categoricalscores to the server.
The server then adaptively and diversely decompresses the neighbors of theselected centroids.
The main idea is illustrated in Fig.
Our major contributions are summarized as follows:
• New problem and insight :
Motivated by the strong demands for efﬁcient and conﬁdential outsourcing, using public data in place of the client data is an attractive solution.
However, the impact ofheterogeneous sources of the public data, namely open-source data, is rarely studied in existing works.
Our empirical study shows the potential challenges due to such heterogeneity.•
New sampling paradigm : We propose a new uniﬁed sampling paradigm, where the server onlysends very few query data to the client and requests very few responses that efﬁciently and privatelyguide the cloud for various learning settings on open-source data.
To our best knowledge, our methodenables efﬁcient cloud outsourcing under the most practical assumption of open-source public data,and does not require accessing raw client data or executing cumbersome local training.• Compelling results : In all three practical learning scenarios, our method improves the modelaccuracy with pseudo, manual or pre-trained supervisions.
Besides, our method shows competitiveefﬁciency in terms of both communication and computation.
Related Work
There are a series of efforts studying how to leverage the data and computation resources on thecloud to assist client model training, especially when client data cannot be shared [42, 38].
Wecategorize them as follows: 1) Feature sharing : Methods like group knowledge transfer [16], splitlearning [37] and domain adaptation [13, 12] transfer edge knowledge by communicating featuresextracted by networks.
To provide a theoretical guarantee of privacy protection, [28] proposed anadvanced information removal to disentangle sensitive attributes from shared features.
In the notionof rigorous privacy deﬁnition, Liu et al.
leverage public data to assist private information release [23].
Earlier, data encryption was used for outsourcing which however is too computation-intensive for aclient and less applicable for large-scale data and deep networks [8, 20].
Federated Learning (FL)considered the same constraint on data sharing but introduced the burdens of training to clients [26].2) Private labeling: PATE and its variants were proposed to generate client-approximated labels forunlabeled public data, on which a model can be trained [30, 31].
Without training multiple modelsby clients, Private kNN was a more efﬁcient alternative which explored the private neighborhood ofpublic images for labeling [44].
These approaches are based on a strong assumption of the availabilityof public data that is iid as the local data.
This paper considers a more practical yet challengingsetting where public data are from multiple agnostic sources with heterogeneous features.
Sampling from public data has been explored in central settings.
For example, Xu et al.
[41] used afew target-domain samples as a seed dataset to ﬁlter the open-domain datasets by positive-unlabeledlearning [25].
In self-supervised contrastive learning, model-aware K -center (MAK) used a modelpre-trained on the seed dataset to ﬁnd desired-class samples from open-world dataset [18].
Thoughthese methods provided effective sampling, they are less applicable when the seed dataset is placedat the low-energy edge, because the private seed data at the edge cannot be shared with the cloudfor ﬁltering and the edge device is incapable of computation-intensive training.
To address thesechallenges, we develop a new sampling strategy requiring only light-weight computation at the edge.
Outsourcing Model Training With Open-Source Data
3.1 Problem Setting and Challenges
[Table]
Motivated in Section 1, we aim to outsource the training process from computation-constraineddevices to the powerful cloud server, where a proxy public dataset without privacy concerns isused in place of the client dataset for cloud training.
One solution is (private) client labeling byk-nearest-neighbors (kNN) [44], where the client and cloud server communicate the pseudo-label ofa public dataset privately and the server trains a classiﬁer by the labeled and unlabeled samples in asemi-supervised manner.
The success of this strategy depends on the key assumption that public datain the cloud and private data in the client are iid , which is rather strong in practice and thus prevents itfrom many real-world applications.
In this work, we make a more realistic assumption that the publicdatasets are as accessible as open-source data.
The open-source data consists of biased features frommultiple heterogeneous sources (feature domains), and therefore includes not only in-distribution(ID) samples similar to the client data but also multi-domain OoD samples.
[Table]
The immediate question is how the OoD samples affect the outsourced training.
In Table 1, weempirically study the problem by using a 5-domain dataset, Digits, where 50% of one domain is usedon the client and the remained 50% together with the other 4 domains serve as the public dataset onthe cloud.
To conduct the cloud training, we leverage the client data to generate pseudo labels for theunlabeled public samples, following [44].
It turns out that the presence of OoD samples in the cloudgreatly degrades the model accuracy.
The inherent reason for the degradation is that the distributionalshift of data [33] compromised the transferability of the model to the client data [40].
Problem formulation by sampling principles .
Given a client dataset D p and an open-source dataset
D q , the goal of open-source sampling is to ﬁnd a proper subset S from D q , whose distribution matches
D p .
In [18], Model-Aware K-center (MAK) formulated the sampling as a principled optimization:
[Equation]
where  ( S, D p ) :
=
E x 0 2 D p [min x 2 S k x  x 0 k 2 ] measures proximity as the set difference between
S and D p , and the latter H ( S [ D p ; D q ) :
= max x 0 2 D q min x 2 S [ D p k x  x 0 k 2 measures diversityby contradicting S [ D p and D q (suppose D q is the most diverse set).
Solving Eq.
(1) resultsin an NP-hard problem that is intractable [9], and MAK provides an approximated solution by acoordinate-wise greedy strategy.
It ﬁrst pre-trains the model representations on D p and ﬁnds a largecandidate set with the best proximity to extracted features.
Then, it incrementally selects the mostdiverse samples from the candidate set until the sampling budget is used up.
Though MAK is successful in the central setting, it is not applicable when D p is isolated from cloudopen-source data and is located at a resource-constrained client for two reasons: 1) Communicationinefﬁciency .
Uploading client data may result in privacy leakage, sending public data to the client is adirect alternative but the cost can be prohibitive.
2) Computation inefﬁciency .
Pre-training a modelon D p or proximal sampling (which computes the distances between paired samples from D q and
D p ) induces unaffordable computation overheads for the low-energy client.
3.2 Proposed Solution: Efﬁcient Collaborative Open-Source Sampling (ECOS)
To address the above challenges, we design a new strategy that 1) uses compressed queries to reducethe communication and computation overhead and 2) uses a novel principled objective to effectivelysample from open-source data with the client responses of the compressed queries.
Construct communication-efﬁcient and an informative query set ˆ D q at cloud .
Let d be thenumber of pixels of an image, the communication overhead of transmitting D q to the client is givenby O
( d | D q | ) .
For communication efﬁciency, we optimize the following two factors:
i) Data dimension d .
First, we transmit extracted features instead of images to reduce the commu-
nication overhead to O ( d e | D q | ) where d e is a much smaller embedding dimension.
For accurate
estimation of the distance  , a discriminative feature space is essential.
Depending on resources, one
may consider hand-crafted features such as HOG [10]
or an off-the-shelf pre-trained model such asResNet pre-trained on ImageNet.ii) Data size | D q | .
Even with compression, sending all data for querying is inefﬁcient due to the large
size of open-source data | D q | .
Meanwhile, too many queries would cast unacceptable privacy costs
to the client.
As querying on similar samples leads to redundant information in querying, we propose
to reduce such redundancy by selecting informative samples.
We use the classic clustering method
KMeans [15] for compressing similar samples by clustering them, and collect the R mean vectors or
centroids into ˆ D q = { c r } Rr =
We denote R as the compression size .
The selected centroids can bedecompressed by the cluster assignment into corresponding original samples with rich features.
New sampling objective .
We propose to optimize a communication-efﬁcient surrogate loss:
[Equation]
where ˆ S (or ˆ D q ) is the compressed centroid substitute of S (or D q ).
ˆ D p consists of the featuresof D p .
Different from Eq.
(1), we decompose the proximity term into two in order to facilitate communication efﬁciency leveraging ˆ D q .
We solve the optimization problem in a greedy manner bytwo steps at the client and the cloud, respectively:i)
At the client step , we optimize  ( ˆ S, ˆ D p ) to ﬁnd a set of centroids ( ˆ S ⇢ ˆ D q ) that are proximal tothe client set ˆ D p .
Noticing that ˆ D q contains the cluster centroids, we take advantage of the propertyto deﬁne a novel proximity measure of the cluster r : Centroid Coverage (CC), denoted as v r .
Uponreceiving centroids from the cloud, the client uses them to partition the local data into {C pr } Rr =1 where
C pr denotes the r -th cluster partition of local data.
We compute the CC score by the cardinality ofthe neighbor samples of the centroid r , i.e., v r =
|C pr | .
To augment the sensitivity to the proximalclusters, we scale the CC score by a non-linear function v 0 r =
s ( |C pr | ) , where the scale functions ( x ) =
x s is parameterized by s .ii)
At the cloud step , we optimize the proximity of S w.r.t.
the proxy set ˆ S , i.e.,  ( ˆ S, S ) , and removeredundant and irrelevant samples from the candidate set to encourage diversity, i.e.,  H ( S ; D q ) .
Assamples among clusters are already diversiﬁed by KMeans, we only need to promote the in-clusterdiversity .
To this end, we reduce the sample redundancy per cluster at cloud by K-Center [34], whichheuristically ﬁnd the most diverse samples.
Such design transfers the diversity operation to cloudand thus reduces the local computation overhead.
To maintain the proximity , the K-Center is appliedwithin each cloud cluster and the sampling budget per cluster is proportional to their vote numbersand the original cluster sizes.
With the normalized scores, we compute the sampling budget percluster which is upper bounded by the ratio of the cluster in the cloud set.
Algorithm
Input: Client
We summarize our algorithm in Algorithm 1, which enables the clients to enjoy better computationefﬁciency than local training and better communication efﬁciency than the centralized sampling (e.g.,
MAK).
1) Computation efﬁciency .
Since our method only requires inference operations on the clientdevice, which should be efﬁciently designed for the standard predictive functions of the device, and istraining-free for the client, the major complexity of ECOS is on computing centroid coverage and ismuch lower than gradient-based algorithms whose complexity scales with the model size and trainingiterations.
As computing the CC scores only requires the nearest centroid estimation and ranking, theﬁltering can be efﬁciently done.
The total time complexity is O ( C  | D p | +( d e +1)
R | D p | ) , dominatedby the ﬁrst term, where C  is the complexity of extracting features depending on the speciﬁc method.
The second term ( d e + 1) R | D p | is for computing the pair-wise distances between ˆ D p and ˆ D q andestimating the nearest centroids per sample (or partitioning client data).
In a brief comparison,the complexity of local T -iteration gradient-descent training could be approximately O
( TC  |
D p | ) which is much more expensive since typically
TC
d e .
To complete the analysis, the spacecomplexity is O ( C 0  + ( d + d e ) | D p | + d e R + | D p | R ) for the memory footprint of  , the images( d ) or features ( d e ) of client and cloud centroid data, and the distance matrix.
2) Communicationefﬁciency .
The downloading complexity will be O ( d e R ) for R d e -dimensional centroid featuresand the uploading complexity is O
( R ) including indexes of samples.
Thus, the data that will becommunicated between the client and cloud is approximately of O ( d e R + R ) complexity in total.
Incomparison, downloading the whole open-source dataset by central sampling (e.g., MAK) requires
O ( d | D q | + B ) complexity.
As R < d e R ⌧ d e | D q | ⌧ d | D q | , our method can signiﬁcantly scaledown the computation cost.
Privacy protection and accountant .
When the cloud server is compromised by an attacker, uploading
CC scores leak private information of the local data samples, for example, the presence of anidentity [35].
To mitigate the privacy risk, we protect the uploaded scores by a Gaussian noisemechanism, i.e., ˜ v r = v r + N (0 ,  2 ) , and account for the privacy cost in the notion of differentialprivacy (DP) [14].
DP quantiﬁes the numerical inﬂuence of the absence of a private sample on[ v 1 , · · · , v R ] , which is connected to the chance of exposing the sample to the attacker.
To obtain atight bound on the privacy cost, we utilize the tool of Rényi Differential Privacy (RDP) [27] andleverage the Poisson sampling to further amplify the privacy [43].
With the noise mechanism governedby  , the resultant privacy cost in the sense of ( ✏ ,  ) -DP can be accounted as ✏ =
O (  p
[Equation]
)where  is the Poisson subsampling rate and  is a user-speciﬁed parameter.
A larger ✏ implies higherrisks of privacy leakage in the probability of  .
Formal proofs can be found in Appendix A.3.
Empirical Results
Datasets.
We use datasets from two tasks: digit recognition and object recognition.
Distinct from priorwork [44], in our work, the open-source data contains samples out of the client’s distribution.
Withthe same classes as the client dataset, we assume open-source data are from different environmentsand therefore include different feature distributions, for example, DomainNet [32] and Digits [21].
DomainNet includes large-sized 244 ⇥ 244 everyday images on 6 domains and Digits consists of28 ⇥ 28 digit images on 5 domains.
Instead of using an overly large volume of data from a singledomain like [44], we tailor each-domain subset to contain fewer images than standard digit datasets,for instance, MNIST with 7438 images, which was previously adopted in the distributed learningsetting [21] and mitigates the hardness of collecting enormous data.
In practice, collecting tens ofthousands of images by a single client, e.g., 50000 images from MNIST domain, will be unrealistic.
Similarly, DomainNet will be tailored to only include 10 classes with 2000-5000 images per domain.
Splits of client and cloud datasets .
For Digits, we use one domain for the client and the rest domainsfor the cloud as open-source set.
For DomainNet, we randomly select 50% samples from one domainfor the client and leave the rest samples together with all other domains to the cloud.
The differenceof conﬁgurations for the two datasets is caused by their different domain gaps.
Even without ID data,it is possible for Digits to transfer the knowledge across domains.
Baselines.
For a fair comparison, we compare our method to baselines with the same samplingbudget.
Each experiment case is repeated for three times with seeds { 1 , 2 , 3 } .
We account for theprivacy cost by Poisson-subsampling RDP [43] and translate the cost to the general privacy notion,( ✏ ,  ) -DP when  = 10  5 .
Here, we use the random sampling as a naive baseline.
We also adopta coreset selection method, K-Center [34], to select informative samples within the limited budget.
Both baselines are perfectly private without accessing private information from clients.
Details ofhyper-parameters are deferred to Appendix B.1.
4.1 Evaluations on Training Outsourcing
To demonstrate the general applicability of ECOS, we present three practical use cases of outsourcing,categorized by the form of supervisions.
The conceptual illustrations are in Fig.
Per use case, wetrain a model on partially labeled cloud data (outsourced training) and accuracy on the client test setare reported together with standard deviations.
We present the results in Tables 2 to 4 case by case,
[Figure] [Figure]
[Table]
where we vary the domain of the client by columns.
In each column, we highlight the best resultunless the difference is not statistically signiﬁcant.
Case 1: Selective manual labeling .
We assume that the cloud will label the ﬁltered in-domainsamples by using a third-party label service, e.g.,
Amazon Mechanical Turk [29], or by asking theend clients for manual labeling.
As the selected samples are non-private, they can be freely sharedwith a third parity.
But the high cost of manual labeling or service is the pain point, which should becarefully constrained within a ﬁnite budget of demanded labels.
Given a speciﬁed sampling budget, we compare the test accuracy of semi-supervised learning(FixMatch) on sampled data in Table 2.
Since the ECOS tends to select in-distribution samples, itease the transfer of cloud-trained model to the client data.
On the Digits dataset, we ﬁnd that ourmethod attains more accuracy gains as budget increases, demonstrating that more effective labelsare selected.
On the DomainNet dataset, our method outperforms baselines on 5 out of 6 domainsand is stable in most domains given a small budget (1000) and is superior on average.
Given higherbudgets, the accuracy of all methods is improved, when our method is outstanding on Infograph and
Quickdraw domains and is comparable to the baselines on other domains
[Table]
Sampling
Clipart
Infograph
Painting
[Table]
Case 2: Adaptive model compression .
Due to the large volume of the open-source dataset, alarger model is desired for better capturing the various features, which however is so inefﬁcient toﬁt into the resource-constrained client devices or specialize for the data distribution of the client.
Confronting this challenge, model compression [7] is a conventional idea to forge a memory-efﬁcientmodel by transferring knowledge from large models to small ones.
Speciﬁcally, we ﬁrst pre-train alarge teacher model f t on all cloud data by the supervised learning, assuming labels are availablein advance.
Still, we use an ImageNet-pre-trained model to initialize the feature extractor  ( · ) of astudent model f s .
Then the client will use the downloaded feature extractor  to ﬁlter samples.
Here,we utilize knowledge distillation [17] to ﬁnetune the f s with an additional classiﬁer head upon the  .
On the selected samples, we train a linear classiﬁer head for 30 epochs under the supervision of truelabels and the teacher model f t , and then ﬁne-tune the full network f s for 500 epochs.
The majorchallenge comes from distributional biases between the multi-source open-source data and the clientdata.
Leveraging ECOS, we may sample data near the client distribution and reduce the bias in thefollow-up compression process.
We simulate the case on the large-sized image dataset, DomainNet, which is demanding for largescale networks, e.g., ResNet50 here, to effectively learn the complicated features.
Here, we compress
ResNet50 into a smaller network, ResNet18, by using an adaptively selected subset of the clouddataset.
We omit the experiment for digit images where a large model may not be necessary forsuch a small image size.
In Table 3, we present the test accuracy of compressed ResNet18 using3000 samples from DomainNet in ﬁnetuning.
With a small portion of privacy cost ( ✏ < 0 . 6 ), ourmethod improves the accuracy on Clipart and Real domains against the baselines.
Note that themodel accuracy here is lower than label outsourcing in Table 2, and reason is that the supervisionsfrom the larger models are just an approximation of the full dataset.
Without using the full dataset forcompression, the training can be completed fast and responsively on the demand of a client
[Table] [Table]
Case 3: Automated client labeling .
When the client obtained labeled samples, for example, photoslabeled by phone users, the cloud only needs to collect an unlabeled public dataset.
Therefore, wemay automate the labeling process leveraging the client supervision knowledge to reduce cost orusers’ efforts on manual labeling.
To be speciﬁc, we let the client generate pseudo labels for thecloud data based on their neighbor relation, as previously studied by [44] (private kNN).
However,the private kNN assumes that the client and the cloud follow the same distribution, which weakens itsapplicability confronting the heterogeneity and the large scale of open-source data.
Thus, we utilize
ECOS as a pre-processing of open-source data before being labeled by private kNN.
Therefore, wehave two rounds of communication for transferring client knowledge: proximal-data sampling by the
ECOS and client labeling by the private kNN [44].
To compose the privacy costs from these two steps,we utilize the analytical moment accountant technique to get a tight privacy bound [39].
Interestedreaders can refer to Appendix A.1 for a brief introduction to private kNN and our implementations.
In experiments, we use the state-of-the-art private pseudo-labeling method, private kNN [44], tolabel the subsampled open-source data with the assistance from the labeled client dataset.
To reducethe sensitivity of private kNN w.r.t.
the threshold, we instead enforce the number of the selectedlabels to be 600 and balance the sizes by selecting the top-60 samples with the highest conﬁdenceper class.
Then, we adopt the popular semi-supervised learning method, FixMatch [36], to train theclassiﬁer.
In Table 4, we report the results when cloud features are distributionally biased from theclient ones but they share the same class set.
For each domain choice of client data, we will use the other domains as the cloud dataset.
Distinct from prior studies, e.g., in [44] or [31], we assume80-90% of the cloud data are out of the client distribution and are heterogeneously aggregated fromdifferent domains, casting greater challenges in learning.
On the Digits, we eliminate all ID data fromthe cloud set to harden the task.
Both on Digits and DomainNet datasets, our method consistentlyoutperforms the two sampling baselines under the similar privacy costs.
The variance of privacy costsis mainly resulted from the actual sampling sizes.
Though simply adopting K-Center outperformsthe random sampling, it still presents larger gaps compared to our method in multiple domains.
Forinstance, in Quickdraw domain, given 5000 sampling budget, the K-Center method performs poorlyand is even worse than a random sampling.
4.2 Qualitative Study
[Figure]
To better understand the proposed method, we conduct a seriesof qualitative studies on client labeling.
We use two DomainNetdatasets as a representative benchmark in the studies.
1) Ablation study .
In Table 5, we conduct ablation studies to evaluatethe effect of different objectives introduced by ECOS, following the client labeling benchmark with a 3000 sample budget.
Without proximity and diversity objectives, we let the baselinebe the naive random sampling.
We ﬁrst include the proximityobjective, where we greedily select samples from top-scoredclusters until the budget is fulﬁlled.
However, we ﬁnd that thenaive proximity objective results in a quite negative effect compared to the random baseline.
The failure can be attributed tothe nature of clustering that will include more similar samples,namely lacking diversity.
When diversity is encouraged andcombined with the proximal votes, we ﬁnd the performanceis improved signiﬁcantly in multiple domains and on average.
Now with diversity, the proximityobjective can further improve the sampling in Painting, Quickdraw, and Sketch domains signiﬁcantly.2) Visualize cluster selection .
In Fig.
3, we demonstrate that the CC can effectively reject OoDcentroids.
Also, it is interesting to observe that when multiple centroids are distributed closely, thenthey will compete with each other and reject the redundant ones consequently.
3) Efﬁciency .
In
Section 3, we studied the communication efﬁciency theoretically.
Empirically, ECOS only need toupload 100 bytes of the CC scores in all experiments, while traditional outsourcing needs to uploadthe dataset, which is about 198MB for the lowest load in DomainNet (50% of Sketch domain data forclient).
Even counting the download load, ECOS only needs to download 45MB of the pre-trained
ResNet18 feature extractor together with 51KB data of centroid features, which is much less thandata uploading.
More detailed evaluation of the efﬁciency is placed at Appendix B.4.
[Table]
Conclusion
In this paper, we explore the possibility of outsourcing model training without access to the clientdata.
To reconcile the data shortage from the target domain, we propose to ﬁnd a surrogate datasetfrom the source agnostic public dataset.
We ﬁnd that the heterogeneity of the open-source data greatlycompromises the performance of trained models.
To tackle this practical challenge, we propose acollaborative sampling solution, ECOS, that can efﬁciently and effectively ﬁlter open-source samplesand thus beneﬁts follow-up learning tasks.
We envision this work as a milestone for the private andefﬁcient outsourcing from low-power and cost-effective end devices.
More discussions on the socialimpacts and limitations of this work are enclosed in Appendix A.4.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] Will releaseupon acceptance.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
a) If your work uses existing assets, did you cite the creators?
[Yes]
Did you mention the license of the assets?
[Yes]
) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Did you describe any potential participant risks, with links to Institutional Review
Board
) approvals, if applicable?
[N/A]
Did you include the estimated hourly wage paid to participants and the total amount
spent
on participant compensation?
[N/A]