A Deep Learning Dataloader with Shared Data Preparation
Author(s)
AffiliationAddress
Abstract
Parallelly executing multiple training jobs on overlapped datasets is a commonpractice in developing deep learning models.
By default, each of the parallel jobsprepares (i.e., loads and preprocesses) the data independently, causing redundantconsumption of I/O and computations.
Although a centralized cache componentcan reduce the redundancies by reusing the data preparation work, each job’srandom data shuffling results in a low sampling locality causing heavy cachethrashing.
Prior work tries to improve the sampling locality by enforcing all thetraining jobs loading the same dataset in the same order and pace.
However, sucha solution is only efficient under strong constraints: all jobs are trained on thesame dataset with the same starting moment and training speed.
In this paper, wepropose a new data loading method for efficiently training parallel DNNs with muchflexible constraints.
Our method is still highly efficient when different trainingjobs use different but overlapped datasets and have different starting momentsand training speeds.
To achieve this, we propose a dependent sampling algorithm(DSA) and a domain-specific cache policy.
Moreover, a novel tree data structure isdesigned to efficiently implement DSA.
Based on the proposed technologies, weimplemented a prototype, named J OADER , which can share data preparation workas long as the datasets are overlapped for different training jobs.
We evaluate theproposed J OADER , showing a greater versatility and superiority of training speedimprovement (up to 200% in ResNet18) without affecting the accuracy.
Introduction
The rapid development of deep learning frameworks and tools [3, 22] has enabled and facilitatedresearchers and practitioners in a multitude of disciplines (e g., physics [28], chemistry
[18],biology [12] and Earth science [23] in addition to computer science) to start developing
their own
DNN models solving various problems at hand.
In such cases, they usually need to simultaneouslyrun multiple training jobs on the same dataset or overlapped datasets 1 , due to the purpose of modelselection [15], hyper-parameter tuning [6], network architecture searching [25], etc.
Although widely used, the training of DNNs is usually time-consuming and tricky, which may affectthe development efficiency, especially for general users from fields other than artificial intelligence.
In literature, various approaches have been proposed to reduce the DNN training time such asdata optimization [4], communication overhead reduction [10, 9, 29], GPU memory optimization [8, 16, 26], and compiler-based operator optimization [7, 13, 17]
Different from the above existing work, our focus is on the data preparation aspect, or morespecifically, the data loading aspect, in the context of parallel DNN training.
On the one hand,
Mohan et al.
[19] have observed that data preparation, which consists of data loading and thesubsequent data preprocessing, occupies a significant portion of parallel DNN training time.
This isdue to the fact that multiple training jobs on the same data copy essentially create redundant datapreparation efforts.
Ideally, to reduce the load of hardware and boost the training efficiency, theprepared data should be cached in memory and serve for multiple jobs.
However, the cache maythrash with a high probability as each job adopts a fully random shuffling strategy that independentlyshuffles the dataset in each epoch.
On the other hand, it is until recently have some studies investigatedthe shuffling process itself [20, 19, 21].
These studies investigate to what extent the fully randomshuffling is required and have observed that various shuffling strategies breaking the full randomnessstill yield competitive results.
We name such randomness as correlated randomness as opposed tothe full randomness.
For example, prior work [19, 20] proposed to solve the cache thrashing issue byenforcing that all jobs iterate over the dataset in the same order and pace, and observed no accuracyloss.
However, such a solution maximizes its performance under some strong constraints, i.e., alltraining jobs have to start simultaneously, use the same dataset, and have similar training speeds.
This paper proposes a new data loading method for training multiple parallel jobs.
Specifically, weaim to relax the constraints of prior work in three aspects: 1) all jobs can start and end freely, 2)different datasets can be used as long as they are partially overlapped, and 3) jobs’ training speedsmay vary widely.
For the above purposes, we first propose the dependent sampling algorithm
(DSA)to schedule the sampling for each job while ensuring their correlated randomness.
DSA is inspiredby the correlated sampling theory [5] and it mainly involves three steps: 1) divide the dataset intotwo parts of intersection set and difference set, 2) correlate the selections between intersection anddifference, and 3) share some operations for the intersection set to improve locality Furthermore, wedesign a domain-specific cache policy RefCnt that is tailored for parallel DNN training.
It is aware ofthe fact that all jobs iterate over the dataset once in an epoch, and thus it can evict data that is leastlikely to be used in the near future.
Finally, we design a novel data structure dependent sampling treeto efficiently implement the DSA by elaborately organizing the intersection sets and difference sets.
Based on the above technologies, we implement a prototype system named J OADER and integrate itinto PyTorch for evaluation.
The evaluation results show that J OADER can boost training efficiency inmore flexible cases.
For example, it achieves up to 200% training speedup when models of differentsizes are parallelly trained without affecting the accuracy.
The main contributions of this paper inlcude:
• A new dataloader for parallel DNN training on overlapped datasets.
• A sampling algorithm to increase the sampling locality with guaranteed randomness.
The algorithm
reaches the global optima when there are two training jobs.
• A domain-specific cache policy and a novel tree-based data structure for efficient implementation.
• Experimental evaluations demonstrating the performance improvements.
The rest of this paper is organized as follows.
Section 2 introduces the background information.
Section 3 describes the dependent sampling algorithm and our cache policy, followed by thedescription of the dependent sampling tree in Section 4.
Section 5 presents the evaluation ofthe implemented dataloader J OADER .
The paper is concluded in section 6.
Background2.1 Problem Statement
In each epoch, a DNN training job sends data requests to the storage device and passes throughthe dataset in random order.
For the brevity of the presentation, we assume that each data requestinvolves only one data point/element, and we name the data access order over all the data elementsas the access path for each epoch.
Consider an example in Figure 1, where we assume two training
[Figure]
jobs are running, and the two sampled data elements (‘e1’ and ‘e2’) requested by the two jobs arepackaged and prepared together in one round .
Cache can hold the prepared data to serve the future duplicated requests.
To fully utilize the cachewhen multiple DNN models are being trained simultaneously in a server, one could generate acache-friendly access path by encouraging all jobs to request the same data within a short period, i.e.,improving the sampling locality in each round.
Meanwhile, we still need to ensure the correlatedrandomness for each job.
To this end, we formulate the problem as follows:
Problem 1 Assume there are n training jobs on overlapped datasets, where the i -th job J i trains amodel on dataset D i with cardinality | D i | .
In each round, p ( e ij ) = 1 | D i | indicates the probability ofdata element e j being picked via sampling uniformly at random by J i .
For a given round, the goal isto maximize p ( e 1 =
e
= ...
=
e n ) , while maintaining p ( e i ) = 1 | D i | .
2.2 Related Work
Dataloader for multiple training jobs.
Cerebro [20] is proposed to avoid redundant data preparationwork for the task of model selection.
It partitions the dataset across the servers into clusters and hopesthe models iterate over data from one server to another instead of shuffling data.
Although it can helpavoid redundant data preparation work across servers, the work is still repeated on a single server.
CoorDL [19] provides a more general solution for hyper-parameter optimization.
In one turn, itdistributes one batch to each job and caches the batch into the buffer for future usage.
The batches areevicted until all jobs have consumed them in an epoch.
However, CoorDL still has some limitations:
1) all
jobs must be set up with the same batch size; 2) when the jobs have different training speeds,the faster jobs must wait for the slower ones; 3) CoorDL cannot handle situations where the jobsarrive at different moments or are trained on different datasets.
Domain-specific caching.
Crafting a cache for a specific domain is not a new idea [24].
For example,
Quiver [14] uses local solid-state drive (SSD) caches to eliminate the impact of slow reads from theremote storage.
However, it is too expensive to prepare enough memory against cache thrashing for
DNN training, especially considering the fact we need to cache not only the previous reading but alsothe results of data preprocessing.
In this paper, we propose a well-designed sampling algorithm toimprove the sampling locality with a small cache.
Correlated sampling.
As a theoretical problem, correlated sampling [5] aims to minimize theprobability of two sampling results being equal, given that the two players are sampling uniformly atrandom from two subsets of the same set.
Some sampling strategies [27] give the optimal solution fortwo players.
In this paper, we practically solve a related problem by designing an algorithm and anovel data structure for efficient implementation while enjoying their theoretical results.
Dependent Sampling Algorithm
3.1 Algorithm Design
To solve Problem 1, we propose a dependent sampling algorithm
(DSA) to maximize the samplinglocality while ensuring correlated randomness.
In the following, we first present the algorithm intwo-job case and then extend it to the n-job case.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Two-job Case.
The proposed dependent sampling algorithm (DSA) involves three steps: 1)intersection calculating, 2) dependent selecting, and 3)
shared sampling.
For a better illustration, weassume two jobs J 1 and J 2 are being trained on two overlapped datasets D 1 (the yellow circle) and
D 2 (the green circle), respectively, as shown in Figure 2.
First, we divide the datasets into three subsets: intersection set I =
D
∩ D 2 and two differencesets D d 1 = D 1 \
I, D d 2 = D 2 \ I .
For the default independent sampling algorithm (ISA), which iscurrently used by PyTorch, it then processes the data sampling uniformly at random in the datasetfor each job independently.
As shown in Figure 2a, the ISA algorithm can be divided into two steps:selecting and sampling procedures.
In the selecting procedure, each job selects between its differenceand intersection sets for sampling.
In the sampling procedure, each job randomly picks an element efrom the subset chosen in the previous step.
This two-step process maintains the uniform distributionof sampling for each job.
Then, the proposed DSA takes a careful modification in the sampling procedure.
As shown in
Figure 2c, if the two jobs both select the intersection set, one shares the sampling result to another.
We call this modification shared sampling .
Based on shared sampling, the probability that two jobsselecting the same element is equal to that both jobs selecting the intersection set.
To further improve locality, we need to improve the probability of both two jobs selecting theintersection, which can be achieved according to the conditional probability, as shown in Figure 2b.
The independent events in the selecting procedure can be transformed into the dependent ones,making the probability of the second job conditional.
That is, J 2 makes decisions conditioned on thedecision of J 1 .
To maximize the sampling locality, we formulate the conditional probabilities of dependent selectingin case of | D 1 | < | D 2 | .
For J 1 , the dependent selecting procedure is the same as that in ISA: selecting
I with the probability of | I | | D 1 | , and D d 1 with the probability of | D d 1 | | D 1 | .
The selecting procedure of
J 2 makes the decision according to J 1 .
If J 1 selects D d 1 , then J 2 must select another differenceset D d 2 .
If J 1 select I , then J 2 selects I with the probability of | D 1 | | D 2 | and D d 2 with probability| D 2 |−| D 1 | | D 2 | .
Figure 2b shows the above selecting procedure.
For the case of | D 2 | < | D 1 | , we just needto exchange the order of J 1 and J 2 to apply the above rules.
According to above conditional probabilities, the probability of p ( e 1 j = e 2 i ) is | I | max( | D 2 | , | D 1 | ) , which 147 is also the theoretical upper bound as discussed in Theorem 1.
The proof of this theorem can be foundin Proof 1.
Theorem 1 Assume there are 2 jobs J 1 , J 2 sampling uniformly at random from two datasets D 1 , D 2respectively, and the elements they sampled are e 1 j , e 2 i .
The probability of these two elements beingequal cannot be larger than | I | max( | D 2 | , | D 1 | ) where I =
D 1 ∩ D 2 .
Intersection calculating
Dependent selecting
Shared sampling except the first job
D
I
J
J
J1 J2 J
I
I J
I' J2 J difference intersection job selects difference job selects Intersection
D J job samplesin D
J
D
J
D
J
D
J
D
J
Dd
J
D
J
D
J
Dd
J
D
J Dd
J
D
J
D
J
Dd
I J
Dd
I J
Dd
I J
Dd
I' J
Dd3' I' J
Dd2'
I J
Dd
I J
Dd
I J
Dd
I' J
Dd3' I' J
Dd2'
Dd
J
I' J2 J
I J
Dd
I J
Dd
I J
Dd
N-job Case.
DSA of n -jobs is similar to two-jobs at the beginning.
The difference happens whensome jobs select difference sets.
We need to go through the above procedure recursively for thesejobs rather than sampling in the current difference set directly, as shown in Figure 3.
The reason forrecursive execution is that some data may only be shared by part of the jobs.
In the extreme case, ifall datasets are mutually disjoint, then the intersection in the first recursion is empty and the DSA isuseless.
Therefore, we need to further check whether there are shared data for part of the jobs.
The conditional probability for n-jobs that are sorted by the cardinalities of their datasets is formulatedas follows.
The first job J 1 selects the intersection with the probability of | I | | D 1 | while selecting thedifference with the probability of | D d 1 | | D 1 | .
If the job J i − 1 selects the difference set, then job J i mustselect the difference set.
If job J i − 1 selects the intersection set, then job J i selects the intersectionwith the probability of | D i − 1 | | D i | while selecting the difference set with the probability of | D i |−|
D
i − 1 | | D i | .
For the above algorithm, we can observe that if one job selects the difference, then all subsequentjobs will choose the difference, which is an important feature for the data structure in Section 4.
The pseudo code of DSA are shown in Appendix B for completeness.
Here, we provide the theoreticalresult as follows, whose proofs can be found in Appendix D.2.
Theorem 2 Assume there are multiples jobs with the i -th job J i trained on dataset D i .
Theprobability of any element e being picked for J i via DSA is p ( e )
= 1 | D i | .
3.2 RefCnt Cache Policy
If we allow jobs to vary greatly in speed, then some data may be evicted before they are consumedby all jobs.
Therefore, we need a cache policy that can take out the data that is least likely to beused (i.e., the data that least jobs will request) in the near future.
Classical cache policies (e.g., LRUand LFU) are not suitable for deep learning scenarios, where each training job reads each data onlyonce in an epoch.
So the total number of request to each data equals to the number of jobs using thedata for training.
To this end, if we could record the count of data request, the number of request tothis data in near future could be derived.
The fewer jobs that access the data, the higher the evictionpriority of that data.
Based on the above understanding, we design the caching policy RefCnt .
We maintain a referencecount for each data element.
If a job will request the element in the future, we increase its reference count; if a job reads the data, we decrease its reference count.
When the cache is full, the data withthe lowest reference count will be evicted.
Dependent Sampling Tree
The time complexity of DSA is O ( nm ) , where n is the number of jobs and m is the maximumcardinality of the datasets.
The worst case happens if all jobs select the difference set in each recursion,which means we need n recursions for n jobs.
Each recursion has the time complexity of O ( m ) .
To reduce the time complexity of sampling, we can maintain the intersections for DSA to avoid theintersection calculation in each recursion, which can make each recursion cost constant time.
In theworst case of DSA, the complexity of DSA is reduced from O ( nm ) to O ( n ) .
We propose the dependent sampling tree for organizing the intersections.
Although the number ofintersections is 2 n − 1 for n sets, only n − 1 intersections are valuable.
In DSA, we only need tocalculate the intersection of datasets for those jobs that select the difference, as shown in Figure 3.
When the jobs are sorted by the cardinality of their datasets, the job selecting the difference willmake all following jobs select the difference sets.
Therefore, the possible combinations for jobsthat select the difference must be the suffixes of the sorted jobs array.
For example, if there arethree sorted jobs [ J 1 , J 2 , J 3 ] , the groups of jobs choosing the difference set can only be one of thefollowing three sets, including [ J 1 , J 2 , J 3 ] , [ J 2 , J 3 ] , and [ J 3 ] .
Thus, the sets to be calculated are{ D 1 ∩ D 2 ∩ D 3 , D 2 ∩ D 3 } .
4.1 Tree Definition
For n datasets { D 1 , D 2 , ..., D n } in ascending order w.r.t.
| D 1 | < | D 2 | < ... <
| D n | , we needto calculate n − 1 intersections { D 1 ∩ ...
∩ D n , D 2 ∩ ...
∩ D n , ..., D n − 1 ∩ D n } .
We build adependent sampling tree, where each intersection is an internal vertex, and each difference set isa leaf.
Meanwhile, all vertices in a path from the root to a leaf represent a complete dataset.
Thedefinition of dependent sampling tree is as follows
Definition 1 Dependent sampling tree is a binary tree that stores a collection of datasets{ D 1 , D 2 , ..., D n } , with | D 1 | < | D 2 | < ... <
| D n | .
In the tree, the root is the intersection ofall datasets
I
=
D 1 ∩ D 2 ...
∩ D n .
The right child is the difference of the smallest dataset D 1 \ I .
The left child is a new dependent sampling tree of { D 2 \ I, D 3 \ I, ..., D n \ I } .
When there is onlyone dataset, the tree has one vertex that contains the dataset.
If there are n datasets, the height of the dependent sampling tree is n − 1 .
Figure 4a shows anexample with three sets { D 1 , D 2 , D 3 } with | D 1 | < | D 2 | < | D 3 | .
In this tree, vertex A contains
D A
= D 1 ∩ D 2 ∩ D 3 , and vertex B contains D B
= ( D 2 \ D A )
∩
( D 3 \ D A ) .
For the leaf vertices
C , E , and F , they maintain the other differences.
4.2 Tree Operations
When executing the DSA algorithm, dependent sampling tree should maintain the datasets intointersection and difference sets dynamically.
Furthermore, dependent sampling tree should alsosupport the situations when a job finishes its execution or a new job starts.
In this part, we discuss theoperations provided by dependent sampling tree, which are sampling , deletion , and insertion .
Sampling.
The intersections of the flow in Figure 3 can be represented by one or multiple vertexes ofthe rightest path in the dependent sampling tree.
Thus, the process of DSA is to traverse the pathfrom the root to the most-right leaf in the tree.
When traversing to an internal vertex, the jobs will
The intersection calculation needs to traverse the dataset
so its complexity is linear to the cardinality of the dataset. 2)
In the worst case, only the first job needs to make the decision, and others follow it.
Therefore, dependent selecting needs constant time.
Sampling only needs to pick a random number between 0 and the size of the dataset, which needs constant time.
A
B
C
E
F
D D
D (a) Dependent sampling tree of three datasets { D 1 , D 2 , D 3 }
A
B+F
C
E
D D
D (b)
Delete a dataset D 2 from the dependent sampling tree
A' B
C
E
F
D D
D D
X (c) Insert a smaller dataset D ( | D | ≤ |
D 1 | ) in dependent sampling tree
A'+B
A'+C
E
F
D D
D D
D'
X insert (d) Insert a larger dataset D ( | D | > | D 1 | ) in dependent sampling tree decide whether to select this vertex according the conditional probability.
For those jobs that do notselect this vertex, they will make decisions in the right child tree.
Sometimes the intersection can be composed of multiple vertexes.
For example, if only the job of D 3selects difference at A in Figure 4a, then it can sample both in B and F .
In these cases, the algorithmneeds to first select a vertex that belongs to the intersection and then sample.
Dataset Deletion.
When a job finishes training or is killed, we should delete its dataset from the tree.
The deletion involves two steps: 1) deleting the corresponding leaf vertex, and 2) merging its parentvertex with its sibling vertex.
For example, if we want to delete D 2 in Figure 4a, we need to deletethe leaf vertex E and merge vertex B with F , as shown in Figure 4b.
The time complexity is O
(1)since deleting and merging both cost constant time.
Dataset Insertion.
The insertion operation depends on the cardinality of the new dataset D .
When Dis smaller than the smallest set in the sampling tree, the procedure involves three steps: 1) calculatingthe intersection of it and the root of the tree, 2) setting the new intersection as the new root of thedependent sampling tree, and 3) setting the difference set of the old root as the right child and thedifference of D as the left child for the new root.
For example, Figure 4c shows the new tree afterinserting a smaller dataset D .
The new root X is the intersection of D and the set in vertex A .
Theold tree becomes the right sub-tree, and the difference of the dataset D becomes the left leaf.
When the new dataset D is larger than the smallest dataset in the sampling tree, we need to insert itrecursively: 1) calculating the intersection of D and root A and set it as new root X , 2) adding thedifference of A to its children, 3) deleting the old root A and adding their children to the new root, 4)recursively executing step 1 on X ’s right child with the difference set of D until the difference set isthe smallest in the current sub-tree.
An example of this procedure is shown in Figure 4d.
In the worstcase, the time complexity of insertion is O ( n | D | ) if the new dataset D is the largest 3 .
Evaluation
In this section, we evaluate J OADER on ImageNet with the family of ResNet models.
We denote thedefault dataloader strategy in PyTorch as the ‘Baseline’ method, and further compare J OADER withthe state-of-the-art method CoorDL [19].
Due to the space limit, we mainly show two experiments inthe paper and more experimental results can be found in the appendix.
First, we profile the serverwhen there are multiple training jobs to show the bottlenecks and compare J OADER with CoorDLand Baseline in synchronous case.
Second, we evaluate J OADER in asynchronous cases: 1) multiplemodels trained at different speeds, 2) multiple jobs arrived at different moments, and 3) multipledatasets only partially overlapped.
For the results in the appendix, we summarize them as follows: 1)ablation studies show that both DSA and RefCnt improve the training efficiency , 2) the I/O speed isnot the bottleneck for data loading , and 3) we train ResNet18 with PyTorch and J OADER separatelyto show J OADER does not affect the accuracy.
The evaluated models are the basic models with their default settings in torchvision [1], and trainedon top of the PyTorch 1.6.0 DL framework.
The experiments were conducted on a GPU server with
[Figure]
CPU quickly becomes the bottleneck for the
Baseline dataloader of PyTorch, while both
CoorDL and J OADER can greatly reduce CPU utilization.
[Figure]
J OADER can reduce the CPU utilization and keep each job’s own speed.
In contrast, fast job must wait for the slow job for CoorDL (e.g., ResNet18 costs the same time with
ResNet152).
[Figure]
[Figure] [Figure]
Intel Xeon Gold 5118 CPUs @ 2.30GHz (24 physical cores and 48 threads), 500GB RAM, and
6 TITAN RTX GPUs.
The server ran Ubuntu 18.04 with GNU/Linux kernel 4.15.0.
The disk is
Symbios Logic MegaRAID SAS-3 3316 of 1GB/s read speed.
5.1 Performance in Synchronous Cases
In this experiment, we start training multiple ResNet18 models with different hyper-parameters.
Wetry to make fair comparison by forcing the jobs training on the GPUs of the same version to get thetheoretically same training speeds.
All jobs train the same model on the same dataset and start at thesame time.
We start from training 1 model to training 6 models on ImageNet, and profile the server for differentdataloaders.
The experiments show that the CPU utilization tends to be the bottleneck, as shown in Figure 5.
The CoorDL and J OADER both can reduce the CPU utilization greatly by sharing the data preparation.
Additionally, it shows that J OADER can achieve close performance to CoorDL when all training jobsstart at the same time and have a similar speed.
Note that CoorDL is designed for such synchronouscases while J OADER can also handle asynchronous cases as we will later show.
The time cost of each epoch is shown in Table 1.
With 6 training jobs, J OADER saves 44 .
8% training time with over 40%less CPU utilization, compared the default Baseline in PyTorch.
[Table]
5.2 Performance in Asynchronous Cases
Next, we consider three asynchronous cases when multiple models differ in training speeds, arrivingmoments, and datasets.
Multiple models with different training speeds.
We first simultaneously train five models(ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152) with different training speeds, andshow the lifetime and CPU utilization results in Figure 6.
With the Baseline dataloader providedby PyTorch, each job preprocesses the data individually so that the CPU load is extremely heavy.
CoorDL enforces multiple jobs requesting the data in lockstep to share the data preparation work, andit achieves less CPU load and higher training speed than Baseline.
However, the faster jobs (smallermodels) must wait for the slower jobs, making CPU idle during the training period.
J OADER can bothreduce the CPU load by sharing the preprocessing work and keep these jobs’ own training speeds.
For example, ResNet18 runs 4 × faster using J OADER compared with CoorDL.
Furthermore, thisexperiment suggests that our system has less profitable for larger models because GPU tends to bethe bottleneck for those models.
Multiple models arriving at different moments.
Next, we consider the situation that multiple DNNtraining jobs start at different moments.
Since CoorDL must wait for all the jobs to be ready fortraining, we only compare J OADER with Baseline in this experiment.
Figure 7a shows the CPUutilization when four ResNet18 training jobs arrive at different moments.
J OADER takes up less CPUload than the Baseline.
This is due to the fact Baseline repeats the data preparation work for newjobs, whereas J OADER reuses the prepared data from previous jobs.
As a result, J OADER saves 17.1%training time and 8.7% CPU utilization.
Multiple models with partially overlapping datasets.
Finally, we show the CPU utilization whenthe two datasets only partially overlap in Figure 7b.
The datasets are re-sampled from ImageNet.
We use the ratio between intersection size and dataset size to measure the overlapping degree.
Thelarger the intersection, the more data to share.
We still compare with the Baseline method only since
CoorDL requires all jobs having the same access path over datasets.
From the figure, we can observethat the CPU utilization of the same dataset is only 60% of that of different datasets.
Additionally,the CPU utilization always decrease as the intersection size increases, showing that J OADER is validwith some overlapping elements.
The reason is that the DSA algorithm can get the same samplingresults with the max probability and the cache policy evicts the useless data with higher priority.
Conclusion
Accelerating DNN training is a fundamental challenge in deep learning.
This paper presents anapproach to boosting training efficiency by caching data prep work and reusing them to paralleltraining jobs.
To prevent cache thrashing, we propose DSA to improve sampling locality with ensuredrandomness.
We also propose a novel tree-based data structure to efficiently implement the proposedalgorithm.
Furthermore, a domain-specifical cache policy is proposed for evicting data that is leastused in the near future.
At last, we evaluate the proposed approach and the results show significantimprovement in asynchronous cases while achieving close performance in synchronous cases to
CoorDL which is specially designed for such cases.
A Evaluation
[Figure]
In this section, we first present the I/O speed in the synchronous cases and asynchronous cases toshow J OADER can reduce the redundant I/O work.
Then, we evaluate the algorithm DSA and RefCntseparately for the ablation test. Finally, we show the loss and accuracy trace of ResNet18 in 40epochs to demonstrate J OADER does not affect convergence speed. We denote the default dataloaderstrategy in PyTorch as the ‘Baseline’ method, and further compare JOADER with the state-of-the-artmethod CoorDL. At last, we show the cost of the operation in DSA .
A.1 I/O speed
In this part, we show the I/O speed in the synchronous and asynchronous cases.
Note that I/
O isnot the bottleneck in our server (Maximum I/O speed of the server is about 1GB/s).
Figure 8 showsthe
I/O from training one job to training six jobs.
The I/O speed of Baseline tends to increase whilethat of CoorDL and J OADER tends to decrease, because CoorDL and J OADER can reuse the datain memory.
When we train 5 jobs (ResNet18, ResNet34, ResNet50, ResNet101, ResNet152) withdifferent speed, the I/O speed is shown in Figure 9.
Although the I/O speed of CoorDL is far lessthan J OADER , the fast job must wait for a slow job causing inefficiency.
J OADER can make jobs runat their own speeds.
Figure 10a show the I/O speed for four jobs that start at different moments.
Dueto the redundant I/O work, the baseline has a more considerable I/O speed.
For J OADER , the moredata is shared, the smaller the I/O speed, as shown in Figure 10b.
A.2 Algorithm evaluation
In this section, we first compare ISA, the default sampling algorithm in PyTorch, with DSA to showthe performance in two cases: 1) datasets overlap partially, and 2) datasets vary in size.
Then wefurther compare the RefCnt with the generic cache policy in the above cases.
We evaluate thesealgorithms in different cache sizes and the evaluation metric is the count of cache misses , indicatingthe number of elements not read from the cache.
For n datasets D 1 , D 2 , ..., D n , the maximum countof cache misses is | D 1 | + | D 2 | + ... + | D n | that all the data is read from the storage, while theminimum is | D 1 ∪ D 2 ∪ ... ∪ D n | that all the repeated data is hit in cache.
We construct the following cases for evaluation 4 in Table 2:
[Equation]
D
[Equation]
sample ([0 , 13333] , 10000) means sample a subset D with 10000 of size from [0 , 13333] uniformly
random
[Figure]
[Figure]
[Figure]
Baseline
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Table]
SamplingAlgorithm overlappingpartially
[Table]
varyinginsize
[Table]
CachePolicy overlappingpartially
[Table]
varying in [Table] Dependent sampling algorithm. In this part, we show DSA can significantly reduce the cachemisses in various of scenes.
To simulate various scenes, we generate multiple datasets with differentsizes and different intersections.
We use DSA to access them in shuffling order from the storage withthe cache of different sizes (e.g., from one-slot to holding all datasets) to evaluate the number ofmisses, compared with the experimental results with ISA (shuffling independently).
We start the the evaluation in two jobs training on dataset 1 and dataset 2 , respectively.
We set upeight different configurations for the two datasets, which can be divided into two types: the differentsizes of intersections and different sizes of the datasets, as shown in Table 2.
For different intersection sizes, Figure 11a plots the relations between cache size and the count ofcache misses.
For ISA, the misses count is 2x higher than the size of dataset in the case of one-slotcache, shows that all elements are missed and it decreases linearly as the cache size increase.
Unlessthe cache can hold all the data of intersection, the misses count could be minimized.
However, DSAcan always get the minimum cache misses regardless of the cache size.
The reason is that in theselecting procedure, all jobs must select the same subset when the datasets’ sizes are equal, due to theconditional probability of 1.
Thus, their sampling results would be the same, meaning that all jobswill access the same element simultaneously when the datasets with the same size.
Figure 11b plots the evaluation results on for the datasets with different sizes.
For one-slot cache,
DSA can reduce 50% cache misses compared with ISA when the dataset sizes differ by 25%.
Thecache held 30% dataset could get the best performance for DSA while ISA needs to cache 75%dataset to reach optimal.
DSA needs to cache 25% data because the size difference brings someuncertainty in the selecting procedure.
Then, we evaluate the DSA algorithm on four DNN training jobs to show the outstanding performanceof the DSA algorithm on multiple datasets.
Multiple datasets bring more complexity to the intersectionas there can be 2 n − 1 intersection sets for n sets.
To simulate an utterly random scene, werandom sample 10,000 elements from 13,333 elements four times, making four different datasets forevaluation.
Compared ISA in the random scene, DSA can reduce 50% misses with a one-slot cache(20000 misses out of 40000 misses), as shown in Figure 13a.
DSA also provides good performance for multiple jobs training on datasets with different sizes.
Compared to ISA in Figure 13b, one-slot cache can reduce 30% cache misses (16000 misses out of22000 misses) with DSA when four jobs are training on four datasets, with a 25% different size.
RefCnt cache policy.
In this part, we compare cache policy RefCnt with the generic cache policies
LRU, FIFO, and random replacement (RR) to show 10% misses reduction with the same cache size.
We evaluate these policies in above two situations of four jobs: utterly random scene and differentdataset sizes.
Figure 12 shows the results, which can be summarized as two improvements: 1) RefCnt
[Figure]
cache policy can reduce 10% misses in the same cache size compared with general cache policies,and 2) RefCnt cache policy can get the best performance with only caching the 60% dataset while thegeneric cache policies require holding the whole dataset.
A.3 Correctness experiments
To demonstrate J OADER does not affect convergence speed w.r.t.
accuracy and loss, we train ResNet18on ImageNet-1K with J OADER and PyTorch 5 .
Figure 13 shows that the loss descent trajectory andaccuracy trajectory are almost the same.
A.4 Cost of operation in sampling tree
We have conducted an additional experiment by randomly inserting 128 datasets into our system and 502 then evaluating the time cost of dataset operations.
The size of each dataset is between 1,000,000 and2,000,000 elements (note that ImageNet contains 1,400,000 images).
The average cost of inserting each 504 dataset is 0.57 seconds.
And the time cost of the operations remains nearly constant and does not changeas the number of data increases.
Deletion and sampling are also very efficient in our algorithm.
Specifically, sampling takes an average of0.000054 seconds for each element, and deletion takes an average of 0.00000105 seconds.
We will add thecorresponding description and experiment in our rebuttal revision.
B Dependent Sampling Algorithm
In this section, we show the pseudo-code of the DSA, which involves two functions: selecting andsampling.
The selecting algorithm is shown in Algorithm 1, and the sampling algorithm is shown in
Algorithm 2.
C Sampling in Dependent Sampling Tree
The sampling operation involves two steps: 1) Deciding: choose a vertex for each job, and 2)
Sampling: select an element in the corresponding vertex for each job randomly.
In deciding , we start the procedure from the root of the tree.
In the beginning, all jobs are marked asundecided .
In each vertex, all undecided jobs need to decide whether to select the current vertex, i.e.,the intersection I in the DSA algorithm. If the job chooses the current vertex, we will remove themfrom the undecided job set. Then, the remaining undecided jobs are put down to the child and makedecisions recursively until the undecided job set is empty, like Algorithm 1. Data: n jobs J 1 , ..., J n and n datasets D 1 , ..., D n
Result:
The subset that each job selects 1 JOB ← [ J 1 , ..., J n ] ; // Array of all jobs 2 DS ← [ D 1 , ..., D n ] ; // Array of all datasets sorted by their cardinality 3 S ← ∅ ; // Set of data that are not selected 4 while JOB ̸
= ∅ do
I ← (cid:84)
D
i − S, ∀ D i ∈ DS ; c ← | I | ; for D , J in DS , JOB do // c
| D |−| S | is the conditional probability |
D i − 1 | | D i | if rand (0 , 1) ≤ c | D |−| S | then
J selects I ;
DS ← DS − D ;
JOB ← JOB − J ; c ← | D | ; else break; if There is no job selecting I then
JOB [0] selects D [0] − I ;
DS ← DS − D [0] ;
JOB ← JOB − JOB [0] ;
S ← S ∪ I ;
Algorithm Data: n jobs J 1 , ..., J n and k subsets S 1 , ..., S k
Result:
The sampling results of each job 1 JOB ← [ { J i , ..., J j } , ... ] ; //
The job set where job selects the correspond subset 2 SUBSET ← [ S 1 , ..., S k ] ; //
The subset that the correspond job set selects 3 for i in 0..
k do subset ← SUBSET [ i ] ; job _ set ← JOB [ i ] ;
Sample e uniformly at random in subset ; for job in job _ set do job picks e ;
However, in deciding , there may be an intersection that contains multiple vertices of the samplingtree.
As shown in Figure 4c, if job J 2 with D 2 and job J 3 with D 3 do not choose the root X in thefirst recursion, then they should make decision in the intersection ( D 2 − D X )
∩
( D 3 − D X ) , whichcontains the datasets in vertices A and B .
In this case, we should combine these vertices into a bigset first and start the deciding procedure once.
If someone chooses the big set, it should choose avertex from the big set randomly again.
After the deciding procedure, each job is assigned to a vertex randomly.
Then, we need to select anelement in the corresponding vertex randomly.
Based on the united sampling discussed in Section 3,those jobs that are assigned to the same vertex should share the same sampling result.
Each data item in the dataset should be read exactly once for a job in an epoch.
Intuitively, the dataitem should be deleted from the vertex to avoid being read by the job repeatedly.
This process raisesthe issue that other jobs may need the deleted element that some have read.
For example, suppose inthe Figure 4a, job J 1 fetches the element e 1 from vertex
A while job J 2 and job J 3 fetch the element [Table] S Set of elements that are not selected from other vertices.
Thus, for J 1 , we should remove e 1 from vertex A , but J 2 and J 3 still need theelement e 1 .
The compensation procedure is introduced in sampling to solve this issue, which adds the deletedelement to the child vertex of the job that still needs it.
To decide which vertex needs compensation,we construct a job set for each vertex that contains all the jobs that will sample in this vertex.
Forexample, the job set of vertex A is { J 1 , J 2 , J 3 } and the set of vertex B is { J 2 , J 3 } in Figure 4a.
Andafter deleting some elements, we should construct a compensation set containing all the jobs thatstill need the deleted elements.
If the compensation set includes the job set, the deleted elements areadded to the corresponding vertex.
Partially Sampling.
When some jobs are much faster than other jobs, we only need the samplingresults for these fast jobs.
The sampling procedures are the same as the above.
However, the treecannot be ordered after sampling because these datasets decrease fast and can be less than the abovedatasets in the tree soon.
Therefore, we need to reorder the tree, just like the intersection procedure.
D Proof of Algorithm
In this section, we present the derivation and the proof for the proposed algorithm.
Proposition 1shows the first principle for randomness.
Table 3 presents some notations for the following proof.
Proposition 1 For any job J i , the probability of choosing any element from dataset D
i is 1 | D i | .
D.1 Derivation in Two-job Case
There are 2 jobs J 1 and J 2 which training upon datasets D 1 and D 2 , while the intersection is
I =
D 1 ∩ D 1 and two difference sets are D d 1 = D 1 −
I, D d 2 = D 2 − I .
Their cardinalities are| D 1 | , | D 2 | , | I | , | D d 1 | , and | D d 2 | .
The event that J 1 selects the intersection is J I 1 , while the event that
J 1 selects the difference set is J D 1 .
Due to the randomness, the constraints that we need to maintain are
[Equation]
Since J 2 makes the decision conditioned on J 1 , we can the formulate the equations according to thelaw of total probability in below
[Equation]
After shared sampling , our target is to maximize the probability of J 1 and J 2 both choose theintersection, which is positive correlated with p ( J I
|
J I 1 ) .
The p ( J I 2 |
J
I 1 ) is
[Equation]
Therefore, we can get the conditional probability in two cases
[Equation]
When | D 1 | ≤ | D 2 | , the other conditional probabilities of J 2 conditioned on J 1 can be derived asfollows
[Equation]
In dependent selecting procedure, the probability of both choosing intersection set is p ( J I 1 ) ∗ p ( J I
| J I 1 ) .
With data partition , we can get
[Equation]
while e 1 j , e 2 i is the sampling results of J 1 , J 2 in one round.
We then prove the the proposed dependent algorithm gives the optimal solution for the scenario oftwo-jobs.
Proof 1 Assume that the distribution of the event exists, and the probability of p ( e 1 j = e 2 i ) is unknown.
When e 1 j and e 2 i are equal, they must belong to the intersection set D i .
Therefore, p ( e 1 j = e 2 i ) is equalto p ( e 1 j = e 2 i , e 1 j ∈ D
i , e 2 i ∈
D i ) .
Due to the definition of joint probability, we can getp ( e 1 j = e 2 i ) ≤ p ( e 1 j ∈ D i ) , and p ( e 1 j = e 2 i ) ≤ p ( e 2 i ∈ D i ) .
Due to the constraint of randomness, e 1 j and e 2 i are both sampled under the uniform distribution, sop ( e 1 j ∈ D i ) =
| D i | | D 1 | and p ( e 2 i ∈
D i ) =
| D i | | D 2 | .
Then p ( e 1 j = e 2 i ) ≤ | D i | | D 1 | and p ( e 1 j = e 2 i ) ≤ | D i | | D 2 | bothhold.
Thus, for any distribution of the event, it holds that
[Equation]
D.2 Derivation in N-job Case
Assume there are n jobs { J 1 , ..., J n } and n datasets { D 1 , ..., D n } , while the intersection I = 577 (cid:84) n i =1 D i .
For the k -th job where k > 1 , the probability of the job J k chooses the intersection is
[Equation]
(7) where x is the sum of probabilities of J k chooses intersection in other conditions.
With the similarprocess in Proof 1, we can get the result for k -jobs, where the maximum probability of they all choose intersection is also p ( J I 1 ...J Ik ) ≤ | I | max [Equation] D 1 ,...,D k ) .
Then, we can get
[Equation]
Assume that the datasets are sorted in ascending order w.r.t.
their cardinalities, then the formula canbe written as for the maximum probability
[Equation]
The probability of n jobs all choose intersection I is
[Equation]
which is the theoretical maximum according to a similar Proof 1.
D.3 Proof of Randomness
We prove the Proposition 1 for Algorithm 1.
Although multiple jobs share the sampling results in thesampling procedure, the random sampling is not changed.
Therefore, to prove Proposition 1, we onlyneed to prove the subsets are randomly selected for each job.
The Lemma is stated as follow
Lemma
In each loop of selecting procedure, the intersection I is selected with the probability of| I | | D i | for every job J i .
In the line 8-12 of Algorithm 1, the job J i chooses the intersection only if the previous job has chosenthe intersection I , that the probability is
[Equation]
Therefore, the Lemma 1 is satisfied only if the probability of entering this loop is | D i |−| S | | D i | , that is
Lemma
For each loop, the set of elements that are not selected is S , and the probability of enteringthis loop is | D i |−| S | | D i | for job J i .
Proof 2 We prove the Lemma 2 by induction on loop index k .
Base case.
Show Lemma 2 holds for the first loop.
The first loop ( k = 1 ) must be entered, while S is an empty set and | D i |−| S | | D i | = 1 .
Thus Lemma 2 issatisfied.
Induction step.
Show that for any k ≥ 1 , if the k -th recursion holds Lemma 2, then ( k + 1) -th loopalso holds.
The probability of entering the k -th loop is | D i |−| S | | D i | , and if job J i does not select the intersection in the k -th loop, then it entering the ( k + 1) -th loop.
The probability of J i does not select intersection is
[Equation]
− [Equation] where J k is the first job in job List JOB , then we can get the probability of entering the next loop is| D i |−| S |−|
I | | D i | while new S is S ∪ I , which also satisfies the Lemma 2.
Conclusion.
Since both the base case and the inductive step have been proved as true, the Lemma 2holds for every loop by mathematical induction.
E System Implementation
In this section, we describe the implementation of J OADER , which is a data loading managementsystem for multiple DNN training jobs.
The proposed dependent sampling algorithm applied with thedependent sampling tree is the core in J OADER .
In details, we describe the system overview, threemain components, and the adaption for distributed DNN training.
E.1 Overview
API.
J OADER API allows users to do two things: 1) creating and altering datasets, and 2) registering
DNN training jobs.
Before training, the user needs to create the dataset with a name in J OADER first.
Then, to execute the training job on the dataset, the job should register itself to J OADER with thedataset’s name.
If multiple jobs are training on that dataset, J OADER will attach the dataset to thejobs for sharing reading and preprocessing.
Each job in an epoch will be assigned to a unique id in
J OADER , and J OADER will dispatch data according to the job id.
Architecture.
J OADER includes a frontend and a backend.
The backend is implemented in Rust [2]that accesses and processes data efficiently, while the frontend is implemented in Python to provide auser-friendly API.
They communicate with each other in RPC.
Figure 14 shows the architecture of
J OADER , which consists of three components: Sampler, Loader, and Cache.
E.2 Sampler and Loader
The Sampler is responsible for data sampling in J OADER .
An instance of the Sampler consists of asampling tree for a dataset.
The datasets are organized in tables like the relational database.
Eachdata tuple has a unique integer id (i.e., primary key) to identify itself, consisting of multiple elements,e.g., image, label, and the bounding boxes in ImageNet.
Each job registered to the sampler needs to specify the job’s name, dataset’s name, names of theneeded columns in the table, and a predicate for filtering.
Then, the sampler collects the ids of datatuples that meet the requirements and inserts the ids into the sampling tree corresponding to thedataset.
Each sampling tree should pick the element uniformly at random in the DS algorithm.
Theseelements (id of data tuple) will be transformed into the data requests according to the columns ofeach job needed.
If the elements and the columns are the same, these data requests will be mergedinto one request.
The Loader is responsible for reading and processing data w.r.t.
the data request from Sampler.
In
J OADER , each loader corresponds to one type of storage, e.g., POSIX file system, key-value database,or distributed storage.
Loaders encapsulate different storages and provide a unified interface to
Sampler to present good scalability and compatibility.
E.3 Specific Cache Implementation
The Cache is used to store and share data between J OADER and the DNN training jobs, which isimplemented by shared memory to avoid the cost of memory copying, serialization, and network.
Tomanage data and reduce data race, the layout of cache is slotted, making multiple slots in the head ofthe cache, as shown in Figure 14.
Each element in the cache is managed by a slot containing the startaddress, length, and read bit of the data.
The read bit is used to determine whether the data has beenconsumed.
When Loader tries to load data, it needs to require slots and a contiguous block memory for storingthis data.
Then, the Loader dispatches the slot ids to training jobs that need the data.
These jobs willaccess the data soon and set the read bit.
When there is not enough memory, the reclamation programshould be triggered.
Data can be classified as consumed and unconsumed in the cache according tothe read bit.
Because those unconsumed data were sampled earlier than the data that Loader is tryingto require memory, they will be accessed earlier than the new data and cannot be evicted.
We can only reclaim those consumed elements, which are prioritized in different levels.
For example,there are two training jobs J 1 and J − 2 all want to read four data { d 1 , d 2 , d 3 , d 4 } .
In the first roundof sampling, d 1 is consumed by both jobs.
In the second round, job J 1 consumed d 2 while job J 2consumed d 3 .
Then the memory reclamation program is triggered.
Considering the constraint thateach job should traverse the dataset once.
In this time, data d 1 , d 2 , d 3 are in cache while data d 1 is nolonger needed anymore while d 2 , d 3 are still needed.
Therefore, d 1 will be prioritized for eviction.
In J OADER , we reclaim data according to the number of reference of data, as shown in section 3.2.
E.
In our Joader, the time complexity of dataset operations, i.e., sampling, deletion, and insertion, are
O (1) , O (1) , and O ( | D | ) , respectively, for each dataset.
Moreover, the dependent sampling treemanages the index of each input rather than the input itself, and thus executing the operations isextremely fast.
In our implementation, we use a bitmap to represent the dataset in our dependentsampling tree. The bitmap is very efficient for these operations. In practice, the operations only countup to a minor fraction of time consumption, which takes a tiny proportion of a total training epoch. To further answer the question, we have conducted an additional experiment by randomly inserting128 datasets into our system and then evaluating the time cost of dataset operations.
The size of eachdataset is between 1,000,000 and 2,000,000 elements (note that ImageNet contains 1,400,000 images).
The average cost of inserting each dataset is 0.57 seconds.
And the time cost of the operations remainsnearly constant and does not change as the number of data increases.
Deletion and sampling are also very efficient in our algorithm.
Specifically, sampling takes an averageof 0.000054 seconds for each element, and deletion takes an average of 0.00000105 seconds.
E.5 Towards Distributed Training
In model parallelism, the model is segmented into different parts that can run concurrently in differentnodes.
Only one node is responsible for data preparation.
Therefore, J OADER need not changeanything to apply to the distributed training in model parallelism.
In data parallelism, the dataset is divided into several partitions, where the number of partitions isequal to the total number of available nodes in the cluster.
The model is replicated to the workernodes.
Each worker operates on its subset of the dataset to train the model locally.
For each workernode, we need to set up a J OADER for sharing data preparation to multiple DNN training jobs in thisnode.
However, the sampling work should be done globally to avoid redundant sampling.
To solve the above issue, we set multiple J OADER for multiple nodes, while only one leader J OADERcan sample.
The leader needs to dispatch the sampling results to the followers, as shown in Figure 15.
During training, each sub-process of the distributed training job in data parallelism needs to registeritself to the local J OADER in the same worker node with the same name and the assigned id.
Meanwhile, the local J OADER should register a sub-sampler in leader J OADER .
The local J OADERshould keep fetching sampling results from the leader J OADER and load them to the local cache.
By doing this, only leader J OADER is responsible for sampling and dispatching sampling results tofollowers.
Notice that the cache is also distributed in different worker nodes, storing more data than asingle cache.
The same data is sent to the same worker node when the leader dispatches sampling results.
PyTorchuses the hash partition algorithm to dispatch data.
For example, suppose a worker node array nodes =[ node 1 , node 2 , ..., node n ] .
The data with integer id is sent to nodes [ id % n ] .
However, things willbe more complicated when dealing with multiple training jobs.
For example, two training jobs areon the worker node arrays [ node 1 , node 2 , ..., node n − 1 ] and [ node 2 , node 3 , ..., node n ] , respectively.
Due to the offset between the two arrays, the same data cannot be sent to the same worker node forthe two jobs by one hash partition.
In J OADER , we solve it by hashing the data id twice.
In the first hash, we try to dispatch the data toall the worker node in cluster.
Therefore, the same data can be sent to the same node.
However, sometraining jobs are not training on some worker nodes and the data can not be sent to these nodes forthese jobs.
Therefore, in the second hash, for the data dispatched wrongly, we will try to dispatch itto the nodes locally w.r.t.
the configuration of each job.