A Deep Learning Dataloader with Shared Data Preparation
Abstract
Parallelly executing multiple training jobs on overlapped datasets is a common practice in developing deep learning models.
By default, each of the parallel jobs prepares (i.e., loads and preprocesses) the data independently, causing redundant consumption of I/O and CPU.
Although a centralized cache component can reduce the redundancies by reusing the data preparation work, each job’s random data shuffling results in a low sampling locality causing heavy cache thrashing.
Prior work tries to improve the sampling locality by enforcing all the training jobs loading the same dataset in the same order and pace.
However, such a solution is only efficient under strong constraints: all jobs are trained on the same dataset with the same starting moment and training speed.
In this paper, we propose a new data loading method for efficiently training parallel DNNs with much flexible constraints.
Our method is still highly efficient when different training jobs use different but overlapped datasets and have different starting moments and training speeds.
To achieve this, we propose a dependent sampling algorithm (DSA) and a domain-specific cache policy.
Moreover, a novel tree data structure is designed to efficiently implement DSA.
Based on the proposed techniques, we implemented a prototype, named J OADER , which can share data preparation work as long as the datasets are overlapped for different training jobs.
We evaluate the proposed J OADER , showing a greater versatility and superiority of training speed improvement (up to 200% on ResNet18) without affecting the accuracy.
Introduction
The rapid development of deep learning frameworks and tools [2, 21] has enabled and facilitated researchers and practitioners in a multitude of disciplines (e.g., physics [27], chemistry [17], biology [11] and Earth science [22] in addition to computer science) to start developing their own
DNN models solving various problems at hand.
In such cases, they usually need to simultaneously run multiple training jobs on the same dataset or overlapped datasets 2 , due to the purpose of model selection [14], hyper-parameter tuning [5], network architecture searching [24], etc.
Although widely used, the training of DNNs is usually time-consuming and tricky, which may affect the development efficiency, especially for general users from fields other than artificial intelligence.
In literature, various approaches have been proposed to reduce the DNN training time such as data preparation optimization [3], communication overhead reduction [9, 8, 28], GPU memory optimization [7, 15, 25], and compiler-based operator optimization [6, 12, 16].
Different from the above existing work, our focus is on the data preparation aspect, or more specifically, the data loading aspect, in the context of parallel DNN training.
On the one hand,
Mohan et al.
[18] have observed that data preparation, which consists of data loading and the subsequent data preprocessing, occupies a significant portion of parallel DNN training time.
This is due to the fact that multiple training jobs on the same data copy essentially create redundant data preparation efforts.
Ideally, to reduce the load of hardware and boost the training efficiency, the prepared data should be cached in memory and serve for multiple jobs.
However, the cache may thrash with a high probability as each job adopts a fully random shuffling strategy that independently shuffles the dataset in each epoch.
On the other hand, it is until recently have some studies investigated the shuffling process itself [19, 18, 20].
These studies investigate to what extent the fully random shuffling is required and have observed that various shuffling strategies breaking the full randomness still yield competitive results.
We name such randomness as correlated randomness as opposed to the full randomness.
For example, prior work [18, 19] proposed to solve the cache thrashing issue by enforcing that all jobs iterate over the dataset in the same order and pace, and observed no accuracy loss.
However, such a solution maximizes its performance under some strong constraints, i.e., all training jobs have to start simultaneously, use the same dataset, and have similar training speeds.
This paper proposes a new data loading method for training multiple parallel jobs.
Specifically, we aim to relax the constraints of prior work in three aspects: 1) all jobs can start and end freely, 2) different datasets can be used as long as they are partially overlapped, and 3) jobs’ training speeds may vary widely.
For the above purposes, we first propose the dependent sampling algorithm
(DSA) to schedule the sampling for each job while ensuring their correlated randomness.
DSA is inspired by the correlated sampling theory [4] and it mainly involves three steps: 1) divide the dataset into two parts of intersection set and difference set, 2) correlate the selections between intersection and difference, and 3) share some operations for the intersection set to improve locality Furthermore, we design a domain-specific cache policy RefCnt that is tailored for parallel DNN training.
It is aware of the fact that all jobs iterate over the dataset once in an epoch, and thus it can evict data that is least likely to be used in the near future.
Finally, we design a novel data structure dependent sampling tree to efficiently implement the DSA by elaborately organizing the intersection sets and difference sets.
Based on the above technologies, we implement a prototype system named J OADER and integrate it into PyTorch for evaluation.
The evaluation results show that J OADER can boost training efficiency in more flexible cases.
For example, it achieves up to 200% training speedup when models of different sizes are parallelly trained without affecting the accuracy.
The main contributions of this paper inlcude:
• A new dataloader for parallel DNN training on overlapped datasets.
• A sampling algorithm to increase the sampling locality with guaranteed randomness.
The algorithm reaches the global optima when there are two training jobs.
• A domain-specific cache policy and a novel tree-based data structure for efficient implementation.
• Experimental evaluations demonstrating the performance improvements.
The rest of this paper is organized as follows.
Section 2 introduces the background information.
Section 3 describes the dependent sampling algorithm and our cache policy, followed by the description of the dependent sampling tree in Section 4.
Section 5 presents the evaluation of the implemented dataloader J OADER .
The paper is concluded in section 6.
Background 2.1 Problem Statement
In each epoch, a DNN training job sends data requests to the storage device and passes through the dataset in random order.
For the brevity of the presentation, we assume that each data request involves only one data point/element, and we name the data access order over all the data elements as the access path for each epoch.
Consider an example in Figure 1, where we assume two training jobs are running, and the two sampled data elements (‘e1’ and ‘e2’) requested by the two jobs are packaged and prepared together in one round .
Cache can hold the prepared data to serve the future duplicated requests.
To fully utilize the cache when multiple DNN models are being trained simultaneously in a server, one could generate a cache-friendly access path by encouraging all jobs to request the same data within a short period, i.e., [Figure] improving the sampling locality in each round.
Meanwhile, we still need to ensure the correlated randomness for each job.
To this end, we formulate the problem as follows:
Problem 1 Assume there are n training jobs on overlapped datasets, where the i -th job J i trains a model on dataset D i with cardinality | D i | .
In each round, p ( e ij ) = 1 | D i | indicates the probability of data element e j being picked via sampling uniformly at random by J i .
For a given round, the goal is to maximize p ( e 1 = e
= ...
=
e n ) , while maintaining p ( e i ) = 1 | D i | .
2.2 Related Work
Dataloader for multiple training jobs.
Cerebro [19] is proposed to avoid redundant data preparation work for the task of model selection.
It partitions the dataset across the servers into clusters and hopes the models iterate over data from one server to another instead of shuffling data.
Although it can help avoid redundant data preparation work across servers, the work is still repeated on a single server.
CoorDL [18] provides a more general solution for hyper-parameter optimization.
In one turn, it distributes one batch to each job and caches the batch into the buffer for future usage.
The batches are evicted until all jobs have consumed them in an epoch.
However, CoorDL still has some limitations: 1) all jobs must be set up with the same batch size; 2) when the jobs have different training speeds, the faster jobs must wait for the slower ones; 3) CoorDL cannot handle situations where the jobs arrive at different moments or are trained on different datasets.
Domain-specific caching.
Crafting a cache for a specific domain is not a new idea [23].
For example,
Quiver [13] uses local solid-state drive (SSD) caches to eliminate the impact of slow reads from the remote storage.
However, it is too expensive to prepare enough memory against cache thrashing for
DNN training, especially considering the fact we need to cache not only the previous reading but also the results of data preprocessing.
In this paper, we propose a well-designed sampling algorithm to improve the sampling locality with a small cache.
Correlated sampling.
As a theoretical problem, correlated sampling [4] aims to minimize the probability of two sampling results being equal, given that the two players are sampling uniformly at random from two subsets of the same set.
Some sampling strategies [26] give the optimal solution for two players.
In this paper, we practically solve a related problem by designing an algorithm and a novel data structure for efficient implementation while enjoying their theoretical results.
Dependent Sampling Algorithm 3.1 Algorithm Design
To solve Problem 1, we propose a dependent sampling algorithm
(DSA) to maximize the sampling locality while ensuring correlated randomness.
In the following, we first present the algorithm in two-job case and then extend it to the n-job case.
Two-job Case.
The proposed dependent sampling algorithm (DSA) involves three steps: 1) intersection calculating, 2) dependent selecting, and 3) shared sampling.
For a better illustration, we assume two jobs J 1 and J 2 are being trained on two overlapped datasets D 1 (the yellow circle) and
D 2 (the green circle), respectively, as shown in Figure 2.
First, we divide the datasets into three subsets: intersection set I =
D
∩ D 2 and two difference sets D d 1 = D 1 \ I, D d 2 = D 2 \ I .
For the default independent sampling algorithm (ISA), which is [Figure] currently used by PyTorch, it then processes the data sampling uniformly at random in the dataset for each job independently.
As shown in Figure 2a, the ISA algorithm can be divided into two steps: selecting and sampling procedures.
In the selecting procedure, each job selects between its difference and intersection sets for sampling.
In the sampling procedure, each job randomly picks an element e from the subset chosen in the previous step.
This two-step process maintains the uniform distribution of sampling for each job.
Then, the proposed DSA takes a careful modification in the sampling procedure.
As shown in
Figure 2c, if the two jobs both select the intersection set, one shares the sampling result to another.
We call this modification shared sampling .
Based on shared sampling, the probability that two jobs selecting the same element is equal to that both jobs selecting the intersection set.
To further improve locality, we need to improve the probability of both two jobs selecting the intersection, which can be achieved according to the conditional probability, as shown in Figure 2b.
The independent events in the selecting procedure can be transformed into the dependent ones, making the probability of the second job conditional.
That is, J 2 makes decisions conditioned on the decision of J 1 .
To maximize the sampling locality, we formulate the conditional probabilities of dependent selecting in case of | D 1 | < | D 2 | .
For J 1 , the dependent selecting procedure is the same as that in ISA: selecting
I with the probability of | I | | D 1 | , and D d 1 with the probability of | D d 1 | | D 1 | .
The selecting procedure of
J 2 makes the decision according to J 1 .
If J 1 selects D d 1 , then J 2 must select another difference set D d 2 .
If J 1 select I , then J 2 selects I with the probability of | D 1 | | D 2 | and D d 2 with probability | D 2 |−| D 1 | | D 2 | .
Figure 2b shows the above selecting procedure.
For the case of | D 2 | < | D 1 | , we just need to exchange the order of J 1 and J 2 to apply the above rules.
According to above conditional probabilities, the probability of p ( e 1 j = e 2 i ) is | I | max( | D 2 | , | D 1 | ) , which is also the theoretical upper bound as discussed in Theorem 1.
The proof of this theorem can be found in Proof 1.
Theorem 1 Assume there are 2 jobs J 1 , J 2 sampling uniformly at random from two datasets D 1 , Drespectively, and the elements they sampled are e 1 j , e 2 i .
The probability of these two elements being equal cannot be larger than [Equation] N-job Case.
DSA of n -jobs is similar to two-jobs at the beginning.
The difference happens when some jobs select difference sets.
We need to go through the above procedure recursively for these jobs rather than sampling in the current difference set directly, as shown in Figure 3. The reason for recursive execution is that some data may only be shared by part of the jobs. In the extreme case, if all datasets are mutually disjoint, then the intersection in the first recursion is empty and the DSA is useless. Therefore, we need to further check whether there are shared data for part of the jobs.
Intersection calculating
Dependent selecting
Shared sampling except the first job
D
I
J
J
J1 J2 J
I
I J
I' J2 J difference intersection job selects difference job selects Intersection
D J job samplesin D
J
D
J
D
J
D
J
D
J
Dd
J
D
J
D
J
Dd J D
J
Dd
J D J D
J
Dd
I J
Dd
I J
Dd
I J Dd I' J Dd3' I' J
Dd2'
I J
Dd
I J
Dd
I J
Dd
I' J
Dd3' I' J
Dd2'
Dd
J
I' J2 J
I J
Dd
I J
Dd
I J
Dd
The conditional probability for n-jobs that are sorted by the cardinalities of their datasets is formulated as follows
The first job J 1 selects the intersection with the probability of | I | | D 1 | while selecting the difference with
the probability of | D d 1 | | D 1 | . If the job J i − 1 selects the difference set, then job J i must select the difference set.
If job J i − 1 selects the intersection set, then job J i selects the intersection with the probability of | D i − | | D i | while selecting the difference set with the probability of | D i |−|
D
i − 1 | | D i | .
For the above algorithm, we can observe that if one job selects the difference, then all subsequent jobs will choose the difference, which is an important feature for the data structure in Section 4.
The pseudo code of DSA are shown in Appendix B for completeness.
Here, we provide the theoretical result as follows, whose proofs can be found in Appendix D.2.
Theorem 2 Assume there are multiples jobs with the i -th job J i trained on dataset D i .
The probability of any element e being picked for J i via DSA is p ( e )
= 1 | D i | .
3.2 RefCnt Cache Policy
If we allow jobs to vary greatly in speed, then some data may be evicted before they are consumed by all jobs.
Therefore, we need a cache policy that can take out the data that is least likely to be used (i.e., the data that least jobs will request) in the near future.
Classical cache policies (e.g., LRU and LFU) are not suitable for deep learning scenarios, where each training job reads each data only once in an epoch.
So the total number of request to each data equals to the number of jobs using the data for training.
To this end, if we could record the count of data request, the number of request to this data in near future could be derived.
The fewer jobs that access the data, the higher the eviction priority of that data.
Based on the above understanding, we design the caching policy RefCnt .
We maintain a reference count for each data element.
If a job will request the element in the future, we increase its reference count; if a job reads the data, we decrease its reference count.
When the cache is full, the data with the lowest reference count will be evicted.
Dependent Sampling Tree
The time complexity of DSA is O ( nm ) , where n is the number of jobs and m is the maximum cardinality of the datasets.
The worst case happens if all jobs select the difference set in each recursion, which means we need n recursions for n jobs.
Each recursion has the time complexity of O ( m ) .
To reduce the time complexity of sampling, we can maintain the intersections for DSA to avoid the intersection calculation in each recursion, which can make each recursion cost constant time.
In the worst case of DSA, the complexity of DSA is reduced from O ( nm ) to O ( n ) .
We propose the dependent sampling tree for organizing the intersections.
Although the number of intersections is 2 n − 1 for n sets, only n − 1 intersections are valuable.
In DSA, we only need to calculate the intersection of datasets for those jobs that select the difference, as shown in Figure 3.
When the jobs are sorted by the cardinality of their datasets, the job selecting the difference will make all following jobs select the difference sets.
Therefore, the possible combinations for jobs that select the difference must be the suffixes of the sorted jobs array.
For example, if there are three sorted jobs [ J 1 , J 2 , J 3 ] , the groups of jobs choosing the difference set can only be one of the following three sets, including [ J 1 , J 2 , J 3 ] , [ J 2 , J 3 ] , and [ J 3 ] .
Thus, the sets to be calculated are { D 1 ∩ D 2 ∩ D 3 , D 2 ∩ D 3 } .
4.1 Tree Definition
For n datasets { D 1 , D 2 , ..., D n } in ascending order w.r.t.
| D 1 | < | D 2 | < ... <
| D n | , we need to calculate n − 1 intersections { D 1 ∩ ...
∩ D n , D 2 ∩ ...
∩ D n , ..., D n − 1 ∩ D n } .
We build a dependent sampling tree, where each intersection is an internal vertex, and each difference set is a leaf.
Meanwhile, all vertices in a path from the root to a leaf represent a complete dataset.
The definition of dependent sampling tree is as follows
Definition 1 Dependent sampling tree is a binary tree that stores a collection of datasets { D 1 , D 2 , ..., D n } , with | D 1 | < | D 2 | < ... <
| D n | .
In the tree, the root is the intersection of all datasets I
=
D 1 ∩ D 2 ...
∩ D n .
The right child is the difference of the smallest dataset D 1 \ I .
The left child is a new dependent sampling tree of { D 2 \ I, D 3 \ I, ..., D n \ I } .
When there is only one dataset, the tree has one vertex that contains the dataset.
If there are n datasets, the height of the dependent sampling tree is n − 1 .
Figure 4a shows an example with three sets { D 1 , D 2 , D 3 } with | D 1 | < | D 2 | < | D 3 | .
In this tree, vertex A contains
D A
= D 1 ∩ D 2 ∩ D 3 , and vertex B contains D B
= ( D 2 \ D A )
∩
( D 3 \ D A ) .
For the leaf vertices
C , E , and F , they maintain the other differences.
4.2 Tree Operations
When executing the DSA algorithm, dependent sampling tree should maintain the datasets into intersection and difference sets dynamically.
Furthermore, dependent sampling tree should also support the situations when a job finishes its execution or a new job starts.
In this part, we discuss the operations provided by dependent sampling tree, which are sampling , deletion , and insertion .
Sampling.
The intersections of the flow in Figure 3 can be represented by one or multiple vertexes of the rightest path in the dependent sampling tree.
Thus, the process of DSA is to traverse the path from the root to the most-right leaf in the tree.
When traversing to an internal vertex, the jobs will decide whether to select this vertex according the conditional probability. For those jobs that do not select this vertex, they will make decisions in the right child tree. Sometimes the intersection can be composed of multiple vertexes. For example, if only the job of Dselects difference at A in Figure 4a, then it can sample both in B and F .
In these cases, the algorithm needs to first select a vertex that belongs to the intersection and then sample. Dataset Deletion. When a job finishes training or is killed, we should delete its dataset from the tree.
The deletion involves two steps: 1) deleting the corresponding leaf vertex, and 2) merging its parent vertex with its sibling vertex. For example, if we want to delete D 2 in Figure 4a, we need to delete the leaf vertex E and merge vertex B with F , as shown in Figure 4b. The time complexity is O (1) since deleting and merging both cost constant time. The intersection calculation needs to traverse the dataset so its complexity is linear to the cardinality of the dataset. 2)
In the worst case, only the first job needs to make the decision, and others follow it.
Therefore, dependent selecting needs constant time.
Sampling only needs to pick a random number between 0 and the size of the dataset, which needs constant time.
A
B
C
E
F
D D
D (a) Dependent sampling tree of three datasets { D 1 , D 2 , D 3 }
A
B+F
C
E
D D
D (b)
Delete a dataset D 2 from the dependent sampling tree
A' B
C
E
F
D D
D D
X (c) Insert a smaller dataset D ( | D | ≤ |
D 1 | ) in dependent sampling tree
A'+B
A'+C
E
F
D D
D
D
D'
X insert (d) Insert a larger dataset D ( | D | > | D 1 | ) in dependent sampling tree
Dataset Insertion.
The insertion operation depends on the cardinality of the new dataset D .
When D is smaller than the smallest set in the sampling tree, the procedure involves three steps: 1) calculating the intersection of it and the root of the tree, 2) setting the new intersection as the new root of the dependent sampling tree, and 3) setting the difference set of the old root as the right child and the difference of D as the left child for the new root.
For example, Figure 4c shows the new tree after inserting a smaller dataset D .
The new root X is the intersection of D and the set in vertex A .
The old tree becomes the right sub-tree, and the difference of the dataset D becomes the left leaf.
When the new dataset D is larger than the smallest dataset in the sampling tree, we need to insert it recursively: 1) calculating the intersection of D and root A and set it as new root X , 2) adding the difference of A to its children, 3) deleting the old root A and adding their children to the new root, 4) recursively executing step 1 on X ’s right child with the difference set of D until the difference set is the smallest in the current sub-tree.
An example of this procedure is shown in Figure 4d.
In the worst case, the time complexity of insertion is O ( n | D | ) if the new dataset D is the largest 4 .
Evaluation
In this section, we evaluate J OADER on ImageNet with the family of ResNet models.
We denote the default dataloader strategy in PyTorch as the ‘Baseline’ method, and further compare J OADER with the state-of-the-art method CoorDL [18].
Due to the space limit, we mainly show two experiments in the paper and more experimental results can be found in the appendix.
First, we profile the server when there are multiple training jobs to show the bottlenecks and compare J OADER with CoorDL and Baseline in synchronous case.
Second, we evaluate J OADER in asynchronous cases: 1) multiple models trained at different speeds, 2) multiple jobs arrived at different moments, and 3) multiple datasets only partially overlapped.
For the results in the appendix, we summarize them as follows: 1) ablation studies show that both DSA and RefCnt improve the training efficiency , 2) the I/O speed is not the bottleneck for data loading , and 3) we train ResNet18 with PyTorch and J OADER separately to show J OADER does not affect the accuracy.
The evaluated models are the basic models with their default settings in torchvision [1], and trained on top of the PyTorch 1.6.0 DL framework.
The experiments were conducted on a GPU server with two Intel Xeon Gold 5118 CPUs @
2.30GHz (24 physical cores and 48 threads), 500GB RAM, and 6 TITAN RTX GPUs.
The server ran Ubuntu 18.04 with GNU/Linux kernel 4.15.0.
The disk is
Symbios Logic MegaRAID SAS-3 3316 of 1GB/s read speed.
5.1 Performance in Synchronous Cases
In this experiment, we start training multiple ResNet18 models at the same time but with different hyper-parameters.
We start from training 1 model to training 6 models on ImageNet, and profile the server for different dataloaders.
The experiments show that the CPU utilization tends to be the bottleneck, as shown in Figure 5.
The CoorDL and J OADER both can reduce the CPU utilization greatly by sharing the data preparation.
Additionally, it shows that J OADER can achieve close performance to CoorDL when all training jobs start at the same time and have a similar speed.
Note that CoorDL is designed for such synchronous cases while J OADER can also handle asynchronous cases as we will later show.
The time cost of each [Figure]
CPU quickly becomes the bottleneck for the
Baseline dataloader of PyTorch, while both
CoorDL and J OADER can greatly reduce CPU utilization.
[Figure]
J OADER can reduce the CPU utilization and keep each job’s own speed.
In contrast, fast job must wait for the slow job for CoorDL (e.g., ResNet18 costs the same time with
ResNet152).
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
epoch is shown in Table 1.
With 6 training jobs, J OADER saves 44 .
8% training time with over 40% less CPU utilization, compared the default Baseline
PyTorch when training multiple ResNet18 models in an epoch.
[Table]
5.2 Performance in Asynchronous Cases
Next, we consider three asynchronous cases when multiple models differ in training speeds, arriving moments, and datasets.
Multiple models with different training speeds.
We first simultaneously train five models (ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152) with different training speeds, and show the lifetime and CPU utilization results in Figure 6.
With the Baseline dataloader provided by PyTorch, each job preprocesses the data individually so that the CPU load is extremely heavy.
CoorDL enforces multiple jobs requesting the data in lockstep to share the data preparation work, and it achieves less CPU load and higher training speed than Baseline.
However, the faster jobs (smaller models) must wait for the slower jobs, making CPU idle during the training period.
J OADER can both reduce the CPU load by sharing the preprocessing work and keep these jobs’ own training speeds.
For example, ResNet18 runs 4 × faster using J OADER compared with CoorDL.
Furthermore, this experiment suggests that our system has less profitable for larger models because GPU tends to be the bottleneck for those models.
Multiple models arriving at different moments.
Next, we consider the situation that multiple DNN training jobs start at different moments.
Since CoorDL must wait for all the jobs to be ready for training, we only compare J OADER with Baseline in this experiment.
Figure 7a shows the CPU utilization when four ResNet18 training jobs arrive at different moments.
J OADER takes up less CPU load than the Baseline.
This is due to the fact Baseline repeats the data preparation work for new jobs, whereas J OADER reuses the prepared data from previous jobs.
As a result, J OADER saves 17.1% training time and 8.7% CPU utilization.
Multiple models with partially overlapping datasets.
Finally, we show the CPU utilization when the two datasets only partially overlap in Figure 7b.
The datasets are re-sampled from ImageNet.
We use the ratio between intersection size and dataset size to measure the overlapping degree.
The larger the intersection, the more data to share.
We still compare with the Baseline method only since
CoorDL requires all jobs having the same access path over datasets.
From the figure, we can observe that the CPU utilization of the same dataset is only 60% of that of different datasets.
Additionally, the CPU utilization always decrease as the intersection size increases, showing that J OADER is valid with some overlapping elements.
The reason is that the DSA algorithm can get the same sampling results with the max probability and the cache policy evicts the useless data with higher priority.
Conclusion
Accelerating DNN training is a fundamental challenge in deep learning.
This paper presents an approach to boosting training efficiency by caching data prep work and reusing them to parallel training jobs.
To prevent cache thrashing, we propose DSA to improve sampling locality with ensured randomness.
We also propose a novel tree-based data structure to efficiently implement the proposed algorithm.
Furthermore, a domain-specifical cache policy is proposed for evicting data that is least used in the near future.
At last, we evaluate the proposed approach and the results show significant improvement in asynchronous cases while achieving close performance in synchronous cases to
CoorDL which is specially designed for such cases.
Discussion
Limitations.
Although J OADER proved its effectiveness in classic scenarios, it inevitably works under certain constraints.
For the Dependent Sampling Algorithm, superior locality as it brings, it breaks the strict independence across workloads, which means for correlated workloads, e.g., parallel tasks aimed for variance reduction by by ensembles, Joader won’t achieve an as good result as workloads without strict independence constraints.
Bottleneck.
For efficiency of the Dependent Sampling Tree implementation, the bottleneck can come from CPU or disk I
/O. Based on our experiment in Appendix A,
disk I/O cost during training process would be masked by CPU cost if the disk
I/
O speed is over 350MB/s. Otherwise if the disk
I/O read speed is under 350MB/s, the bottleneck would come from I/O. We tested our implementation under a
SSD with disk I/
O read speed at 1GB/s, which is a pretty common spec, any disk with better read speed should achieve better results.
Cost of Dependent Sampling Tree.
J OADER introduce the Dependent Sampling Tree compared with primeval dataloader, which can bring more operations for datasets, such as calculating intersection.
To reduce them, the bitmap index are used to represented the dataset.
To evaluate the time cost of dataset operations, we have conducted an additional experiment by randomly inserting 128 datasets into our system.
The size of each dataset is between 1,000,000 and 2,000,000 elements (note that
ImageNet contains 1,400,000 images).
The average cost of inserting each dataset is 0.57 seconds.
Other machine learning tasks.
J OADER can be applied to all machine learning tasks which need to iteratively traverse the dataset in random order.
However, the benefit depends on the cost of reading and processing.
In computer vision, image is usually large, and image decoding takes much calculation.
Thus, J OADER can get better benefits.
In contrast, if the data is small, e.g., each sample in the NLP dataset only tasks several bytes, and the cost of processing is little.
In this case, our system cannot speed up training too much.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their helpful comments.
This work was supported in part by the Key-Area Research & Development Program of Guangdong Province (No. 2020B010164003), the National Natural Science Foundation of China (No. 62025202, 62172199) and the Collaborative
Innovation Center of Novel Software Technology and Industrialization.
Hanghang Tong is partially supported by NSF (1947135)
A Evaluation
In this section
[Figure]
we first present the I/O speed in the synchronous cases and asynchronous cases to show J OADER can reduce the redundant I/O work.
Then, we evaluate the algorithm DSA and RefCnt separately for the ablation test.
Finally, we show the loss and accuracy trace of ResNet18 inepochs to demonstrate J OADER does not affect convergence speed.
We denote the default dataloader strategy in PyTorch as the ‘Baseline’ method [Figure] and further compare JOADER with the state-of-the-art method CoorDL.
A.1 I/O speed
In this part, we show the I/O speed in the synchronous and asynchronous cases.
Note that I/O is not the bottleneck in our server (Maximum I/O speed of the server is about 1GB/s).
Figure 8 shows the I/O from training one job to training six jobs.
The I/O speed of Baseline tends to increase while that of CoorDL and J OADER tends to decrease, because CoorDL and J OADER can reuse the data in memory.
When we train 5 jobs (ResNet18, ResNet34, ResNet50, ResNet101, ResNet152) with different speed, the I/O speed is shown in Figure 9.
Although the I/O speed of CoorDL is far less than J OADER , the fast job must wait for a slow job causing inefficiency.
J OADER can make jobs run at their own speeds.
Figure 10a show the I/O speed for four jobs that start at different moments.
Due to the redundant I/O work, the baseline has a more considerable I/O speed.
For J OADER , the more data is shared, the smaller the I/O speed, as shown in Figure 10b. [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] A.2 Algorithm evaluation In this section, we first compare ISA, the default sampling algorithm in PyTorch, with DSA to show the performance in two cases: 1) datasets overlap partially, and 2) datasets vary in size.
Then we further compare the RefCnt with the generic cache policy in the above cases.
We evaluate these algorithms in different cache sizes and the evaluation metric is the count of cache misses , indicating the number of elements not read from the cache.
For n datasets D 1 , D 2 , ..., D n , the maximum count of cache misses is | D 1 | + | D 2 | + ... + | D n | that all the data is read from the storage, while the minimum is | D 1 ∪ D 2 ∪ ... ∪ D n | that all the repeated data is hit in cache.
We construct the following cases for evaluation 5 in Table 2:
Dependent sampling algorithm.
In this part, we show DSA can significantly reduce the cache misses in various of scenes.
To simulate various scenes, we generate multiple datasets with different sizes and different intersections.
We use DSA to access them in shuffling order from the storage with the cache of different sizes (e.g., from one-slot to holding all datasets) to evaluate the number of misses, compared with the experimental results with ISA (shuffling independently).
We start the the evaluation in two jobs training on dataset 1 and dataset 2 , respectively.
We set up eight different configurations for the two datasets, which can be divided into two types: the different sizes of intersections and different sizes of the datasets, as shown in Table 2.
5 1) D =
[0 , 1 e 4] means dataset D constains all number from 0 to 10000.
2) D = sample ([0 , 13333] , 10000) means sample a subset D with 10000 of size from [0 , 13333] uniformly at random
[Table]
SamplingAlgorithm overlappingpartially [Table] varyinginsize [Table]
CachePolicy overlappingpartially [Table] For different intersection sizes, Figure 11a plots the relations between cache size and the count of cache misses. For ISA, the misses count is 2x higher than the size of dataset in the case of one-slot cache, shows that all elements are missed and it decreases linearly as the cache size increase.
Unless the cache can hold all the data of intersection, the misses count could be minimized.
However, DSA can always get the minimum cache misses regardless of the cache size.
The reason is that in the selecting procedure, all jobs must select the same subset when the datasets’ sizes are equal, due to the conditional probability of 1.
Thus, their sampling results would be the same, meaning that all jobs will access the same element simultaneously when the datasets with the same size.
Figure 11b plots the evaluation results on for the datasets with different sizes.
For one-slot cache,
DSA can reduce 50% cache misses compared with ISA when the dataset sizes differ by 25%.
The cache held 30% dataset could get the best performance for DSA while ISA needs to cache 75% dataset to reach optimal.
DSA needs to cache 25% data because the size difference brings some uncertainty in the selecting procedure.
Then, we evaluate the DSA algorithm on four DNN training jobs to show the outstanding performance of the DSA algorithm on multiple datasets.
Multiple datasets bring more complexity to the intersection as there can be 2 n − 1 intersection sets for n sets.
To simulate an utterly random scene, we random sample 10,000 elements from 13,333 elements four times, making four different datasets for evaluation.
Compared ISA in the random scene, DSA can reduce 50% misses with a one-slot cache (20000 misses out of 40000 misses), as shown in Figure 13a.
DSA also provides good performance for multiple jobs training on datasets with different sizes.
Compared to ISA in Figure 13b, one-slot cache can reduce 30% cache misses (16000 misses out of 22000 misses) with DSA when four jobs are training on four datasets, with a 25% different size.
RefCnt cache policy.
In this part, we compare cache policy RefCnt with the generic cache policies
LRU, FIFO, and random replacement (RR) to show 10% misses reduction with the same cache size.
We evaluate these policies in above two situations of four jobs: utterly random scene and different dataset sizes.
Figure 12 shows the results, which can be summarized as two improvements: 1) RefCnt cache policy can reduce 10% misses in the same cache size compared with general cache policies, and 2) RefCnt cache policy can get the best performance with only caching the 60% dataset while the generic cache policies require holding the whole dataset.
A.3 Correctness experiments
To demonstrate J OADER does not affect convergence speed w.r.t.
accuracy and loss, we train ResNet on ImageNet-1K with J OADER and PyTorch 6 .
Figure 13 shows that the loss descent trajectory and accuracy trajectory are almost the same.
[Figure]
[Figure]
[Figure]
B Dependent Sampling Algorithm
In this section, we show the pseudo-code of the DSA, which involves two functions: selecting and sampling.
The selecting algorithm is shown in Algorithm 1, and the sampling algorithm is shown in
Algorithm 2.
Algorithm
Data: n jobs J 1 , ..., J n and n datasets D 1 , ..., D n
Result:
The subset that each job selects 1 JOB ← [ J 1 , ..., J n ] ; // Array of all jobs 2 DS ← [ D 1 , ..., D n ] ; // Array of all datasets sorted by their cardinality 3 S ← ∅ ; // Set of data that are not selected 4 while JOB ̸
= ∅ do
C Sampling in Dependent Sampling Tree
The sampling operation involves two steps: 1) Deciding: choose a vertex for each job, and 2)
Sampling: select an element in the corresponding vertex for each job randomly.
In deciding , we start the procedure from the root of the tree.
In the beginning, all jobs are marked as undecided .
In each vertex, all undecided jobs need to decide whether to select the current vertex, i.e.,
Data: n jobs J 1 , ..., J n and k subsets S 1 , ..., S k
Result:
The sampling results of each job 1 JOB ← [ { J i , ..., J j } , ... ] ; //
The job set where job selects the correspond subset 2 SUBSET ← [ S 1 , ..., S k ] ; //
The subset that the correspond job set selects 3 for i in 0..
k do subset ← SUBSET [ i ] ; job _ set ← JOB [ i ] ;
Sample e uniformly at random in subset ; for job in job _ set do job picks e ; the intersection I in the DSA algorithm.
If the job chooses the current vertex, we will remove them from the undecided job set.
Then, the remaining undecided jobs are put down to the child and make decisions recursively until the undecided job set is empty, like Algorithm 1.
However, in deciding , there may be an intersection that contains multiple vertices of the sampling tree.
As shown in Figure 4c, if job J 2 with D 2 and job J 3 with D 3 do not choose the root X in the first recursion, then they should make decision in the intersection ( D 2 − D X )
∩
( D 3 − D X ) , which contains the datasets in vertices A and B .
In this case, we should combine these vertices into a big set first and start the deciding procedure once.
If someone chooses the big set, it should choose a vertex from the big set randomly again.
After the deciding procedure, each job is assigned to a vertex randomly.
Then, we need to select an element in the corresponding vertex randomly.
Based on the united sampling discussed in Section 3, those jobs that are assigned to the same vertex should share the same sampling result.
Each data item in the dataset should be read exactly once for a job in an epoch.
Intuitively, the data item should be deleted from the vertex to avoid being read by the job repeatedly.
This process raises the issue that other jobs may need the deleted element that some have read.
For example, suppose in the Figure 4a, job J 1 fetches the element e 1 from vertex
A while job J 2 and job J 3 fetch the element from other vertices.
Thus, for J 1 , we should remove e 1 from vertex A , but J 2 and J 3 still need the element e 1 .
The compensation procedure is introduced in sampling to solve this issue, which adds the deleted element to the child vertex of the job that still needs it.
To decide which vertex needs compensation, we construct a job set for each vertex that contains all the jobs that will sample in this vertex.
For example, the job set of vertex A is { J 1 , J 2 , J 3 } and the set of vertex B is { J 2 , J 3 } in Figure 4a.
And after deleting some elements, we should construct a compensation set containing all the jobs that still need the deleted elements.
If the compensation set includes the job set, the deleted elements are added to the corresponding vertex.
Partially Sampling.
When some jobs are much faster than other jobs, we only need the sampling results for these fast jobs.
The sampling procedures are the same as the above.
However, the tree cannot be ordered after sampling because these datasets decrease fast and can be less than the above datasets in the tree soon.
Therefore, we need to reorder the tree, just like the intersection procedure.
D Proof of Algorithm
In this section, we present the derivation and the proof for the proposed algorithm.
Propositionshows the first principle for randomness.
Table 3 presents some notations for the following proof.
Proposition 1 For any job J i , the probability of choosing any element from dataset D
i is 1 | D i | .
[Table]
[Equation]
[Table]
[Equation]
[Table]
D.1 Derivation in Two-job Case
There are 2 jobs J 1 and J 2 which training upon datasets D 1 and D 2 , while the intersection is
I =
D 1 ∩ D 1 and two difference sets are D d 1 = D 1 −
I, D d 2 = D 2 − I .
Their cardinalities are | D 1 | , | D 2 | , | I | , | D d 1 | , and | D d 2 | .
The event that J 1 selects the intersection is J I 1 , while the event that
J 1 selects the difference set is J D 1 .
Due to the randomness, the constraints that we need to maintain are
[Equation]
Since J 2 makes the decision conditioned on J 1 , we can the formulate the equations according to the law of total probability in below
[Equation]
After shared sampling , our target is to maximize the probability of J 1 and J 2 both choose the intersection, which is positive correlated with p ( J I
|
J I 1 ) .
The p ( J I 2 |
J
I 1 ) is
[Equation]
Therefore, we can get the conditional probability in two cases
[Equation]
When | D 1 | ≤ | D 2 | , the other conditional probabilities of J 2 conditioned on J 1 can be derived as follows
[Equation]
In dependent selecting procedure, the probability of both choosing intersection set is p ( J I 1 ) ∗ p ( J I
| J I 1 ) .
With data partition , we can get
[Equation]
while e 1 j , e 2 i is the sampling results of J 1 , J 2 in one round.
We then prove the the proposed dependent algorithm gives the optimal solution for the scenario of two-jobs.
Proof 1 Assume that the distribution of the event exists, and the probability of p ( e 1 j = e 2 i ) is unknown.
When e 1 j and e 2 i are equal, they must belong to the intersection set D i .
Therefore, p ( e 1 j = e 2 i ) is equal to p ( e 1 j = e 2 i , e 1 j ∈ D
i , e 2 i ∈
D i ) .
Due to the definition of joint probability, we can get p ( e 1 j = e 2 i ) ≤ p ( e 1 j ∈ D i ) , and p ( e 1 j = e 2 i ) ≤ p ( e 2 i ∈
D i ) .
Due to the constraint of randomness, e 1 j and e 2 i are both sampled under the uniform distribution, so p ( e 1 j ∈ D i ) =
| D i | | D 1 | and p ( e 2 i ∈
D i ) =
| D i | | D 2 | .
Then p ( e 1 j = e 2 i ) ≤ | D i | | D 1 | and p ( e 1 j = e 2 i ) ≤ | D i | | D 2 | both hold.
Thus, for any distribution of the event, it holds that
[Equation]
D.2 Derivation in N-job Case
Assume there are n jobs { J 1 , ..., J n } and n datasets { D 1 , ..., D n } , while the intersection I = (cid:84) ni =1 D i .
For the k -th job where k > 1 , the probability of the job J k chooses the intersection is
[Equation]
where x is the sum of probabilities of J k chooses intersection in other conditions.
With the similar process in Proof 1, we can get the result for k -jobs, where the maximum probability of they all choose intersection is also p ( J I 1 ...J Ik ) ≤ | I | max ( D 1 ,...,D k ) .
Then, we can get
[Equation]
Assume that the datasets are sorted in ascending order w.r.t.
their cardinalities, then the formula can be written as for the maximum probability
[Equation]
The probability of n jobs all choose intersection I is
[Equation]
which is the theoretical maximum according to a similar Proof 1.
D.3 Proof of Randomness
We prove the Proposition 1 for Algorithm 1.
Although multiple jobs share the sampling results in the sampling procedure, the random sampling is not changed.
Therefore, to prove Proposition 1, we only need to prove the subsets are randomly selected for each job.
The Lemma is stated as follow
Lemma
In each loop of selecting procedure, the intersection I is selected with the probability of | I | | D i | for every job J i .
In the line 8-12 of Algorithm 1, the job J i chooses the intersection only if the previous job has chosen the intersection I , that the probability is
[Equation]
Therefore, the Lemma 1 is satisfied only if the probability of entering this loop is | D i |−| S | | D i | , that is
Lemma
For each loop, the set of elements that are not selected is S , and the probability of entering this loop is | D i |−| S | | D i | for job J i .
Proof 2 We prove the Lemma 2 by induction on loop index k .
Base case.
Show Lemma 2 holds for the first loop.
The first loop ( k = 1 ) must be entered, while S is an empty set and | D i |−| S | | D i | = 1 .
Thus Lemma 2 is satisfied.
Induction step.
Show that for any k ≥ 1 , if the k -th recursion holds Lemma 2, then ( k + 1) -th loop also holds.
The probability of entering the k -th loop is | D i |−| S | | D i | , and if job J i does not select the intersection in the k -th loop, then it entering the ( k + 1) -th loop.
The probability of J i does not select intersection is
[Equation]
where J k is the first job in job List JOB , then we can get the probability of entering the next loop is | D i |−| S |−|
I | | D i | while new S is S ∪ I , which also satisfies the Lemma 2.
Conclusion.
Since both the base case and the inductive step have been proved as true, the Lemmaholds for every loop by mathematical induction.