Repository: facebookresearch/DomainBed
Issue: Slow training speed for custom dataset (million images)

My dataset have 2 domains, the first domain is for training, which contains 1.35 million images.
And the other domain contains about 10,000 images.

I modified the source code in domainbed/datasets.py so that I can load my dataset

I trained with the following command:
```bash
CUDA_VISIBLE_DEVICES=2 python3 -m domainbed.scripts.train --data_dir=../mydataset/ --algorithm CORAL --dataset mydataset
```
However, the screen stuck after the following text output
```
Environment:
        Python: 3.6.7
        PyTorch: 1.7.1+cu92
        Torchvision: 0.8.2+cu92
        CUDA: 9.2
        CUDNN: 7603
        NumPy: 1.19.5
        PIL: 8.3.1
Args:
        algorithm: CORAL
        checkpoint_freq: None
        data_dir: ../mydataset/
        dataset: mydataset
        holdout_fraction: 0.2
        hparams: None
        hparams_seed: 0
        output_dir: train_output
        save_model_every_checkpoint: False
        seed: 0
        skip_model_save: False
        steps: None
        task: domain_generalization
        test_envs: [1]
        trial_seed: 0
        uda_holdout_fraction: 0
HParams:
        batch_size: 256
        class_balanced: False
        data_augmentation: True
        lr: 5e-05
        mmd_gamma: 1.0
        nonlinear_classifier: False
        resnet18: True
        resnet_dropout: 0.0
        weight_decay: 0.0
```

Is there any method to handle such a big dataset? or anything I can do for this situation?

Just found that the training finally done, but how can I test the performance of the model?

I don't quite understand the meaning of the following number from the output log:
```
0.9977020161  0.9976049940  0.4900733707  0.4831606218  138.10962451  0.0042221369  5.5169997215  0.0000000000  5000          0.3844062908 
```

See #53 

@e96031413, I have also successfully loaded my dataset although it is not as big as yours. It has still been stuck for a long time. Approximately how long did you have to wait? Is there anything you did to get the output?

I will paste my output as well:


Environment:
        Python: 3.10.12
        PyTorch: 1.12.1+cu102
        Torchvision: 0.13.1+cu102
        CUDA: 10.2
        CUDNN: 7605
        NumPy: 1.22.4
        PIL: 10.3.0
Args:
        algorithm: ERM
        checkpoint_freq: None
        data_dir: None
        dataset: CustomDataset
        holdout_fraction: 0.2
        hparams: None
        hparams_seed: 0
        output_dir: train_output
        save_model_every_checkpoint: False
        seed: 0
        skip_model_save: False
        steps: None
        task: domain_generalization
        test_envs: [0]
        trial_seed: 0
        uda_holdout_fraction: 0
HParams:
        batch_size: 32
        class_balanced: False
        data_augmentation: True
        dinov2: False
        freeze_bn: False
        lars: False
        linear_steps: 500
        lr: 5e-05
        nonlinear_classifier: False
        resnet18: False
        resnet50_augmix: True
        resnet_dropout: 0.0
        vit: False
        vit_attn_tune: False
        vit_dropout: 0.0
        weight_decay: 0.0
Using GPU


@RohitSingh1226 
Because it was several years ago, I've forgotten the exact time it took.

@e96031413 do you have record of your Multiple Domain Custom Dataset file? My training has been stuck for some time now.



@RohitSingh1226 No.

> ，我也成功加载了我的数据集，尽管它没有你的数据集那么大。它仍然卡住了很长时间。您大约需要等待多长时间？你做了什么来获得输出吗？
> 
> 我也会粘贴我的输出：
> 
> 环境： Python： 3.10.12 PyTorch： 1.12.1+cu102 Torchvision： 0.13.1+cu102 CUDA： 10.2 CUDNN： 7605 NumPy： 1.22.4 PIL： 10.3.0 参数： 算法： ERM checkpoint_freq： 无 data_dir： 无 数据集： CustomDataset holdout_fraction： 0.2 hparams： 无 hparams_seed： 0 output_dir： train_outputsave_model_every_checkpoint： 假种子： 0 skip_model_save： 假步骤： 无 任务： domain_generalization test_envs： [0] trial_seed： 0 uda_holdout_fraction： 0 HParams： batch_size： 32 class_balanced： 假 data_augmentation： 真 dinov2： 假 freeze_bn： 假 lars： 假 linear_steps： 500 lr： 5e-05 nonlinear_classifier： False resnet18： False resnet50_augmix： True resnet_dropout： 0.0 vit： False vit_attn_tune： False vit_dropout： 0.0 weight_decay： 0.0 Using GPU

Have you finally completed the training? If this happens to me, the task will be terminated