Repository: pytorch
Issue: Python3 Depletes 2021 M1 Mac Memory Running Training Ops For Model's M, L and X
Labels: triaged,module: macos

### 🐛 Describe the bug

Hi Admin exec/s,

I am experiencing a strange issue where Python3 is hogging all of the resources while running the training operations for the yolov5 model's M, L and X.  It is pointless to deploy the m6 models, for examples yolov5m6.pt or .ymal, l6, x6.

My research lead me to initialise the GPU/MPS on the silicon MacBook Pro 2021 14" MAX 32 cores. But this proved futile as the training will not allow the GPU to be initialised, so I followed PyTorch's nightly installation and that was futile as well. 

Though it was not allowing the gpu/mps processing, somehow the yolov5m.pt model was able to work as well as from scratch, but not the larger models. This (m) took 4 days to train on the minuscule batch and epoch sizes, it took approximately 2hrs+ for 1 epoch iteration cycle.

Everyday the initialisation of the Training Operations is constantly prompting that yolov5 is out of date and I have to pull the necessary files to get the process to work as it should based on my knowledge.

![Screenshot 2022-08-24 at 08 12 08](https://user-images.githubusercontent.com/50293021/186354382-89d2427f-3ded-4a99-814f-4f0c3450e3dc.png)

What I did was:
1. Create an environment  ML
2. Then install the requirements as per the instructions from @Glenn-Jocher downloaded in the folder of files from the clone process.
3. Install packages as prompted that required upgrading and then initialise the run commands below:

Packages:
```
Package                 Version
----------------------- --------------------
absl-py                 1.2.0
appnope                 0.1.3
asttokens               2.0.8
backcall                0.2.0
cachetools              5.2.0
certifi                 2022.6.15
charset-normalizer      2.1.1
coremltools             5.2.0
cycler                  0.11.0
decorator               5.1.1
executing               0.10.0
fonttools               4.36.0
google-auth             2.11.0
google-auth-oauthlib    0.4.6
grpcio                  1.47.0
idna                    3.3
ipython                 8.4.0
jedi                    0.18.1
kiwisolver              1.4.4
Markdown                3.4.1
MarkupSafe              2.1.1
matplotlib              3.5.3
matplotlib-inline       0.1.6
mpmath                  1.2.1
natsort                 8.1.0
numpy                   1.23.2
oauthlib                3.2.0
opencv-python           4.6.0.66
packaging               21.3
pandas                  1.4.3
parso                   0.8.3
pexpect                 4.8.0
pickleshare             0.7.5
Pillow                  9.2.0
pip                     22.2.2
prompt-toolkit          3.0.30
protobuf                3.19.4
psutil                  5.9.1
ptyprocess              0.7.0
pure-eval               0.2.2
pyasn1                  0.4.8
pyasn1-modules          0.2.8
Pygments                2.13.0
pyparsing               3.0.9
python-dateutil         2.8.2
pytz                    2022.2.1
PyYAML                  6.0
requests                2.28.1
requests-oauthlib       1.3.1
rsa                     4.9
scipy                   1.9.0
seaborn                 0.11.2
setuptools              63.2.0
six                     1.16.0
stack-data              0.4.0
sympy                   1.10.1
tensorboard             2.10.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
thop                    0.1.1.post2207130030
torch                   1.12.1
torchaudio              0.12.1
torchvision             0.13.1
tqdm                    4.64.0
traitlets               5.3.0
typing_extensions       4.3.0
urllib3                 1.26.11
wcwidth                 0.2.5
Werkzeug                2.2.2
wheel                   0.37.1
```

Initialisation Code: For Both M and M6 Models
I swapped between 412, 640 and 1280 image sizes to reduce the drag on resources. Then I tried increasing and decreasing the batch size from 4-10-16-32 and 64. I also tried --hyp low, med, high but this created more drag and nothing worked, see the images below:

COMMANDS:
1. Python3 train.py --data coco128.yaml --epoch 30 --batch 32 --weights  yolov5m6.pt --img 640 --cache 

2. Python3 train.py --data coco128.yaml --epoch 30 --batch 32 --weights  yolov5m.pt --img 640 --cache

All additional unnecessary APPS via Activity Monitor WERE STOPPED(Shut Down) 

Image of issue:
![image](https://user-images.githubusercontent.com/50293021/186357492-3653c155-d28a-4008-af74-b768afa74a7c.jpeg)
![image](https://user-images.githubusercontent.com/50293021/186357512-9a7a03db-5cf4-479c-9329-ab40e536ceb7.jpeg)


THEN, I also Tried Initialising the MPS as it was installed but a different set of errors persist.

1. Python3 train.py --data coco128.yaml --epoch 30 --batch 32 --weights yolov5m6.pt --img 640 --cache --device mps

2. Python3 train.py --data coco128.yaml --epoch 30 --batch 32 --weights yolov5m.pt --img 640 --cache --device mps


The larger models is my preference due to heavy dependency on accuracy, it's no use changing to a large model the resources are being depleted and the operations is too slow. I am aware that pytorch is still testing operations with packages and particulars for efficiency via MAC, but this is way to slow especially for a new machine with all the bells and whistles.

I have been stuck at this stage for 2 weeks now and not sure how to proceed, please assist me?

THANX LOADs in advance, I really appreciate you for taking the time to acknowledge my digital presence. 

Cheers..

Sym.

 


 

### Versions

`Python3 train.py --data coco128.yaml --epoch 30 --batch 32 --weights yolov5x6.pt --img 640 --cache --device mps
train: weights=yolov5x6.pt, cfg=, data=coco128.yaml, hyp=data/hyps/hyp.scratch-low.yaml, epochs=30, batch_size=32, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, noplots=False, evolve=None, bucket=, cache=ram, image_weights=False, device=mps, multi_scale=False, single_cls=False, optimizer=SGD, sync_bn=False, workers=8, project=runs/train, name=exp, exist_ok=False, quad=False, cos_lr=False, label_smoothing=0.0, patience=100, freeze=[0], save_period=-1, seed=0, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: up to date with https://github.com/ultralytics/yolov5 ✅
YOLOv5 🚀 v6.2-51-ge6f54c5 Python-3.10.6 torch-1.12.1 MPS

hyperparameters: lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
Weights & Biases: run 'pip install wandb' to automatically track and visualize YOLOv5 🚀 runs in Weights & Biases
ClearML: run 'pip install clearml' to automatically track, visualize and remotely train YOLOv5 🚀 in ClearML
TensorBoard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/
Downloading https://github.com/ultralytics/yolov5/releases/download/v6.2/yolov5x6.pt to yolov5x6.pt...
ERROR: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Re-attempting https://storage.googleapis.com/ultralytics/yolov5/v6.2/yolov5x6.pt to yolov5x6.pt...
############################################################################################################################################################################################################################################ 100.0%

Overriding model.yaml nc=80 with nc=13

                 from  n    params  module                                  arguments                     
  0                -1  1      8800  models.common.Conv                      [3, 80, 6, 2, 2]              
  1                -1  1    115520  models.common.Conv                      [80, 160, 3, 2]               
  2                -1  4    309120  models.common.C3                        [160, 160, 4]                 
  3                -1  1    461440  models.common.Conv                      [160, 320, 3, 2]              
  4                -1  8   2259200  models.common.C3                        [320, 320, 8]                 
  5                -1  1   1844480  models.common.Conv                      [320, 640, 3, 2]              
  6                -1 12  13125120  models.common.C3                        [640, 640, 12]                
  7                -1  1   5531520  models.common.Conv                      [640, 960, 3, 2]              
  8                -1  4  11070720  models.common.C3                        [960, 960, 4]                 
  9                -1  1  11061760  models.common.Conv                      [960, 1280, 3, 2]             
 10                -1  4  19676160  models.common.C3                        [1280, 1280, 4]               
 11                -1  1   4099840  models.common.SPPF                      [1280, 1280, 5]               
 12                -1  1   1230720  models.common.Conv                      [1280, 960, 1, 1]             
 13                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 14           [-1, 8]  1         0  models.common.Concat                    [1]                           
 15                -1  4  11992320  models.common.C3                        [1920, 960, 4, False]         
 16                -1  1    615680  models.common.Conv                      [960, 640, 1, 1]              
 17                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 18           [-1, 6]  1         0  models.common.Concat                    [1]                           
 19                -1  4   5332480  models.common.C3                        [1280, 640, 4, False]         
 20                -1  1    205440  models.common.Conv                      [640, 320, 1, 1]              
 21                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 22           [-1, 4]  1         0  models.common.Concat                    [1]                           
 23                -1  4   1335040  models.common.C3                        [640, 320, 4, False]          
 24                -1  1    922240  models.common.Conv                      [320, 320, 3, 2]              
 25          [-1, 20]  1         0  models.common.Concat                    [1]                           
 26                -1  4   4922880  models.common.C3                        [640, 640, 4, False]          
 27                -1  1   3687680  models.common.Conv                      [640, 640, 3, 2]              
 28          [-1, 16]  1         0  models.common.Concat                    [1]                           
 29                -1  4  11377920  models.common.C3                        [1280, 960, 4, False]         
 30                -1  1   8296320  models.common.Conv                      [960, 960, 3, 2]              
 31          [-1, 12]  1         0  models.common.Concat                    [1]                           
 32                -1  4  20495360  models.common.C3                        [1920, 1280, 4, False]        
 33  [23, 26, 29, 32]  1    173016  models.yolo.Detect                      [13, [[19, 27, 44, 40, 38, 94], [96, 68, 86, 152, 180, 137], [140, 301, 303, 264, 238, 542], [436, 615, 739, 380, 925, 792]], [320, 640, 960, 1280]]
[W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.
Model summary: 733 layers, 140150776 parameters, 140150776 gradients, 209.0 GFLOPs

Transferred 955/963 items from yolov5x6.pt
AMP: checks failed ❌, disabling Automatic Mixed Precision. See https://github.com/ultralytics/yolov5/issues/7908
optimizer: SGD(lr=0.01) with parameter groups 159 weight(decay=0.0), 163 weight(decay=0.0005), 163 bias
train: Scanning '/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/datasets/coco128/images/Merge_Discussions-Stab_Knife_Deploy/train/labels.cache' images and labels... 5730 found, 0 missing, 240 empty, 0 corrupt: 100%|██████████| 573
train: Caching images (7.0GB ram): 100%|██████████| 5730/5730 [00:03<00:00, 1739.79it/s]                                                                                                                                                           
val: Scanning '/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/datasets/coco128/images/Merge_Discussions-Stab_Knife_Deploy/val/labels.cache' images and labels... 474 found, 0 missing, 20 empty, 0 corrupt: 100%|██████████| 474/474 [
val: Caching images (0.6GB ram): 100%|██████████| 474/474 [00:00<00:00, 1035.74it/s]                                                                                                                                                               

AutoAnchor: 5.83 anchors/target, 0.972 Best Possible Recall (BPR). Anchors are a poor fit to dataset ⚠️, attempting to improve...
AutoAnchor: WARNING: Extremely small objects found: 64 of 30526 labels are < 3 pixels in size
AutoAnchor: Running kmeans for 12 anchors on 30526 points...
AutoAnchor: Evolving anchors with Genetic Algorithm: fitness = 0.7549: 100%|██████████| 1000/1000 [00:08<00:00, 124.95it/s]                                                                                                                        
AutoAnchor: thr=0.25: 0.9926 best possible recall, 7.51 anchors past thr
AutoAnchor: n=12, img_size=640, metric_all=0.351/0.756-mean/best, past_thr=0.497-mean: 17,15, 87,77, 198,124, 131,192, 263,176, 162,319, 384,151, 369,272, 260,481, 530,247, 488,420, 582,587
Traceback (most recent call last):
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/yolov5/train.py", line 630, in <module>
    main(opt)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/yolov5/train.py", line 526, in main
    train(opt.hyp, opt, device, callbacks)
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/yolov5/train.py", line 222, in train
    check_anchors(dataset, model=model, thr=hyp['anchor_t'], imgsz=imgsz)  # run AutoAnchor
  File "/Users/symbadian/Desktop/LIRIS_new_dataset/WITH_BG_WITHAUGs/yolov5/utils/autoanchor.py", line 58, in check_anchors
    anchors = torch.tensor(anchors, device=m.anchors.device).type_as(m.anchors)
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
(/Users/symbadian/miniforge3/ml) Matthews-MacBook-Pro:yolov5 symbadian$ 
`

cc @malfet @albanD