Repository: mlcommons/inference
Issue: about multiple GPUs on DLRM with pytorch
Root cause: Resource Mishandling

Dear sir or madam, 

    When i run the inference of DLRM, I meet some problems.
    I use the tb0875_10M.pt (12GB) model on fake dataset(made by tools)
    When I launch the test on ONE NVIDIA T4 (16GB) , it will cause "CUDA run out of memory" fault.
    Then I change my machine to 4*T4 (64GB),  it still say that run out of memory.
    So I turn to your code, in mlcommons-inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py 

(https://github.com/octoml/mlcommons-inference/blob/9e9a57c66ebe931e1da6e7d760305a55db523de9/recommendation/dlrm/pytorch/python/backend_pytorch_native.py)

I find some prolems:

1. self.ndevices = min(ngpus, mini_batch_size, ln_emb.size) 
    the "mini_batch_size" is not a parameter and is set to 1 by MLPerf. So, even the ngpus = 4, the self.ndevices will  always equal to 1.  In line 62, the dlrm.ndevices = 1 , so the line of 63  and line66 will never be executed and step into line 71.

Question : How can I use multiple GPUs to run this test ?


2. I set the mini_batch_size to a bigger number, the self.ndevices = 4,  then line 63 will be executed , but new error occurs:

File "/workdir/xieyunyao/mlcomons/inference/recommendation/dlrm/pytorch/python/backend_pytorch_native.py", line 93, in load
    dlrm.load_state_dict(ld_model["state_dict"],strict='False')
  File "/home/hadoop-hdp/miniconda3/envs/pytop/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DLRM_Net:
	Unexpected key(s) in state_dict: "emb_l.0.weight", "emb_l.1.weight", "emb_l.2.weight", "emb_l.3.weight", "emb_l.4.weight", "emb_l.5.weight", "emb_l.6.weight", "emb_l.7.weight", "emb_l.8.weight", "emb_l.9.weight", "emb_l.10.weight", "emb_l.11.weight", "emb_l.12.weight", "emb_l.13.weight", "emb_l.14.weight", "emb_l.15.weight", "emb_l.16.weight", "emb_l.17.weight", "emb_l.18.weight", "emb_l.19.weight", "emb_l.20.weight", "emb_l.21.weight", "emb_l.22.weight", "emb_l.23.weight", "emb_l.24.weight", "emb_l.25.weight".

Question : How can I use multiple GPUs to run this test ? Should I modify the mini_batch_size and make the self.ndevices euqal to my device count?

Thank you!


Hi @Water2style , do you plan to run DLRM on T4 server for MLPerf-Inference result submission? Or are you just running the tests for local experiments?

> Hi @Water2style , do you plan to run DLRM on T4 server for MLPerf-Inference result submission? Or are you just running the tests for local experiments?

Hi, at present I am running for local experiments, but in the future , our company may consider join the MLcomons for submission

I see. I think the second error may be caused by different PyTorch version. Could you try rolling back to older version of PyTorch?

> I see. I think the second error may be caused by different PyTorch version. Could you try rolling back to older version of PyTorch?

Thanks for your suggestions.  But it did not work when i turn pytorch from 1.7 to 1.4.
And it's very strange that :
I set self.ndevices bigger than 1, it occurs error, i set it equal to 1, it works. Eventhough they use same code "ld_model = torch.load(model_path)" without any map_location

Hi, I see this issue in torch 2.2.1 and dlrm. 
I cannot use single gpu because of memory issues. I have set the self.ndevices  to value >1 to use multiple gpus and i see the error 
`   raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DLRM_Net:
        Unexpected key(s) in state_dict: "emb_l.0.weight", "emb_l.1.weight", "emb_l.2.weight", "emb_l.3.weight", "emb_l.4.weight", "emb_l.5.weight", "emb_l.6.weight", "emb_l.7.weight", "emb_l.8.weight", "emb_l.9.weight", "emb_l.10.weight", "emb_l.11.weight", "emb_l.12.weight", "emb_l.13.weight", "emb_l.14.weight", "emb_l.15.weight", "emb_l.16.weight", "emb_l.17.weight", "emb_l.18.weight", "emb_l.19.weight", "emb_l.20.weight", "emb_l.21.weight", "emb_l.22.weight", "emb_l.23.weight", "emb_l.24.weight", "emb_l.25.weight". 
`
Is there any resolution? Thanks.