Repository: mlcommons/inference
Issue: DLRM ONNX support for the reference code
Root cause: External Dependency Breakage

Background below:
---
From: Spandan Tiwari <sptiwari@microsoft.com>
Date: Thursday, July 2, 2020 at 10:18 AM
To: "Rong, Zhong" <aaron.zhong@alibaba-inc.com>, "Gattupalli, Srujana" <srujana.gattupalli@intel.com>, Guenther Schmuelling <guschmue@microsoft.com>, Ashwini Khade <askhade@microsoft.com>
Cc: "Chukka, Ramesh N" <ramesh.n.chukka@intel.com>, "Cheng, Christine" <christine.cheng@intel.com>
Subject: RE: [EXTERNAL] Re: DLRM ONNX setup instructions

Embedding_bag is not supported in PyTorch exporter, but nn.embedding is. As this issue suggests, may be you can try exporting using nn.embedding instead of embedding_bag?
 
Also, as can be seen here, the ATen op is exported in only the ONNX_ATEN_FALLBACK mode, which is NOT the mode for ONNX export. The default mode for torch.onnx.export API is ONNX, but ATen ops should not be exported (ideally) in that mode. Therefore, a couple of questions:
1)	Are you using the nightly version of PyTorch?
2)	Is the export API call explicitly using ONNX_ATEN_FALLBACK or default ONNX?
Thanks,
-Spandan
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, July 2, 2020 10:08 AM
To: Gattupalli, Srujana <srujana.gattupalli@intel.com>; Guenther Schmuelling <guschmue@microsoft.com>; Ashwini Khade <askhade@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>; Cheng, Christine <christine.cheng@intel.com>
Subject: Re: [EXTERNAL] Re: DLRM ONNX setup instructions
 
Just use Netron to take a simple look of the dumped onnx model. The Aten op is used to implement EmbeddingBag pytorch op.
 
Google online and found this issue
https://github.com/facebookresearch/dlrm/issues/12
 
probably we can try Dan's solution.
 
Aaron
 
 
 
 
------------------------------------------------------------------
From:Gattupalli, Srujana <srujana.gattupalli@intel.com>
Sent At:2020 Jul. 2 (Thu.) 09:58
To:Aaron <aaron.zhong@alibaba-inc.com>; Guenther Schmuelling <guschmue@microsoft.com>; Ashwini Khade <askhade@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>; "Cheng, Christine" <christine.cheng@intel.com>
Subject:RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
+Christine
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, July 2, 2020 9:53 AM
To: Guenther Schmuelling <guschmue@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>; Ashwini Khade <askhade@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: [EXTERNAL] Re: DLRM ONNX setup instructions
 
And I was expecting intel would implement the onnx backend in v0.5/recommendation/python/ if you claim onnx is working. But i don't see the script there. I assume intel verified the model on pytorch backend? this would explain why ATen op is still working.
 
Aaron
 
------------------------------------------------------------------
From:Guenther Schmuelling <guschmue@microsoft.com>
Sent At:2020 Jul. 1 (Wed.) 19:12
To:Aaron <aaron.zhong@alibaba-inc.com>; "Gattupalli, Srujana" <srujana.gattupalli@intel.com>; Ashwini Khade <askhade@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
You want to export the model with opset_versoin=11 and you want to use a newer version of pytorch (ie. nightly).
But if I remember for dlrm it was working for intel. Is this really the same model intel used ?
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Wednesday, July 1, 2020 6:33 PM
To: Gattupalli, Srujana <srujana.gattupalli@intel.com>; Ashwini Khade <askhade@microsoft.com>; Guenther Schmuelling <guschmue@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: [EXTERNAL] Re: DLRM ONNX setup instructions
 
Hi,
 
I tried with the onnx model downloaded from https://dlrm.s3-us-west-1.amazonaws.com/models/tb00_40M.onnx.tar on onnxruntime and got the same error
 
>>> import onnxruntime as ort
>>> m = ort.InferenceSession('dlrm_s_pytorch_0505.onnx')
Warning: ATen was a removed experimental ops. In the future, we may directly reject this operator. Please update your model as soon as possible.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/ssd/aaronzhong/dlrm/py36/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 23, in __init__
    self._load_model()
  File "/mnt/ssd/aaronzhong/dlrm/py36/lib/python3.6/site-packages/onnxruntime/capi/session.py", line 35, in _load_model
    self._sess.load_model(self._path_or_bytes, providers)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from dlrm_s_pytorch_0505.onnx failed:Fatal error: ATen is not a registered function/op
>>> 
 
Best Regards,
Aaron
------------------------------------------------------------------
From:Gattupalli, Srujana <srujana.gattupalli@intel.com>
Sent At:2020 Jun. 30 (Tue.) 10:32
To:Aaron <aaron.zhong@alibaba-inc.com>; Ashwini Khade <askhade@microsoft.com>; Guenther Schmuelling <guschmue@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
Hi Aaron,
 
Following up on this, I see the load issue is resolved. Were you able to run inference on the model with onnx runtime? We haven’t tested that path after onnx load.
 
Best,
Srujana
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, June 11, 2020 6:36 PM
To: Ashwini Khade <askhade@microsoft.com>; Guenther Schmuelling <guschmue@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: [EXTERNAL] Re: DLRM ONNX setup instructions
 
Ashwini,
 
I tried load_external_data_for_model and it works.
 
Best Regards,
Aaron
 
 
 
 
 
------------------------------------------------------------------
From:Ashwini Khade <askhade@microsoft.com>
Sent At:2020 Jun. 11 (Thu.) 18:12
To:Guenther Schmuelling <guschmue@microsoft.com>; Spandan Tiwari <sptiwari@microsoft.com>; Aaron <aaron.zhong@alibaba-inc.com>; "Gattupalli, Srujana" <srujana.gattupalli@intel.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
By large model you mean a model > 2GB which has tensor data stored in a separate external file right? If yes then yes onnx can load such a model.
The support was added in this PR: https://github.com/onnx/onnx/pull/678
 
Usage: 
import onnx
onnx_model = onnx.load(<path to model>)
 
OR
import onnx
from onnx.external_data_helper import load_external_data_for_model
 
onnx_model = onnx.load(<path to model>, load_external_data=False)
load_external_data_for_model(onnx_model, <path>)
 
 
onnx checker currently has a bug and there is a fix in progress after which the checker should also work for a large model. 
 
Thanks,
Ashwini
 
From: Guenther Schmuelling <guschmue@microsoft.com> 
Sent: Thursday, June 11, 2020 5:53 PM
To: Spandan Tiwari <sptiwari@microsoft.com>; Ashwini Khade <askhade@microsoft.com>; Rong, Zhong <aaron.zhong@alibaba-inc.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
@Ashwini Khade – on the original onnx issue: does the onnx python helper support the large format ?
 
From: Spandan Tiwari <sptiwari@microsoft.com> 
Sent: Thursday, June 11, 2020 5:49 PM
To: Ashwini Khade <askhade@microsoft.com>; Rong, Zhong <aaron.zhong@alibaba-inc.com>; Guenther Schmuelling <guschmue@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
If you are using the latest PyTorch nightly, no aten namepsace ops should be exported in ONNX graphs. Only ops in ONNX spec should be exported, and if there’s some op that is not supported, you should see an error.
 
One recommendation is to set opset_version=11 in the export API call. If you still see this error, can you share your export and model code so we can take a look.
 
Thanks,
-Spandan
 
 
From: Ashwini Khade <askhade@microsoft.com> 
Sent: Thursday, June 11, 2020 5:02 PM
To: Rong, Zhong <aaron.zhong@alibaba-inc.com>; Guenther Schmuelling <guschmue@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>; Spandan Tiwari <sptiwari@microsoft.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
+ Spandan
I believe Aten op gets added when some op is not present in the onnx exporter. @Spandan Tiwari can add more details regarding pytorch exporter.
 
 
Thanks,
Ashwini
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, June 11, 2020 3:38 PM
To: Ashwini Khade <askhade@microsoft.com>; Guenther Schmuelling <guschmue@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: [EXTERNAL] Re: DLRM ONNX setup instructions
 
The pytorch version i used is preview version (nightly) downloaded from pytorch homepage. I guess this should be the latest nightly built for pytorch.
Anyway, I think I will just wait for either DLRM tf version or official onnx model download link and instruction from Srujana.
 
Aaron
 
 
------------------------------------------------------------------
From:Ashwini Khade <askhade@microsoft.com>
Sent At:2020 Jun. 11 (Thu.) 15:32
To:Aaron <aaron.zhong@alibaba-inc.com>; Guenther Schmuelling <guschmue@microsoft.com>; "Gattupalli, Srujana" <srujana.gattupalli@intel.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: [EXTERNAL] Re: DLRM ONNX setup instructions
 
Onnx does support loading large files. If the external data is not in the same directory as the model file then you need to call “load_external_data_for_model”and specify the path.
 
Regarding the error you are seeing with ORT. Aten is not supported by ort. This was an experimental op in ONNX and has been removed. Per my understanding Aten was a catch-all op in PyTorch exporter for capturing some ops that did not have an ONNX equivalent. Infact pytorch onnx exporter should not even use this op. Please update pytorch to the latest version too and then try loading the model with ort again.
 
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, June 11, 2020 1:52 PM
To: Guenther Schmuelling <guschmue@microsoft.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>; Ashwini Khade <askhade@microsoft.com>
Subject: [EXTERNAL] Re: DLRM ONNX setup instructions
 
I am using onnx master tip code
 
And I also tried onnxruntime, who i just use pip to install and i got this
 
 
(py3) aaronzhong@alics07:/mnt/ssd/aaronzhong/dlrm/training/recommendation/dlrm/onnx$ python
Python 3.5.2 (default, Oct  8 2019, 13:06:37) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime as ort
>>> m = ort.InferenceSession('dlrm_s_pytorch.onnx')
Warning: ATen was a removed experimental ops. In the future, we may directly reject this operator. Please update your model as soon as possible.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/ssd/aaronzhong/dlrm/py3/lib/python3.5/site-packages/onnxruntime/capi/session.py", line 158, in __init__
    self._load_model(providers)
  File "/mnt/ssd/aaronzhong/dlrm/py3/lib/python3.5/site-packages/onnxruntime/capi/session.py", line 177, in _load_model
    self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Fatal error: ATen is not a registered function/op
>>> 
 
Aaron
 
 
 
 
 
------------------------------------------------------------------
From:Guenther Schmuelling <guschmue@microsoft.com>
Sent At:2020 Jun. 11 (Thu.) 13:48
To:"Gattupalli, Srujana" <srujana.gattupalli@intel.com>; Aaron <aaron.zhong@alibaba-inc.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>; Ashwini Khade <askhade@microsoft.com>
Subject:RE: DLRM ONNX setup instructions
 
I think this might be the onnx python helper not supporting the ‘large’ format (I never tried, maybe Ashwini knows).
This is onnx-1.7?
You can try with onnxruntime:
import onnxruntime as ort
m= ort.InferenceSession(model_path)
 
 
From: Gattupalli, Srujana <srujana.gattupalli@intel.com> 
Sent: Thursday, June 11, 2020 12:37 PM
To: Rong, Zhong <aaron.zhong@alibaba-inc.com>; Guenther Schmuelling <guschmue@microsoft.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: [EXTERNAL] RE: DLRM ONNX setup instructions
 
+1 Guenther
Adding Guenther for expertise on ONNX model loading. To provide context, Aaron has converted 100GB pytorch model to ONNX using the same steps as we did for conversion. He has below error on model load.
 
Hi Aaron,
 
We were able to load the model on a CPU instance on AWS (r5.16xlarge). This seems like a protobuf memory issue which we haven’t encountered.
 
Regards,
Srujana
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Thursday, June 11, 2020 11:28 AM
To: Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: DLRM ONNX setup instructions
 
Srujana,
 
I tried to load the model by doing
 
(py3) aaronzhong@alics07:/mnt/ssd/aaronzhong/dlrm/training/recommendation/dlrm/onnx$ python
Python 3.5.2 (default, Oct  8 2019, 13:06:37) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnx
>>> onnx.load('dlrm_s_pytorch.onnx')
[libprotobuf ERROR google/protobuf/io/zero_copy_stream_impl_lite.cc:155] Cannot allocate buffer larger than kint32max for StringOutputStream.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unable to convert message to str
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
>>> quit()
 
Do you have any suggestion about this error?
 
Aaron
 
 
 
 
------------------------------------------------------------------
From:Gattupalli, Srujana <srujana.gattupalli@intel.com>
Sent At:2020 Jun. 2 (Tue.) 17:05
To:Aaron <aaron.zhong@alibaba-inc.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: DLRM ONNX setup instructions
 
Hi Aaron,
 
Yes, this looks valid.
 
Thanks,
Srujana
 
From: Rong, Zhong <aaron.zhong@alibaba-inc.com> 
Sent: Tuesday, June 2, 2020 4:26 PM
To: Aaron <aaron.zhong@alibaba-inc.com>; Gattupalli, Srujana <srujana.gattupalli@intel.com>
Cc: Chukka, Ramesh N <ramesh.n.chukka@intel.com>
Subject: Re: DLRM ONNX setup instructions
 
Srujana,
 
I followed your instruction to dump the onnx model. The pytorch nightly still pop up ValueError about 2GB limit, but the onnx files is saved as below. 
Can you verify whether this dump is good based on the name and size of the files?
 
Another question is do you have onnx backend implemented for DLRM in MLPerf reference implementation?
 
-rw-r--r-- 1 aaronzhong arch       26624 Jun  2 03:59 bot_l.0.weight
-rw-r--r-- 1 aaronzhong arch      524288 Jun  2 03:59 bot_l.2.weight
-rw-r--r-- 1 aaronzhong arch      131072 Jun  2 03:59 bot_l.4.weight
-rw-r--r-- 1 aaronzhong arch       40122 Jun  2 04:00 dlrm_s_pytorch.onnx
-rw-r--r-- 1 aaronzhong arch 20420815872 Jun  2 03:59 emb_l.0.weight
-rw-r--r-- 1 aaronzhong arch  1512215552 Jun  2 03:59 emb_l.10.weight
-rw-r--r-- 1 aaronzhong arch   206513152 Jun  2 03:59 emb_l.11.weight
-rw-r--r-- 1 aaronzhong arch        5120 Jun  2 03:59 emb_l.12.weight
-rw-r--r-- 1 aaronzhong arch     1130496 Jun  2 03:59 emb_l.13.weight
-rw-r--r-- 1 aaronzhong arch     6112256 Jun  2 03:59 emb_l.14.weight
-rw-r--r-- 1 aaronzhong arch       79360 Jun  2 03:59 emb_l.15.weight
-rw-r--r-- 1 aaronzhong arch      499712 Jun  2 03:59 emb_l.17.weight
-rw-r--r-- 1 aaronzhong arch        7168 Jun  2 03:59 emb_l.18.weight
-rw-r--r-- 1 aaronzhong arch 20469642752 Jun  2 03:59 emb_l.19.weight
-rw-r--r-- 1 aaronzhong arch    19990016 Jun  2 03:59 emb_l.1.weight
-rw-r--r-- 1 aaronzhong arch 13128343040 Jun  2 03:59 emb_l.20.weight
-rw-r--r-- 1 aaronzhong arch 20308471808 Jun  2 03:59 emb_l.21.weight
-rw-r--r-- 1 aaronzhong arch   299998720 Jun  2 03:59 emb_l.22.weight
-rw-r--r-- 1 aaronzhong arch     6641664 Jun  2 03:59 emb_l.23.weight
-rw-r--r-- 1 aaronzhong arch       55296 Jun  2 03:59 emb_l.24.weight
-rw-r--r-- 1 aaronzhong arch       18432 Jun  2 03:59 emb_l.25.weight
-rw-r--r-- 1 aaronzhong arch     8851968 Jun  2 03:59 emb_l.2.weight
-rw-r--r-- 1 aaronzhong arch     3799040 Jun  2 03:59 emb_l.3.weight
-rw-r--r-- 1 aaronzhong arch    10374656 Jun  2 03:59 emb_l.4.weight
-rw-r--r-- 1 aaronzhong arch     3645440 Jun  2 03:59 emb_l.6.weight
-rw-r--r-- 1 aaronzhong arch      790016 Jun  2 03:59 emb_l.7.weight
-rw-r--r-- 1 aaronzhong arch       32256 Jun  2 03:59 emb_l.8.weight
-rw-r--r-- 1 aaronzhong arch 19728870912 Jun  2 04:00 emb_l.9.weight
-rw-r--r-- 1 aaronzhong arch     1961984 Jun  2 04:00 top_l.0.weight
-rw-r--r-- 1 aaronzhong arch     4194304 Jun  2 04:00 top_l.2.weight
-rw-r--r-- 1 aaronzhong arch     2097152 Jun  2 04:00 top_l.4.weight
-rw-r--r-- 1 aaronzhong arch      524288 Jun  2 04:00 top_l.6.weight
 
Best Regards,
Aaron
 
 
 
------------------------------------------------------------------
From:Aaron <aaron.zhong@alibaba-inc.com>
Sent At:2020 May 26 (Tue.) 15:19
To:"Gattupalli, Srujana" <srujana.gattupalli@intel.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:Re: DLRM ONNX setup instructions
 
Will Try. Thanks
 
Aaron
 
------------------------------------------------------------------
From:Gattupalli, Srujana <srujana.gattupalli@intel.com>
Sent At:2020 May 26 (Tue.) 13:32
To:Aaron <aaron.zhong@alibaba-inc.com>
Cc:"Chukka, Ramesh N" <ramesh.n.chukka@intel.com>
Subject:RE: DLRM ONNX setup instructions
 
Hi Aaron,
 
Below are the instructions for DLRM Pytorch to ONNX conversion:
 
1.	The code is here:
https://github.com/facebookresearch/dlrm/blob/4705ea122d3cc693367f54e937db28c9c673d71b/dlrm_s_pytorch.py#L1186
2.	Pytorch nightly should have the large model export. Remember to set use_external_data_format=True for the torch.onnx.export call.
Edits to the conversion code:
1.      The code is using internal torch.onnx._export api instead of public api torch.onnx.export. Thus export might be missing important config setup.
2.      The code is passing an onnx file for argument “f”, however for the case of large model export, a string is required.
        dlrm_pytorch_onnx_file = 'dlrm_s_pytorch.onnx'
        (X, lS_o, lS_i, _) = train_data[0]  # get first batch of elements
        torch.onnx.export(
            dlrm, (X, lS_o, lS_i), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True
        )
3.	./bench/run_and_time.sh “--inference-only --load-model=tb00_40M.pt  --save-onnx”. Path to Terabyte dataset and preprocessed files would be needed here.
Regards,
Srujana
 
 
From:mlperf-inference-submitters@googlegroups.com <mlperf-inference-submitters@googlegroups.com> On Behalf Of Rong, Zhong
Sent: Wednesday, May 20, 2020 5:44 PM
To: mlperf-inference-submitters <mlperf-inference-submitters@googlegroups.com>
Subject: DLRM ONNX setup instructions
 
Hi,
 
As I heard onnx dump for DLRM is working, so I tried to sync https://github.com/mnaumovfb/inference.git to tip. But I can't find any setup instructions about how to dump onnx model from pytorch one in the README.md at https://github.com/mnaumovfb/inference/blob/master/v0.5/recommendation/README.md. Could someone tell me how to do that?
 
Aaron
 
-- 
You received this message because you are subscribed to the Google Groups "MLPerf Inference Submitters" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlperf-inference-submitters+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mlperf-inference-submitters/51bc29ee-d8cf-4842-8b95-7fdd031a31db.aaron.zhong%40alibaba-inc.com.
 
 
 
 
 
 



Two workarounds:
1) sub embedding bag with embedding
2) work with the pytorch exporter process to ensure embedding bag is supported. (right solution) Break embedding bag into 2 operators. 

The Aten op should disappear with these solutions, which will mapped to existing ONNX operators. 

@guschmue 

solution 3: wait for TF model. 
solution 4: hack the ONNX model to replace the ops as a one time hack. 


PR for embedding_bag support in onnx exporter:
https://github.com/pytorch/pytorch/pull/41234


Waiting for PR to merge to PyTorch. The next step is to test the converted model. @mnaumovfb  please help with the PT merge request. 

I build pytorch latest master with pytorch/pytorch#41234 and tried converting DLRM 10GB model from Pytorch to onnx,

I get Using CPU...
Reading pre-processed data=/home/ubuntu//drive/dlrm_data/day
Sparse features= 26, Dense features= 13
Reading pre-processed data=/home/ubuntu//drive/dlrm_data/day
Sparse features= 26, Dense features= 13
Loading saved model tb0875_10M.pt
Saved at: epoch = 0/1, batch = 307200/315310, ntbatch = 840
Training state: loss = 0.416358, accuracy = 81.459 %
Testing state: loss = 0.421776, accuracy = 81.071 %
time/loss/accuracy (if enabled):
Finished inference it 315310/315310 of epoch 0, 7.57 ms/it, loss 0.414235, accuracy 81.583 %
Traceback (most recent call last):
  File "dlrm_s_pytorch.py", line 1191, in <module>
    dlrm, (X_onnx, lS_o_onnx, lS_i_onnx), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 230, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 538, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 392, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 188, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 263, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 799, in _run_symbolic_function
    return symbolic_fn(g, *inputs, **attrs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 129, in wrapper
    return fn(g, *args)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_opset9.py", line 413, in embedding_bag
    return sym_help._onnx_unsupported('embedding_bag')
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 181, in _onnx_unsupported
    'Please open a bug to request ONNX export support for the missing operator.'.format(op_name))
**RuntimeError: Unsupported: ONNX export of operator embedding_bag. Please open a bug to request ONNX export support for the missing operator.** @guschmue Please let me know if there is a workaround or missing something other than pytorch/pytorch#41234.


Following up on above - Pytorch version 1.7.0a0+ef59f58

We are almost there, but I don't think the PR is merged https://github.com/pytorch/pytorch/pull/41234

We need to wait for the merged to be finished and pickup the next nightly build of PyTorch.

I added the PR to master before build using git pull origin pull/41234/head and got above error while using that build to convert the model.

I see, then we need to ask @guschmue about it.

hm, something must be wrong since it still complains about a missing embedding_bag op. After 41234 it should be no longer there.

The PR https://github.com/pytorch/pytorch/pull/41234 has been merged. Please give it another try.

@gsrujana89 will try again. 

@guschmue @mnaumovfb I have pulled latest master and run 
```
git pull
git submodule sync
git submodule update --init --recursive
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install

```
**Is there a build flag to ensure this feature is on?**

Also, tried nightly build:
**python -c "import torch; print(torch.__version__)"
1.7.0.dev20200720**

Still get the below error with both. Is there a way to confirm the feature is enabled in the pytorch before trying the conversion again? I do see the changes in the pytorch repo that I am building there are script changes related to this PR. (For e.g. test_pytorch_onnx_onnxruntime.py  has  ln 2898)
```
def test_embedding_bag_dynamic(self):
        class EmbeddingModel(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.embeddingbag = torch.nn.EmbeddingBag(40, 12, mode='sum'))
```


**Console log of error:**

  lj = torch.tensor([j for i in range(nj) for j in range(i + offset)])
Using CPU...
Reading pre-processed data=/home/ubuntu//drive/dlrm_data/day
Sparse features= 26, Dense features= 13
Reading pre-processed data=/home/ubuntu//drive/dlrm_data/day
Sparse features= 26, Dense features= 13
Loading saved model tb0875_10M.pt
Saved at: epoch = 0/1, batch = 307200/315310, ntbatch = 840
Training state: loss = 0.416358, accuracy = 81.459 %
Testing state: loss = 0.421776, accuracy = 81.071 %
time/loss/accuracy (if enabled):
Finished inference it 315310/315310 of epoch 0, 7.80 ms/it, loss 0.414235, accuracy 81.583 %
Traceback (most recent call last):
  File "dlrm_s_pytorch.py", line 1191, in <module>
    dlrm, (X_onnx, lS_o_onnx, lS_i_onnx), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 230, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 538, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 392, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 188, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 263, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 799, in _run_symbolic_function
    return symbolic_fn(g, *inputs, **attrs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 129, in wrapper
    return fn(g, *args)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_opset9.py", line 413, in embedding_bag
    return sym_help._onnx_unsupported('embedding_bag')
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 181, in _onnx_unsupported
    'Please open a bug to request ONNX export support for the missing operator.'.format(op_name))
**RuntimeError: Unsupported: ONNX export of operator embedding_bag. Please open a bug to request ONNX export support for the missing operator.**

When exactly does this error happen? Did you save the model again or are you loading the model saved earlier?

During saving the model again, used the same script as used before for saving 10GB model, only difference is the pytorch build
`python dlrm_s_pytorch.py --arch-sparse-feature-size=64 --arch-mlp-bot="13-512-256-64" --arch-mlp-top="512-512-256-1" --max-ind-range=10000000 --data-generation=dataset --data-set=terabyte --raw-data-file=/home/ubuntu//drive/dlrm_data/day  --processed-data-file=/home/ubuntu//drive/dlrm_data/day  --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=2048 --print-freq=1024 --print-time --test-mini-batch-size=16384 --test-num-workers=16 --load-model=tb0875_10M.pt --save-onnx --inference-only --test-freq=10240 --memory-map --data-sub-sample-rate=0.875`

This step used to work fine before, right?

Yes, this is the same command that generated https://dlrm.s3-us-west-1.amazonaws.com/models/tb0875_10M.onnx.tar. Thanks

@guschmue Can you please double check why embedding_bag is flagged as unsupported?

@gsrujana89 I'm wondering if you have to install the latest version of ONNX (besides PyTorch). Can you try using the nightly version of ONNX?

hm, should not need a newer onnx version, opset-10 is sufficient (at least the ut in the pr said so).
In the pr there was one ut with:
```
@unittest.skip("Enable once jit trace Tensor.numel as constant is fixed.")
```
I wonder if we hit this.

@guschmue thanks, could you elaborate on how to use/ run this ut? I can try and run the test.
Ran default tests `python test_pytorch_onnx_onnxruntime.py `
Ran 4600 tests in 2348.680s
OK (skipped=920)

maybe add ```TestONNXRuntime.test_embedding_bag``` to the command line to just run this specific test.

@guschmue tried that 
`python test_pytorch_onnx_onnxruntime.py TestONNXRuntime.test_embedding_bag`
Fail to import hypothesis in common_utils, tests are not derandomized
s
Ran 1 test in 0.001s

OK (skipped=1)

@guschmue @mnaumovfb On commenting out @unittest.skip get the following
`python test_pytorch_onnx_onnxruntime.py test_embedding_bag`
Fail to import hypothesis in common_utils, tests are not derandomized
E

**ERROR: test_embedding_bag (unittest.loader._FailedTest)
AttributeError: module '__main__' has no attribute 'test_embedding_bag'**

Ran 1 test in 0.000s

FAILED (errors=1).

Please help on how to add that module. Thanks

I have created a small repro to this bug. Please give it a try @gsrujana89 and let us know if it fails. If so, can we have somebody from @guschmue team to take a look at it?

```
# Copyright (c) Facebook, Inc. and its affiliates.                                                                                                                                                                                                                                                                                                                                                                                                                 
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
# This source code is licensed under the MIT license found in the                                                                                                                                                                                                                                                                                                                                                                                                  
# LICENSE file in the root directory of this source tree.                                                                                                                                                                                                                                                                                                                                                                                                          
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                  

import torch
import torch.nn as nn
import onnx

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.emb = nn.EmbeddingBag(10, 3, mode='sum')

    def forward(self, indices, offsets):
        return self.emb(indices, offsets)

net = Net()

indices = torch.LongTensor([1,2,4,5,4,3,2,9])
offsets = torch.LongTensor([0,4])
predict = net(indices, offsets)
print(predict)

torch.onnx.export(net, (indices, offsets), 'net.onnx', verbose=True, use_external_data_format=True)
net_onnx = onnx.load("net.onnx")
onnx.checker.check_model(net_onnx)
```

above fails on 1.7.0.dev20200724.
fyi @BowenBao,  @spandantiwari 

  File "/opt/anaconda3/lib/python3.7/site-packages/torch/onnx/__init__.py", line 263, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/onnx/utils.py", line 801, in _run_symbolic_function
    return symbolic_fn(g, *inputs, **attrs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/onnx/symbolic_helper.py", line 129, in wrapper
    return fn(g, *args)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/onnx/symbolic_opset9.py", line 413, in embedding_bag
    return sym_help._onnx_unsupported('embedding_bag')
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/onnx/symbolic_helper.py", line 181, in _onnx_unsupported
    'Please open a bug to request ONNX export support for the missing operator.'.format(op_name))
RuntimeError: Unsupported: ONNX export of operator embedding_bag. Please open a bug to request ONNX export support for the missing operator.


add ```opset_version=10``` to torch.onnx.export and it will work.

Yes, it works with opset_version=10 on Pytorch 1.7.0.dev20200720. I will add this to dlrm model conversion script and try. 

@guschmue @mnaumovfb Edited this in dlrm_s_pytorch.py for conversion

```
torch.onnx.export(dlrm, (X_onnx, lS_o_onnx, lS_i_onnx), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True, opset_version=10)
```
Got this on run,
testing state: loss = 0.421776, accuracy = 81.071 %
time/loss/accuracy (if enabled):
Finished inference it 315310/315310 of epoch 0, 8.09 ms/it, loss 0.414235, accuracy 81.583 %
Traceback (most recent call last):
  File "dlrm_s_pytorch.py", line 1191, in <module>
    dlrm, (X_onnx, lS_o_onnx, lS_i_onnx), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True, opset_version=10
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 230, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 538, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 392, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 188, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 263, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 799, in _run_symbolic_function
    return symbolic_fn(g, *inputs, **attrs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 129, in wrapper
    return fn(g, *args)
  **File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/onnx/symbolic_opset10.py", line 202, in embedding_bag
    if len(list(indices.node().inputs())) > 0 and indices.node().inputs().__next__().type().sizes() is not None \
RuntimeError: r INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1595228937984/work/aten/src/ATen/core/jit_type.h":170, please report a bug to PyTorch.** Please advise. Thanks.

Please make sure `X_onnx, lS_o_onnx, lS_i_onnx` are [set on this line](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L924), then print & let us know the dimensions of these tensors at the ONNX [call point](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L1235).

@mnaumovfb  Seems like ls_o_onnx is not defined at call point.

Saved at: epoch = 0/1, batch = 307200/315310, ntbatch = 840
Training state: loss = 0.416358, accuracy = 81.459 %
Testing state: loss = 0.421776, accuracy = 81.071 %
time/loss/accuracy (if enabled):
**X_onnx, lS_o_onnx, lS_i_onnx = X, lS_o, lS_i set**  --- print from [this line.](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L924)
Finished inference it 315310/315310 of epoch 0, 8.36 ms/it, loss 0.414235, accuracy 81.583 %
Print from[ call point](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L1235)
X_onnx tensor([[ 2.56495,  6.27664,  0.00000,  ...,  1.79176,  9.21294,  0.00000],
        [ 0.00000,  6.97448,  0.00000,  ...,  2.56495,  7.98242,  0.00000],
        [ 0.00000,  0.69315,  0.00000,  ...,  0.00000,  0.69315,  0.00000],
        ...,
        [ 1.09861,  7.36138,  0.69315,  ...,  1.60944,  7.81682,  0.69315],
        [ 1.09861,  4.15888,  1.60944,  ...,  0.69315, 10.46039,  1.60944],
        [ 0.00000,  6.90776,  0.00000,  ...,  1.09861,  9.97030,  0.00000]])
Traceback (most recent call last):
  File "dlrm_s_pytorch.py", line 1192, in <module>
    print("ls_o_onnx", ls_o_onnx)
**NameError: name 'ls_o_onnx' is not defined**

@mnaumovfb will try to investigate the issue. Now the onnx support seems to be there.

@mnaumovfb  to add to my previous comment: ls_o_onnx is available at [this line](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L924) and have printed it. It becomes null on call point.

Saved at: epoch = 0/1, batch = 307200/315310, ntbatch = 840
Training state: loss = 0.416358, accuracy = 81.459 %
Testing state: loss = 0.421776, accuracy = 81.071 %
time/loss/accuracy (if enabled):
X_onnx, lS_o_onnx, lS_i_onnx = X, lS_o, lS_i set
lS_o_onnx from ln 924 tensor([[   0,    1,    2,  ..., 2045, 2046, 2047],
        [   0,    1,    2,  ..., 2045, 2046, 2047],
        [   0,    1,    2,  ..., 2045, 2046, 2047],
        ...,
        [   0,    1,    2,  ..., 2045, 2046, 2047],
        [   0,    1,    2,  ..., 2045, 2046, 2047],
        [   0,    1,    2,  ..., 2045, 2046, 2047]]) <- print from t[his line](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L924)


Finished inference it 315310/315310 of epoch 0, 8.05 ms/it, loss 0.414235, accuracy 81.583 %

At[ call point](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L1235)
X_onnx tensor([[ 2.56495,  6.27664,  0.00000,  ...,  1.79176,  9.21294,  0.00000],
        [ 0.00000,  6.97448,  0.00000,  ...,  2.56495,  7.98242,  0.00000],
        [ 0.00000,  0.69315,  0.00000,  ...,  0.00000,  0.69315,  0.00000],
        ...,
        [ 1.09861,  7.36138,  0.69315,  ...,  1.60944,  7.81682,  0.69315],
        [ 1.09861,  4.15888,  1.60944,  ...,  0.69315, 10.46039,  1.60944],
        [ 0.00000,  6.90776,  0.00000,  ...,  1.09861,  9.97030,  0.00000]])
Traceback (most recent call last):
  File "dlrm_s_pytorch.py", line 1235, in <module>
print("ls_o_onnx", ls_o_onnx)
**NameError: name 'ls_o_onnx' is not defined** Thanks

First of all with respect to prints. You are printing the wrong variable name, it should be "lS_o_onnx" not "ls_o_onnx". That's why you see the not defined error. Please print these variables right after they are set on [line 953](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L953) and let us know the dimensions of these tensors.

Second, the embeddingBag operator seems to be implemented correctly in ONNX.  However, there is another "index" operator that is missing from ONNX and is used in DLRM on [line 319](https://github.com/facebookresearch/dlrm/blob/master/dlrm_s_pytorch.py#L319). We can see the corresponding error by running the following simple command (@guschmue to advise whether it can be mapped to an existing ONNX operator):
```
> pytorch dlrm_s_pytorch.py --save-onnx
...
  %83 : Long(6:1, requires_grad=0, device=cpu) = onnx::Constant[value= 0  0  1  0  1  2 [ CPULongType{6} ]]()
  **%84 : Float(1:6, 6:1, requires_grad=1, device=cpu) = onnx::ATen[operator="index"](%80, %81, %82, %83) # .../dlrm_s_pytorch.py:529:0**
  %85 : Float(1:8, 8:1, requires_grad=1, device=cpu) = onnx::Concat[axis=1](%23, %84) # .../dlrm_s_pytorch.py:531:0
  %86 : Float(1:4, 4:1, requires_grad=1, device=cpu) = onnx::Gemm[alpha=1., beta=1., transB=1](%85, %14, %15) # .../torch/nn/functional.py:1682:0
 ...
  %91 : Float(1:1, 1:1, requires_grad=1, device=cpu) = onnx::Sigmoid(%90) # .../torch/nn/modules/activation.py:299:0
  return (%91)

**Warning: ATen was a removed experimental ops. In the future, we may directly reject this operator. Please update your model as soon as possible.**
```

Here is a small repro for ONNX team (@guschmue to advise)
```
# Copyright (c) Facebook, Inc. and its affiliates.                                                                                                                                                                                                                                                    
#                                                                                                                                                                                                                                                                                                     
# This source code is licensed under the MIT license found in the                                                                                                                                                                                                                                     
# LICENSE file in the root directory of this source tree.                                                                                                                                                                                                                                             
#                                                                                                                                                                                                                                                                                                     

import torch
import torch.nn as nn
import onnx

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.emb = nn.EmbeddingBag(10, 3, mode='sum')

    def forward(self, indices, offsets):
        y = self.emb(indices, offsets)
        ni, nj = y.shape
        li = torch.tensor([i for i in range(ni) for j in range(i)])
        lj = torch.tensor([j for i in range(nj) for j in range(i)])
        return y[li, lj]

net = Net()

indices = torch.LongTensor([1,2,4,5,4,3,2,9])
offsets = torch.LongTensor([0,2,4])
predict = net(indices, offsets)
print(predict)

torch.onnx.export(net, (indices, offsets), 'net.onnx', verbose=True, use_external_data_format=True, opset_version=10)
net_onnx = onnx.load("net.onnx")
onnx.checker.check_model(net_onnx)
```

I was finally able to reproduce @gsrujana89 crash https://github.com/mlperf/inference/issues/645#issuecomment-663718038

Here is a workaround for it, insert the following code prior to call to `torch.onnx.export`
```
if torch.is_tensor(lS_i_onnx):
    lS_i_onnx = [lS_i_onnx[j] for j in range(len(lS_i_onnx))]
```

The onnx-exporter fix for EmbeddingBag is not in pytorch-1.6. You need to use nightly.
If I do this (1.7.0.dev20200805) the sample script above works fine.


@guschmue Can you please share the log of your [run](https://github.com/mlperf/inference/issues/645#issuecomment-667559049)? In my run I see the following error 

graph(%input : Long(8:1, requires_grad=0, device=cpu),
      %offsets : Long(3:1, requires_grad=0, device=cpu),
      %emb.weight : Float(10:3, 3:1, requires_grad=1, device=cpu)):
  %3 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %4 : Tensor = onnx::Concat[axis=0](%offsets, %3)
  %5 : Tensor = onnx::Constant[value={0}]()
  %6 : Tensor = onnx::Gather[axis=0](%4, %5)
  %7 : Tensor = onnx::Unsqueeze[axes=[0]](%6)
  %8 : Tensor = onnx::Constant[value={1}]()
  %9 : Tensor = onnx::Gather[axis=0](%4, %8)
  %10 : Tensor = onnx::Unsqueeze[axes=[0]](%9)
  %11 : Tensor = onnx::Constant[value={0}]()
  %12 : Tensor = onnx::Slice(%input, %7, %10, %11)
  %13 : Tensor = onnx::Gather(%emb.weight, %12)
  %14 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%13)
  %15 : Tensor = onnx::Unsqueeze[axes=[0]](%14)
  %16 : Tensor = onnx::Constant[value={1}]()
  %17 : Tensor = onnx::Gather[axis=0](%4, %16)
  %18 : Tensor = onnx::Unsqueeze[axes=[0]](%17)
  %19 : Tensor = onnx::Constant[value={2}]()
  %20 : Tensor = onnx::Gather[axis=0](%4, %19)
  %21 : Tensor = onnx::Unsqueeze[axes=[0]](%20)
  %22 : Tensor = onnx::Constant[value={0}]()
  %23 : Tensor = onnx::Slice(%input, %18, %21, %22)
  %24 : Tensor = onnx::Gather(%emb.weight, %23)
  %25 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%24)
  %26 : Tensor = onnx::Unsqueeze[axes=[0]](%25)
  %27 : Tensor = onnx::Constant[value={2}]()
  %28 : Tensor = onnx::Gather[axis=0](%4, %27)
  %29 : Tensor = onnx::Unsqueeze[axes=[0]](%28)
  %30 : Tensor = onnx::Constant[value={3}]()
  %31 : Tensor = onnx::Gather[axis=0](%4, %30)
  %32 : Tensor = onnx::Unsqueeze[axes=[0]](%31)
  %33 : Tensor = onnx::Constant[value={0}]()
  %34 : Tensor = onnx::Slice(%input, %29, %32, %33)
  %35 : Tensor = onnx::Gather(%emb.weight, %34)
  %36 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%35)
  %37 : Tensor = onnx::Unsqueeze[axes=[0]](%36)
  %38 : Float(3:3, 3:1, requires_grad=1, device=cpu) = onnx::Concat[axis=0](%15, %26, %37) # …/torch/nn/functional.py:2011:0
  %39 : Long(3:1, requires_grad=0, device=cpu) = onnx::Constant[value= 1  2  2 [ CPULongType{3} ]]()
  %40 : Long(3:1, requires_grad=0, device=cpu) = onnx::Constant[value= 0  0  1 [ CPULongType{3} ]]()
  **%41 : Float(3:1, requires_grad=1, device=cpu) = onnx::ATen[operator="index"](%38, %39, %40) # /home/mnaumov/onnx_index_test.py:22:0**
  return (%41)
 
**Warning: ATen was a removed experimental ops. In the future, we may directly reject this operator. Please update your model as soon as possible.**


graph(%input : Long(8:1, requires_grad=0, device=cpu),
      %offsets : Long(3:1, requires_grad=0, device=cpu),
      %emb.weight : Float(10:3, 3:1, requires_grad=1, device=cpu),
      %54 : Long(1:1, requires_grad=0, device=cpu)):
  %3 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %4 : Tensor = onnx::Concat[axis=0](%offsets, %3)
  %5 : Tensor = onnx::Constant[value={0}]()
  %6 : Tensor = onnx::Gather[axis=0](%4, %5)
  %7 : Tensor = onnx::Unsqueeze[axes=[0]](%6)
  %8 : Tensor = onnx::Constant[value={1}]()
  %9 : Tensor = onnx::Gather[axis=0](%4, %8)
  %10 : Tensor = onnx::Unsqueeze[axes=[0]](%9)
  %11 : Tensor = onnx::Constant[value={0}]()
  %12 : Tensor = onnx::Slice(%input, %7, %10, %11)
  %13 : Tensor = onnx::Gather(%emb.weight, %12)
  %14 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%13)
  %15 : Tensor = onnx::Unsqueeze[axes=[0]](%14)
  %16 : Tensor = onnx::Constant[value={1}]()
  %17 : Tensor = onnx::Gather[axis=0](%4, %16)
  %18 : Tensor = onnx::Unsqueeze[axes=[0]](%17)
  %19 : Tensor = onnx::Constant[value={2}]()
  %20 : Tensor = onnx::Gather[axis=0](%4, %19)
  %21 : Tensor = onnx::Unsqueeze[axes=[0]](%20)
  %22 : Tensor = onnx::Constant[value={0}]()
  %23 : Tensor = onnx::Slice(%input, %18, %21, %22)
  %24 : Tensor = onnx::Gather(%emb.weight, %23)
  %25 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%24)
  %26 : Tensor = onnx::Unsqueeze[axes=[0]](%25)
  %27 : Tensor = onnx::Constant[value={2}]()
  %28 : Tensor = onnx::Gather[axis=0](%4, %27)
  %29 : Tensor = onnx::Unsqueeze[axes=[0]](%28)
  %30 : Tensor = onnx::Constant[value={3}]()
  %31 : Tensor = onnx::Gather[axis=0](%4, %30)
  %32 : Tensor = onnx::Unsqueeze[axes=[0]](%31)
  %33 : Tensor = onnx::Constant[value={0}]()
  %34 : Tensor = onnx::Slice(%input, %29, %32, %33)
  %35 : Tensor = onnx::Gather(%emb.weight, %34)
  %36 : Tensor = onnx::ReduceSum[axes=[0], keepdims=0](%35)
  %37 : Tensor = onnx::Unsqueeze[axes=[0]](%36)
  %38 : Float(3:3, 3:1, requires_grad=1, device=cpu) = onnx::Concat[axis=0](%15, %26, %37) # /opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py:2011:0
  %39 : Tensor = onnx::Shape(%38)
  %40 : Tensor = onnx::Constant[value={1}]()
  %41 : Tensor = onnx::Gather[axis=0](%39, %40)
  %42 : Tensor = onnx::Flatten[axis=2](%38)
  %43 : Long(3:1, requires_grad=0, device=cpu) = onnx::Constant[value= 1  2  2 [ CPULongType{3} ]]()
  %44 : LongTensor = onnx::Mul(%43, %41)
  %45 : Long(3:1, requires_grad=0, device=cpu) = onnx::Constant[value= 0  0  1 [ CPULongType{3} ]]()
  %46 : LongTensor = onnx::Add(%45, %44)
  %47 : Tensor = onnx::Gather[axis=0](%42, %46)
  %48 : Tensor = onnx::Shape(%46)
  %51 : Tensor = onnx::Reshape(%47, %54)
  %52 : Tensor = onnx::Concat[axis=0](%48)
  %53 : Float(3:1, requires_grad=1, device=cpu) = onnx::Reshape(%51, %52) # embeddingbag-test.py:17:0
  return (%53)



Thank you @guschmue. This trace looks good.  
 
@gsrujana89 please try to reproduce this trace with https://github.com/mlperf/inference/issues/645#issuecomment-667559049.
 
Then, try the small model as shown here https://github.com/mlperf/inference/issues/645#issuecomment-667506387
 
Finally, add a workaround https://github.com/mlperf/inference/issues/645#issuecomment-668830932 and try converting the full model used for MLPerf inference.
 

The new model inputs are inconsistent with previously generated model

Below is the difference in the 10GB properties for old (with Aten) vs new (without Aten ops)
 
On printing out layers with below python script:
  import onnx
  model = onnx.load("/home/ubuntu/dlrm/dlrm/dlrm_onnx10gb/dlrm_s_pytorch_10GB.onnx")
  output =[node.name for node in model.graph.output
  input_all = [node.name for node in model.graph.input]
  input_initializer =  [node.name for node in model.graph.initializer]
  net_feed_input = list(set(input_all)  - set(input_initializer))
  print('Inputs: ', net_feed_input)
  print('Outputs: ', output)
  
get below old 10GB onnx model (with Aten layers)
**Inputs:  ['input.1', 'lS_o', 'lS_i']
Outputs:  ['299']**
 
new 10GB model
**Inputs:  ['input.11', 'input.7', 'input.23', 'input.15', 'lS_o', 'input.22', 'input.1', 'input.16', 'input.32', 'input.8', 'input.21', 'input.24', 'input.9', 'input.17', 'input.19', 'input.10', 'input.26', 'input.12', 'input.25', 'input.13', 'input.18', 'input.27', 'input.14', 'input.28', 'input.31', 'input.20', 'input.30', 'input.29']
Outputs:  ['585974']**

**@guschmue @mnaumovfb** Where in the code does enabling support for enbedding bag change model inputs? The conversion command and parameters used are same for both models. Please advice on how to set correct input shape with the new changes. Thanks.

The workaround https://github.com/mlperf/inference/issues/645#issuecomment-668830932 switches the use of a single tensor to a list of indices. This is likely why you are seeing the list of inputs.

In general both a single tensor a list should work as an input, but right now the tensor is causing a crash https://github.com/mlperf/inference/issues/645#issuecomment-663718038, that's why I suggested the above workaround. 

We have discussed an approach to handle it in the related email thread. This should not prevent the model from running. 

Main issue on using this model is how to set correct ls_i index to onnx inputs, 

Inputs: ['input.11', 'input.7', 'input.23', 'input.15', 'lS_o', 'input.22', 'input.1', 'input.16', 'input.32', 'input.8', 'input.21', 'input.24', 'input.9', 'input.17', 'input.19', 'input.10', 'input.26', 'input.12', 'input.25', 'input.13', 'input.18', 'input.27', 'input.14', 'input.28', 'input.31', 'input.20', 'input.30', 'input.29']

Based on these inputs, would this be the correct mapping?

ONNX model inputs (input.) | Index
-- | --
11 | X
7 | lS_i 1
23 | lS_i 2
15 | lS_i 3
ls_o | ls_o
22 | lS_i 4
1 | lS_i 5
16 | lS_i 6
32 | lS_i 7
8 | lS_i 8
21 | lS_i 9
24 | lS_i 10
9 | lS_i 11
17 | lS_i 12
19 | lS_i 13
10 | lS_i 14
26 | lS_i 15
12 | lS_i 16
25 | lS_i 17
13 | lS_i 18
18 | lS_i 19
27 | lS_i 20
14 | lS_i 21
28 | lS_i 22
31 | lS_i 23
20 | lS_i 24
30 | lS_i 25
29 | lS_i 26
Thanks.


Hello,

I am working on DLRM ONNX interface and I have some issues running the predict. 

The model (Critio Kaggle, 7 days of data) is converted with the following code from dlrm_s_pytorch.py
if args.save_onnx:
        dlrm_pytorch_onnx_file = "dlrm_kaggle.onnxruntime"
        print(X_onnx, lS_o_onnx, lS_i_onnx)
        torch.onnx.export(
            dlrm, (X_onnx, lS_o_onnx, lS_i_onnx), dlrm_pytorch_onnx_file, verbose=True, use_external_data_format=True, opset_version=11
        )
        # recover the model back
        dlrm_pytorch_onnx = onnx.load("dlrm_kaggle.onnxruntime")
        # check the onnx model
        onnx.checker.check_model(dlrm_pytorch_onnx)

The results of conversion are in file: 
[convert_kaggle_model.log](https://github.com/mlperf/inference/files/5093469/convert_kaggle_model.log)

I get an error when I call predict:

def predict(self, batch_dense_X, batch_lS_o, batch_lS_i):
        return self.sess.run(output_names=self.outputs, input_feed=[{self.inputs[0]:batch_dense_X.numpy()}, {self.inputs[1]:batch_lS_o.numpy()}, {self.inputs[2]:batch_lS_i.numpy()}])

See the log file:
[onnx-predict.log](https://github.com/mlperf/inference/files/5093491/onnx-predict.log)

Could you advise me on how to run the predict correctly. All inputs and output names came from the converted model.

The code is located at: https://github.com/dkorchevgithub/inference/tree/master/v0.5/recommendation, ONNX related code was taken and modified from the classification and detection.

I use the 
pytorch                   1.7.0.dev20200817 py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch-nightly
onnx                      1.7.0                    pypi_0    pypi

Thank you,
Dmitriy


There is a bug in ONNX run time or exporter to allow for variable batch size. Once ONNX is fixed, we can get it working.

Here is the PR https://github.com/mlperf/inference/pull/728 that addresses this issue as well as the issue  https://github.com/mlperf/inference/issues/725.

Please merge when you have a chance.

fixed and merged