zhiminy/EvalEng huggingface dataset:
Repository: mlcommons/inference
Issue: DLRM: provide a single script for generating a fake dataset
Root cause: Architectural Constraint

To generate a fake dataset, I've used the following command sequence:
```
$ cd $WORKSPACE/inference/v0.5/recommendation
$ python -m pip install numpy --user
$ cd tools/
$ ./make_fake_criteo.sh terabyte0875
$ mv fake_criteo/ ../
```

Here, [`make_fake_criteo.sh`](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/tools/make_fake_criteo.sh) calls [`quickgen.py`](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/tools/quickgen.py) which is also in the `tools/` directory. Hence, the need to descend there first.

However, all the `make_fake_criteo.sh` script does is it creates a new directory if it doesn't exist, checks that the only argument is one of the `kaggle|terabyte0875|terabyte` and passes it to `quickgen.py`:
```
python quickgen.py --num-samples=4096 --profile=$QUICKGEN_PROFILE --output-dir=$OUTPUT_DIR
```
Actually, it even gets in the way because it forcibly uses `python` not `python3` (which are different on e.g. Ubuntu 16.04). For example, I didn't have NumPy installed for Python 2, hence had to install it first.

This functionality can be folded into `quickgen.py` itself.

Here's a [good spot](https://github.com/mlperf/inference/blob/db0b7eb368f009adf703d073765aa212d1f3a026/v0.5/recommendation/tools/quickgen.py#L65) to error if the profile argument is not one of the `kaggle|terabyte0875|terabyte` (including the default value).

closing this @psyhtest  for now and please reach out @mnaumovfb if needed.
Repository: mlcommons/inference
Issue: DLRM: any hints for graph analysis?
Root cause: Documentation Deficiency

Given the size of the DLRM models (~10 GB, ~100 GB), it's challenging to apply tried and tested analysis tools to graph analysis.

For example, trying to load the ~10 GB model into [Netron](https://github.com/lutzroeder/netron) failed on a machine with 32 GB RAM:
- from the browser: `File read error '0'.`
- from the command line: `Error loading model. Unsupported file content for extension '.pt' in 'tb0875_10M.pt'.` (memory usage shoot up to 27 GB)

I must admit I'm not that familiar with PyTorch and its ecosystem of tools. Any hints @mnaumovfb @naghshineh ?

In particular, I'm interested how to separate out the bottom MLP subgraph (and feed to it representative dense features) and the top MLP subgraph (and feed to it representative feature interactions).

![dlrm](https://user-images.githubusercontent.com/6597818/82664866-9294b480-9c2a-11ea-9a7e-00d05d09efda.gif)

(Or am I wrong in thinking that the bottom MLP requires only dense features? The diagram does not make it particularly clear.)

I would suggest trying with a smaller model. You can always create (train and save) a smaller model from a subset of the data. Also, you can use --max-ind-size during training to limit the number of vectors in the embedding tables (you must pre-process the dataset with this parameter as well). 

The bottom MLP does require only dense features. I don't see how separating the MLPs from the rest of the model would help, but if you look at the python code, you will see calls to `create_mlp` and `apply_mlp`. I think that these are the calls you are interested in playing with.

Thanks Maxim, this is very helpful.
Repository: mlcommons/inference
Issue: DLRM: Unable to export to ONNX
Root cause: Validation Gap

I'm trying to export using `--save-onnx` and hitting problems.

In dlrm_s_pytorch.py, the training data isn't in the right size and shape to be fed into the onnx exporter. 

`            (X, lS_o, lS_i, _) = train_data[0]  # get first batch of elements`

I couldn't figure out how to massage that data into the correct format so I've worked around this by caching one of the instances of X, lS_o, lS_i used to test the trained model and passing those instead. That seems to hang together, but I can't see obviously how to fix this correctly. It may be something simple for someone in the know?



the issue should be resolved in the latest reference. please check again and re-open the issue if needed
Repository: mlcommons/inference
Issue: RNNT: Running inference on CUDA fails.
Root cause: Validation Gap

When CUDA is enabled, inference fails.

The lambda function set up to call `cuda()` on each of the objects passed into the `audio_processor` fails as one of the objects is of type `list`. (the list containing the translations for the batch)

`eval_transforms.append(lambda xs: [x.cuda() for x in xs])`

I tried addressing this by making the call to `cuda()` conditional on the object type as I don't think the translations should be passed to the GPU? But after making that change there's a further issue with the following lambda functions receiving `NoneType` as an argument.

` eval_transforms.append(lambda xs: [*audio_preprocessor(xs[0:2]), *xs[2:]])`

All works correctly when run on the CPU.


@nvpohanh I think you have successfully run RNN-T on T4? Any hints?

now you can look at the NV submission code on RNNT. closing this issue for now and please re-open if needed.

I am experiencing the same issue originally identified by G4V.  When '--cuda' is not specified, the workload runs but only on CPU.  The launching [script](https://github.com/mlcommons/inference/blob/db0b7eb368f009adf703d073765aa212d1f3a026/v0.7/speech_recognition/rnnt/pytorch/inference.py) hits the following error at line 186: `AttributeError: 'list' object has no attribute 'cuda'`

Any help resolving the bug would be greatly appreciated.
Repository: mlcommons/inference
Issue: ssd resnet34 model
Root cause: Documentation Deficiency

Hi,
I am try to run the ssd-resnet34 model in pytorch, but I got many issues on this model.
1.  the out_size here cause miss understand. https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L22
if the input size is 1200x1200, the out_size is 150, is 38 also works on 150 in function _build_additional_features
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L44

2. is the stride in loc and conf always 3?
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L36

3. the loc and conf shape miss match the comments, For SSD 300, shall return nbatch x 8732 x {nlabels, nlocs} results
https://github.com/mlperf/inference/blob/master/cloud/single_stage_detector/pytorch/ssd_r34.py#L143
the value I got is 
`(Pdb) print(ploc.shape)`
`torch.Size([1, 4, 1098])`
`(Pdb) print(plabel.shape)`
`torch.Size([1, 81, 1098])
`

1.	The input size is 50x50 for 1200x1200 (with default strides) – anyway I removed the unnecessary conditions.
2.	Yes, the current pre-trained model used stride=3 for all loc/conf layers. Perhaps changing that can aid to the accuracy. If you manage to run this experiment (retrain the model) and achieved better results please let me know.  Note that we wish to change the dataset and the model for 0.6 submission 
3.	The 300x300 should run with strides 1,1,2,2,2,1 (as was done in the original single_stage_detector 
 repo from mlperf/training) I added it to the comment

Additional concerns? - I want to close this issue


@itayhubara thanks for your reply. Did not have any question currently, I will close this issue.
Repository: mlcommons/inference
Issue: DLRM: Can't run recommendation by real data
Root cause: Configuration Error

Hi,

I tried to run inference model of recommendation.
https://github.com/mlperf/inference/tree/master/v0.5/recommendation

I ran the model by fake data successfully, but failed when running terabyte 0.875 by real data.
The root cause is that some of the sparse inputs are larger than the index of embedding table.

For example, the dimension of the first embedding table is 9980333*64, but input data is out of range (larger than 9980333).
In my opinion, the range of input data is limited, such as, in this case, is 0~9980332.
Is there anything wrong?
Is there any solution?

Lot of thanks.
BR,
chiungliang

I found that pre-processed data is different under different configuration. Delete all pre-processed data and run again, it works.
Repository: mlcommons/inference
Issue: [v0.8 logging] Single log file for each run
Root cause: Unimplemented Feature Gap

Currently, LoadGen generates 4 files for each run: summary, detail, trace, and accuracy.json. Following the Training convention, each run should only generate one single log file, which contains all the information needed for result showing and validation.

We can think about how to maintain the nice human-readability while doing this. For example, we can still print summary at the end of the log without the logging prefix format so that human can read them while the validator will ignore them.
Repository: mlcommons/inference
Issue: [v0.8 logging] Change logging format to match Training
GitHub labels: inference v1.0

Log in unified format. See:

- Training logging spec: https://docs.google.com/document/d/1u15ouRbvd2xZkMEHNtmtg-yGzw0Zs6KVxQG_5WjTz7o/edit
- (WIP) Inference logging spec: https://docs.google.com/document/d/1o2qp_wMHZ1KkmfA7h9nhrqx1kA901UOQvF_-G34tfl8/edit#

The format looks something like:
```
:::MLLOG {"namespace": NAMESPACE, "time_ms": TIMESTAMP, "event_type": EVENTTYPE, "key": KEY, "value": VALUE, "metadata": METADATA}
```
Repository: mlcommons/inference
Issue: [v0.8 logging] Add/Modify file-level utility checker
Root cause: Unimplemented Feature Gap

Add/Modify file-level utility checker which does the following:

- P0 - Prints individual errors found
- P0 - Indicates if the individual file is correct or not
- P1 - Extract out the loadgen settings
- P0 - Verifier looks at loadgen settings compared against valid configuration
- P1 - Verifier looks at system description and verifies (using D. Aronchick tools)
Repository: mlcommons/inference
Issue: [v0.8 logging] Add/Modify package-level utility checker
Root cause: Unimplemented Feature Gap

Add/Modify package-level utility checker which does:

- P0 - Prints individual verification errors found
- P0 - Indicates of the entire package is correct or not
- P0 - prints out the results (as appears in the final results table)
- P1 - summarizes the contents of the package, including hyperparameters and results
- P1 - Given a package of logs, summarizes the contents including average scores, etc.


@nvpohanh  is this still a request for v1.0? 

It's something that logging WG is working on. I think we can keep this open to track, but we don't need to go through these in main Inference WG.

@nvpohanh @guschmue Is there anything here to be done still?

I think this was done.
Repository: mlcommons/inference
Issue: DLRM: Indicate which model is the reference in README 

DLRM: Indicate which model is the reference in README 

Make it clear that the other two models are not the reference models for MLPerf Inference v0.7

@mnaumovfb Could you help to fix this? Thanks!

we are talking about the table with 3 models in the readme here. @mnaumovfb 

The readme already states right under the table "The official model is the largest model on the order of 100GB, while interface to other models is only provided to facilitate debugging."

What else would you add?

Hi @mnaumovfb,
How about for the 3 models in the name column, instead of having 'dlrm' for all of them, can we name them something like dlrm-S, dlrm-M, dlrm-L(MLPerf official model)? The table always gets people's attention first :)

I have added a debugging or official qualification for each model in parenthesis https://github.com/mlperf/inference/pull/649

Please review and accept when you have a chance.

done
Repository: mlcommons/inference
Issue: BERT calibration dataset
Root cause: Unimplemented Feature Gap

Discuss if and who's going to contribute to a BERT calibration dataset.  Will you be able to do it? @itayhubara 

Keeping the following folks in the loop
@nvpohanh @DilipSequeira 

There is an INT8 quantized model shared with the reference. There is no plan for this. Please volunteer before 7/20

no calibration dataset for BERT in v0.7 per WG decision on 7/21
Repository: mlcommons/inference
Issue: Tracking 3D UNET onnxruntime dev work
Root cause: External Dependency Breakage

@nvpohanh need to integrate with onnxruntime nightly and try the ONNX model again.

#627 added onnxruntime path using onnxruntime nightly build, but got zero accuracy. @guschmue could you give it a try to see if there is any obvious issue? Thanks

@guschmue  will find help in identifying the onnxruntime issue

@christ1ne Could we change the issue title to "ONNXRuntime" to emphasize that this is an onnxruntime issue, NOT an ONNX issue? Thanks

I tried to look at it but how does one get the dataset? If I want to register to download I end up here:
https://www.med.upenn.edu/cbica/affiliated-faculty.html


@tonyreina can you help @guschmue  here? Thanks!

Looks like the wrong link on the webpage. I've notified Spyros to see if he can correct the link for the registration.



https://www.med.upenn.edu/cbica/brats2019/registration.html  @guschmue 

let's see if we can conclude on this in the next few days. @guschmue 

@guschmue  any updates?

currently blocked by the dataset licensing problem.
Repository: mlcommons/inference
Issue: BERT -  Link to download Wikipedia and BookCorpus datasets
Root cause: Environment Incompatibility

Are there alternative links to download Wikipedia and BookCorpus datasets?. It seems to be a known issue for other teams:

https://github.com/mlperf/training/issues/377

https://github.com/NVIDIA/DeepLearningExamples/issues/536

@vilmara Can you clarify why do you need this? The inference code only requires the Squad dataset per my understanding.

Hi @christ1ne, you are right these datasets are required only for training code. The link to download is broken and wanted to know if somebody in this group has a working link,  MLPerf train group is also looking for it https://github.com/mlperf/training/issues/377

closing since English Wikipedia is still available, ignoring BookCorpus dataset
Repository: mlcommons/inference
Issue: Gnmt dataset - ascii' codec can't decode byte
Root cause: Environment Incompatibility

Hi , I'm following those instructions: https://github.com/mlperf/inference/tree/master/cloud/translation/gnmt/pytorch
While in this command :"bash download_dataset.sh"
I'm having this Error:"
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6727: ordinal not in range(128)"

Any Idea?

Thanks.

POSIX default is not UTF-8
add something like

export LC_CTYPE="en_EN.UTF-8"

to your env.


‪On Tue, Apr 9, 2019 at 2:22 AM ‪barak battach‬‏‎ <notifications@github.com>
wrote:‬

> Hi , I'm following those instructions:
> https://github.com/mlperf/inference/tree/master/cloud/translation/gnmt/pytorch
> While in this command :"bash download_dataset.sh"
> I'm having this Error:"
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6727:
> ordinal not in range(128)"
>
> Any Idea?
>
> Thanks.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/65>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIUuTwkyfspGmOKqU94eJYyGgqvqbR9-ks5vfFvJgaJpZM4ckBWC>
> .
>
Repository: mlcommons/inference
Issue: Add option to accuracy-dlrm.py to parse targets (labels) from dataset directly
Root cause: Unimplemented Feature Gap

Add option to `accuracy-dlrm.py` to parse targets (labels) from dataset directly

Currently, `accuracy-dlrm.py` assumes that the targets (i.e. labels, 0 or 1) are part of the loadgen accuracy logs. However, in some submissions, the harness wouldn't know the labels. Proposal is to add a flag `--day_23_txt_file` and when it's present, the accuracy script gets the labels from `day_23.txt` and assume that the loadgen accuracy logs only contain predictions, no targets.

@nvpohanh will have the action item to add this.

done
Repository: mlcommons/inference
Issue: How to run cloud language modeling inference with GPU support
Root cause: Documentation Deficiency

hi all, how can I run cloud Language Modeling inference benchmark with GPU support?. Thanks, Vilmara

Solved. I have modified the requirements.txt file to run the installation of TensorFlow with gpu support : `tensorflow-gpu==1.12.0 `

To get even better performance running the benchmark with GPU support, use the latest Nvidia docker image  `nvcr.io/nvidia/tensorflow:19.04-py3`. The throughput increased by 21%

do you know what libraries they are using?
d


On Thu, May 2, 2019 at 8:24 AM Vilmara <notifications@github.com> wrote:

> To get even better performance running the benchmarks with GPU support,
> use the latest Nvidia docker image nvcr.io/nvidia/tensorflow:19.04-py3.
> The throughput increased by 21%
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/66#issuecomment-488717097>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACCS4T56IIXIYPI5LIYUTL3PTMBSFANCNFSM4HETODRA>
> .
>


Hi @David-Levinthal , here is the link with the description of the full software stack included in the container version 19.04-py3: https://docs.nvidia.com/deeplearning/dgx/tensorflow-release-notes/rel_19.04.html#rel_19.04

thanks
d


On Thu, May 2, 2019 at 9:53 AM Vilmara <notifications@github.com> wrote:

> Hi @David-Levinthal <https://github.com/David-Levinthal> , here is the
> link with the description of the full software stack included in the
> container version 19.04-py3:
> https://docs.nvidia.com/deeplearning/dgx/tensorflow-release-notes/rel_19.04.html#rel_19.04
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/mlperf/inference/issues/66#issuecomment-488749043>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACCS4TZVIIFJMHRGZFZRNU3PTMMCNANCNFSM4HETODRA>
> .
>
Repository: mlcommons/inference
Issue: Add MD5 hash of the official DLRM models
Root cause: Documentation Deficiency

@mnaumovfb Could we add MD5 has of the official DLRM models (PyT and ONNX) to README? That will be super useful for us to verify if our downloaded model is sane. Thanks!

@psyhtest @christ1ne FYI

Sure, we can do that. Give me a little bit of time to generate and add it.

FWIW, this is the MD5 for the PyTorch model we downloaded `2d49a5288cddb37c3c64860a06d79bb9`

Ok, we can use it. This for the largest model, right? 

Since you already generated it for one of the models. How hard would it be to provide the MD5 for other two models as well?

I only downloaded this one... Do you happen to have models somewhere in your scratch space? Or do you know someone has? 

@psyhtest Do you have the DLRM ONNX model that you can provide MD5 for? 

Downloading a 100GB file is quite painful...

Sorry, when you say this one, which one is it? The README has a link to 2 pyTorch and 2 ONNX models https://github.com/mlperf/inference/tree/master/v0.5/recommendation

Let's not worry about ONNX models right now, because they are being re generated with ONNX support for EmbeddingBag.

Ah, mine is official PyT (100GB one). Thanks

So, if you could provide the MD5 for the other PyTorch debugging model, roughly ~10GB in size, that would help.

I have created MD5 hash for both models. I'll add it to the README.
```
> md5sum tb0875_10M.pt
b7cacffcf75f767faa9cb2af397723aa  tb0875_10M.pt
> md5sum tb00_40M.pt
2d49a5288cddb37c3c64860a06d79bb9  tb00_40M.pt
```

The following PR https://github.com/mlperf/inference/pull/674 adds the MD5 hash for models. 

Thanks! This is super useful
Repository: mlcommons/inference
Issue: Does Loadgen Support Distributed Inference?
Root cause: Unimplemented Feature Gap

We tested our system with reference code and found that the final log is generated from this function: `lg.StartTest(sut, qsl, settings)`.

The linked is given by: https://github.com/mlperf/inference/blob/55bfef23f63edbe2d9450fa24fd897c03cea3cab/v0.5/classification_and_detection/python/main.py#L515

So, the question is how to modify the code if we are going to use distrubuted system for inference? Thanks!

right now loadgen does not support distributed inference.


We noticed the results in inference V0.5. submitted by Google with a system with a maximum of 32xCloud TPU v3. How did they summarise the logs generated by loadgen on different nodes? Thanks!

There was only one node. In Google Cloud, TPUs are connected to hosts over a network. You can connect an arbitrary number of TPUs to a single host. You can see the source code for the Google submission in the https://github.com/mlperf/inference_results_v0.5 repository.

closing for now. We always welcome PRs to enable new functionality. @SIATHICAS
Repository: mlcommons/inference
Issue: Why LoadGen use C++

I'm curious why using C++ within Loadgen not other language. What's the advantage of using C++ instead of python and C?

The LoadGen needs to be very low overhead in order to measure the maximum throughput of large systems and to avoid overburdening the CPU in small systems. Python's overheads are too high. Choosing between C and C++, there are more programmers comfortable writing well-modularized code in C++ than C.
Repository: mlcommons/inference
Issue: nnUnet-3D benchmark improvements
Root cause: Architectural Constraint

Hi!

While working on OpenVino SUT I found what existing approach not let to achieve best performance. In current implementation SUT code do sample request, runs inference, applying Softmax, converting data to fp16 and report back to framework. I measured time of inference and total time for single sample for all steps mentioned above and found what inference time on my system is just 40% of total time, so framework spend more time on data conversion and serialization than on inference. 

For accuracy mode accuracy-brats script loads mlperf log, calling nnUnet function save_segmentation_nifti_from_softmax to restore cropped image to original size and apply Argmax. Why not do Argmax right at SUT code, this will save a time? This will be even more fair benchmark as in a real workload one will apply Argmax to output.

I've changed approach, now SUTs applying Argmax after Softmax, my branch is available here: https://github.com/mlperf/inference/compare/master...Drizshko:unet_argmax 

Another question - why to do conversion to fp16 and create byte array in benchmark mode as data not serialized in this mode? Accuracy log is empty if you are not using --accuracy flag. In a real workload i prefer to handle data in fp32 and not to loose a time on data conversion to fp16, and fp16 needed only by mlperf just to save storage space and improve save/load time.
Repository: mlcommons/inference
Issue: Accuracy Log storage

3D UNet accuracy logs are 8GB in fp16, twice that in fp32.  We should think about how to manage these. 

Would one of the CSPs be able to contribute some storage?

yeah, a normal github account will not like it. And we hope to receive many of them.
Even if you gzip it still might be too much for github.

It's about 340MB per file zipped (it will compress better if we change the model along the lines of https://github.com/mlperf/inference/issues/670 to output the class per pixel rather than the entire softmax.)   

Since Nvidia is planning to submit multiple configs, we expect it to add up significantly. And we have a lot more submitters this time around.

If I remember correctly, the repo size in free github was 1GB and 100MB per file, so this will become a pretty big issue.

@alexkarargyris please follow up. the backup is to use Google cloud storage somewhere. 

WG decided to update a script. To be done by @guschmue 

We have a resolution, closing in WG.