Repository: mlcommons/inference
Issue: SSD-ResNet34 with cpu. Errors with Server and Offline scenarios on accuracy mode
Root cause: Resource Mishandling

hi all, I am running object detection as `./run_and_time.sh pytorch ssd-resnet34 cpu` on accuracy mode and getting the below errors:

python version: 3

**TestScenario.Server**
```
INFO:main:starting TestScenario.Server, latency=0.01
python3: /opt/conda/conda-bld/pytorch-cpu_1556653114183/work/third_party/ideep/mkl-dnn/src/common/memory_tracking.hpp:229: void* mkldnn::impl::memory_tracking::registry_t::get(const key_t&, void*) const: Assertion `size() == 0' failed.
```

**TestScenario.Offline**
```
Traceback (most recent call last):
  File "python/main.py", line 527, in <module>
    main()
  File "python/main.py", line 504, in main
    lg.StartTest(sut, qsl, settings)
  File "python/main.py", line 445, in issue_queries
/mlperf/run_helper.sh: line 33: 12944 Segmentation fault      (core dumped) python3 python/main.py $opts --output /output/results_Offline.json --scenario Offline --accuracy
```



@briandersn any thoughts? or @christ1ne or @guschmue do you have any pointers?

I have not tried any PyTorch code yet. Is the TF model working for you? Are you only interested in the PyTorch code @vilmara?
Thanks! 

Hi @vilmara , can you upload the loadgen log files you get in each case?

The Server error doesn't look loadgen related.

The Offline error looks like it originates from the loadgen though. Can you make sure you are running with the latest code then upload a stack trace that includes the C++ frames? You may need to attach gdb.

I have not tried the pytorch for some time (have the wrong pytorch version on my box). If I remember this correctly the pytorch model might use a batch size of 1 but by default we pass in 128 and for offline we'd use that batch size. 
You could add a ```--max-batch-size 1``` , if that works I can make that the default for this model. I'd not expect that it crashes, but ...

@briandersn is this fixed with #278?

> is this fixed with #278?

As far as I know, #278 only fixed a hang and not a segfault. I haven't been able to reproduce this error to understand the root cause.

hi @briandersn / @guschmue I have used the latest code and the offile scenario is still with issues on both modes


`$ git log -1`
```
commit 2749668398d63fd9078eaaad8818b4e8442ac51a (HEAD -> master, origin/master, origin/HEAD)
Merge: e37ce8a 3ee4877
Author: jveejay <jvijay@gmail.com>
Date:   Thu Jul 18 18:25:11 2019 -0400

    Merge pull request #274 from briandersn/lg_typos

    Make python and C++ linters happy
```

**Trace log Performance mode**
```
INFO:main:Namespace(accuracy=False, backend='pytorch-native', cache=0, count=None, data_format=None, dataset='coco-1200-pt', dataset_list=None, dataset_path='/home/user1/CK-TOOLS/dataset-coco-2017-val', inputs=['image'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/user1/mlperf_models/resnet34-ssd1200.pytorch', output='/output/results_Offline_acc.json', outputs=['bboxes', 'labels', 'scores'], profile='ssd-resnet34-pytorch', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=80, time=None)
INFO:coco:loaded 201 images, cache=0, took=0.4sec
/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'models.ssd_r34.SSD_R34' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torchvision.models.resnet.BasicBlock' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
INFO:main:starting TestScenario.Offline
/mlperf/run_helper.sh: line 19:    10 Killed                  python3 python/main.py $opts --output /output/results_Offline_acc.json --scenario Offline
ENDING RUN AT 2019-07-24 06:45:48 PM
```

**Tracelog Accuracy mode:** 
```
INFO:main:Namespace(accuracy=True, backend='pytorch-native', cache=0, count=None, data_format=None, dataset='coco-1200-pt', dataset_list=None, dataset_path='/home/user1/CK-TOOLS/dataset-coco-2017-val', inputs=['image'], max_batchsize=128, max_latency=[0.01, 0.05, 0.1], model='/home/user1/mlperf_models/resnet34-ssd1200.pytorch', output='/output/results_Offline_acc.json', outputs=['bboxes', 'labels', 'scores'], profile='ssd-resnet34-pytorch', qps=10, queries_multi=24576, queries_offline=24576, queries_single=1024, scenario=[TestScenario.Offline], threads=80, time=None)
INFO:coco:loaded 5000 images, cache=0, took=0.5sec
/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'models.ssd_r34.SSD_R34' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torchvision.models.resnet.BasicBlock' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
INFO:main:starting TestScenario.Offline
/mlperf/run_helper.sh: line 19:    10 Killed                  python3 python/main.py $opts --output /output/results_Offline_acc.json --scenario Offline --accuracy
```

[mlperf_log_detail_accuracy_mode.txt](https://github.com/mlperf/inference/files/3428378/mlperf_log_detail_accuracy_mode.txt)
[mlperf_log_detail.txt](https://github.com/mlperf/inference/files/3428379/mlperf_log_detail.txt)



@vilmara please let us know if this issue still persist. 

Hi @christ1ne / @briandersn the problem still persists only for Offline scenario on accuracy mode

@vilmara: I can't see anything immediately wrong with the settings chosen. Can you run this through gdb and get a stack trace?