Repository: mlcommons/inference
Issue: as python's threading is one core multithreading, so only one core is working for inference.
Root cause: Documentation Deficiency

class QueueRunner(RunnerBase):
    def __init__(self, model, ds, threads, post_proc=None, max_batchsize=128):
        ...
        for _ in range(self.threads):
            **worker = threading.Thread(target=self.handle_tasks, args=(self.tasks,))**
            worker.daemon = True
            self.workers.append(worker)
            worker.start()
From code in "inference/vision/classification_and_detection/python/main.py" , the QueueRunner is designed to use multi cores. But as threading of python is single core mutithreading. so only 1 core is working.
Does anybody know the reason?  Or will the threading be changed to mutiprocessing.

Please feel free to file a PR to improve it.

'so only 1 core is working' is not entirely correct.
What happens is that as soon you call the runtime predict() the runtime will release the gil (I'm 100% sure for tensorflow and onnxruntime). The runtime itself will utilize all threads (hopefully).
Because the gil is released, a python thread can go and schedule additional requests.

If you look at cpu utilization when running inference on cpu you should see all cpu's fully loaded (a least that is what I see).

I'm not saying that I'm a fan of using threading. It would be interesting to try multiprocessing (which is possible of course) but I fear the overhead is higher than plain threading.
The main issue with threading is that for cheap models (say mobilenet) and super fast hardware (say a100) the request rates get so high that python runs into gil contention in scheduling and post processing for the requests.

![image](https://user-images.githubusercontent.com/15189794/98885933-ec858100-24cd-11eb-8130-b0c81fb2679b.png)
there's 3 point that could apply multi core. The 1st is mlperf, as described in this loop, but python's threading can't apply mutl core because of GIL lock. The 2nd point is runtime(tensorflow/pytorch/onnxruntime), I can't make sure how the runtime apply multi core,  maybe divide a model into multi matrix operation, and schedule a matrix multiply into one core. but as the figure here, pytorch don't do such parallel. the 3th point is do parallel in backend math lib(mkl/openblas/eigen), as mathlib only recieve matrix operation, so the mathlib could only divide a matrix to many parts and schedule to many cores. 
the best way is do parallel in mlperf, this could make sure only lestest schedule overhead is introduced. 
also paste a flame graph of multhread applied by openblas, the schedule overhead  almost consumed more 80% cpu 
![image](https://user-images.githubusercontent.com/15189794/98886824-b517d400-24cf-11eb-81ec-a5f0dabcb929.png)


Your term window above - you are running SingleStream. That scenario does not use threads:
https://github.com/mlperf/inference/blob/master/vision/classification_and_detection/python/main.py#L469

if the runtime doesn't do things in parallel ... perf is not going to be good so I don't think you'll find many runtimes that run matrix ops on a single thread. Unless of course you care more about resource than perf which I would expect for embedded stuff and maybe mobile.
But as long the runtime releases the gil before running inference the mlperf app should run on other threads. The runtimes I know will do this, for example here:
https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/onnxruntime_pybind_state.cc#L1715


onnxruntime really could use multi core correctly , so close