Repository: pytorch/benchmark
Issue: Investigate apex for yolov3
Root cause: Architectural Constraint

yolov3 used to modify its behavior based on if apex was installed (see #677), which led to benchmarking different things based on the environment.  We should investigate if it is a good idea to switch this model to requiring apex.

#677 fixes this by always disabling apex, and I am fine with it. However, keeping benchmark consistent with environment could also mean we simply can't support many environments.

For example, after enabling half support as suggested in https://github.com/pytorch/benchmark/issues/657, the benchmark can't run on GPUs that don't support half precision. Similarly, current batch sizes, which saturates the GPU on our CI GPU, will cause CUDA OOM on smaller GPUs which are used by hobbyists (https://github.com/pytorch/benchmark/issues/665).

Maybe a better solution is to:
1. at the beginning of the benchmarking, detect the environment and generate a benchmark metadata object (e.g. `apex: false`, `batch_size: small`, etc)
2. write the benchmark metadata object into result json, possibly attach a hash of the metadata object, and only compare the result when the hashes match

@wconstab @eircfb any feedbacks?


IMO we should avoid rewriting benchmarks with the goal of increasing GPU utilization.  I get that this helps performance, but it also is a slippery slope where soon we are measuring something different from what our users are actually doing.

We should optimize the models people write, not the models we wish they wrote.



@jansel  I agree that we should keep the code as close as the original model in general.

However, there are a few issues in we've met in practice, and here are our current solutions:
1. The default model configuration is not representative (e.g. doesn't enable half when it is available https://github.com/pytorch/benchmark/issues/657)
   In this case, we tune the model config to make it representative (e.g. enable half by default when available).
2. The models don't have a reference implementation that suggests what is the right batch size to use (e.g. torchvision models)
   For train, we use the "batch size per device" value used in the paper (divide the original batch size by the number of GPU/TPU used in the paper). For inference, the batch size is usually not provided in the paper. Therefore, we use the smallest batch size that saturate our CI machine's GPU.
3. The reference implementation doesn't run on our CI environment because of hardware limitation (e.g. CUDA OOM)
   For train, we disable the test and are working to support more hardware this half. This is because we can't modify train batch size without having correctness checks.
   For inference, we downscale the batch size until it fits on our current GPU.
4. The reference implementation exists, but it doesn't have good GPU utilization
   We stick to the original batch size even if they don't have good GPU utilization. We plan to add labels to such tests in the repository. Ideally, we should identify the reason of GPU idleness and submit patches to upstream (either model or pytorch domain packages).

> IMO we should avoid rewriting benchmarks with the goal of increasing GPU utilization. I get that this helps performance, but it also is a slippery slope where soon we are measuring something different from what our users are actually doing.
> 
> We should optimize the models people write, not the models we wish they wrote.

I agree it is important to have a body of actual user code to benchmark, a library of curiosities, but we also need to hold our stack accountable to perf.

The danger with benchmarking badly written loads is that we're rewarding the optimization of yesterdays ineptitudes, or in this case years old problems. In many cases the loads have low utilization from idleness due to communication latency. Is not something that a naive compiler can fix. Quite often we see sizes that were tuned on years old hardware and they're ridiculous on modern stuff that is 10x more powerful.

Example: A base criteria is to insure that inference time increases with batch size. In many cases with default sizes the batch is so small that gpu work isn't not hidden by cpu dispatch. Sure we can optimize the dispatch but no way can that scale with the expected improvements in gpu capacity. The work size is often too small to keep gpu busy. If a benchmark is an idle inference then it bottlenecks on dispatch instead of compute and isn't sensitive to the thing that competent customers care about.

I think to sensibly track cq we need to benchmark well written loads and align our opt efforts to what competent customers should write.

We are now addressing this problem in https://github.com/pytorch/benchmark/pull/776.
First, we don't support nvidia apex (at least for now).
Second, we support `amp` for all models, but don't use it for default for now (until the concerns in the PR are addressed)
Finally, we plan to enable `amp` for part of the models (e.g., defined in `metadata.yml`)

Resolved by https://github.com/pytorch/benchmark/pull/776