Repository: pytorch/benchmark
Issue: Invalid Literal error using non-server GPUs
Root cause: Validation Gap

Doing "non-accurate" benchmark testing to verify all models are working with new binaries I compiled. Benchmark was run on an AMD Ryzen 7950 with Nvidia MSI 4090 Gaming X trio. Due to GPU not being "server" based, some options are not available.

One Option missing with non-server GPUs is 'clocks.applications.graphics'
```
$ nvidia-smi --query-gpu="clocks.applications.graphics" --format=csv
clocks.applications.graphics [MHz]
[N/A]
```
The __[N/A]__ from the nvidia-smi causes the following in a good amount of the benchmark tests.
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
torchbenchmark/util/machine_config.py:295: in get_machine_state
    state['nvidia_gpu_clocks'] = get_nvidia_gpu_clocks()
torchbenchmark/util/machine_config.py:120: in get_nvidia_gpu_clocks
    return [int(clock) for clock in clocks]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <list_iterator object at 0x7f001788b400>

>   return [int(clock) for clock in clocks]
E   ValueError: invalid literal for int() with base 10: '[N/A]'

torchbenchmark/util/machine_config.py:120: ValueError
```
I still get the results of the test, but with all the errors from this one issue. Example of Resnet50
```
--------------------------------------------------------------------------------------- benchmark 'hub': 4 tests --------------------------------------------------------------------------------------
Name (time in ms)                       Min                 Max                Mean            StdDev              Median               IQR            Outliers       OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_eval[resnet50-cuda-jit]         5.0603 (1.0)        6.1184 (1.0)        5.1609 (1.0)      0.1355 (1.0)        5.1156 (1.0)      0.0543 (1.0)         15;31  193.7653 (1.0)         191           1
test_eval[resnet50-cuda-eager]       6.4079 (1.27)      11.0721 (1.81)       6.8307 (1.32)     0.9952 (7.34)       6.4930 (1.27)     0.2411 (4.44)        10;18  146.3974 (0.76)        156           1
test_eval[resnet50-cpu-jit]        277.4333 (54.83)    283.6096 (46.35)    279.7809 (54.21)    2.5651 (18.92)    279.1325 (54.57)    4.0045 (73.80)         1;0    3.5742 (0.02)          5           1
test_eval[resnet50-cpu-eager]      440.5987 (87.07)    448.1738 (73.25)    444.2787 (86.09)    3.0295 (22.35)    443.8730 (86.77)    4.8589 (89.55)         2;0    2.2508 (0.01)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
Have forked the repo and made a branch with the fix for non-server Nvidia GPUs. Will be running all benchmarks tomorrow to insure no other issues.

Not sure if you want the fix with it being "non-server" but I have it. In effect, any value in that array that is __[N/A]__ will be __0__ instead. giving the needed integer for the return. My fix does support multi-gpu setups.

Thanks for your report! I don't think returning 0 is a good idea. Maybe we should use `nvmlDeviceGetClockInfo` in pynvml to obtain GPU clock when `nvidia_smi_query` doesn't work. or completely use pynvml rather than `nvidia_smi_query`. what do you think about it? @xuzhao9 


Oh, it's not about which tool we use. GeForce GPUs don't support `clocks.applications.graphics`. 
> "clocks.applications.graphics" or "clocks.applications.gr"
>  User specified frequency of graphics (shader) clock.




@FindHao fun thing with Nvidia. I can see about getting clock information from another point for GeForce cards. I don't think we can use a -1 as notice the data is missing as I think that would not be considered an integer. In this case, we might need to rethink the forcing of the integer. Looks like the update in my branch solved my issues. Mainly dealing with "Numpy is not available", which is not part of this issue.

The ideal way is to find the corresponding API available on Geforce GPUs in pynvml lib or nvidia-smi. The `clocks.applications.graphics` means when applications are executed, the GPU will be set to that frequency to run the applications. It's not the current GPU clock. 

Going deeper into this. Looks like the application graphics clock can be set and is set, by default, on AWS instances. As you stated, Nvidia does not support this on GeForce.
```
# nvidia-smi -i 0 -ac 5001,2910
Setting applications clocks is not supported for GPU 00000000:01:00.0.
Treating as warning and moving on.
All done.
```
With this, the only real option would be "clocks.current.graphics", being the entire GPU or leave as "0" when the value shows [N/A]
```
# nvidia-smi --query-gpu=clocks.current.graphics -i 0 --format=csv
clocks.current.graphics [MHz]
210 MHz
```
As we are talking consumer usage here, which would not be good for a "true" benchmark, I feel either option would work. We still need to add ```--ignore_machine_config```, stating we realize this is not the best benchmarking platform. Please, let me know what you think.

`clocks.current.graphics` can't describe the clock when applications are executed. This clock frequency is obtained before the model executions. So it is misleading if we use this metric. For now, looks like return 0 for GeForce GPUs is the best choice. What do you think? @xuzhao9 

You will need `--ignore_machine_config` even if it supports GeForce GPU, because the machine config we set will only work on the AWS CI instance, it has not only GPU frequency requirements, but also CPU isolation requirements.

I believe always adding the `--ignore_machine_config` in your benchmark environment is the best effort.

I don't think we pose any constraint on the type of this field, so I think either N/A or 0 works in this case.

@xuzhao9 and @FindHao, will put in a PR and see what you think then.

Thank you for the Merge. Closing the Issue.