Repository: pytorch/benchmark
Issue: Readme Example is broken
Root cause: Documentation Deficiency

# Summary

Steps to reproduce

```Shell
python3 install.py
pip install . # add -e for an editable installation
```

```Python
import torchbenchmark.models.densenet121
model, example_inputs = torchbenchmark.models.densenet121.Model(test="eval", device="cuda", batch_size=1).get_module()
model(*example_inputs)
```

```Shell
Traceback (most recent call last):
  File "/home/drisspg/meta/benchmark/torchbench_loop.py", line 14, in <module>
    main()
  File "/home/drisspg/meta/benchmark/torchbench_loop.py", line 6, in main
    model, example_inputs =  models.densenet121.Model(test="eval", device="cuda", batch_size=1).get_module()
AttributeError: module 'torchbenchmark.models' has no attribute 'densenet121
```


Having Torchbench not be package though makes the path shennagins kind of hard to reason about so not sure if the why the  model is not being picked up.

Sorry I could not reproduce this error:

```
(base) ➜  benchmark git:(main) python -c "import torchbenchmark.models.densenet121; print(torchbenchmark.models.densenet121.Model)"
/home/xz/miniconda3/lib/python3.11/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
<class 'torchbenchmark.models.densenet121.Model'>
```

This will also work:
```
python -c 'from torchbenchmark.models import densenet121; print(densenet121.Model)'
/home/xz/miniconda3/lib/python3.11/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
<class 'torchbenchmark.models.densenet121.Model'>
```

Can you please try with `import torchbenchmark.models.densenet121`, and then use
`torchbenchmark.models.densenet121.Model()` instead of `models.densenet121.Model()`?


--
I think I can reproduce when importing with `torchbenchmark` or `torchbenchmark.models`:

```
python -c 'import torchbenchmark; print(torchbenchmark.models.densenet121.Model)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'torchbenchmark.models' has no attribute 'densenet121'
```

I can look into how to support directly instantiate the model with importing `torchbenchmark.models` or `torchbenchmark` module.

Yeah, I was able to work around this by making sure I was running the script from with the torchbench top level structure, I do think this is very counter-intutitive. 

Script I got working, all though many models threw failures with the --precision=bf16 args
https://gist.github.com/drisspg/402855d460dbf130013b9230b35699c2#file-torchbench_loop-py

Feel free to close this issue if you think its not reproducible

I will look into how to instantiate models by directly importing from `torchbenchmark.models`.

Can we have a list of models that don't support bf16? We fixed `BERT_pytorch` in https://github.com/pytorch/benchmark/pull/2185 but I hope we can have a complete list for the track of record.

Using this script:
```Python
from transformer_nuggets.utils.shape_trace import ShapeLog
import torch
from pathlib import Path
from tqdm import tqdm
import logging
import json

logging.basicConfig(level=logging.INFO)

def main():
    import torchbenchmark.models as models
    models = []
    success_count = 0
    failure_count = 0
    model_failures = {}
    for file in Path("torchbenchmark/models/").iterdir():
        if file.is_dir():
            models.append(file.name)
    for model_name in tqdm(models, desc="Logging models", aunit="model"):
        try:
            module = __import__(f"torchbenchmark.models.{model_name}", fromlist=[model_name])
            model, example_inputs = module.Model(test="train", device="cuda", extra_args=["--precision=bf16",]).get_module()
            model(*example_inputs)
            success_count += 1
        except Exception as e:
            tqdm.write(f"Failed to log {module}: {e}")
            failure_count += 1
            model_failures[model_name] = str(e)

    tqdm.write(f"Successfully logged {success_count} models")
    tqdm.write(f"Failed to log {failure_count} models")
    with open("model_failures_bf16.txt", "w") as f:
        json.dump(model_failures, f)



if __name__ == "__main__":
    main()

```

returns the following model failures;

https://www.internalfb.com/intern/paste/P1197341526/



Thanks for sharing the result, I created a new issue https://github.com/pytorch/benchmark/issues/2203 tracking the bf16 precision support of the models.

The package import problem has been fixed, so I am closing this issue now.