Repository: pytorch/benchmark
Issue: pplbench_beanmachine benchmark doesn't use pytorch
Root cause: Unimplemented Feature Gap

I ran it with the pytorch profiler, and it has zero ops in it.

Looking at the source code.  
```
    def get_module(self):
        return self.model, self.example_inputs
```

Leads to
```
        self.model = RobustRegression()
```

Which comes from:
```
from pplbench.models.robust_regression import RobustRegression
```

Looking for the string "torch" in the `pplbench.models.*` folder:
```
$ grep "torch" .../site-packages/pplbench/models/*.py
$
```
Has no matches.

Looking at robust_regression.py, it seems like it primarily uses `xarray` not PyTorch.

I'd suggest deleting this benchmark.  Or finding an alternate version based on PyTorch.



This model was suggested by @anijain2305  and @Chillee to address dynamic shapes. @anijain2305 do you think we should keep this model, or remove it? Is this because we don't correctly implement this model?

Other parts of BeamMachine might use PyTorch, just not `pplbench.models.robust_regression.RobustRegression()`.

I'm pretty sure the benchmark isn't correctly implemented - will ping the relevant folk.

Hi @jansel , you're right that `pplbench.models.robust_regression` does not utilize PyTorch at all :). This is because that module is actually for [data generation](https://github.com/facebookresearch/pplbench/blob/main/pplbench/models/robust_regression.py#L65-L106) and [evaluating](https://github.com/facebookresearch/pplbench/blob/main/pplbench/models/robust_regression.py#L108-L128) the performance given the samples/predictions from an inference.

The part that actually uses PyTorch is `pplbench.ppls.beanmachine` module, which defines the model and run inference in our [Bean Machine](https://github.com/facebookresearch/beanmachine) library (which is written in PyTorch). So If you're running the profiler, [this section of TorchBench](https://github.com/pytorch/benchmark/blob/97efd8b0dbec4aaae5f95a76dfee97f6859556e6/torchbenchmark/models/pplbench_beanmachine/__init__.py#L46-L47) might be what you actually want to look into instead :).

@horizon-blue we are using the `get_module()` API here: https://github.com/pytorch/benchmark/blob/97efd8b0dbec4aaae5f95a76dfee97f6859556e6/torchbenchmark/models/pplbench_beanmachine/__init__.py#L76 -- Which doesn't use any of the code you linked.

Perhaps we should pre-compute the `robust_regression` stuff and exclude it from benchmark timing.  There is not much PyTorch can do to speed up code that doesn't use PyTorch.



> @horizon-blue we are using the `get_module()` API here:
> 
> https://github.com/pytorch/benchmark/blob/97efd8b0dbec4aaae5f95a76dfee97f6859556e6/torchbenchmark/models/pplbench_beanmachine/__init__.py#L76
> 
> -- Which doesn't use any of the code you linked.
> Perhaps we should pre-compute the `robust_regression` stuff and exclude it from benchmark timing. There is not much PyTorch can do to speed up code that doesn't use PyTorch.

@horizon-blue @jansel 
My understanding is: if `self.test` is set to "train", then the model returned by `get_module()` API uses PyTorch. If `self.test` is set to "eval", then no PyTorch code is used.

Therefore, I submitted https://github.com/pytorch/benchmark/pull/909 to disable the eval test, but keeps the train test.


I belive the `get_module()` API doesn't use pytorch in either mode.  It always just returns the xarray robust_regression.   

Either way, a huge chunk of this benchmark is measuring non-PyTorch code.  We should refactor it to not measure non-PyTorch models.

> I belive the `get_module()` API doesn't use pytorch in either mode. 

I'm not very familiar with TorchBench, so apologize if this is a stupid question, but is `get_module()` the only thing that's being measured at the moment?

Calling `model, example_inputs = self.get_module()` itself does not involve any PyTorch operations. However, if we are in the "train" mode, then running the returned model (i.e. calling `model(*inputs)` will invoke the MCMC inference algorithm that's written in PyTorch.

> > I belive the `get_module()` API doesn't use pytorch in either mode.
> 
> I'm not very familiar with TorchBench, so apologize if this is a stupid question, but is `get_module()` the only thing that's being measured at the moment?
> 
> Calling `model, example_inputs = self.get_module()` itself does not involve any PyTorch operations. However, if we are in the "train" mode, then running the returned model (i.e. calling `model(*inputs)` will invoke the MCMC inference algorithm that's written in PyTorch.

TorchDynamo uses `get_module()` for both eval/train and doesn't use the `train()` function.



@anijain2305  and I decided that this benchmark is not very useful because only train uses pytorch and only works on cpu. We probably need a more systematic plan to deal with dynamic shapes, which will be addressed later.