Repository: pytorch/benchmark
Issue: [Feature Request] Benchmarking OCR Models

Currently, trying out torchdynamo on a few OCR models, with several different backends on GPU. Not able to achieve the shown speedups on a few benchmarks you published. Currently, getting almost the same (or sometimes slower) inference time with `@torch._dynamo.optimize`


can you point us to some open-source OCR models that you are interested?

Currently, testing dynamo on few detection & recognition models, mostly open-source. [DocTR](https://github.com/mindee/doctr) is quite a powerful OCR tool (especially on difficult documents and some edge cases which I tried out before). Might be any other tool with similar backbones. The main idea is to actually achieve the speedups.

I can recreate the benchmarking with vanilla BERT and dynamo is quite good there, which is awesome. But looks like dynamo finds it harder to optimize the python bytecode coming from some 'custom' models like DocTR.

I have seen this [benchmark](https://github.com/pytorch/pytorch/issues/93794) which shows 

``` 
+----------------+------------+-------------+-------------+
|    Compiler    | torchbench | huggingface | timm_models |
+----------------+------------+-------------+-------------+
|     eager      |   1.00x    |    1.01x    |    1.00x    |
|   aot_eager    |   1.00x    |    1.00x    |    1.00x    |
| aot_cudagraphs |   1.09x    |    1.00x    |    1.00x    |
|  aot_nvfuser   |   1.16x    |    0.0x     |    1.20x    |
|    inductor    |   1.70x    |    2.17x    |    1.30x    |
+----------------+------------+-------------+-------------+
```

From here, it is hard to say that dynamo boosts performance on most of those back-ends. Is it right to say that it is expected to not see any speedups on OCR models as well? @soumith . 

Btw, I have played with DocTR before and can help on getting that set up for you so that any of you guys can implement dynamo on it. Hopefully, eventually, it gets to somewhere near promised speedup.

it kinda depends -- we hope to see a speedup, but if we're not seeing as much speed as we should then it has to be investigated.

If you would be willing to spend some time on figuring out why, please look at this section: https://github.com/pytorch/torchdynamo/blob/main/documentation/FAQ.md#why-am-i-not-seeing-speedups

And try some of the things suggested there

Thanks for the info. Yes, I have tried the profiling functionalities and a few other stuff mentioned on that FAQ page. Apparently, OCR models are kinda custom as opposed to regular BERT, and dynamo's tracing method cannot figure out any way to optimize it, thus, always falling back to native PyTorch. 


can you post some logs if you don't mind.
What does `.explain()` give, what other logs did you see that shows fallbacks and graph breaks...

Haven't played with `.explain()` but checked `profiler.report`
```
torch._dynamo.config.verbose=True
torch._dynamo.config.suppress_errors = True
prof = dynamo.utils.CompileProfiler()
```

which logs out:
```
Torchdynamo Profiler Report

The following conditions caused torchdynamo to break out of tracing and fall back to python.
You may gain additional insight by passing `nopython=True` to torch._dynamo.optimize, to break on the first condition.
Graph Break Reason                           Count
-----------------------------------------  -------
call_method NNModuleVariable() eval [] {}        8No cache-limited recompilations detected.

```

Created PR https://github.com/pytorch/benchmark/pull/1315 which adds the doctr model. 
It will be better if we can use torch.tensor instead of numpy.array as inputs.
Also, please help confirm the way I set batch size is correct, thanks!

Thanks.
Commented there few things. We usually test out detection and recognition models separately to see how much speedup each model gets. However, both approaches should be fine. Will wait for approval from @ryanchesler on `checking the correctness of the output tensor`.

> Thanks. Commented there few things. We usually test out detection and recognition models separately to see how much speedup each model gets. However, both approaches should be fine. Will wait for approval from @ryanchesler on `checking the correctness of the output tensor`.

I have updated and addressed comments from both of you, please help take another look, thanks.