sharjeelyunus/github-issues-dataset huggingface dataset:
Repository: pytorch
Issue: [inductor] Add more matmul configurations to `TORCHINDUCTOR_MAX_AUTOTUNE=1` mode
Labels: triaged,enhancement,oncall: pt2,module: inductor

https://github.com/pytorch/pytorch/pull/90738 contains 12 configurations for mm/addmm/bmm/etc:

https://github.com/pytorch/pytorch/blob/2b04caf71f09728cb655ccd9c7569d04ecb8b883/torch/_inductor/ops/mm_common.py#L22-L57

While writing this PR I observed speedups by adding/tweaking configurations, and I still think there is room for even more improvements.

When run with `TORCHINDUCTOR_MAX_AUTOTUNE=1`,
those 12 configurations are benchmarked, and the fastest configuration is chosen (at compile time) here:
https://github.com/pytorch/pytorch/blob/2b04caf71f09728cb655ccd9c7569d04ecb8b883/torch/_inductor/select_algorithm.py#L576

Upstream Triton has some similar configurations:
https://github.com/openai/triton/blob/8650b4d1cbc750d659156e2c17a058736614827b/python/triton/ops/matmul.py#L30-L51
plus 100+ more that are generated in `get_configs_io_bound()`.

It then prunes those configurations via a performance model:
https://github.com/openai/triton/blob/8650b4d1cbc750d659156e2c17a058736614827b/python/triton/ops/matmul_perf_model.py#L33

We might be able to use a similar perf model + pruning approach, though we should spend time benchmarking  -- espeically for the mm variants like bmm that upstream Triton doesn't have ops for.

Note we will need to filter out `SPLIT_K>1` configurations, since our template doesn't allow that in order to do epilogue fusions.

We could also try making other tweaks to the templates, such as different load orders (see the GROUP_M reordering).  For small sizes we could also try alternate approaches such as removing the loop and loading things into a single Triton block.



cc @ezyang @msaroufim @bdhirsh @anijain2305 @zou3519 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @amjames @desertfire @soumith @wconstab @ngimel
Repository: pytorch
Issue: A Simple Function Causing Graph Break
Labels: triaged,bug

### 🐛 Describe the bug

An error raised when calling `torch._dynamo.optimize` when optimizing huggingface GPT2 and enforcing a single optimized graph for the whole model.

```
  File "conda/dexport/lib/python3.9/site-packages/torch/_dynamo/variables/user_defined.py", line 243, in call_function
    return super().call_function(tx, args, kwargs)
  File "conda/dexport/lib/python3.9/site-packages/torch/_dynamo/variables/base.py", line 230, in call_function
    unimplemented(f"call_function {self} {args} {kwargs}")
  File "conda/dexport/lib/python3.9/site-packages/torch/_dynamo/exc.py", line 67, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function UserDefinedObjectVariable(_lru_cache_wrapper) [] {}

from user code:
   File "dexport/tests/loop.py", line 12, in forward
    if transformers.is_torch_tpu_available():

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
```

Below is a minimal repro. If `transformers.is_torch_tpu_available()` is removed, then the code works fine.

```python
import torch
import torch._dynamo
from torch import nn
import transformers

class ToyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 10)

    def forward(self, x):
        if transformers.is_torch_tpu_available():
            x = x + 1.0
        else:
            x = x + 2.0
        return self.linear(x)


model = ToyModel()
x = torch.rand(10, 10, dtype=torch.float32)
y = model(x)

class GraphCaptureCompiler:
    def __init__(self):
        self.captured_graph = None
        self.captured_graph_count =  0
    def compile(self, gm, _):
        assert self.captured_graph_count == 0
        self.captured_graph = gm
        self.captured_graph_count += 1
        return gm
compiler = GraphCaptureCompiler()
y_optimized = torch._dynamo.optimize(compiler.compile, nopython=True)(model)(x)
```

I'd expect dynamo can just trace through this simple function.

### Error logs

_No response_

### Minified repro

_No response_
Repository: pytorch
Issue: If minifier test fails, stderr/stdout of subprocess calls is not printed
Labels: low priority,triaged,bug

### 🐛 Describe the bug

```
    # Runs the minifier launcher script in `repro_dir`, patched with `patch_code`.
    def _run_minifier_launcher(self, patch_code, repro_dir):
        self.assertIsNotNone(repro_dir)
        launch_file = os.path.join(repro_dir, "minifier_launcher.py")
        self.assertTrue(os.path.exists(launch_file))
        launch_code = self._inject_code(patch_code, launch_file)

        launch_proc = subprocess.run(
            ["python3", launch_file],
            capture_output=True,
            cwd=repro_dir,
        )

        return launch_proc, launch_code

```

this captures output, but then there is nothing that ensures this output is printed if, e.g., something else fails

### Error logs

_No response_

### Minified repro

_No response_
Repository: pytorch
Issue: tensor.to_sparse() handling indices incorrectly under dynamo/fake tensor
Labels: triaged,bug,oncall: pt2,module: dynamic shapes

### 🐛 Describe the bug

to_sparse() is returning a FakeTensor where the indices attribute has the wrong shape/size.

### Error logs

_No response_

### Minified repro



Repro 1:
```python
import torch
import torch._dynamo
import logging

# torch._dynamo.config.verbose = True
# torch._dynamo.config.log_level = logging.DEBUG

def fn():
    x = torch.rand((7, 3))
    return x.to_sparse().indices().stride()

print(fn()) # returns (21, 1)
print(torch._dynamo.optimize("eager")(fn)()) # returns (1, 1)
```

Repro 2:
```python
import torch
import torch._dynamo
import logging

# torch._dynamo.config.verbose = True
# torch._dynamo.config.log_level = logging.DEBUG

fake_mode = torch._subclasses.fake_tensor.FakeTensorMode()

print("~~ fake mode below")
with fake_mode:
    x = torch.rand((7, 3))
    y = x.to_sparse()

print(f"fake mode indices size: {y.indices().size()}") # torch.Size([2, 0]); should be [2, 21]
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Repository: pytorch
Issue: Torchdynamo with onnxrt backend generating fake tensor errors
Labels: module: onnx,triaged,bug,oncall: pt2

### 🐛 Describe the bug

Hi,
  I'm trying to use torchdynamo with onnxrt (both onnxrt_cuda and onnxrt_cpu backends) following the resnet example described in  https://pytorch.org/tutorials/intermediate/dynamo_tutorial.html but I'm hitting an error

`torch._dynamo.exc.BackendCompilerFailed: onnxrt_cuda raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(16, 3, 128, 128)), cuda:0), Parameter containing:
tensor([[[[...]]]], device='cuda:0', requires_grad=True),`

I've tried many examples from torchbench and all are hitting same error. It does not happen with other backends e.g. aot_cudagraphs etc. I've also force set the FakeTensorMode in fake_tensor.py but that hit another error down the line. Am I missing something obvious here?

Torch version: 2.0.0a0+gitdf46ba4
onnxrt version: 1.13.1

I was thinking something obvious is missing here before I dig into this in detail.

### Error logs

`torch._dynamo.exc.BackendCompilerFailed: onnxrt_cuda raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(16, 3, 128, 128)), cuda:0), Parameter containing:
tensor([[[[...]]]], device='cuda:0', requires_grad=True),`

### Minified repro

```python
import torch
import torch._dynamo as dynamo

# Returns the result of running `fn()` and the time it took for `fn()` to run,
# in seconds. We use CUDA events and synchronization for the most accurate
# measurements.
def timed(fn):
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    result = fn()
    end.record()
    torch.cuda.synchronize()
    return result, start.elapsed_time(end) / 1000

# Generates random input and targets data for the model, where `b` is
# batch size.
def generate_data(b):
    return (
        torch.randn(b, 3, 128, 128).to(torch.float32).cuda(),
        torch.randint(1000, (b,)).cuda(),
    )

from torchvision.models import resnet18
def init_model():
    return resnet18().to(torch.float32).cuda()

def eval(mod, inp):
    return mod(inp)
torch._dynamo.config.verbose=True
model = init_model()
eval_opt = dynamo.optimize("onnxrt_cuda")(eval)

inp = generate_data(16)[0]
print("eager:", timed(lambda: eval(model, inp))[1])
print("dynamo:", timed(lambda: eval_opt(model, inp))[1])
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @Chillee @samdow @kshitij12345 @janeyx99 @soumith @ngimel
Repository: pytorch
Issue: Triton Autotuning Cache-Clearing Adds 256MB Memory Overhead
Labels: module: memory usage,triaged,oncall: pt2,module: inductor

### 🐛 Describe the bug

In inductor autotuning, we invoke triton's do_bench which internally allocates a [256mb tensor](https://github.com/openai/triton/blob/259f4c5f7d4ecb8c64c519fe669318a15d6f75f2/python/triton/testing.py#L162) that is used to clear the cache.

From a practical standpoint, 256 MB is insignificant on a 40GB machine, however for a 12GB RTX 3080 this is a 2% overhead. Additionally, it's a non-linear factor, which throws off memory usage stats for small batch sizes. It has a big effect on the torchbench performance dashboard because low batch sizes are being used. E.g. `resnet18` is at .69 compression with autotuning, and ~1 without.

We could pass a handle to other allocated memory that is not being used in the autotuned kernel to avoid the additional allocation.

### Error logs

_No response_

### Minified repro

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @soumith @ngimel
Repository: pytorch
Issue: TorchInductor CPU Performance Dashboard
Labels: triaged,oncall: pt2,oncall: cpu inductor

Dashboard to track the performance of torchinductor on CPU.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @soumith @ngimel @chauhang
Repository: pytorch
Issue: torchdynamo.export doesn't work with data-dependent control flow
Labels: triaged,enhancement

### 🐛 Describe the bug

```
def f(x):
    x_bool = torch.isfinite(x)
    if x.all():
        return x.cos() 
    return x.sin()

torchdynamo.export(f, torch.ones(6, 4), aten_graph=True, tracing_mode="symbolic")
```

### Error logs

torch._dynamo.exc.Unsupported: generic_jump TensorVariable()

### Minified repro

_No response_
Repository: pytorch
Issue: AssertionError: Unknown expression s2
Labels: triaged,bug

### 🐛 Describe the bug

```
TORCHDYNAMO_DYNAMIC_SHAPES=1 AOT_DYNAMIC_SHAPES=0 time python benchmarks/dynamo/huggingface.py  --accuracy --backend eager --training --only AllenaiLongformerBase
```

on master

```
  File "/private/home/ezyang/.local/lib/python3.8/site-packages/sympy/printing/printer.py", line 331, in _print
    return printmethod(expr, **kwargs)
  File "/private/home/ezyang/.local/lib/python3.8/site-packages/sympy/printing/str.py", line 779, in _print_Relational
    return '%s(%s, %s)' % (charmap[expr.rel_op], self._print(expr.lhs),
  File "/private/home/ezyang/.local/lib/python3.8/site-packages/sympy/printing/printer.py", line 331, in _print    return printmethod(expr, **kwargs)
  File "/raid/ezyang/pytorch-scratch2/torch/_dynamo/guards.py", line 526, in _print_Symbol
    assert expr in self.expr_to_tensor_ref, f"Unknown expression {expr}"
AssertionError: Unknown expression s2
```

### Error logs

_No response_

### Minified repro

_No response_
Repository: pytorch
Issue: [Inductor] Support deterministic parallel reduction in CPP backend
Labels: triaged

CPP backend uses OMP to do parallel reduction which cannot guarantee a deterministic reduction order and might produce different run-to-run result while determinism is important to ease the debugging. We should replace omp parallel reduction with deterministic parallel reduction.

Relevant issue/PR: https://github.com/pytorch/pytorch/issues/93542, https://github.com/pytorch/pytorch/pull/87259
Repository: pytorch
Issue: Turning on minifier causes bug to go away (on DebertaForMaskedLM)
Labels: low priority,triaged,bug

### 🐛 Describe the bug

1. Check out and build pytorch at e4a8661ab84022c1bff622c6d2f6e679180b1df5 (Oct 29, before breaking commit was reverted)
2. Run `python benchmarks/dynamo/huggingface.py  --accuracy --backend inductor --training --only DebertaForMaskedLM`, it fails
3. Run `TORCHDYNAMO_REPRO_AFTER=dynamo python benchmarks/dynamo/huggingface.py  --accuracy --backend inductor --training --only DebertaForMaskedLM`, it passes

### Error logs

https://www.internalfb.com/intern/paste/P543371878/

### Minified repro

minifier did not work
Repository: pytorch
Issue: Minifier doesn't work on DebertaForQuestionAnswering
Labels: low priority,triaged,bug

### 🐛 Describe the bug

Build pytorch e4a8661ab84022c1bff622c6d2f6e679180b1df5

Apply this patch

```
diff --git a/torch/_dynamo/optimizations/backends.py b/torch/_dynamo/optimizations/backends.py
index 660e7a5ca56..d53733bec73 100644
--- a/torch/_dynamo/optimizations/backends.py
+++ b/torch/_dynamo/optimizations/backends.py
@@ -552,7 +552,8 @@ def cudagraphs_inner(model, inputs, copy_outputs=True):
 def aot_autograd(subgraph, **kwargs):
     def _wrapped_bw_compiler(*args, **kwargs):
         # stop TorchDynamo from trying to compile our generated backwards pass
-        return disable(bw_compiler(*args, **kwargs))
+        bwd_func = disable(bw_compiler)(*args, **kwargs)
+        return disable(bwd_func)
 
     bw_compiler = kwargs.get("bw_compiler") or kwargs["fw_compiler"]
     kwargs["bw_compiler"] = _wrapped_bw_compiler
```

Deberta failures accuracy with aot_eager, but the minifier doesn't do anything

```
$ pp PYTHONUNBUFFERED=1 TORCHDYNAMO_REPRO_AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4  python benchmarks/dynamo/huggingface.py  --accuracy --backend aot_eager --training --only DebertaForQuestionAnswering
[2022-10-29 19:30:33,226] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
cuda train DebertaForQuestionAnswering        [2022-10-29 19:30:33,509] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:33,626] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:33,731] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:54,042] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:54,050] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.00016, (ref-fp64): 0.00000 and shape=torch.Size([50265, 768])
[2022-10-29 19:30:54,050] torch._dynamo.utils: [ERROR] Accuracy failed for key name deberta.embeddings.word_embeddings.weight.grad
```

### Error logs

_No response_

### Minified repro

_No response_
Repository: pytorch
Issue: Don't store example_value on FX node meta
Labels: triaged,bug

### 🐛 Describe the bug

Storing it in the meta dict implies that we will pass it to the backend compiler. But actually we strip all of the example values from the FX graph when we're done. So this is really just an internal concept to Dynamo. I propose we store it on TensorVariable instead.

BTW, you don't want to actually pass this metadata on to the backend, because in the presence of metadata mutation it can be misleading. E.g.:

```
@torch._dynamo.optimize(my_backend)
def f(x):
    y = x.clone()
    y.transpose_(0, 1)
    return y

f(torch.randn(4, 5))
```

What do you expect the example value of y to be?

cc @eellison @Chillee @voznesenskym @jansel 

### Error logs

_No response_

### Minified repro

_No response_
Repository: pytorch
Issue: [Inductor] Input Buffers Should Be Representable As Storage And Layout
Labels: feature,triaged,oncall: pt2,module: inductor,internal ramp-up task

### 🐛 Describe the bug

[InputBuffer](https://github.com/pytorch/pytorch/blob/master/torch/_inductor/ir.py#L1897)'s currently do not have an associated storage. This prevents correct handling of aliased inputs - see [TODO here](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/graph.py#L637). 

When calling [require_stride_order](https://github.com/pytorch/pytorch/blob/master/torch/_inductor/ir.py#L2465), because `is_storage_and_layout` returns False for InputBuffers we fall through to an unnecessary copy. 


A shorter term fix would be to handle InputBuffers in `require_stride_order`.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @soumith @ngimel
Repository: pytorch
Issue: Explore TorchInductor optimization pass to reorder kernel bodies
Labels: feature,triaged,oncall: pt2,module: inductor

See pytorch/torchdynamo#934 for more context.   We found an example of a 10% performance difference from very similar kernels, where the biggest difference seemed to be instruction ordering.

Interesting, my first guess looking at the two kernels is just the ordering of ops.  The loads on the faster kernel are "spread out" while the loads in the slow kernel are "bunched up".

Perhaps we should explore a compiler pass that reorders ops within a kernel.

Our current inductor kernels usually look like:
```
<all of the loads>
<all of the compute>
<all of the stores>
````

When you have indirect loads, it moves those indirect loads into the "compute" section, because they must come after the address computation.  Thus allowing that spread out pattern to be generated.

My thinking of doing that ordering was it makes compiler analysis easier for Triton/LLVM.  I may have been wrong there... especially if Triton doesn't yet have an instruction reordering pass.

This is just one theory though, we should test it.

_Originally posted by @jansel in https://github.com/pytorch/torchdynamo/issues/934#issuecomment-1221583598_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @soumith @ngimel
Repository: pytorch
Issue: explain() has confusing explanation of graph breaks
Labels: triaged,oncall: pt2,module: dynamo

For full repro, see: n2409035.

The output of explain:
```
Dynamo produced 12 graphswith 11 graph break and 24 ops
 Break reasons: 

1. generic_jump TensorVariable() 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/models/transformer/transformer_encoder.py", line 207, in forward_scriptable
    x, encoder_embedding = self.forward_embedding(src_tokens, token_embeddings)
 
2. generic_jump TensorVariable() 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/models/transformer/transformer_encoder.py", line 127, in forward_embedding
    x = embed + self.embed_positions(src_tokens)
 
3. generic_jump 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/modules/sinusoidal_positional_embedding.py", line 71, in forward
    if self.weights is None or max_pos > self.weights.size(0):
 
4. generic_jump 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/models/transformer/transformer_encoder.py", line 210, in forward_scriptable
    if has_pads:
 
5. generic_jump 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/models/transformer/transformer_encoder.py", line 225, in forward_scriptable
    x, encoder_padding_mask=encoder_padding_mask if has_pads else None
 
6. generic_jump 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/modules/multihead_attention.py", line 506, in forward
    assert (
 
7. generic_jump 
   File "/mnt/xarfuse/uid-572259/de1cb585-seed-nspid4026531836_cgpid105054292-ns-4026531840/fairseq/modules/multihead_attention.py", line 506, in forward
    assert (
```
Two immediate issues:
- 11 graph breaks are quoted, but the list only has seven elements.
- Breaks 1 and 2 point to generic_jump but do not actually reference a control flow statement. They appear to be the callstack for break 3? If I fix break 3, breaks 1 and 2 go away.

Some more thoughts, but require more fleshing out to be actionable:
- I don't think a string is the right way to present this information; ideally it's queryable, (so I can do something like `breaks[2].source_location` in order to get more context then is provided here, including a full stack trace).
- The break reason is quite opaque if you don't understand the dynamo implementation.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @soumith @ngimel
Repository: pytorch
Issue: Capture scalar outputs / dynamically sized outputs by default, partition graphs for backends that can't handle it
Labels: feature,triaged,ezyang's list,oncall: pt2,module: dynamic shapes,module: inductor

Capturing scalar outputs improves perf e.g. by capturing the optimizer. Now that the fx subgraph capability matcher has landed, it should be easy for backends which do not support scalars to compile subgraphs even with this config turned on. Additionally, having a config option be off by default means its test coverage is extremely limited. 

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @soumith @ngimel
Repository: pytorch
Issue: Can we rewrite numpy operators to pytorch operators?
Labels: good first issue,triaged,module: numpy

Currently, numpy operations cause graph break, while often exist in training / inference code. But many operators have same semantic in torch operators. Like [YOLOP](https://github.com/hustvl/YOLOP/blob/084d0bb19fa2db4ba40a6e1d6b120f0153d2f335/lib/core/evaluate.py#L230), it would get more than 5x speedup during inference, if we manual replace all the possible `numpy` to `torch`, so they can run in gpu.
So can we dynamicly rewrite these numpy code to torch code using torchdynamo, rewrite pattern may like:
`x = tensor.numpy(); b = np.a(x)` to  `x = torch.a(tensor); b = x.numpy()`
or rewrite all support numpy operator `y = numpy.a(x)` to `x = torch.from_numpy(x); y = torch.a(x); y = y.numpy()` and then erase all the redundant `y = y.numpy(); y = torch.from_numpy(y)`.



cc @mruberry @rgommers