Repository: pytorch
Issue: torch.compile support for SeamlessExpressivity/SeamlessM4T in fairseq2
Labels: triaged,actionable,oncall: pt2,module: dynamic shapes,module: inductor,module: dynamo,module: guards

### 🐛 Describe the bug

Internal notes:
* https://fb.workplace.com/notes/389433230075029/?comment_id=362284632838743&reply_comment_id=332571542802227&notif_id=1699490653585208&notif_t=comment_mention&ref=notif
* https://docs.google.com/document/d/1PVy4KibfljirQDoijOwyHCV97B67r_iElWqFh7h1Acc/edit#heading=h.r8e5eoeu8v9e

Setup notes:

* It does NOT work in conda, you must do a venv
* Despite the instructions, you can build it against CUDA 12
* fmt is incompatible with PyTorch main, see https://github.com/facebookresearch/fairseq2/issues/170 for WAR

With the stock repo `python scripts/m4t/benchmark/benchmark.py  --apply_compile --dynamic` errors. https://github.com/pytorch/pytorch/pull/114362 solves the immediate problem but we end up with no generations after we slowly hit cache limit everywhere:

```
[2023-11-22 08:45:41,648] [23/47] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 447
[[]]
Traceback (most recent call last):
  File "/data/users/ezyang/seamless_communication/scripts/m4t/benchmark/benchmark.py", line 457, in <module>
    main(args)
  File "/data/users/ezyang/seamless_communication/scripts/m4t/benchmark/benchmark.py", line 362, in main
    res = run_inference(example)
  File "/data/users/ezyang/pytorch-venv/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/users/ezyang/seamless_communication/scripts/m4t/benchmark/benchmark.py", line 281, in apply_model
    return translator.predict(
  File "/data/users/ezyang/pytorch-venv/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/users/ezyang/seamless_communication/seamless_communication/models/inference/translator.py", line 256, in predict
    text_output, unit_output = self.get_prediction(
  File "/data/users/ezyang/seamless_communication/seamless_communication/models/inference/translator.py", line 162, in get_prediction
    return generator(
  File "/data/users/ezyang/pytorch-venv/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/users/ezyang/seamless_communication/seamless_communication/models/unity/generator.py", line 178, in __call__
    text_output = self.s2t_generator.generate_ex(
  File "/data/users/ezyang/fairseq2/src/fairseq2/generation/text.py", line 152, in generate_ex
    return self._do_generate(source_seqs, source_padding_mask)
  File "/data/users/ezyang/pytorch-venv/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data/users/ezyang/fairseq2/src/fairseq2/generation/text.py", line 83, in _do_generate
    sentences = [self.token_decoder(b[0].seq) for b in gen_output.results]
  File "/data/users/ezyang/fairseq2/src/fairseq2/generation/text.py", line 83, in <listcomp>
    sentences = [self.token_decoder(b[0].seq) for b in gen_output.results]
IndexError: list index out of range
```

So there's an accuracy problem, and there's also a recompilation problem.

### Versions

main

cc @chauhang @penguinwu @bobrenjc93 @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @aakhundov @anijain2305 @msaroufim @bdhirsh @zou3519 @wconstab