Repository: pytorch
Issue: PyTorch Profiler fails recording functions
Labels: oncall: profiler

### 🐛 Describe the bug

Using PyTorchProfiler with PyTorch 2.0 for training GPT2LMHeadModel from hugging face, mostly fails. Kineto init does not seem to throw error but nvtx captures fails with the following error and the runs ends with a report `RuntimeError: Can't disable Kineto profiler when it's not running` when `_disable_profiler()` is called on teardown.  

The profiler works fine for other networks I have tried. 

```
[W record_function.cpp:497] Exception in RecordFunction callback: state_ptr INTERNAL ASSERT FAILED at "../torch/csrc/profiler/standalone/nvtx_observer.cpp":116, please report a bug to PyTorch. Expected profiler state set
Exception raised from updateOutputTensorTracker at ../torch/csrc/profiler/standalone/nvtx_observer.cpp:116 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb9cbcba4d7 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb9cbc8436b in /wks/.venv/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, char const*) + 0x43 (0x7fb9cbcb8413 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x524314e (0x7fb9f708e14e in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #4: at::RecordFunction::end() + 0x51 (0x7fb9f38d75f1 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x4a12fa6 (0x7fb9f685dfa6 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const + 0x222 (0x7fb9f36d8f62 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, pybind11::args, pybind11::kwargs const&, c10::optional<c10::DispatchKey>) + 0x483 (0x7fba0b5be553 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, c10::Symbol, pybind11::args, pybind11::kwargs const&, bool, c10::optional<c10::DispatchKey>) + 0x5a2 (0x7fba0b5bee42 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x784a07 (0x7fba0b4b3a07 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x3b7050 (0x7fba0b0e6050 in /wks/.venv/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #11: PyCFunction_Call + 0x59 (0x5f5b39 in /wks/.venv/bin/python3)
frame #12: _PyEval_EvalFrameDefault + 0x6465 (0x5717f5 in /wks/.venv/bin/python3)
frame #13: _PyEval_EvalCodeWithName + 0x26a (0x569d8a in /wks/.venv/bin/python3)
frame #14: _PyFunction_Vectorcall + 0x393 (0x5f60c3 in /wks/.venv/bin/python3)
frame #15: /wks/.venv/bin/python3() [0x59cb2e]
frame #16: _PyObject_MakeTpCall + 0x296 (0x5f6706 in /wks/.venv/bin/python3)
frame #17: _PyEval_EvalFrameDefault + 0x5db3 (0x571143 in /wks/.venv/bin/python3)
frame #18: _PyFunction_Vectorcall + 0x1b6 (0x5f5ee6 in /wks/.venv/bin/python3)
frame #19: _PyEval_EvalFrameDefault + 0x851 (0x56bbe1 in /wks/.venv/bin/python3)
frame #20: _PyFunction_Vectorcall + 0x1b6 (0x5f5ee6 in /wks/.venv/bin/python3)
frame #21: _PyEval_EvalFrameDefault + 0x851 (0x56bbe1 in /wks/.venv/bin/python3)
frame #22: _PyFunction_Vectorcall + 0x1b6 (0x5f5ee6 in /wks/.venv/bin/python3)
frame #23: /wks/.venv/bin/python3() [0x50b849]
frame #24: PyObject_Call + 0x62 (0x5f52b2 in /wks/.venv/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x1f2c (0x56d2bc in /wks/.venv/bin/python3)
frame #26: _PyEval_EvalCodeWithName + 0x26a (0x569d8a in /wks/.venv/bin/python3)
```

Still working through why it only happens for GPT2LMHeadModel while others work fine.  Confirming that creasing to recording nvtx emits leads to successful captures for other metrics. 

### Versions

PyTorch 2.0 with cuda 11.7 and cupti 11.7

cc @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov @davidberard98