Repository: material-ui
Issue: [ModalManager] Dialog in Shadow DOM creates scroll jump
Labels: bug 🐛,component: dialog

### Duplicates

- [X] I have searched the existing issues

### Latest version

- [X] I have tested the latest version

### Steps to reproduce 🕹

Link to live example:
https://stackblitz.com/edit/react-aj5fcw?file=index.tsx

Steps:

1. add a dialog using [Dialog demo](https://mui.com/material-ui/react-dialog/)
2. add the same dialog in a Shadow DOM following [steps from the docs](https://mui.com/material-ui/guides/shadow-dom/)
3. add lots of `div`s to create overflow
4. open the two dialogs and see the difference in handling the scrollbar


### Current behavior 😯

when the dialog is opened in Shadow DOM the scrollbar is removed (`overflow: hidden`) however the space is not filled with a padding-right, as it is done for a dialog outside of shadow DOM, see the column of text on the right on the example:

![image](https://github.com/mui/material-ui/assets/3442879/5cd080b3-ddff-4b1e-b997-007b3a01d6bd)

### Expected behavior 🤔

the scrollbar's width must be replaced with a padding-right

### Context 🔦

We use Shadow DOM to isolate styles for components we inject into other pages. We add a component on a long page that opens a dialog, but because its added in a Shadow DOM it doesn't replace the scrollbar width with padding so there's a jump on the page.

here's where I think the issue is happening in MUI:
In `ModalManager` `isOverflowin` function returns `false` because `container` is not the same as `body` and both its `clientHeight` and `scrollHeight` are 0

https://github.com/mui/material-ui/blob/64c48b50612fec38d29606868de45ce4ebb09019/packages/mui-base/src/unstable_useModal/ModalManager.ts#L12-L20

so `handleContainer` will not add `paddingRight`
https://github.com/mui/material-ui/blob/64c48b50612fec38d29606868de45ce4ebb09019/packages/mui-base/src/unstable_useModal/ModalManager.ts#L101-L123

### Your environment 🌎

<details>
  <summary><code>npx @mui/envinfo</code></summary>

```
  System:
    OS: Windows 10 10.0.22621
  Binaries:
    Node: 18.18.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 9.8.1 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 118.0.5993.117
    Edge: Chromium (118.0.2088.69)
  npmPackages:
    @emotion/react:  11.11.1
    @emotion/styled:  11.11.0
    @mui/base:  5.0.0-beta.21
    @mui/codemod:  5.14.15
    @mui/core-downloads-tracker:  5.14.15
    @mui/docs:  5.14.15
    @mui/envinfo:  2.0.12
    @mui/icons-material:  5.14.15
    @mui/internal-waterfall:  1.0.0
    @mui/joy:  5.0.0-beta.12
    @mui/lab:  5.0.0-alpha.150
    @mui/markdown:  5.0.0
    @mui/material:  5.14.15
    @mui/material-next:  6.0.0-alpha.107
    @mui/private-theming:  5.14.15
    @mui/styled-engine:  5.14.15
    @mui/styled-engine-sc:  6.0.0-alpha.3
    @mui/styles:  5.14.15
    @mui/system:  5.14.15
    @mui/types:  7.2.7
    @mui/utils:  5.14.15
    @mui/x-charts:  6.0.0-alpha.15
    @mui/x-data-grid:  6.16.2
    @mui/x-data-grid-generator:  6.16.2
    @mui/x-data-grid-premium:  6.16.2
    @mui/x-data-grid-pro:  6.16.2
    @mui/x-date-pickers:  6.16.2
    @mui/x-date-pickers-pro:  6.16.2
    @mui/x-license-pro:  6.10.2
    @mui/x-tree-view:  6.0.0-alpha.1
    @mui/zero-jest:  0.0.1-alpha.0
    @mui/zero-next-plugin:  0.0.1-alpha.0
    @mui/zero-runtime:  0.0.1-alpha.0
    @mui/zero-tag-processor:  0.0.1-alpha.0
    @mui/zero-vite-plugin:  0.0.1-alpha.0
    @types/react: ^18.2.32 => 18.2.33
    react:  18.2.0
    react-dom:  18.2.0
    styled-components:  6.0.8
    typescript: ^5.1.6 => 5.1.6
```
this was tested in Chrome
</details>