Repository: stable-diffusion-webui
Issue: [Bug]: Linux/Pop_os (22.04 LTS NVidia) - Hires.fix problem
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

I switched a few days ago from Windows 10 to Linux (PopOs 22.04 LTS NVidia). I installed Webui by running webui.sh, and generally everything works pretty well except for Hires.fix.

It turns out that Hires.fix only works correctly if "Upscale by" <= 1.5. Above this value, the generated image is dominated by artifacts that I have not experienced on Windows 10. Below are some examples.

Base image:
![4285218448-7-15 vinteprotogenmix_V20 - before hires_fix](https://user-images.githubusercontent.com/122713558/230699938-86577e2a-5e76-43e9-bec4-89b56d99ba75.png)

Upscaled by 1.5
![4285218448-7-15 vinteprotogenmix_V20 - upscaled by 1 5](https://user-images.githubusercontent.com/122713558/230699956-7798fea4-239c-49f6-81c8-6ff98e9d0a1a.png)

Starting point of upscaling by 2.0
![image](https://user-images.githubusercontent.com/122713558/230700135-5ed4df60-ddde-4ed0-88c1-2daa0373beb2.png)
![image](https://user-images.githubusercontent.com/122713558/230700649-9f676c80-cad0-402a-9643-c4863edf09c8.png)


Upscaled by 2.0
![4285218448-7-15 vinteprotogenmix_V20 - upscaled by 2 0](https://user-images.githubusercontent.com/122713558/230699964-20e61fbc-ddfb-44bc-92b7-e917976c2198.png)

And if go any higher than 2.0 the problem is even worse.

Upscaled by 1.5
![4285218462-4285218444-7-15 vinteprotogenmix_V20 - upscaled by 1 5](https://user-images.githubusercontent.com/122713558/230700177-05f152bb-08ba-4bb4-bdec-109b14efd68b.png)

Upscaled by 2.5
![4285218461-4285218444-7-15 vinteprotogenmix_20 - upscaled by 2 5](https://user-images.githubusercontent.com/122713558/230700193-6ffd1b61-3a48-4e0b-9538-af38f87e86cd.png)



### Steps to reproduce the problem

1. Go to txt2img tab
2. Check Hires.fix
3. Set Upscaler setting to "Latent" (model, sampler, steps, base resolution does not matter)
4. Try to generate images upscaled by factors: 1.5, 2.0, 2.5, 3.0


### What should have happened?

The generated noise should not distort the details which base image has clearly visible. 

### Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/22bcc7be428c94e9408f589966c2040187245d81

### What platforms do you use to access the UI ?

Linux

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

### Command Line Arguments

```Shell
Command line parameters does really matter. Already tried no arguments at all, to many possible combinations of: --medvram, --opt-split-attention, --no-half, --xformers

Anyway, all these images were generated using: "--medvram --opt-split-attention"
```


### List of extensions

Yes and no. Already tried disabling all of the custom extensions no with result.

### Console logs

```Shell
asm@pop:/mnt/ssd/stable-diffusion-webui$ ./webui.sh

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on asm user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
Commit hash: 22bcc7be428c94e9408f589966c2040187245d81
Installing requirements for Web UI

Installing sd-dynamic-prompts requirements.txt

Launching Web UI with arguments: --medvram --opt-split-attention
No module 'xformers'. Proceeding without it.
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
Civitai Helper: Get Custom Model Folder
Civitai Helper: Load setting from: /mnt/ssd/stable-diffusion-webui/extensions/Stable-Diffusion-Webui-Civitai-Helper/setting.json
Civitai Helper: No setting file, use default
Error loading script: patch_fixer.py
Traceback (most recent call last):
  File "/mnt/ssd/stable-diffusion-webui/modules/scripts.py", line 256, in load_scripts
    script_module = script_loading.load_module(scriptfile.path)
  File "/mnt/ssd/stable-diffusion-webui/modules/script_loading.py", line 11, in load_module
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/mnt/ssd/stable-diffusion-webui/extensions/sd_auto_fix/scripts/patch_fixer.py", line 16, in <module>
    from modules.sd_hijack_inpainting import do_inpainting_hijack, should_hijack_inpainting
ImportError: cannot import name 'should_hijack_inpainting' from 'modules.sd_hijack_inpainting' (/mnt/ssd/stable-diffusion-webui/modules/sd_hijack_inpainting.py)

Loading weights [f2f29d2617] from /mnt/ssd/stable-diffusion-webui/models/Stable-diffusion/vinteprotogenmix_V20.safetensors
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/safetensors/torch.py:99: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  with safe_open(filename, framework="pt", device=device) as f:
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/storage.py:899: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  storage = cls(wrap_storage=untyped_storage)
Creating model from config: /mnt/ssd/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying cross attention optimization (Doggettx).
/mnt/ssd/stable-diffusion-webui/modules/safe.py:30: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return TypedStorage()
Textual inversion embeddings loaded(42): easynegative, emb-rrf2, ng_deepnegative_v1_75t, ng_deepnegative_v1_4t, nervous512, bad-picture-chill-1V-5601, ng_deepnegative_v1_2t, style-pnmagic, atompunkstylesd15, Style-Moana-neg, rfktr_bontrex-150, defiance512, bad_prompt, bad-picture-chill-32V-449, smile512, Style-Kitchen-neg, sad512, ng_deepnegative_v1_16t, shock512, style-rustmagic-neg, bad-picture-chill-75v-2501, araknope, bad_prompt_version2, laugh512, Style-Necromancy, cloudport, lowangleview-3000, happy512, eonn, emb-nuke2, grin512, Style-Neeko, bad-artist, ng_deepnegative_v1_32t, Style-Goblinmode, style-paintmagic, style-sylvamagic, ng_deepnegative_v1_64t, angry512, bad-artist-anime, style-nebmagic-neg, style-psycho
Textual inversion embeddings skipped(6): nfixer, nrealfixer, anthro, nartfixer, UrbanJungle, DrD_PNTE768
Model loaded in 2.1s (load weights from disk: 0.2s, create model: 0.6s, apply weights to model: 0.8s, apply half(): 0.4s).
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Row, please remove them: {'scale': 0.5}
  warnings.warn(
/mnt/ssd/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Button, please remove them: {'full_width': True}
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 7.1s (import torch: 0.9s, import gradio: 0.8s, import ldm: 0.3s, other imports: 1.1s, load scripts: 0.6s, load SD checkpoint: 2.2s, create ui: 0.7s, gradio launch: 0.3s).
Total progress: 0it [00:00, ?it/s]
Data shape for DDIM sampling is (1, 4, 64, 96), eta 1
Running DDIM Sampling with 16 timesteps
DDIM Sampler: 100%|██████████████████████████████████████████████████████| 16/16 [00:09<00:00,  1.70it/s]
Running DDIM Sampling with 14 timesteps█████████▋                        | 16/30 [00:14<00:05,  2.34it/s]
Decoding image: 100%|████████████████████████████████████████████████████| 14/14 [00:28<00:00,  2.03s/it]
Loading CLiP model ViT-L/14 █████████████████████████████████████████████| 30/30 [00:42<00:00,  1.83s/it]
/mnt/ssd/stable-diffusion-webui/modules/safe.py:30: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return TypedStorage()
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:53<00:00,  1.78s/it]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:53<00:00,  1.83s/itData shape for DDIM sampling is (1, 4, 64, 96), eta 1                              | 0/15 [00:00<?, ?it/s]
Data shape for DDIM sampling is (1, 4, 64, 96), eta 1
Running DDIM Sampling with 16 timesteps
DDIM Sampler: 100%|██████████████████████████████████████████████████████| 16/16 [00:06<00:00,  2.46it/s]
Running DDIM Sampling with 14 timesteps                                                                  
Decoding image: 100%|████████████████████████████████████████████████████| 14/14 [00:59<00:00,  4.24s/it]
Total progress: 46it [01:34,  2.06s/it]
Total progress: 46it [01:34,  3.93s/it]                                                                 Data shape for DDIM sampling is (1, 4, 64, 96), eta 1                              | 0/15 [00:00<?, ?it/s]
Running DDIM Sampling with 16 timesteps
DDIM Sampler: 100%|██████████████████████████████████████████████████████| 16/16 [00:07<00:00,  2.15it/s]
Running DDIM Sampling with 14 timesteps█████████▋                        | 16/30 [00:07<00:05,  2.60it/s]
Decoding image: 100%|████████████████████████████████████████████████████| 14/14 [00:12<00:00,  1.13it/s]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:22<00:00,  1.33it/s]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:22<00:00,  1.33it/sData shape for DDIM sampling is (1, 4, 64, 109), eta 1                             | 0/15 [00:00<?, ?it/s]
Running DDIM Sampling with 16 timesteps
DDIM Sampler: 100%|██████████████████████████████████████████████████████| 16/16 [00:07<00:00,  2.10it/s]
Running DDIM Sampling with 14 timesteps█████████▋                        | 16/30 [00:08<00:06,  2.13it/s]
Decoding image: 100%|████████████████████████████████████████████████████| 14/14 [00:34<00:00,  2.49s/it]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:46<00:00,  1.54s/it]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:46<00:00,  2.28s/itData shape for DDIM sampling is (1, 4, 64, 109), eta 1                             | 0/15 [00:00<?, ?it/s]
Running DDIM Sampling with 16 timesteps
DDIM Sampler: 100%|██████████████████████████████████████████████████████| 16/16 [00:07<00:00,  2.15it/s]
Running DDIM Sampling with 14 timesteps█████████▋                        | 16/30 [00:07<00:06,  2.10it/s]
Decoding image: 100%|████████████████████████████████████████████████████| 14/14 [00:34<00:00,  2.44s/it]
Total progress: 100%|████████████████████████████████████████████████████| 30/30 [00:44<00:00,  1.49s/it]
```


### Additional information

Already tried:
- lots of different image resolutions
- different models
- different samplers
- different "Sampler parameters" settings
- enabling / disabling: "Enable quantization in K samplers for sharper and cleaner results. This may change existing seeds. Requires restart to apply."
- running clean installation of webui
- and who knows what else :)