Repository: stable-diffusion-webui
Issue: [Bug]: In sketch and inpaint sketch mode, FPS drops
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

In sketch and inpaint sketch mode, the FPS drops and it is impossible to draw - the movements become abrupt! Only refreshing the page helps.

https://user-images.githubusercontent.com/31959570/218333718-08ab1a31-1c1a-4a53-a8ab-a2b3b3656d5d.mp4



### Steps to reproduce the problem

1. Go to - export the image to inpaint, then to Sketch / Inpaint Sketch mode
2. Press left mouse button to draw the mask.


### What should have happened?

Masking should work as smoothly as inpaint mode.

### Commit where the problem happens

ea9bd9fc7409109adcd61b897abc2c8881161256

### What platforms do you use to access the UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
Launching Web UI with arguments: --api --xformers --autolaunch
```


### List of extensions

stable-diffusion-webui-Prompt_Generator
ultimate-upscale-for-automatic1111

### Console logs

```Shell
venv "venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: ea9bd9fc7409109adcd61b897abc2c8881161256
Installing requirements for Web UI

Launching Web UI with arguments: --api --xformers --autolaunch
Checkpoint  not found; loading fallback deliberate_v11.safetensors [d8691b4d16]
Loading weights [d8691b4d16] from E:\SD_Portable\models\Stable-diffusion\deliberate_v11.safetensors
Creating model from config: E:\SD_Portable\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(1): CharTurner
Textual inversion embeddings skipped(7): AnimeScreenCap, AnimeScreenCap-90, Chadeisson, Kipaki-150, UlukInkSketch2, VikingPunk-63, VikingPunk
Model loaded in 2.9s (create model: 0.2s, apply weights to model: 0.4s, apply half(): 0.5s, move model to device: 0.3s, load textual inversion embeddings: 1.4s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:04<00:00,  5.18it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 22/22 [00:03<00:00,  6.28it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:02<00:00,  7.58it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 22/22 [00:02<00:00,  7.66it/s]
Closing server running on port: 7860
Restarting UI...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:02<00:00,  7.50it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 22/22 [00:02<00:00,  7.61it/s]
Closing server running on port: 7860
Restarting UI...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Closing server running on port: 7860
Restarting UI...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Closing server running on port: 7860
Restarting UI...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Loading weights [9e2c6ceff3] from E:\SD_Portable\models\Stable-diffusion\f222.ckpt
Applying xformers cross attention optimization.
Weights loaded in 32.3s (load weights from disk: 31.6s, apply weights to model: 0.2s, move model to device: 0.5s).
100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:02<00:00, 10.93it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 11.11it/s]
```


### Additional information

_No response_