Repository: stable-diffusion-webui
Issue: [Bug]: Prompt scheduling affects earlier steps than it should
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Prompt scheduling syntax like "[foo:X]" or "[foo:bar:X]" still alters steps prior to X.

### Steps to reproduce the problem

The following example is particularly easy to spot, but I've reproduced something like this across different checkpoints, samplers, parameters, and optimization types. All non-standard extensions were disabled. (I accidentally had CodeFormer face restoration on, but get the same results with it off.)

Using these settings with --xformers:
`Steps: 50, Sampler: DPM++ 2M SDE Exponential, CFG scale: 6.1, Seed: 111, Face restoration: CodeFormer, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, VAE hash: 63aeecb90f, VAE: sdxl_vae.safetensors, Version: v1.6.0`
the prompt "simple geometric pattern" produces this:
![00157-111](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/13244175/fbe6a256-c164-4060-aab2-8e6bd98c6517)

"simple [beautiful:49] geometric pattern" produces this:
![00158-111](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/13244175/bb1d2896-98b7-4ee8-b3a8-e59bf36d1ad0)

The easiest difference to spot is the missing vertical black line near the left edge. This change can be clearly seen in live previews by step 15 or earlier, even though nothing should change until step 49 of 50.

In fact, even "[simple:simple:49] geometric pattern" produces a different result, also visible by step 15:
![00159-111](https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/13244175/38333b67-ac64-416b-a3e1-6f4bddf6b73b)
Of course the syntax in that one is redundant, but I'd expect no change at all from the first, since the change actually changes nothing. Other variants such as "[simple:beautiful:49] geometric pattern" also cause changes much earlier than step 49. This also happens when using the decimal syntax like "[beautiful:0.9]".

Prompt scheduling could be incredibly useful to modify the prompt at later steps without disrupting the overall image composition in earlier steps. But if the earlier steps are still affected, this benefit is lessened. I've run into this several times already when trying to use it for that purpose.

### What should have happened?

Prompt changes scheduled for later steps should not affect earlier steps.

- "simple geometric pattern", "simple [beautiful:49] geometric pattern", and "[simple:beautiful:49] geometric pattern" should not diverge until step 49.
- also, "simple geometric pattern" and "[simple:simple:49] geometric pattern" should be identical (although nobody would need that syntax, it suggests something is not working correctly).

### Sysinfo

[sysinfo-2023-09-19-11-24.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/12659491/sysinfo-2023-09-19-11-24.txt)


### What browsers do you use to access the UI ?

Google Chrome

### Console logs

```Shell
venv "C:\Users\justi\auto1111\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Launching Web UI with arguments: --ckpt-dir C:\users\justi\newsdnext\automatic\models\Stable-diffusion --vae-dir C:\users\justi\newsdnext\automatic\models\vae --lora-dir C:\users\justi\newsdnext\automatic\models\lora --medvram --no-half-vae --xformers
Loading weights [0b76532e03] from C:\users\justi\newsdnext\automatic\models\Stable-diffusion\crystalClearXL_ccxl.safetensors
Running on local URL:  http://127.0.0.1:7860
Creating model from config: C:\Users\justi\auto1111\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml

To create a public link, set `share=True` in `launch()`.
Startup time: 13.6s (prepare environment: 2.8s, import torch: 3.5s, import gradio: 1.0s, setup paths: 0.9s, initialize shared: 0.2s, other imports: 0.7s, setup codeformer: 0.1s, load scripts: 1.1s, create ui: 0.5s, gradio launch: 2.6s).
Loading VAE weights specified in settings: C:\users\justi\newsdnext\automatic\models\vae\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 13.7s (load weights from disk: 1.2s, create model: 0.5s, apply weights to model: 7.2s, apply half(): 0.2s, load VAE: 0.4s, calculate empty prompt: 4.0s).
Reusing loaded model crystalClearXL_ccxl.safetensors [0b76532e03] to load sd_xl_base_1.0.safetensors [31e35c80fc]
Loading weights [31e35c80fc] from C:\users\justi\newsdnext\automatic\models\Stable-diffusion\sd_xl_base_1.0.safetensors
Loading VAE weights specified in settings: C:\users\justi\newsdnext\automatic\models\vae\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
Weights loaded in 12.2s (send model to cpu: 0.5s, load weights from disk: 1.1s, apply weights to model: 9.8s, load VAE: 0.6s, hijack: 0.2s).
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:47<00:00,  1.06it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:48<00:00,  1.02it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:45<00:00,  1.09it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:47<00:00,  1.06it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:46<00:00,  1.09it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:47<00:00,  1.04it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 50/50 [00:47<00:00,  1.13it/s]
```


### Additional information

When investigating I noticed some oddities that might help in tracking this down. I can't guarantee these patterns always hold, but they did over several tests.

These all produce identical results to each other, though not to the original prompt:
- [simple:beautiful:49] geometric pattern
- [simple:ugly:49] geometric pattern
- [simple:wizard:49]... or whatever -- for all "[simple:-single token-:49] geometric pattern", it seems they come out the same.

But with more than one token in the replacement part, that no longer holds. These produce clearly different results well before step 49:
- [simple:very beautiful:49]...
- [simple:super beautiful:49]...
- [simple:very ugly:49]...

These are also different:
- simple [beautiful:49]...
- simple [ugly:49]...
- simple [colorful:49]... etc.