Repository: stable-diffusion-webui
Issue: [Bug]:I found that when using the `txt2img` API, I can successfully pass the required parameters for the `controlnet` extension plugin via `always_scripts` for the first time. However, when I no longer use `controlnet` for the second time and do not pass the `always_scripts` parameter, or if I change the `enabled` value in the previous `controlnet` parameters to `false`, the program throws an error. 
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

that when using the `txt2img` API, I can successfully pass the required parameters for the `controlnet` extension plugin via `always_scripts` for the first time. However, when I no longer use `controlnet` for the second time and do not pass the `always_scripts` parameter, or if I change the `enabled` value in the previous `controlnet` parameters to `false`, the program throws an error. 

### Steps to reproduce the problem

1. Use the `txt2img` API with `controlnet` parameters via `always_scripts`. (Success)
2. Use the `txt2img` API again without `controlnet` or with `controlnet` parameters where `enabled` is set to `false`. (Error occurs)


### What should have happened?

The program should handle the absence of `controlnet` or the change in the `enabled` value gracefully without errors.

### Sysinfo

[api parameters.txt](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/12603402/api.parameters.txt)


### What browsers do you use to access the UI ?

Google Chrome, Other

### Console logs

```Shell
** API error: POST: http://127.0.0.1:7860/sdapi/v1/txt2img {'error': 'NansException', 'detail': '', 'body': '', 'errors': 'A tensor with all NaNs was produced in Unet. This could be either because there\'s not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.'}
    Traceback (most recent call last):
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\streams\memory.py", line 98, in receive
        return self.receive_nowait()
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\streams\memory.py", line 93, in receive_nowait
        raise WouldBlock
    anyio.WouldBlock
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
        message = await recv_stream.receive()
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\streams\memory.py", line 118, in receive
        raise EndOfStream
    anyio.EndOfStream
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\api\api.py", line 153, in exception_handling
        return await call_next(request)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 108, in __call__
        response = await self.dispatch_func(request, call_next)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\api\api.py", line 117, in log_and_time
        res: Response = await call_next(req)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\cors.py", line 92, in __call__
        await self.simple_response(scope, receive, send, request_headers=headers)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\cors.py", line 147, in simple_response
        await self.app(scope, receive, send)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
        await responder(scope, receive, send)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
        await self.app(scope, receive, self.send_with_gzip)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
        raise exc
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
        await self.app(scope, receive, sender)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
        raise e
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
        await self.app(scope, receive, send)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\routing.py", line 718, in __call__
        await route.handle(scope, receive, send)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\routing.py", line 276, in handle
        await self.app(scope, receive, send)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\routing.py", line 66, in app
        response = await func(request)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\fastapi\routing.py", line 237, in app
        raw_response = await run_endpoint_function(
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
        return await run_in_threadpool(dependant.call, **values)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
        return await anyio.to_thread.run_sync(func, *args)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
        return await future
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
        result = context.run(func, *args)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\api\api.py", line 346, in text2imgapi
        processed = process_images(p)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\processing.py", line 642, in process_images
        res = process_images_inner(p)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\processing.py", line 765, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\processing.py", line 1019, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\sd_samplers_kdiffusion.py", line 439, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\sd_samplers_kdiffusion.py", line 278, in launch_sampling
        return func()
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\sd_samplers_kdiffusion.py", line 439, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\_alpha_v1\resources\app\lib\miniconda3\envs\dxc-sd140\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\sd_samplers_kdiffusion.py", line 190, in forward
        devices.test_for_nans(x_out, "unet")
      File "C:\_alpha_v1\resources\app\lib\stable_diffusion_webui\modules\devices.py", line 157, in test_for_nans
        raise NansException(message)
    modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.
```


### Additional information

Observe that the issue is resolved when switching the model or restarting the WebUI.Please tell me how to fix this bug. I can't wait for your team to fix it; the waiting time is too long.