Repository: stable-diffusion-webui
Issue: [Bug]: API Error  (Only while using API)    CUDA error: unspecified launch failure after torch._C._cuda_emptyCache()
Labels: bug-report

### Checklist

- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [X] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

I am using automatic1111 api, it was working perfect for past 3 months. i haven't update anything but suddenly i got this error. If i restart the automatic1111 it works and give error after few minutes.

```
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side 
```
I am using Agent Schedular, i suspected it but this error occurred after disabling it and i also completely removed it. 
I have done fresh installation. It works in webui and when i use API i am getting the error. I can't find when this error occurs because it occurs randomly. I have attached the log

Please someone help me. i am restarting it every 15 to 20 min for past 3 days.





### Steps to reproduce the problem

-

### What should have happened?

-

### What browsers do you use to access the UI ?

Other

### Sysinfo

I am using Ubuntu 20 LTS, 

### Console logs

```Shell
Jan 05 14:10:23 automaticserver bash[40174]: *** API error: POST: /sdapi/v1/txt2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: unspecified launch failure\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
Jan 05 14:10:23 automaticserver bash[40174]:     Traceback (most recent call last):
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
Jan 05 14:10:23 automaticserver bash[40174]:         return self.receive_nowait()
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
Jan 05 14:10:23 automaticserver bash[40174]:         raise WouldBlock
Jan 05 14:10:23 automaticserver bash[40174]:     anyio.WouldBlock
Jan 05 14:10:23 automaticserver bash[40174]:     During handling of the above exception, another exception occurred:
Jan 05 14:10:23 automaticserver bash[40174]:     Traceback (most recent call last):
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 78, in call_next
Jan 05 14:10:23 automaticserver bash[40174]:         message = await recv_stream.receive()
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 118, in receive
Jan 05 14:10:23 automaticserver bash[40174]:         raise EndOfStream
Jan 05 14:10:23 automaticserver bash[40174]:     anyio.EndOfStream
Jan 05 14:10:23 automaticserver bash[40174]:     During handling of the above exception, another exception occurred:
Jan 05 14:10:23 automaticserver bash[40174]:     Traceback (most recent call last):
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/modules/api/api.py", line 186, in exception_handling
Jan 05 14:10:23 automaticserver bash[40174]:         return await call_next(request)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
Jan 05 14:10:23 automaticserver bash[40174]:         raise app_exc
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive_or_disconnect, send_no_error)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         response = await self.dispatch_func(request, call_next)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/modules/api/api.py", line 150, in log_and_time
Jan 05 14:10:23 automaticserver bash[40174]:         res: Response = await call_next(req)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
Jan 05 14:10:23 automaticserver bash[40174]:         raise app_exc
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive_or_disconnect, send_no_error)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive, send)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 26, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive, send)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         raise exc
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive, sender)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         raise e
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive, send)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
Jan 05 14:10:23 automaticserver bash[40174]:         await route.handle(scope, receive, send)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
Jan 05 14:10:23 automaticserver bash[40174]:         await self.app(scope, receive, send)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
Jan 05 14:10:23 automaticserver bash[40174]:         response = await func(request)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
Jan 05 14:10:23 automaticserver bash[40174]:         raw_response = await run_endpoint_function(
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function
Jan 05 14:10:23 automaticserver bash[40174]:         return await run_in_threadpool(dependant.call, **values)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
Jan 05 14:10:23 automaticserver bash[40174]:         return await anyio.to_thread.run_sync(func, *args)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
Jan 05 14:10:23 automaticserver bash[40174]:         return await get_asynclib().run_sync_in_worker_thread(
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
Jan 05 14:10:23 automaticserver bash[40174]:         return await future
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
Jan 05 14:10:23 automaticserver bash[40174]:         result = context.run(func, *args)
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/modules/api/api.py", line 381, in text2imgapi
Jan 05 14:10:23 automaticserver bash[40174]:         shared.state.end()
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/modules/shared_state.py", line 128, in end
Jan 05 14:10:23 automaticserver bash[40174]:         devices.torch_gc()
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/modules/devices.py", line 61, in torch_gc
Jan 05 14:10:23 automaticserver bash[40174]:         torch.cuda.empty_cache()
Jan 05 14:10:23 automaticserver bash[40174]:       File "/var/www/sd/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/cuda/memory.py", line 133, in empty_cache
Jan 05 14:10:23 automaticserver bash[40174]:         torch._C._cuda_emptyCache()
Jan 05 14:10:23 automaticserver bash[40174]:     RuntimeError: CUDA error: unspecified launch failure
Jan 05 14:10:23 automaticserver bash[40174]:     CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
Jan 05 14:10:23 automaticserver bash[40174]:     For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Jan 05 14:10:23 automaticserver bash[40174]:     Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Jan 05 14:10:23 automaticserver bash[40174]: ---
```


### Additional information

_No response_