Repository: stable-diffusion-webui
Issue: [Bug]: API  Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions
Labels: bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

*** API error: POST: http://127.0.0.1:8085/sdapi/v1/img2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
    Traceback (most recent call last):
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
        return self.receive_nowait()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
        raise WouldBlock
    anyio.WouldBlock

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 78, in call_next
        message = await recv_stream.receive()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 118, in receive
        raise EndOfStream
    anyio.EndOfStream

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 187, in exception_handling
        return await call_next(request)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
        raise app_exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
        response = await self.dispatch_func(request, call_next)
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 151, in log_and_time
        res: Response = await call_next(req)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
        raise app_exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 24, in __call__
        await responder(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 44, in __call__
        await self.app(scope, receive, self.send_with_gzip)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
        raise exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
        await self.app(scope, receive, sender)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
        raise e
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
        await route.handle(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
        response = await func(request)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
        raw_response = await run_endpoint_function(
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function
        return await run_in_threadpool(dependant.call, **values)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
        return await anyio.to_thread.run_sync(func, *args)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
        return await future
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
        result = context.run(func, *args)
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 443, in img2imgapi
        shared.state.end()
      File "/home/sd/stable-diffusion-webui/modules/shared_state.py", line 128, in end
        devices.torch_gc()
      File "/home/sd/stable-diffusion-webui/modules/devices.py", line 51, in torch_gc
        torch.cuda.empty_cache()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/torch/cuda/memory.py", line 133, in empty_cache
        torch._C._cuda_emptyCache()
    RuntimeError: CUDA error: an illegal memory access was encountered
    CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
    For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
    Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

### Steps to reproduce the problem

Download latest version

### What should have happened?

Start fine.

### Sysinfo

cuda:11.8.0-cudnn8-devel-ubuntu22.04

 torch==2.0.1
 torchvision==0.15.2
 torchaudio==2.0.2
xformers==0.0.21

### What browsers do you use to access the UI ?

_No response_

### Console logs

```Shell
*** API error: POST: http://127.0.0.1:8085/sdapi/v1/img2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
    Traceback (most recent call last):
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
        return self.receive_nowait()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
        raise WouldBlock
    anyio.WouldBlock

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 78, in call_next
        message = await recv_stream.receive()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/streams/memory.py", line 118, in receive
        raise EndOfStream
    anyio.EndOfStream

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 187, in exception_handling
        return await call_next(request)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
        raise app_exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 108, in __call__
        response = await self.dispatch_func(request, call_next)
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 151, in log_and_time
        res: Response = await call_next(req)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 84, in call_next
        raise app_exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 24, in __call__
        await responder(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/gzip.py", line 44, in __call__
        await self.app(scope, receive, self.send_with_gzip)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
        raise exc
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
        await self.app(scope, receive, sender)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
        raise e
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
        await route.handle(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
        await self.app(scope, receive, send)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
        response = await func(request)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
        raw_response = await run_endpoint_function(
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function
        return await run_in_threadpool(dependant.call, **values)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
        return await anyio.to_thread.run_sync(func, *args)
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
        return await future
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
        result = context.run(func, *args)
      File "/home/sd/stable-diffusion-webui/modules/api/api.py", line 443, in img2imgapi
        shared.state.end()
      File "/home/sd/stable-diffusion-webui/modules/shared_state.py", line 128, in end
        devices.torch_gc()
      File "/home/sd/stable-diffusion-webui/modules/devices.py", line 51, in torch_gc
        torch.cuda.empty_cache()
      File "/home/sd/anaconda3/envs/webui/lib/python3.10/site-packages/torch/cuda/memory.py", line 133, in empty_cache
        torch._C._cuda_emptyCache()
    RuntimeError: CUDA error: an illegal memory access was encountered
    CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
    For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
    Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```


### Additional information

_No response_