> > there is something slightly taboo about coordination between workgroups in the same dispatch. Many GPU experts I’ve talked with express skepticism that this can work at all. Even so, interest in this type of pattern is picking up, in part because of advanced rendering engines like Nanite, which also uses atomics to coordinate work between workgroups in a live-running dispatch.

My opinion on the matter...

1. Because workgroups don't necessarily launch with each other. Workgroup #500 may not execute within Workgroup #1's lifetime, so already a lot of coordination between workgroups is limited. In contrast, if pthread_create started up a new thread, its in the running state (maybe not physically running, but it will get some time to run when the CPU is free).

2. Because kernel launch and kernel exit is the "natural" way to synchronize, and is actually decently efficient (especially device-side kernel launches). Even if you're forced for host-side kernel launches, CUDA-streams make async behavior easy and efficient (OpenCL task graphs are also quite efficient).

3. Because L1 caches in GPU-land are non-cohesive IIRC, so read/write barriers are very expensive (aka: you just invalidate L1 cache and wait for L2 cache). As such, #2 tends to be the more efficient approach in many cases.

-------

Simple, relaxed atomics work out very well though (relaxed atomics don't need any memory barriers and therefore execute in whatever order is most efficient for the GPU). I think the SIMD-queue thing also works out well in practice as a shared data-structure because it can rely upon relaxed atomics.

But the minute you start implementing significant blocking behavior (ex: compare-and-swap loops making lock/unlock spinlocks)... well... things continue to __work__ but its not nearly as efficient as you think. You end up thinking about doing #2 instead in many cases. The penalty on those memory-barriers even for acquire/release semantics is really really high, far higher than you'd expect.

(Ex: If you have a SIMD-queue. You could use atomics to implement a lock/unlock for the queue, and then your workgroups can then operate on the queue at their leisure. However: how many workgroups / blocks to spawn? And when they do coordinate, the stiff-penalties associated with memory-barriers really sucks. In contrast, if you just launch a kernel over the _entire_ queue, you get better control over the amount of resources used, and the individual kernels don't have to worry about coordination as much, especially if you have a separate output queue per workgroup that later merges in in a separate kernel)

But if you have to CAS-loop a non-blocking event across a bunch of stuff, modern GPUs _can_ do it. But I'm not convinced that programmers _should_ do it.

In short: GPUs don't seem to MESI with each other. Patterns in CPU-land around spinlocks / interthread communications are simply not as efficient in GPU-land. In contrast, GPU-land workgroup barriers and/or kernel launch/exists are EXTREMELY efficient (while the CPU-equivalent "pthread_create" and "pthread_join" is extremely inefficient)... but only within GPU-specific structures (barriers are efficient only within a workgroup. Kernel-exit is only effective at synchronizing with your own CUDA grid / OpenCL Task).