> Stunt 9: Boast massive speedups with accelerators!

> Always compare the fully parallel accelerator code with a sequential (single-core) CPU version.

> Be sure to rush ahead to the next slide before anyone dares to ask why you don’t use a single GPGPU “core” to make the comparison even more “well-defined.”

> Accelerate only the kernels that are easily portable and report only their performance. Amdahl’s Law and communication overhead will be nothing but smoke and mirrors!

I disagree with this one. The baseline for a parallel program is always the best single-threaded program you can write (which usually means with OpenMP/threading removed to reduce overhead).

The goal of writing parallel programs is to increase the degree of parallelism that a program exposes, even if the total running time of the program increases (i.e. time spent over all cores, not time-per-core). You know going into it that adding synchronization is going to slow you down - even in standard parallel-CPU programs (OpenMP).

Effective GPU parallelization works the same way. For example, you might use counter-based RNGs that are really too slow to be used on CPUs (at least, in the days before AES-NI), but on GPUs math is often "free" because the program is bandwidth limited. Or you might choose to re-compute data in memory rather if possible rather than storing it in memory and costing RAM and bandwidth. You are "wasting cycles" deliberately, to expose a greater degree of parallelism/reduce contention to increase overall program performance.

So running the program on a single GPU core is just as stupid as benchmarking an OpenMP program on a single core - again, we literally know going into it that the single-threaded programs are usually most efficient in terms of total cycles expended, that's not really what we're trying to measure here.

In the text, the author then pivots to "well GPUs have more cores/memory bandwidth!", which is also rather myopic and dismissive of the actual challenges of GPGPU programming. Exploiting that bandwidth is not easy on a processor with massive latency and <100 bytes of SRAM per core to spread between cache, shared memory, and constant pointers across many threads, not to mention the problems of thread divergence, etc.

I do agree that communication overhead needs to be taken into account and total program time is what really counts. This is (IMO) a strong argument for either porting your entire program to run on the GPU, or an absolutely minimal set, so that you minimize communication overhead. The distinction I'm drawing here is the latter is something like GPU-accelerated database searching, where you are not going to store actual rows on the GPU, but just DB indexes that you search and copy target IDs back to the host system.

Either way you approach it, your data set needs to live on the GPU as much as possible, which means minimizing the number of times and amount that it gets copied to-and-fro and incurs communication bottlenecks/latency. In turn - this means absolutely brutal minimization of the program state to reduce memory consumption, because fitting a larger problem into the GPU will usually increase total speedups (for a well-written program).