I'm a graphics programmer.

The only reason to prefer the GPU is because it confers an advantage over the traditional CPU+RAM combination. There's nothing inherently special about a modern GPU. The GPU is a sequence of actions and abilities encoded into hardware, e.g. the ability to automatically perform various kinds of texture filtering transparently to the game developer.

Since the GPU is hardware, and since hardware is less flexible than software, a graphics programmer would always prefer a software-based pipeline to a hardware-based one. The reason hardware pipelines are preferred is strictly because their advantages outweigh their disadvantages. Typically, using a GPU enables graphics programmers to create renderers which are 10-100x more efficient than software-based renderers, so the added flexibility of a software rasterizer tends to be forgotten in the face of massive efficiency enabled by the GPU.

The GPU primarily became popular because (a) it offloaded part of the computation from the CPU to dedicated hardware, freeing up the CPU for other tasks like game logic, AI, and more recently physics computations (though nVidia is trying hard to convince developers that hardware-accelerated physics is a viable concept), (b) GPUs increased the amount of available memory, and (c) GPUs dramatically increased the throughput (memory operations per second) of graphics memory.

Memory latency plays a key role in many modern graphics algorithms, such as voxel-based renderers. It's often the case that an algorithm needs to repeatedly cast rays against a voxel structure until hitting some kind of geometry. Therefore, within an individual pixel of the screen to be rendered, this type of algorithm can be hard to parallelize because typically the raycasting can't be broken up into parallelizable steps. It typically looks like, "While not hit: traceAlongRay();" for each pixel, each frame. I.e. this algorithm can only trace one section of the ray at a time before tracing the next.

That raycasting algorithm is memory-latency-bound because it completes only when it finishes looking up enough memory locations that it detects the ray has intersected some 3D geometry. In other words, by reducing memory latency by 2x, and assuming memory bandwidth is sufficient, then this algorithm will complete twice as fast. This means instead of 24 frames per second, you might get 48 frames per second.

So, all that said, if it becomes common to have 1TB of regular RAM with the latency and bandwidth traditionally offered by GPUs, along with a surplus of available CPU cores to offload computations to, then software renderers will once again become preferable to GPU renderers. A software pipeline will always be more flexible and easier to maintain than a hardware pipeline, simply because the featureset of the software pipeline isn't restricted to the capabilities of the videocard hardware it's executing on. It's also easier to debug and maintain.

All of that means that it'll be easier for art pipelines to produce more complex, more immersive visual experiences than at present. But replacing the traditional GPU-based renderer with a CPU-based software renderer will only be practical if there's a major advance of RAM technology in the future, because current RAM tech can't match the memory bandwidth / latency of a modern GPU. Hence, any major developments in the area of volatile memory tech will be extremely interesting to graphics programmers.