> ...now that the low hanging fruit has long gone.

I'd postulate that the low hanging fruit came in two forms.

The first being advances in general purpose CPUs by increasing the number of transistor. So, we're about to hit the hard limit in that regard.

The second, being the design of software under the assumption that CPU speed will continue to increase linearly.

-----

Hardware:

Currently, the majority of systems architectures are designed to be general purpose. The exceptions being GPUs for parallel processing and ASIC for specialized tasks like encoding/decoding media formats.

The next generation under currently under development is SoC (Systems on a Chip). Ie SoC is the complete hardware infrastructure contained on a single chip. They run cooler, are more power efficient, and are faster in terms of passing data via the internal cache layers.

What may come after is a combination of SoC and dynamically configurable hardware architecture similar to how FPGAs work today. The combination of the cloud gaining mass appeal as well as IoT taking root (despite all the hype), a market will grow for chips with a combination of very predictable performance characteristics and specialized capabilities.

With dynamically configurable hardware you could bootstrap a system and tell it to specialize as a graph database node, a media encoding/decoding cluster node, a Smart TV platform, a smart car CPU, etc. There's already talk of neuromorphic chips that can dynamically change/adapt to a specific environment.

The other direction I see room for growth is communication speed. I've heard some about using optics as a communication bus on multi-layered chips. Material manufacturing seems to be the issue. If somebody were to come up with a solution then enabling on board optical communication busses could have a huge impact on IO performance/latency.

-----

Software:

We're at the tail end of 3 decades of unrestricted software glut. Every system commonly available today runs on a general purpose monolithic architecture. With a ton of legacy glut and a huge surface attack surface for security vulnerabilities.

General purpose OSes won't go away for personal workstations but there's a lot of room for improvement when it comes to special purpose platforms. I've already been mocked on twitter for it but I think Unikernels will be the next major advancement in software for non-general-purpose systems.

Instead of installing a huge bloated software platform and configuring it to run your code. The OS will be reduced in size to the point where it can be versioned in source control and setup as a step in the build process.

NodeOS is one example of this. It uses a very tiny Linux kernel stripped of everything except the absolutely bare minimum foundation. It has a scheduler, dynamic memory management, enough of a HAL to run on a VM, and a basic network stack. Everything else runs on the V8 VM, including POSIX utility equivalents written in JS.

That's just Node. There are unikernels for a wide range of different languages. There's even the Runtime.js project which is working to implement even the underlying kernel from scratch in pure JS.

The Raspberry PI platform has proven beyond a reasonable doubt that there's a gaping middle ground (and potential market) between embedded programming and general purpose platforms.

I think the greatest potential for future performance gains will come from cutting out the old cruft and optimizing on what we use today.