> These cpus are much more than 10x faster than the CPUs you would have used then.

Actually, it's not only about the CPU performance, because NIC performance drew a cosine wave w.r.t. processor power over the years. Earlier NICs had much more machinery inside them and were expensive. Then Realtek came and we witnessed the era of (almost) software LAN adapters, then the silicon became cheap, and some of the stuff moved back into the LAN controller. So modern, higher end cards do not do significantly more offloading, but they handle other things like virtualization of cards between many VMs, or other higher level tasks.

From what I've seen, making a faster CPU is easy, but a faster and wider fabric is harder to weave due to distance, switching related noise and other physics related phenomena. Cache and good memory controllers are also relatively easy, but expensive. Also, RAM and higher voltage doesn't go well together, because RAM uses a lot of energy for its size, so it comes with heat and stability problems and other lifetime related problems (yes, you can fry RAMs).

Storage is another hard technology. Spinning rust is limited by physics, vibration, rotation speed and other mechanical stuff. SSDs came crashing, but before they were cheap and Sun didn't got swallowed by Oracle, they did nice ZFS arrays with some very expensive SSD caches and spinning drives combined. I've seen whattheheckisthat amount of speed from an array with a size of 6-8U total and a couple of IB interfaces fitted to it at the factory. Currently we get that amount of speed from Lustre arrays with some SSDs and a lot of spinning disks.

With the current backbone capacity of a standard desktop computer, even on the high end, 40Gbps network is overkill unless you're going to ingest that data directly at the CPU. Yes, absorbing the data at the disks are possible, for what cost and what use case? 10Gbps is understandable, but I still consider 1Gbps as a sweet spot for general purpose computing. If you're going to transfer vast amounts of data from a storage array to a local system, yes you can go higher, but it's still a niche. Also as you've said, going over 1Gbps is problematic from a signal integrity point of view, and fiber is too fragile and expensive for average customer.

These Mellanox cards get quite hot when they're constantly utilized, so in a little ITX box, both the card and the system will be cooked. The process becomes faster if you use active fiber cables, because connectors sink the heat inside the case via heat sinks attached to the connectors on the cards. Even if you use them in a system room, building a large network with them requires expensive equipment which has adequate bandwidth and cable length. Also, running an IB network needs other software machinery to keep the network up. You can directly run ethernet, or TCP over IB, which kinda defeats the purpose and adds additional penalty and reduces the performance.

All in all, higher speed network is still doesn't bring too much of a value to the home of the average user, and is not still that cheap for the enthusiast. Also the hardware is not scaled down for the home. Cards are big & hot, connectors/cables are bulky and the fabric gear is noisy, bulky and high maintenance for home.

Yes they work rather nicely, but I'd not want them at home. I have enough of them at the system room already.