MMIO means something much narrower than "it behaves like memory". It really only means "its addressable like memory." I can give you a bit of background in one case. This is just an "existence proof" if you will, I don't know how commonplace the situation is. Most of my experience is on microcontrollers that don't have caches, or in userspace.

I'm currently working on a Cortex-A9 platform: Zynq-7000. Somewhat out of date, but not outrageously so. One of this particular SOC's gotchas are that many of the memory masters on the system aren't coherent, either with each other or with the CPUs. Ethernet DMA? Not coherent. General purpose DMA? Not coherent. All but one of the ports from FPGA fabric into the hard processor? Not coherent.

In any case where you want to tell the other peripheral that it should be reading from what the CPU wrote, the CPU must therefore take measures to get it out of the CPU's caches first. Concretely: I want to give the PL330 DMA controller some instructions in the form of a little DMA program. Part of the setup is that I effectively give the DMA the start address of the program, and then the DMA reads the program afterwards. Therefore, I need a happens-before relationship from <constructing the dma program instructions> to <pass instruction pointer to dma engine>.

First idea: I'll use C11 atomics! Nope, no dice: None of the AXI slave ports into peripheral address spaces support an exclusive monitor. Many of the ARMv7 C11 atomics are based on LDREX/STREX (similar to load-linked/store-conditional), and without some bit of hardware to observe the exclusive access, it isn't exclusive. Actually its a bit worse than that: LDREX doesn't have any way to communicate the failure to acquire an exclusive access, so strex never passes. The C11 atomics are based on an assumption of forward progress. So now, `atomic_fetch_or()` is actually `while (true){}`.

In kernel space.

OK, that's unfortunate, but very well, I'll use a barrier instruction, instead! I didn't really need exclusive access anyway, I was just doing that to get a portable release barrier. Well, that's not enough on its own, either. You see, barrier instructions enforce a kind of happens-before relationship, but it isn't the one I need. Yes that peripheral register was in Device memory, so it wasn't cached out from under you. But you placed your little DMA program in Normal memory (aka cacheable SDRAM). The master port on the PL330 doesn't pass through the processor's cache hierarchy at all. So that bit of hardware has no idea what you may have intended to put in the DMA program, it only gets to see whatever parts of it made it out to DRAM by the whimsical choices of the cache replacement policy, phase of the moon, and so on.

OK, fine then, be that way. I'll allocate a buffer for my DMA program from a chunk of memory that is specifically marked in the page tables as Device memory, even though its actually in DRAM. Now, the hardware works, but each individual write of my DMA program takes 10^2 machine cycles to complete. And since the PL330 uses a variable-length instruction set, and I am hold myself to the strict aliasing and alignment rules I must write each instruction one byte at a time. Therefore, I can very easily take more time just sending out each instruction to DRAM that I did forming the entire thing in cache memory in the first place.

Note, that all of this was just consumed for the narrow purpose of describing the DMA transactions to perform. We haven't even gotten to the DMA transactions themselves. As near as I can tell, if a program on the CPU is either a source or sink of the data, I need to use cache maintenance instructions for the job. Fortunately, it isn't super hard to either invalidate or clean-and-invalidate the individual cache lines prior to consuming or right after sending the data. But dammit, I'd just as rather not have to do either one of those things.

So yeah, the difficulties aren't insurmountable. There are moderately standard concoctions of barriers, cache operations, and custom memory allocators that can do the job. But it takes that much more time to deal with. And I couldn't help but notice that the successor generation (Zynq Ultrascale) has far more options to use coherent masters.