Looks like we're on the same page now :-).

> and then be unable to find out why because some other thread reset my context.

Then don't do that. Don't reset the context in another thread without coordination.

What is the definition of a "modern" IO API? It would be unreasonable to argue that streaming APIs like FILE are ancient, or not modern. FILE is a streaming abstraction that allows to read streams (such as files on a filesystem on a hard disk are, for example) in arbitrary chunks, even byte-by-byte. The buffering in a shared (ring-) buffer and the access to it in arbitrary chunks represents an abstraction over the system call level. The (submission) buffers are abstracted away. It would be extremely uneconomical to persist explicit request states, and expect status reports, for 1-byte reads. Instead, the abstraction simply accesses a ringbuffer.

Let me make some concessions, since I loved my role as an adversary in this discussion a little too much last night. There certainly are situations where system calls are just the right level of abstraction. They are obviously what you use if you implement an abstraction on top. Maybe also suitable for some kinds of forensic software. Or if you have specific performance requirements. If you need close control over system calls and need status reports for individual requests, this is the level at which to write your application.

Also, it's obviously true that data that was returned to you from fread() has been transferred from the disk to userspace successfully. I don't know if I tried to dispute that, but if I did I surely shouldn't have attempted. What I was at though is that you aren't well advised to rely on any correlation between fread() calls and system calls (unless you disable the buffering in the FILE handle, btw.). This is simply a different level of abstraction. It might be unwise as well to to try and associate a positive ferror() state to any particular fread() or fwrite().

I reiterate that streaming APIs, whether blocking (like FILE) or non-blocking, are a totally valid and practical abstraction. They allow reusing a generic buffering and OS layer, and enable the design of concise application code. They should be the first abstraction to consider for most types of software. For example, as long as you only care whether a file was completely committed to disk or not, which is an extremely common case, just fopen()/fwrite().../fflush()/ferror()/fclose() sequence is enough and the FILE API serves you perfectly.

> The same could be done with a hypothetical safe write operation: initiate a call to definitely_write(), and get back some kind of handle that allowed us to wait until those particular bytes were physically written to disk (or at the very least to the disk driver, since those now often do their own level of buffering...).

Modern approaches take a different route about synchronization and transaction handling: What you actually need is ordering constraints. Write this only after that other thing was completely committed. Read A only after B was successfully read. Ordering is completely sufficient for these tasks. There is rarely a need to synchronize, which is coarse-grained and very slow. So, things like fflush() and fsync() (the latter is a system call) can maybe considered ancient. I don't know what modern APIs are available to achieve ordering of I/O operations, but this is the way it works for thread synchronization over CPU / memory these days.

In conclusion, again, whenever you wait for an error (or any status result really) from an operation that is executed asynchronously, this is _slooow_. (Caveats apply wrt. FILE because it is in some sense a blocking abstraction over an in-spirit asynchronous operation (OS I/O)). Avoid synchronizing on status values (In the case of FILE, avoid fflush()). In most cases, you can easily read them back asynchronously, at most synchronizing at the very end of a larger "batch" of work.