I think this would be pretty straightforward if O_NONBLOCK was allowed to have an effect for regular files. The rules would be:

1. If a read for a range of bytes on a non-blocking file descriptor can be wholly or partly satisfied from the cache, put that data in the read buffer, and return the amount of it

2. If a read on a non-blocking file descriptor can't be satisfied from the cache, return -1, set errno to EAGAIN, and kick off an asynchronous task to load that range of bytes into the cache (an 'asynchronous load')

2a. If there is already an asynchronous load for that range running, don't start another one

3. When an asynchronous load finishes, mark that range as available

1a (surprise!). If that range of bytes was marked as available, remove the mark

4. If there are a nonzero number of available ranges for a file descriptor, return POLLIN from poll()

The way a program with lightweight threads (Node, Go, etc) would use this:

1. Open files as non-blocking

2. Make reads as usual

3. If a read succeeds, or fails other than with EAGAIN, pass the result to the thread and carry on

4. If a read fails with EAGAIN, suspend the thread, and remember the range it was reading

Meanwhile, in a background IO thread ...

5. Repeatedly poll all file descriptors for which there are remembered reads

6. If a file descriptor shows up as POLLIN, run through all the remembered reads for it, and make them again

7. If a read fails with EAGAIN, cool, leave it remembered and carry on

8. If a read succeeds, or fails other than with EAGAIN, pass the result to the thread and resume it

Would that work?

I haven't really thought about partial results of asynchronous reads. If a read asks for two blocks, and one gets loaded, should poll return POLLIN or not? I lean towards yes, because the program can make use of that data. That raises the question of whether to remove the availability mark after the following read. Again, i lean towards yes, because it means asynchronous reads have the same behaviour as ones which partially hit cache. The OS can carry on reading the rest of the range into the cache, but it won't signal the arrival of that data to the program. That should be fine, as the program will probably try to read it soon anyway.

If a program takes too long to get round to reading data once it is available, it might have been pushed out of the cache by then. Too bad. The read will fail with EAGAIN and the whole cycle repeats.

If two threads in a program independently read the same range, only one asynchronous load will be run, and only one thread will end up removing the mark. Programs will need to be careful not to trip up over this; they probably need to coalesce reads on their side of the API.

It sucks to have to retry every outstanding read for a file when it shows POLLIN, but i can't see a way to return information about which ranges are available through the existing calls. A program could keep its list of outstanding reads in time order, scan it from earliest to latest, and stop when it gets a non-EAGAIN result. If there is more than one available range, poll will return POLLIN again and the program will go back and pick it up. If asynchronous loads complete one at a time in order, this reduces the cost to O(1), with a smooth increase in cost as that is less and less the case.

Basically, what i'm saying is that everyone should just implement the kqueue API from BSD.