[HTC One X] TripNRaVeR:
The kernel is building now, also quick backported something new:

Cross Memory Attach

The basic idea behind cross memory attach is to allow MPI programs doing

intra-node communication to do a single copy of the message rather than a

double copy of the message via shared memory.

The following patch attempts to achieve this by allowing a destination

process, given an address and size from a source process, to copy memory

directly from the source process into its own address space via a system

call. There is also a symmetrical ability to copy from the current

process's address space into a destination process's address space.

- Use of /proc/pid/mem has been considered, but there are issues with

using it:

- Does not allow for specifying iovecs for both src and dest, assuming

preadv or pwritev was implemented either the area read from or

written to would need to be contiguous.

- Currently mem_read allows only processes who are currently

ptrace'ing the target and are still able to ptrace the target to read

from the target. This check could possibly be moved to the open call,

but its not clear exactly what race this restriction is stopping

(reason appears to have been lost)

- Having to send the fd of /proc/self/mem via SCM_RIGHTS on unix

domain socket is a bit ugly from a userspace point of view,

especially when you may have hundreds if not (eventually) thousands

of processes that all need to do this with each other

- Doesn't allow for some future use of the interface we would like to

consider adding in the future (see below)

- Interestingly reading from /proc/pid/mem currently actually

involves two copies! (But this could be fixed pretty easily)

As mentioned previously use of vmsplice instead was considered, but has

problems. Since you need the reader and writer working co-operatively if

the pipe is not drained then you block. Which requires some wrapping to

do non blocking on the send side or polling on the receive. In all to all

communication it requires ordering otherwise you can deadlock. And in the

example of many MPI tasks writing to one MPI task vmsplice serialises the

copying.

There are some cases of MPI collectives where even a single copy interface

does not get us the performance gain we could. For example in an

MPI_Reduce rather than copy the data from the source we would like to

instead use it directly in a mathops (say the reduce is doing a sum) as

this would save us doing a copy. We don't need to keep a copy of the data

from the source. I haven't implemented this, but I think this interface

could in the future do all this through the use of the flags - eg could

specify the math operation and type and the kernel rather than just

copying the data would apply the specified operation between the source

and destination and store it in the destination.

Although we don't have a "second user" of the interface (though I've had

some nibbles from people who may be interested in using it for intra

process messaging which is not MPI). This interface is something which

hardware vendors are already doing for their custom drivers to implement

fast local communication. And so in addition to this being useful for

OpenMPI it would mean the driver maintainers don't have to fix things up

when the mm changes.