Were those 286 Xenix applications or 386 Xenix applications?

Running 286 Xenix applications on a 386 or later Unix or Unix-like system is fairly straightforward as far as the 386 kernel is concerned. 286 Xenix used a different syscall mechanism than 386 Unixes did, so you didn't have the problem of the kernel needing to deal with 286 and 386 syscalls ending up in the same place.

My recollection, from being half the two person team that implemented 286 Unix binary compatibility for System V Release 3.2 on 386 at ISC, and then being half the different two person team that did 286 Xenix binary compatibility, was that this is what was required in the kernel:

1. Adding a mechanism to allow a process to set up 16-bit segments as aliases to portions of its address space.

2. Modifying exec to recognize a 286 Unix or Xenix binary, and turn the exec into an exec of /bin/i286emul or /bin/x286emul, with the path to the 286 binary as an argument.

3. I don't remember if we had to do anything special for the 286 system call mechanism, because I don't actually remember what that mechanism was. I don't remember if what the binaries would try to do for a syscall would simply cause a trapable signal, and then the signal handler would recognize the system call and take care of it, of if the system call mechanism used something like a call gate that the 386 code had to set up, and so we had to add a mechanism to allowed the 386 code to do that.

That was almost all the kernel work. Pretty much everything else could be handled in user mode code. When you would run a 286 Xenix binary, that turned into an exec of /bin/x286emul. x286emul would read the Xenix binary, allocate memory for it and load it, map segments to it, and jump to the code, switching the processor to 16-bit mode.

When the Xenix code did a system call and it ended up in the handler in x286emul, for most system calls it was simply a matter of copying arguments from where the 286 call put them to where the corresponding 386 call expected them, making the 386 system call, then putting results in the right place, and returning to the Xenix code.

For some system calls, more work was needed. Signal handling, for instance. If the 286 code wanted to trap a signal, the 386 code would have to set a trap of its own, and its handler would then have to deal with the stack fiddling and such to deliver it to the 286 code.

Speaking of signal handling, that led to the stupidest, most annoying meeting I've ever had to attend.

ISC had done the official 386 port of System V Release 3 under contract from AT&T, and we did the 286 Unix binary compatibility as part of that. Later, there was a deal between AT&T and Sun and Microsoft to make Unixes more compatible, which included Xenix compatibilty, and we were doing the Xenix compatibility as part of that, under contract from Microsoft.

For the 286 binary compatibility, ISC did the whole thing--all the user mode code in i286emul and all the supporting kernel changes. But for Xenix compatibilty, ISC was just doing x286emul. Microsoft was doing any kernel modifications needed, which wasn't very much because the mods already there for i286emul mostly worked find for x286emul.

There was one difference between Xenix and Unix signal handling that we could not fully address in x286emul. We needed kernel support--some sort of per-process flag that would tell it the processes wanted signals to behave like Xenix signals. We asked Microsoft to add such a flag.

A bit later they got back to us and said there was issues with such a flag that could not be settled by email or phone. We had to have an in-person meeting. So me and the other guy working no x286emul had to fly early one morning from Los Angeles to Seattle, take a rental car to Redmond, and attend a meeting with Microsoft.

At the meeting, we introduced ourselves and the half dozen or so Microsoft people introduced themselves, and then they presented this issue that could not be handled by email or phone: should this flag be controlled by an ioctl, or should it be an optional new parameter for the signal() call [1]. We gave our opinion, that was the end of the meeting, and we headed for the airport.

[1] or something like that. I don't remember the exact second option.