Sure! The sequence is roughly like this (note that DOS extenders are doing quite similar things):

1. From DOS, load your operating system kernel into memory.
How you do this is up to you, I simply use DOS routines to allocate memory and load the kernel from disk into that memory.

2. Before transitioning into protected mode, save away both the real mode stack and the CS:IP where your DOS task should later continue when your OS is sufficiently booted up, and pass that in to your kernel. I call this the “rendezvous” point. This could be any function in your loader program. What I actually did was to let it rendezvous into a routine that performs a “Terminate and Stay Resident” (TSR) DOS call, which returns to the DOS shell (e.g. COMMAND.COM), but keeps parts of the memory that the loader program allocated in memory.

3. Boot up your OS to the point where protected mode, page tables, interrupt and exception handling, etc., are sufficiently set up so that you can start creating and switching to tasks.

4. The most crucial step: Create your DOS task. That task needs to be a vm86 task (v86 in older 386 documentation), indicated by a bit in the EFLAGS register, and set up so that DOS exactly sees what it was seeing before (with exception of any stuff you want to virtualize away), e.g. the first MB of memory should be mapped to it (if you use paging), the stack is what you saved away, and set CS:IP to your rendezvous point.

5. Whenever desirable or necessary, switch to the DOS task, where DOS will just continue running, first starting at your rendezvous point, oblivious to the fact that it’s now just a task. You can then reflect interrupts back into the task if you want DOS to handle them (so that you e.g. don’t need to write any keyboard or hard disk I/O routines yet); you can invoke and communicate with little 16bit handlers to perform some jobs on behalf of your OS (so that you can e.g. use DOS’s file system, or even its memory allocator); or the other way around: You can make your DOS programs perform “system calls” that trap into your OS to perform actions there, implemented e.g. through software interrupts.

One hint that is easily missed: DOS has a “busy” flag that helps you figure out when it is safe to call DOS functions (through INT 21h). Because DOS is not reentrant, you have to finish your INT 21h calls before you can make new ones.

It’s sometimes a bit fiddly since you not only manipulate your DOS task’s registers for it to do your bidding, but also have to emulate interrupts and even quite a few assembly instructions in your general protection fault handler (in a part of it known as the “monitor”) due to the way vm86 works, but it’s not too bad and it was fun for me. Intel’s documentation about their x86 CPUs, even the original 80386 one, does a reasonably good job at explaining it, some of it in quite some detail, e.g. interrupt handling.

There are also many possibilities on which way to handle things from there. For example, your DOS task can be given direct access to the IRQ controller registers and let DOS handle that, or you can write a virtual IRQ controller for DOS in your OS and handle the real one yourself. The same goes for many other aspects, like video memory for example. Essentially you can move stuff into your OS at your own pace, and give DOS a virtual device if it still expects access.

If you want to learn more, assuming you already have a firm understanding of task switching on x86, I suggest reading the v86/vm86 chapter in Intel’s reference.