A journey into himem.sys
Published: 2023-07-16 03:30:26

For some strange reason, I had the need to review some aspects of Intel memory manager unit, rediscovering topics no longer used from many years, due modern operating systems and considering that a lot of my notes have been lost in these years.

My journey began with the idea of writing the HIMEM software directly in C with my oldest compiler!

When I was 14, I spent all my savings to buy a license of Borland Turbo C++ 3.0 with Turbo Vision license, I borrowed from the school library the MS-DOS Programmer's Reference Manual and a friend gave me a printed copy of Intel 386 Programmer's Reference Manual.

I remember that in the past I wrote a library for my programs to use XMS memory through the API provided by DOS device driver HIMEM.SYS.

The HIMEM driver (like EMM386 and many other 386 extenders..) is available for Intel 386 and higher CPU.

The 386 cpu was created to support 32bit addressing, allowing to manage at most 4Gb of physical RAM.

The ancestors of Intel 386 were Intel 8086/8088 and Intel 80286 (an 80186 was also introduced in 1982, but it was not used widely used on PC market, and no innovations on MMU was introduced with 80186).

The 8086 was able to manage 1Mb of RAM (it supported an address of 20bit, but internally all registers of cpu were 16 bit) combing a segment (into one of four segment registers CS, DS, ES, SS) with a general purpose register available (AX, CX, BX, DX, SI, DI, SP, BP). The translation is very simply.

`segment value * 16 + offset = physical address.`

A register can hold a value from 0000 to FFFF allowing to address the same physical address with different segment:offset values since the physical address 1000h is available for 0:1000h, 0001:0ff0, 0002:0fe0 and so on.

The last valid combination of segment:offset was FFF0:0000..000F, and any values upper 1Mb was truncated at 20bit going to read bytes at the beginning of memory.

The first innovation on MMU was introduced on 80286, since the CPU was able to manage at least 16Mb of physical RAM thanks to a first version of "protected mode" widely used on 386. Registers on the 286 were able to store only 16bit values, therefore, to enable the CPU to manage 16Mb of memory segment registers were used as INDEX inside a descriptor table where the physical address of memory was stored. To simplify the use of this new memory, a new service was introduced inside the BIOS accessible through interrupt 15h service 87h (http://vitaly_filatov.tripod.com/ng/asm/asm_026.14.html) because on 80286 there is no way to exit from PROTECTED MODE! You can enter on protected mode through MSW instruction (Machine Status Word) but You can leave it only through a CPU reset! After 25 years I understand why there were present two different kind of reset, a soft-reset and an hard-reset.

To switch back to real mode from protected mode on 80286 first, it's necessary to store at address 0040:0067 the return address of your code in protected mode and from protected mode you can reset the CPU, but before doing this it's necessary to write on CMOS memory a code (tipically 05 or 0A) to signal to the POST bios code to switch back. Without this magic value, the POST procedure will continue the execution of standard BIOS code with a complete bootstrap of operating system.

The 80386 was a modern CPU, with 32bit addressing and 32bit registers and a vast assortment of software written for 8086, so to continue to support the past it was necessary to support the previous memory addressing.

So, when a 386 CPU starts (also on modern CPUs) it starts in real-mode, but it is possible to switch into protected mode by configuring the Global Descriptor Table and setting BIT 0 of CR0 (Control Register 0) to 1. After this, the first instruction must be a JUMP to the code descriptor and when you finish your job on protected mode, you can switch back to real mode clearing bit0 of cr0 and jumping back to previous address like this:

REAL MODE => SET UP YOUR GDT => LOAD GDT REGISTER => CONFIGURE CR0 => JUMP IN PROTECTED MODE => TRANSFER DATA => CONFIGURE CR0 => JUMP IN REAL MODE.

This is what I suppose.

The authors of HIMEM driver put inside the interrupt table a small piece of code responsible to switch to PROTECTED MODE and later switch back on REAL MODE. On protected mode, before using the values stored on GDT you must load a segment register with one valid index inside GDT. So, through Page Fault handler they do this. The handler initialize data registers DS and ES (used implicitly by MOVSx instruction with (E)SI/(E)DI) with a valid value stored on GDT and switches back on real mode to continue execution. In this status the CPU is in "unreal mode", you can take advantage of 32bit addressing directly from real mode.

I'm not sure If I want to use this "hack" when rewriting the driver..