Introduction

Hello, and welcome back to CS631 "Advanced Programming in the UNIX Environment". This is Week 11, and we are starting a section on shared libraries. Remember our discussion of the process layout in memory from Week 06? In this series of videos, we'll revisit just how exactly the process gets from disk into memory. We'll also reach back to Week 05, where we talked about the compiler chain, and how it created different object files as one stage of the process before then combining these object files together with some shared libraries into an executable. While these files created by the compiler are binary files -- meaning, we silly humans can't simply open them in an editor and read them. But that doesn't mean that they're just _random_ streams of bytes. Binary files are well-formatted, too, and have to follow a specific order just the same in order to be able to be read, loaded, and executed. There are several such formats -- a.out, COFF, Mach-O, or the ELF format. No, wait, "ELF format"... that's just like "ATM Machine", right? ELF is the Executable and Linkable Format, so there is no such thing as the Executable and Linkable Format Format. But ok. Now notice that one format to describe executables or object code is "a.out" or "assembler output". The "a.out" format was used by older Unix versions, and as so often, newer tools wanted to retain backwards compatibility or not confuse the user by creating output named differently, and so our compiler to this day produces an executable named "a.out", even though that file nowadays -- on most Unix systems -- is of type ELF. Ok, so aside from the terribly named a.out ELF executable, there are a few other types of files that ELF can be used to describe. For example, core files, which we've seen some good use of when we talked about using the debugger to analyze program failures. ELF describes the process snapshot and its debugging information. Then there are your common object files. These files are referred to as "relocatable" files, as the format allows the information retained in them, the symbols and other data, to be relocated, to be combined with other object files to form either a shared library or an executable. And so of course last we have listed here the shared library file, commonly ending in '.so' for "shared object". These libraries contain so-called position-independent code that can be placed in memory and executes regardless of its absolute address. We'll see examples of each of these files throughout this series of videos, but first let's see what ELF describes or how an ELF file is structured. An ELF file is made up of an ELF header followed by some data. The data includes: - the program header table, containing entries that point at different memory segments, such as the data or text segments, which we've seen in detail in our earlier video on the process layout - as well as the section header table, containing entries that point to select _sections_ The segments pointed to from the program header table are used to provide information required at execution time, while the segments pointed to from the section header table are used at link time. Let's take a look at the ELF header inside... the elf.h header file. Of course. In there, we can find out quite a bit about the structure of an ELF file. We see that the header defines a few pieces of information about the file as well as information about the program header table and section header table etc. But perhaps it'll be a bit more useful for us to look at an actual ELF file...

main.o hexdump(1) output

So let's start with a regular object file, such as when we compile a source file without linking it. The file(1) command can look at the file and tell us what type of file it is: an ELF 64-bit relocatable object file compiled for the x86-64 architecture. How does it do that? If you look at the manual page for the file(1) command, you will find out that it performs a few tests to determine the type of file: First, it may call stat(2) and determine the type of file as reported in the st_mode -- we're quite familiar with that call. But we also know that stat(2) will simply tell us "this is a 'regular' file" for something like the object file 'main.o', so the file(1) command has to have something else up its sleeve. And indeed it does -- the file(1) command uses magic to find out details about the file in question. That is, most well-documented file formats include a special "magic number" that is stored in a particular offset near the beginning of the file, and which you can then map to the type of file by performing a lookup in a file catalog. Since that -- trivial as it sounds -- seems indistinguishable from magic, we call that lookup file the "magic file". The format of that file is documented in the "magic(5)" manual page, and we use the "magic" library functions to perform the lookup. Alright, I honestly only mention this here because I think it's pretty cool that we have a magic library and magic manual pages to use magic to look up a file type. But of course we don't need a magic library, and we like to understand how things work rather than wave a wand and hope for magic to happen. So let's take a look at the actual bytes in the file using our good friend hexdump(1). From the information in the elf.h header file, we know that The first four bytes here identify the ELF header, with the hex 7F followed by the ASCII characters E L and F. The next byte defines the file class -- 64-bit in this case. The next byte defines the data encoding type, the endianness of the system -- little-endian or Least Significant Bit first in this case. This is followed by a single byte specifying the file version -- 1 in this case -- and then we identify the OS ABI -- the Application Binary Interface -- as being the original UNIX System V ABI followed by some null byte padding. After that, at byte 16, the start of the of the non-ident header, we encode the ELF type, where 01 represents a standard, relocatable object file. The next two bytes define the machine type, with 3E hex -- or 62 decimal -- representing the AMD x86-64 architecture, followed by an ELF version indicator and a few null bytes in the case of a simple object file, at least. But ok, reading hex bytes is not terribly convenient, and the file(1) command is a lot cooler, but we may also want to get more information from the ELF file than the file(1) command can provide.

main.o readelf(1) output

So we can use the readelf(1) utility, which presents to use a lot of the same information we just extracted in this neat layout: Again, the first few bytes say "ELF", 64 little-endian version 1 System V ABI ABI version 0 type relocatable file for AMD x86-64 version 1 with entry address 0 0 0 0. So why is that field "entry address" all zero? Well, remember: the file in question is an object file that needs to first be linked into an executable by the linker. Let's take a look at an executable instead: This looks almost the same, but with the difference that the ELF type is now an executable instead of a relocatable object file, and the entry address is now no longer zero. This entry address, which as you can tell is represented here as least-significant bit first -- little endian -- at hex address 0x40 05 70 is where the program will begin executing. That is, this is the location where the standard C library startup routine __start will be. Go back to our week 06, segment 2 video, where we changed the entry address, and if you were to compile a program with an alternate entry point function, you should see the address in the ELF header change. But so far we've only seen the meta information from the ELF file. There's a lot more bytes in the file. Let's see if we can make sense of those, too.

a.out hexdump(1) output

Here's our hexdump of the executable. The grayed out bytes are those we've already seen. These bytes over here tell us that there are 7 program headers in total, starting at offset 64 (or hex 40). The size of the ELF header is 0x38 (or 56 decimal). The Program Header Table has read only permissions and is of size 0x40 (64 bytes). And with that information, we then know that we can find the next header at the offset 0x38 + 0x40 = 0x78 which is down here. There, at the offset 0x78, the next program header begins. It's a program header of type PT_INTERP -- specifying the program interpreter -- with read permissions with its contents to be found at 0x01C8. So if we then look at the data found at that offset, at 0x01C8, we find the path of the interpreter specified as /usr/libexec/ld.elf_so. Again, parsing the hex bytes is tedious, so we can again rely on our friend readelf(1), which can present to us the same information we just extracted: 7 program headers, starting at offset 64, with the program interpreter to be found at offset 0x01C8 being /usr/libexec/ld.elf_so. Now what that interpreter does will be the subject of our next video. For now, you've seen a fair bit about the structure of an ELF file and how this binary format can be inspected and understood.

Exercises

To get a better feel for working with a binary format like ELF, I recommend that you play around with the tools and consider a few of the example exercises shown here: We looked at the .o and a.out files only; repeat the same exercise for a core file and a shared library and see what changed. Try to build a binary for a different architecture -- i386 is easily built and supported on our x86-64 platform -- and see what changes. Build an executable with a different entry point and see how that changes the address in the ELF header. Get familiar with the different tools that we can use to analyze ELF files. In addition to the readelf(1) utility, we've also previously used the objdump(1) tool, and both have somewhat overlapping functionality. See what options each supports that the other doesn't. And of course read through the manual pages and the header files to better understand the ELF format. In our next video, we will then look at the program interpreter, the so-called run-time link-editor ld.elf_so and further explore how shared libraries work. Make sure to check out the various links on the course website and in the slides here at the end. Until next time - thanks for watching. Cheers!