Introduction

Hello, and welcome to CS631 "Advanced Programming in the UNIX Environment", week three. In our previous segments, we focused on file descriptors, but in the process we encountered a number of properties of files, such as the optimal file I/O block size when we looked at boosting the efficiency of our simple cat(1) clone. We will revisit these different properties in much more detail in this and immediately following sessions. So get ready to meet your new best friend: the struct stat.

stat(2)

To obtain information about a file - that is, to get all the meta information, not the contents of the file - we introduce the stat(2) family of system calls. As we will see with the related syscalls to be covered within the context of files and directories, stat(2) comes in three flavors: stat(2), lstat(2), and fstat(2). The fstat(2) version differs from the others in that it accepts a file descriptor, which falls in line with our previous discussions of most I/O being done on file descriptors. But in order to get a file descriptor one first has to open(2) a file. As we will see in a future lecture, the file data and the file metadata are well separated in the standard unix file system, and it often is useful to be able to get information about a file without opening it. So to obtain information about a file, we pass a path name to stat(2), as well as a pointer to a struct stat, which the syscall will then fill in. As we saw in our discussion of the open(2) system call, there's also a need to allow for atomically accessing the relative path name outside of the current working directory, so the fstatat(2) syscall meets this need, similarly to openat(2), taking an additional file descriptor, referencing a directory as an argument. As usual, the return code of stat(2) is zero on success, negative one on error, with errno set to indicate the nature of the error. We are interested in getting all the meta information about a file. The idiomatic way for a function to efficiently provide complex information, as well as follow the convention of a simple meaningful error code, is to take as input a pointer to a data structure that the call then fills in. In this case, the data structure in question is a 'struct stat'. As so often, the details of the 'struct stat' are always version specific, with the lowest common denominator being the fields defined by POSIX and shown here in this slide. Of course your particular Unix version may support additional fields, so - all together now! - check your manual page for details. The fields required by POSIX shown here all help describe the file in detail and expose important information used by many tools, most notably of course the ls(1) command. I promise you'll get very familiar with all of them in the near future.

Adding a disk

Before we look at the struct stat information, let us create a small second disk for our VM to play around with. We'll use this disk throughout the next few video segments to better understand the file system. Of course you can use the VirtualBox UI to add a new disk, but where's the fun in that? We'd like to use the command line, so here we go: let us create a new medium of type 'disk', using the VMDK format. We make it 50 megs in size and place it into the directory alongside our VM's root disk, called disk2. it's a small disk, so it goes fast. Next, we attach the new disk to our VM as an IDE drive. We attach it on port 0 as the second device, device one, and specify the path to our newly created disk image. And that's it. We effectively plugged in a new hard drive into our server. So let's start our vm. Okay, once we have it booted up and log in, we need to create a new file system on the disk. To do that, we use the newfs(8) command and we'll specify a block size of 4096 bytes. Next, we mount the new disk under /mnt, and then we change the ownership to our user, so we don't need super user privileges to write data to the new disk. Let's take a look at the available space. And finally, we'll create a simple, one megabyte file. 'ls -l' then displays the file in the output that we're used to.

'ls -l' output

Okay, so let's take a closer look at the output of 'ls -l': each of the fields in the output is derived from the information in the 'struct stat' that we saw earlier. That is, ls(1) called stat(2) and then displays the data in the given format. The permission string on the left side is derived from the st_mode field; next comes the number of links for this file, that is, the number of names that exist for a file associated with this particular inode. After that comes the owner of the file, a symbolic username shown here that is derived from the st_uid field, just like the group owner name is derived from the st_gid field. Next, ls(1) shows us the file size in bytes, as found in the st_size field. The date shown here on the right, finally, is the last modified timestamp, conveniently formatted in human-friendly form by ls(1) from the st_mtime field, and all the way on the right is of course the file name, which is not found anywhere in the 'struct stat' at all. We'll get back to this in our next lecture, but for now let's remember that the file name is not a property of the file; it's a mapping found in the directory. As we've seen, the output of the ls(1) command includes a lot of information from the 'struct stat', but as so often, there's more than one way to do it. Specifically, most Unix versions nowadays have a stat(1) command that allows you to retrieve and inspect the full 'struct stat', including the fields not exposed by ls(1). Although, we should mention that this stat(1) command is not part of POSIX. As a result, the output and usage of the stat(1) command differs quite a bit between different Unix versions. Okay, so here we see the sample output of the stat(1) invocation on the previously displayed file. By default, stat(1) displays the information in human-friendly output; the '-r' flag changes the display to show us the actual values. Specifically, as shown here, the default output displays the st_dev, st_inode, st_mode, st_nlink, uid, gid, rdev, size, atime, mtime, ctime, birth time, block size, blocks, and st_flags fields. The st_dev field identifies the device id the file resides on, ie the disk. Together with the inode number, this uniquely identifies a file across all mount points in the file system. The st_mode encodes the file type as well as the permissions, which you may recognize here in the familiar octal display. The link count, st_nlink, the numeric uid - st_uid - and group id - st_gid - we've already seen. st_rdev is only used for certain devices, which is why we see it set as -1 for regular files, as in this case. st_size - self explanatory - comes next; the st_atime, mtime, and ctime represent the file's last access, modification, and file status change times, respectively. As we described in our early lecture, times like these are kept as seconds since the epoch. We'll get back to the meaning of these three timestamps in a future video segment. The st_birth time is another timestamp, which is not required by POSIX, and may not be supported by all Unix or file system versions. It represents the time of inode creation. In this case, the file system we created on our extra disk mounted under /mnt is of type FFSv1, which does not support an inode birth time. If you were to run the stat(1) command in your VM on a file on the root file system, which was created with system default as an FFSv2 file system, you should see an actual timestamp here. The next two fields, st_blksize and st_blocks we've already seen. The final st_flags field encodes additional file flags that we'll see in the future, although you may already take a peek at the chflags(1) manual page and the '-o' flag for ls(1), if you're curious.

st_mode

Now as we mentioned, the st_mode includes not only the file permissions, but also the file type. So let's take a closer look at what types of files we might encounter: now obviously, there are "regular" files. This is really what most of us mean when we say "a file". A file is merely a way of stashing a bunch of bytes onto disk, and it's important to remember that Unix does not care what you do with those bytes. That is, there is no difference between a binary, an image file, a spreadsheet, an email attachment, or a shared library. As far as the file system is concerned, they are all regular files without any particular meaning. It is up to the application to interpret the sequence of bytes found on disk. Next, there are directorie,s which is to say that, yes, a directory is a "file" - a special type of file, one that maps symbolic names - strings, which humans tend to find awfully convenient, to inode numbers, allowing the file system to find the file meta information and data blocks associated with that string. Any such directory entry, such a mapping between an inode and a string, is known as a "link" or a "hard link", and on any file system, or even with within one directory, there may be many such mappings of strings to a particular inode. We'll learn more about directories in a later segment, and we'll look at some file system implementation aspects in the next week's topics. Next, there are devices, which are represented as files usually under the /dev directory in the root of the file system. These devices come as either character special devices - terminals, for example - and block devices - disks. Next, there's a type of file called a "fifo", also known as a "named pipe". This is, in a nutshell, an inter-process communication endpoint like the shell pipe that is manifested in the file system to allow unrelated processes to communicate with one another. Similarly, there are files of type 'socket', which are likewise used as rendezvous points for unrelated processes on the same system for inter-process communication. Note that these files of type 'socket' are different from network sockets, although the APIs for communications used here are the same. We'll revisit both of those as well in the future lectures. And finally, there are symbolic links, which are interesting in that they are files that contain as their sole contents the path name of another file, and which, when accessed, simply say "oh, uhm, don't look at me - look at this file over here instead!"

simple-ls.c

Let's take a look at some of the different file types when we encounter them on our file system. Remember our low power clone of ls(1) from week one? All it did was open the directory, then iterate over the entries and print out the names of the files it encountered. Let's improve this program to tell us what type of file it is. To do that, we make the following changes: first, we change our current working directory to the target, so we can then call stat(2) simply by passing the file name, and obviate the need to construct an absolute path name. Then, as we loop over the directory entries, we call stat(2) and have it fill in the struct stat buffer here. Based on the information in the struct stat, we then determine the file type. Our 'getType' function uses the S_IS macros on the st_mode field to identify the type of file, returning a descriptive string for each file type, including accounting for an unknown file type, because, hey, you never know, and we want to write defensive, robust code. After we have identified the file type, we then call lstat(2). lstat(2) behaves just like stat(2), but if the file in question is of type symlink, we get the information about the symlink. Remember, a symlink is a file that says "nope, go look at that file over there", so when we call stat(2) it will look at "that file over there", i.e., whatever file the symlink points to, so if we want to identify the symlink and determine, say, the file ownership of the symlink, we need lstat(2). Okay, now before we run this program, let's create a bunch of files of the different file types. First, a directory. Then, a symbolic link to a character special device... ...as well as one to a block special device... and one to a directory, why not. Let's create a fifo, as well as a symlink to a file of type 'socket'. Next, we create a second hard link for 'file', called 'file2', and finally, a symlink to a file that doesn't even exist. Okay, let's see what this looks like. Let's run 'ls -l'. Here, we see the type of file identified by the first character on the left. 'l' for symbolic link, 'd' for directory etc. 'p' for files of type fifo, as they are a named pipe, hence the 'p'. Regular files don't get the character, and we observe that both 'file' and 'file2' are of identical size and have identical last modified timestamps. Now let's run our simple ls clone. As expected, we get the file type correctly identified as type 'directory' or 'regular file', and files that are symlinks to something else get identified by lstat(2) as symlinks, by regular stat(2) as whatever the file the symlink points to. We note that we can have broken symlinks. That is, a symlink that points to a pathname that doesn't exist. In that case, stat(2) failed, but lstat(2) succeeded, which is why we get the output "symlink to unknown". Now the cool thing about symlinks is that you can unbreak them simply by creating the target. No need to make any changes to the symlink itself; as soon as the target exists, the program will be able to stat(2) it via the symlink. So now let's get back to the other type of link: the hard link. As we showed a minute ago, 'file' and 'file2' look identical. When we manipulate one file, for example by truncating it... ...and then writing the string 'foo' into it, then' file2' is changed as well. We note that the st_link count for 'file' and 'file2' is shown to be 2. That is, there are two things pointing to the single file that now contains 'foo'. Let's ask ls(1) to show us the inode numbers of the files. Here, we see that both files have the inode number '3', meaning they really aren't two files, they are one and the same file. Let's write some data into the file, maybe 10k should be sufficient. If we wanted to read the file as efficiently as possible, we'd want to read it in 4k chunks, as shown in the st_block size member of the struct stat. Remember, when we created the file system, we explicitly specified the block size of 4k, so this number should not come as a surprise here. We can also look at how many actual disk blocks are used by this 10k file via the '-s' flag for ls(1). In this case, we see that 'file' uses 20 blocks. But... wait, 20 blocks and a block size of 4k... that's more than 10k total data! Let's quickly check what 'man ls' has to say about block sizes... Okay, here we are: '-s' displays blocks in units of 512 bytes or BLOCKSIZE from the environment. That is, the 20 blocks reported here are 20 512 byte blocks, which adds up nicely to exactly 10k. But our file system uses 4k blocks, so how many of those does 'file' take up? Math is hard, so let's quickly set the BLOCKSIZE environment variable,so ls(1) can tell us - and there we have it: three blocks. Well, technically, 10k would require 2.5 blocks, but we can't use half a block, so our file will require 3 4K blocks. Note that the default of displaying information in 512 byte blocks is all over the system, such as when running the df(1) command, but with consistency a key part of the Unix environment, these tools can often also account for the same environment variables, so running df(1) with the BLOCKSIZE set to 4k will cause it to display the information in those size chunks. There's more on block sizes on disk versus in the file system coming your way in future lectures - lucky you!

Recap

All right, let's pause at this point. We've already covered a good bit: we've met our new best friend, the struct stat. Trust me, you'll get intimately familiar with it and all the other fields in the very near future. We've also seen how the ls(1) command makes use of the 'struct stat' information, as well as how the stat(1) command provides the remaining fields. It's a useful exercise to compare the implementations of this command. We've revisited the st_blocks and st_blksize members from our discussion on I/O efficiency, and we've improved our simple ls clone to show us what type of file it encounters, including accounting for symbolic links. Not bad for our first segment! But as so often, we've just scratched the surface, and there's a whole lot more to come, so stay tuned for our next segment, when we dive deep into file ownership and permissions. Thanks for watching, and make sure to reproduce all the command exercises shown here on your own systems.

Outro

Until next time - cheers!