Introduction

Hello, and welcome back to CS631 "Advanced Programming in the UNIX Environment". In this second segment for our week 2 materials, we will cover the open(2) and close(2) system calls. As usual, we'll walk through some code examples to illustrate the functionality, but this time things should be a bit more straightforward than the last time, when we dove into the question of how many file descriptors a process can open.

Standard I/O

Are you ready? Let's go! As mentioned in our last video segment, almost all file I/O can be performed using just these five functions: open(2) and close(2), which we'll cover in this video segment, and read(2), write(2), and lseek(2), which we'll cover in our next video segment. As you can tell, all of these are system calls, and so it will come as no surprise to you that a lot of this functionality is wrapped for convenience or with additional functionality provided by numerous library functions. However, at the end of the day, it's these syscalls that do all the work. Okay, so before we can do any file I/O, we probably need to actually have a file, right? So let's create one. To do that, we can call... the creat(2) system call. This call takes a pathname as the first argument and the mode_t, a description of the access permissions, as a second argument, and it will dutifully create the new file, open it in write mode, and return the file descriptor representing this new file. But... wait a second! Didn't I just say that all I/O could be made with these five syscalls? creat(2) isn't in there. Well, the reason for that is as follows: as I mentioned, creat(2) returns to you a file descriptor opened in write only mode, but sometimes you want to create a new file and get back a file handle that allows both reading and writing. In the early days of Unix, when open wasn't able to create files, you had to create the file, close it, then open it again. This posed a race condition, whereby another process could have, for example, unlinked the file and put it in a place a different file, leading to unexpected results. That is, the creation and opening of the file was not an atomic operation. As we discuss process relationships with respect to file descriptors in a future video segment, we'll get back to this concept of atomic operations, but for now, suffice it to say that an atomic operation is one where all steps are guaranteed to complete without the possibility for another process to change state during the operation. So the creat(2) call was obsoleted by open(2), with the flags used as shown here.

open(2)

Which then gets us to the open(2) system call. The open(2) syscall takes as argument a pathname, a bit mask of flags telling open(2) how to behave, as well as, optionally, a third argument: the mode permissions with which a file will be created subject to modifications by the process's 'umask', which we will see in a future lecture. Note that this makes open(2) the only syscall we discuss here that doesn't take a file descriptor, which makes sense, since it's the one call that returns the file descriptor. With the pathname argument being self-explanatory, let's take a look at the oflags: when opening a file, you need to specify whether the file should be opened in read-only mode, in write-only mode, or in read-write mode. This is a useful protection for you against yourself: if you know you will never write to the file in question, open it in read-only mode and you won't be able to even do so by accident. Then there are a few additional flags you can enable by ORing the flags. They are: O_APPEND, which we'll see in practice in a little bit; O_CREAT, which is what allowed us to obsolete the creat(2) syscall; if we'll create a file, then the third argument mode is required. O_EXCL allows for an atomic, exclusive opening of a file; since without this, a test for the existence of the file followed by the call to open it, thereby again introducing a race condition. This is actually a common vulnerability pattern known as "TOCTOU", a time-of-check versus time-of-use race condition. So O_EXCL avoids this problem. O_TRUNC truncates the file and we'll see the other options in the future lecture. Note also that the flags supported by open(2) may differ across platforms and go beyond what POSIX requires. Here's a list of additional flags that may or may not be supported on your particular Unix version. For the most part, we'll be focusing on the flags that are prescribed POSIX, but it's useful to know and be aware of which flags are available. Check your local manual page to figure out which open(2) flags are supported on your particular systems; you may find that there's a couple that we didn't even list here. Finally, many Unix versions nowadays also support the openat(2) call, which is used to handle relative path names from a different working directory in an atomic fashion. This may sound confusing; consider that you can change your working directory and you may wish to resolve a path relative to wherever you started without having to either change back or to have to construct a relative path. Perhaps consider this another exercise for you think of a scenario where openat(2) can help prevent the time-of- check/time-of-reuse race condition, then write some code to confirm your understanding. Now open(2) will return to us a file descriptor on success, but of course things don't always go well. When open(2) fails, it will return -1 and set errno appropriately. Some of the most common errors why open(2) may fail are shown here: if we ask open(2) to create a file with O_EXCL, and the file exists, it will fail with EEXIST. As we've seen in our previous video segment, if we run out of file descriptors, open will fail with EMFILE; if the file does not exist, we get ENOENT. If we don't have permission to open the file as we'd like - read permissions for O_RDONLY or O_RDWR, write permissions for O_WRONLY or O_RDWR - we get EPERM. There are many, many other scenarios under which open(2) may fail; check your manual page. Now given that open(2) may fail for so many reasons, we'll always - always! - check the return code of the open(2) call before trying to use the file that it returns. That is, you never write code like this: but always like this: Closing a file is a bit easier: you pass in the file descriptor and that's about it.

close(2)

We'll note that closing the file descriptor will release any records record locks on that file. We discuss the concept of file- and record locking later in the semester. Now in a simple program, you can even get away with not calling close(2) yourself at all: when your process exits, the kernel will automatically close any open file descriptors you had you had open, but that is (a) sloppy programming, and (b) easily leads to you leaking file descriptors when you refactor your code and all of a sudden your open(2) call is made inside a loop, for example. To prevent this, you should get into the habit of always closing the file descriptor explicitly and within the same scope of your open(2) call.

Code example: scope

To illustrate this concept of opening and closing files within the same scope, let's take a look at a quick example: Imagine we have a bunch of code already written, and we then add logic to do something new with the file. We'll add our open(2) call right here, carefully checking the return code, of course, then, right after this code block, we'll immediately add the call to close(2). After that, we go back up and begin inserting code that does something with the file descriptor in between the open(2) and close(2) calls. This way, we do not forget to add the close(2) call later on, as would be easy to do otherwise. Even better, if we decide to move this whole block into another function, then we can just take everything, we can move the whole thing, and again remain reassured that we didn't leak a file descriptor. Let's see in an example. We'll go back up, declare a new function, pull the code that we need, and then call the function. This simple trick of always writing out the close(2) call right after having typed the open(2) call is something you should get into the habit of and should use, likewise, for any other resources, such as when you, for example, allocate memory or have to free it afterwards. An easy way to remember how to keep them in the same scope is to check the indentation level. Here we have open(2) in one level, and we make sure that close(2) is indented on the same level. Okay, so back to close(2). You might have noticed that we are not checking the return code of close(2) in our code example, even though I've said we always have to check the return codes of all of our functions. Ao why is that? Can close(2) simply not fail? If you look at the manual page, you will find that close(2) can fail: it can fail if the number you gave it was not a file descriptor, or if it received a hardware interrupt. That's about it. So what do you do if your close(2) call fails? Well, you are going to close the file descriptor, so you wouldn't be using it after the close(2) call anyway, so in most cases it is actually permissible to move on even if close(2) failed. However, as careful programmers, we want to make sure that the reader understands that we are not blindly ignoring the return value, so we explicitly cast it to void as shown over here.

Code example: open(2)/close(2)

Okay, now let's take a look at opening files by example: our code example 'openex' illustrates what happens when we create a file, try to create an existing file, fail to create an existing file, open an existing file, fail to open a non-existing file, and truncate an existing file. First, we have a few functions here. Note that we're passing in O_CREAT, but not O_EXCL; if the file doesn't exist, this will create the file. If the file does exist, however, it will simply open it. Since we're creating a file, we have to specify a mode; we do this using the symbolic names from sys/stat.h to ensure owner read- and write- permissions. Now also note that, contrary to our recommendation, we are not closing the file descriptor here within the same scope. But here we are doing this on purpose, as we want to illustrate that opening files will yield incrementing file descriptors. Next, we'll try to create a file that we know already exists; this time we pass O_CREAT | O_EXCL, so we expect open(2) to fail here, meaning our attempt to close(2) an invalid file descriptor in the next step will also fail. This time, we'll print the message to confirm. Next, we'll try to open(2) the file that we know doesn't exist, and we know that will fail, so just like we know that the close(2) will fail, so we can cast it to void over here. Here is the most basic scenario: we open(2) an existing file, this source file, and close(2) it again. And finally, we are creating a copy of the source file that we then open with O_TRUNC to illustrate truncating an existing file upon opening. In 'main', we simply call these different functions with a few seconds pause in between. Let's run it. Okay, we try to create new file, which doesn't exist yet. That works out, and we get back a file descriptor number three. That makes sense, because we know that file descriptor 0, 1, and 2 are already open, so the next available file descriptor is bound to be 3. Now remember, we are intentionally not closing this file descriptor, then trying to make the exact same call again, trying to "create" the file that already exists. This time around, 'newfile' already exists, but since we didn't specify O_EXCL, our open(2) call succeeds anyway, and we get back file descriptor 4, which we also leak. Now we try to open 'newfile' with O_CREAT and O_EXCL. As the file already exists, this fails, and since that failed, the file descriptor we are passing to close(2) is also invalid. Now we open 'openex.c' in read-only mode, which works out: we get back a new file descriptor, fd 5, and we promptly close this file descriptor again. Trying to open a non-existent file fails, as expected. And finally, we copied 'openex' to 'newfile'. ls(1) shows us the size of new file before we call open(2). Now we pass O_TRUNC to open(2); open(2) succeeds, returns to us the next available file descriptor, fd 5, and we can observe the new file was indeed truncated. So in this example, we saw the different ways in which the call to open(2) might have failed, as well as how it might succeed. We note that strerror(3) generates meaningful error messages, and that file descriptors appear to generate in sequential order from the lowest number available. Okay, this brings us to the end of this segment. As usual, please make sure to run the code examples by yourself and to ask questions on the mailing list if anything remains unclear. In our next segment, we'll finally do something more exciting with our file descriptors than just opening and closing them. We'll also find out answers to these questions: can you seek backwards on a pipe? What happens when you try to write data beyond the end of a file? How efficient is our simple-cat.c program from week one? Can we make it read more data faster? Until then - thanks for watching.

Outro

Cheers!