Introduction

Hello, and welcome back to CS631 "Advanced Programming in the UNIX Environment". In this video segment covering material for week two, we will go into the details of the read(2), write(2), and lseek(2) system calls. We'll also do a few weird things with file descriptors and take a look at how we can ensure I/O efficiency when making these system calls. After this segment, we have completed all the five basic I/O syscalls and you'll be ready to write some code.

read(2)

Let's roll! Here, we see the function prototype for read(2). As previously promised, read(2) takes a file descriptor as its first argument, not a path name. In addition, it takes a pointer to a generic - i.e. void - buffer into which it will read the data, as well as the number of bytes it should attempt to read. Now in typical C fashion it is your responsibility to ensure that the buffer your hand to read(2) is big enough to read 'num' bytes into. If the pointer passed as 'but' is invalid, or if it points to a buffer that's too small, you'll get a segmentation violation - a segfault. read(2) will begin reading at the current offset of the file. We'll talk more about the offset in a little bit, but for now consider it a marker, a bookmark in the file that says "you are here". As read(2) reads bytes from the file descriptor, it increments the offset so the subsequent read(2) call continues wherever the previous call left off. If there is no more data - that is when we have reached the end of the file, aka EOF - read(2) will return zero. If read(2) encounters an error, it will return -1 and set errno. For the various possible error conditions, see the manual page as usual. Note that read(2) does not guarantee that it will read exactly 'num' bytes and that reading fewer than 'num' bytes is not an error condition. Specifically, if you ask read(2) to read 256 bytes of data, but there are only 32 bytes of data left, then it will read those 32 bytes. A subsequent read(2) call will then return 0 to indicate EOF. There may not be as much data available as you have requested, even though you have not yet reached EOF. For example, if you're reading from a socket, it's possible that not enough data has arrived yet, so read(2) might return to you however many bytes are available, but upon a subsequent call there may be new data available. The same behavior can be seen in any file descriptor representing record-oriented devices such as magnetic tape, which will always return to your data one record at a time. And finally, you might get interrupted by a signal. We had briefly alluded to signals in an earlier segment, but as promised then, we'll have an entire lecture on the topic of signals and signal handling in the future, so let's skip over this here.

write(2)

Writing data works almost the same way. You again provide a file descriptor to write to, a buffer containing the data you want to write, and the number of bytes you want to be written. Just like with read(2), it's again your responsibility to ensure that the buffer is at least 'num' byte in size, or else you again encounter a segmentation violation, a segfault. Like read(2), write(2) will attempt to write the requested number of bytes, but may not necessarily succeed. In general, write(2) will block until it has completed writing the requested number of bytes. If in non-blocking mode, and if the object represented by the file descriptor is subject to flow control - such as network sockets, for example - then write(2) may write fewer than the requested number, and you'd have to retry writing the remaining bytes yourself. We'll discuss the differences between blocking and non-blocking I/O in a future lecture. Also like read(2), write(2) will begin writing at the current offset and increment the offset by however many bytes were written. However, if the file in question was opened with the O_APPEND flag, then write(2) will not use the current offset and instead go to the end of the file before writing. That is, it will append the data as we'll illustrate in a moment. Writing to a file changes the contents of the file - duh. But that can be of significant meaning if the filing question is an executable and even more so if the file in question is a setuid executable. For this reason, write(2) will clear the setuid-bit on the file. We'll discuss how this setuid-bit works in another lecture, but perhaps try to think through this scenario and create an example execution or attack, whereby a captured writable setuid file descriptor could otherwise be abused. I mentioned a few times before that it's good to get into the habit of writing programs, proofs of concept, to clarify and verify your understanding.

read(2)/write(2) code example

All right, time for an example. Let's look at the file 'rwex.c'. This is a trivial program that will simply read a few bytes from its source file, then append the comment at the end. Our open(2) call using O_APPEND and O_RDWR. Next, our read(2) call, where we read the defined number of bytes into our fixed size buffer ,then write data to standard out. Next, we write the comment we had defined above, close our file descriptor, and exit. Let's run it. Okay, we've read the first 64 bytes. You can count the bytes here, if you like. The one thing to note is that we are reading bytes, so are not bound by lines or any other formatting, and our 64 bytes just happen to end in the middle of a comment here. Next, our program has promised to write some data to append a comment at the end. Let's see. Yep, there it is. What happens if we run it again? The same data is read and if we look at the end of the file, we see our second comment appended. Makes sense. Now, though, let's see what would have happened if we hadn't used O_APPEND. To do that, let's remove the flag from the open call. We'll still read BUFFSIZE bytes, which should leave the current offset in the file at byte 65, and then we write our new comment. Let's just quickly remove the comments we had previously appended here at the end to avoid any confusion. Let's compile and run it again. The read(2) remains the same, no surprise, but now let's look at the beginning of the file. We see that the first 64 bytes remained the same, and our new comment is written starting at byte 65, overwriting whatever data we had there before. Here, let's fix our comment. Can we try to insert our new comment after this comment block over here? Let's see. Let's change our BUFFSIZE. We'll change it to 247. Why 247? Because that's the byte position at which the first comment block ends, so if we want to insert our new comment we can begin writing here. Let's give it a try. Okay, that doesn't look so bad. Our comment was read as we expected, right? But now here's our comment. We inserted it at the right point, but wait... what's this? Remember, write(2) simply writes data to the current location. It will not insert data. That is, it will not keep all the remaining data and shift it backwards by however many data it's written. Instead, write(2) begins writing at the current offset, overwriting whatever data happens to be there. So now we get to fix our code again... and return the O_APPEND flag and ensure that everything is back to normal. Okay, so this code example should have illustrated how read(2) and write(2) work and how they operate on the file offset. Specifically, writing at an offset will overwrite any data that happens to be there, and it will extend the file if we're writing to the current end of file.

lseek(2)

Which brings us to lseek(2). Do any of you know what this thing is that we're showing over here? Does anybody even do mixtapes anymore? Oh, well. The idea of an offset in a file is not quite unlike the concept of a cassette tape, where you can play - i.e., read - or record - i.e., write - data at the current position of the read/write head. Now, another thing you can do with a cassette tape - and if you've ever tried to record songs off the radio - is fast forward or rewind. That is, you are changing the position of the read/write head relative to the tape. Okay, perhaps that's enough of the poor analogies from years long gone by. I hope you get the idea. Just like you can fast forward and rewind on a tape, so can you move the current offset of the file by calling lseek(2). lseek(2), by the wa,y is called 'ell'-seek and not just 'seek' because it used to return a 'long int' prior to being switched to returning an abstract data type, an off_t. Anyway, so lseek(2) takes a file descriptor, an offset to move the offset by, as well as a third argument, "whence". This value "whence" determines how the offset is used: you can begin seeking of the beginning of the file with SEEK_SET; you can seek off the current position with SEEK_CUR; or you can seek off of the end of the file via SEEK_END. Upon completion, lseek(2) will move the current offset as indicated and return to you the new offset. A value of -1 indicates an error, as usual. Now, even though seeking in the file seems reasonably logical - especially with my rather excellent cassette tape analogy in mind! - there are a few odd things you can do: you can seek to a negative offset; this may not actually be quite so weird, because that's really just rewinding our tape. So if you want to step back a few bytes, you specify, say, -32 as the offset and SEEK_CUR as "whence"; or say you wish to go to a position 64 bytes before the end of the file: pass in negative 64 and SEEK_END, and there you go. However what do you think happens when you call lseek(2) with -1 and SEEK_SET? Take a look at what lseek(2) returns on error and write a quick three lines of code to confirm. Next, you can pass in an offset of 0, of course. This can be useful when you want to jump to the beginning of the file -- lseek(...,0, SEEK_SET) - or the end of the file - lseek(..., 0, SEEK_END), but of course you can also, say, go zero bytes from the current position, or, in other words, "don't do anything, stay right there". But doing that is not quite a no-op, as we will see in a second. Another weird thing you can do is seek past the end of the file. Now that's something you can't do with a cassette tape. But what does that mean? We'll find out in a minute.

lseek(2) code example

Okay, quick example to show how we might want to seek zero bytes of the current position: Let's look at sleek.c. By trying to seek zero bytes of the current position, we can determine whether or not the file descriptor we have is seekable at all, because, guess what, seeking is not always possible on all file descriptors. Let's give it a try! We try to seek of the stdin file descriptor. If we run the program just like this... we are told that we can seek on stdin. What is that in right now? Well, we didn't change anything, so it's our terminal. Now let's redirect it in and instead connect it to a regular file. Yep, that, too, works. Now what if we pipe a file into this program? Oh, look, at that, we can't seek. That's because now stdin for our program is a pipe, and the pipe is record oriented, meaning you can't move back and forth on the data as it comes in. You can't seek on a pipe. How about a fifo? Let's create one. Nope, we can't seek on the fifo, either. Well, that too should make sense, since the fifo is really nothing else but a pipe manifested in the file system. We revisit pipes and fifos in our lecture on interprocess communication, so don't worry if this doesn't seem quite obvious to you just yet. Okay, but what about seeking past the end of a file? What does that even mean? Suppose you're at the end of the file and then seek a bunch of bytes beyond that and then write data on the new location - what happens to the gap in between?

Creating a sparse file

Let's give it a try. The file hole.c contains a program that will do just that, thereby creating a file with a hole in it, a so-called sparse file. Let's take a look. First, we create a new file using the obsolete creat(2) system call just to show it in action for once, then we're writing a few bytes to the beginning of the file. With the file offset then left at the beginning of that data, uhm, at the end of that data, next we all seek a significant number of bytes off the current position, meaning we're now in no man's land. Then we write the contents of our other buffer - another 10 bytes. Now before we run this command, let's take a look at how much disk space we have available. Note that the information provided to us by the df(1) command is in 512 byte blocks, so here we see how many blocks are available and how many are used now. We can run our program and look at the newly created file. Okay, that's a big file we created there, which is just around 10 megabytes. How many 512 byte blocks should this file use up? Well, let's do the math. Okay, so that should be just about twenty thousand 512-byte blocks, so we should see this number, the number of total blocks used, get increased by about 20k. Let's run df(1) again and do the math. Uhm... what? 96? Let's ask ls(1), which has a flag to report the number of blocks used by the file in question. 96. How do we fit 10 million 240 thousand and 20 bytes into 96 512-byte blocks? Let's take a look at the raw data using the hexdump(1) utility. What we see here are the first few bytes that we wrote, followed by a whole lot of null bytes, repeating and repeating for so long that hexdump(1) got tired and decided to just not print them, until we get to the very end, where we find our last 10 bytes of data. Okay, so what's going on here? We created a file, wrote some data, used lseek(2) to move the offset 10 megabyte beyond the end of the file, then wrote some more data. So on disk, we really only have these 20 bytes of data that we wrote, but with a whole lot of nothing in between, a hole in the file. Now when we read this file, such as using hexdump(1), the kernel says "well, there's nothing there, but I don't know how to represent nothing, so here, have a bunch of null bytes instead; that's as close to nothing as we can get." That is, the kernel pretends that there's null bytes, but on disk there actually aren't any bytes - it's a sparse file: it has a hole. Weird, huh? But wait, it gets weirder! What happens when we copy this file? Looks just about as expected: both files are the same size How many blocks does each file require? Okay, that's weird: one file takes 96 blocks, the other over 20 000 blocks. Wait... 20 000 blocks - that's just what we expected the file to use up. Let's take a look at the contents of that file. That looks just like the other file, the one with the hole. So those files are identical. Let's ask diff(2). Right, diff(1) says those two files are identical, but they are not: one contains a hole, and the other... Well, remember what I said the kernel does when we ask it for the contents of the file? It gives us a bunch of null bytes, so when we copied the file ,the kernel gave the cp(1) program all those null bytes, and cp(1) dutifully wrote out those null bytes to disk into the file "file.nohole". That is, while "file.hole" contains a hole, which is presented to us as a sequence of null bytes, "file.nohole" does not contain a hole; it contains actual null bytes written to disk. Ah, Unix is fun! Hey, let's see what happens when we do this on a different operating system. Let's give it a spin on our macOS system first. Wait, here our file with a hole does take up twenty thousand blocks? Well, that's because support for sparse files depends on the file system, and the HFS+ file system used on this system does not support sparse files. So when we used lseek(1) to go beyond the end of the file and then wrote our data, the OS wrote actual null bytes into the hole, because the file system doesn't want to hear about all this nonsense of files with holes in them ,where data is not present. What about linux? Okay, so here our large file takes up exactly one disk block, so this system must support sparse files. What happens when we copy the file? Uhm... wait. Now what? file.hole, which a second ago used a single block, now all of a sudden uses 258 blocks, while the copy, which should use up more blocks, only requires 33? What the hell is going on here? Let's run 'ls -ls' again. Okay, now both files use 258 blocks. Let's think this through. Where are we here, anyway? Okay, we're in my home directory. How much disk space do we have here? Oh, look at that! Our home directory is on an NFS share, a network file system. So when we created the sparse file, it started to write data, we observed the actual network lag of that data being written to the remote system before it finally settled on using 258 blocks. Let's try this again on a local file system. Here, in /tmp we can try this once more. Confirm this is a local file system, and begin executing our command. Okay, the newly created sparse file takes up eight blocks. Let's copy it. Now both raw files only take up 8 blocks. Maybe we have a similar situation as before, so let's wait a second and check again. No change. So it looks like the cp(1) command created a sparse file. Let's look at the manual page. Oh, it seems that this version of cp(1) is able to detect sparse files and can create a true copy of it, making the destination file sparse as well. All right, and one more quick thing: we had earlier observed that using cat(1) we can copy a file by simply redirecting file descriptors. Let's do that. Aha! Now we have a difference. cat(1), of course, has no such fancy logic to attempt to detect the sparse fire, and instead did write out the null bytes provided by the kernel as we had expected cp(1) to do. So that was a bit weird, huh? To summarize: you can seek past the end of a file and thereby create a so-called sparse file, a file with a hole. But the file system must support this, or else you just get a file with lots of null bytes written to disk. If the file system supports sparse files, then the kernel will supply null bytes when you read the hole, but they will not actually be on disk. Different versions of cp(1) or other tools may or may not be able to support sparse files, although that is mostly based on the guess that if you encounter a sufficiently large sequence of null bytes the input was probably a sparse file. We'll revisit some of the things we discussed here, especially with respect to disk blocks, in a future lecture as well. For now, let's move on. Let's take a look at our simple-cat.c program from last week.

I/O Efficiency

We are processing data from stdin in a loop, reading it into a buffer of size BUFFSIZE. That means that the smaller BUFFSIZE is, the more iterations of the loop we perform, and the more calls to both read*2( and write(2) we perform. Likewise, as we increase BUFFSIZE, we reduce the number of iterations, reduce the number of syscalls, and thus become more efficient. In our initial version we chose a buffer size of 32k. And we could have picked any value, really. Why 32k bytes? If we want to be really efficient, why not a... pinky on your mouse Dr. Evil style: one million byte sized buffer? And if our efficiency increases as we increase the buffer, is there no upper limit? Now we can't create a buffer of unlimited size, but couldn't we create a buffer that's just about as large as the amount of memory we have available and then read in all the data in one go? Well, the right thing to do is try it out. We have a hypothesis, so now we test it - science for the win! Now to test this, we need a few large files. Using this loop over here we create 10 files named file1 through file10, each containing 100 megabytes of random data. Next, we'll try out the following buffer sizes: 3M, 1M, 32K 16k, 4k, 1k, 256 bytes, 128 bytes, 64 bytes, and finally a single byte buffer. For each buffer size, we compile the program using the -DBUFFSIZE flag and then run the command five times in a row, reading the specified file. Note that by using random data in the input file and different files for each buffer size, we are avoiding having the test be affected by the file system trying to be clever and caching the data from this. Here, let's give it a try. Since i'm lazy, I put the commands to generate the data files, as well as the loop to compile and invoke the command with the different buffer sizes into a Makefile, so I can save myself some typing. Ok, here we are now, creating 10 data files, each 100 megabytes of random data. This illustrates the use of 'make' for more than just building code projects. We'll get back to that in a future lecture, but if you're interested, I've placed the Makefile into the updated code tarball on the course website, you can fetch it from there. Anyway, let's take a look: All right, since we now have all of our data files, let's do a comparison on the different buffer sizes from largest - 3M -- to smallest - one measly byte. Okay, the first few iterations go by pretty quickly. Let's scroll up and see what this says. Looking at these numbers, we should quickly see that there doesn't seem to be much of a difference between using a 3M buffer and a 1M buffer, a 32k buffer, and even a 16k buffer. The speed increase with a larger buffer is insignificant, considering that a 3M buffer is almost 200 times the size of a 16k buffer, but the speed increase is around 100th of a second or so. Note also that for each of the five invocations the first one is always slower than the subsequent invocations. That is because the file system's buffer cache does indeed kick in and deliver the data that was just used faster if it is accessed again immediately. Now the difference between 16k and 4k seems a bit more noticeable, and as we decrease our buffer size we clearly see a degradation in performance. Down here, with a 256 byte buffer, things are really starting to slow down, and using a single byte buffer will actually take us about five minutes for each iteration to copy the data. I'm not going to let it run for this long. Okay, so based on this output it looks like the buffer size beyond 16k has only negligible benefits, but why is that? Shouldn't the bigger buffer be more efficient? Well, the reason that we can't keep gaining efficiency by increasing the buffer has to do with the file system. The file system has a fixed block size in which it reads data from the disk, and no matter how large your buffer is, the file system cannot read more efficiently than whatever that process is. What we see here in our benchmark is this effect: the file system has a block size of 16k, so increasing the buffer size beyond that value doesn't buy us a whole lot. But surely we don't want to go and perform such a benchmark every time we want to copy data! Wouldn't it be nice if there was a way to determine the optimal I/O size, that is, the file system block size of a given file? Well, lucky you! There is: Try running stat(1) against the file. Look at that: 16k Note that the command line syntax for the stat(1) command differs between unix versions. Linux, for example, uses "stat -c '%o'". Always check your manual page. Note also that of course the results of a little benchmark here will differ from file system to file system and operating system to operating system; give it a try. All right, so that was a whole bunch of fun. We encountered some trivial things: opening, closing, reading, and writing files and some pretty weird things, such as creating files with holes in them. Make sure you check out the manual pages and see which options are supported. They differ across operating systems and it's useful to see what new features are available now that they are not prescribed by POSIX. Try to think a bit beyond the examples we provided here in this video, and answer the questions on this last slide, share your answers and findings on the class mailing list. This concludes this video segment. In our next segment, we'll take a closer look at file sharing, and what the implications of a multi-processing system are on file descriptor handling. Thanks for watching - cheers!