Introduction

Hello, and welcome back to CS631 "Advanced Programming in the UNIX Environment". In our last segment, we talked a lot about the /etc/passwd file and the various curious accounts and fields one might encounter, but we didn't really talk about the library functions used to look up account information. We'll rectify this with this video. We'll also take a look at the group database and the library functions to perform lookups against that.

etc/passwd

But let's first remind us what the password database looks like. As a text-based file, performing lookups should be easy, and we could imagine processing the file one line at a time. So what do the library functions to look up user information look like?

getpwuid(2)/getpwnam(2)

Well, we have a few functions. For starters, here we show the two functions used to look up a user's account information by UID and by username. They each will return a struct passwd on success, which contains all the fields we've seen in the file /etc/passwd: Note that on some platforms the struct passwd contains additional fields. For example, on BSD systems, you may also find a login class -- a logical grouping of accounts to assign certain resource limits, for example -- as well as an account expiration date or a password change time. Since this information is not stored in /etc/passwd, where does it live? The answer is the 'master.passwd' file, also found under /etc. This file contains all the fields from /etc/passwd, plus the additional fields. This file also includes the encrypted - well, hashed - password, which we noted is not actually present in /etc/passwd, where we only find the asterisk placeholder. This is because /etc/passwd needs to be readable by all users to allow the getpwuid / getpwnam calls to succeed without super-user privileges.

Examples

For example, consider your ls(1) command: when running 'ls -l', it will print out the usernames of the file owners. But as you know, the struct stat does not include the username, only the UID. That is... You actually only have the numeric IDs, and in order for ls(1) to translate UIDs into names, it needs to call 'getpwuid(3)' for each. So if /etc/passwd was not readable by everybody, ls couldn't do this. But since /etc/passwd has a password field, that'd mean that any user could see the other users' albeit hashed passwords and then stage an offline dictionary attack and crack the passwords in their spare time. That doesn't sound so good, so we moved the password hash into /etc/master.passwd, where we could then also add additional fields without breaking compatibility with the old /etc/passwd format. /etc/passwd is then generated from the /etc/master.passwd file. Other Unix systems may use a similar approach -- Linux, for example, uses /etc/shadow, and getpwuid and friends do not return the additional information.

getspnam(2)

For systems that user /etc/shadow, there are a bunch of separate library functions, including the getpwnam equivalent: getspnam, shown here. Calling this function will likely fail if you are not euid 0, as /etc/shadow, like /etc/master.passwd, is readable only by root for the same reasons. Now with getpwuid / getpwnam, you can look up an individual user's password file data, but what if you want to iterate over all the entries, similar to how you might want to iterate over all directory entries? For that, we have getpwent(3). getpwent(3) will read the password database sequentially, but in an order that is opaque to you. That is, even though it may look to you like it goes in the text order you observe in the file, you should not rely on that. setpwent(3) rewinds the stream back to the beginning, and endpwent(3) closes it.

Code examples

Let's see those calls in action: Here's a simple program to look up a user's information and print it in the format we are used to from /etc/passwd. We allow the program to iterate over the entire database or to look up an individual user. Looping over all entries is rather straight forward: you simply call getpwent(3) until it returns NULL. Printing the information only requires formatting the members of the struct, but to ensure we don't accidentally derefence NULL and segfault, we add a simple assertion. If we're given an argument, we have to decide if it's a username or a uid, so we're borrowing the 'who' function from /usr/bin/id, which will first attempt the lookup as a name and, if that fails, try to convert the string to a number and then look up the user by ID. Ok, now if we run it without arguments, we basically get back our /etc/passwd file. If we look up the user with UID 0, we get back root. and if we look up user 1, we get the devil himself. Likewise, looking up the user by username gives us the expected results. But remember, we had two alices -- which one did we get? As before, we got back UID 1004 -- which shows that getpwname will only give you a single user if multiple users are found, just like getpwuid gave you only root and not root and toor. First match wins. Sadly, we have no avocado. Ok, now let's get my user's information. Note that the password field here is an asterisk, but if we run this program with euid 0, then we get the hashed password. That is, getpwnam will provide you with this information from /etc/master.passwd if you are euid 0, but return to you an asterisk if you are a mere mortal. And yes, you can now pause the video here, steal my password hash and start cracking it. Good luck! Remember fred? Fred didn't have a password, but here we get an asterisk. But /etc/passwd doesn't have a password set! But if we run with euid 0, then we get no password. Again, the asterisk is not actually retrieved from the password file, but handed to you by getpwnam based on your euid: if it's 0, you get whatever is in /etc/master.passwd, otherwise, you get an asterisk regardless of what is in that field in /etc/master.passwd or /etc/passwd.

etc/group

Ok, so much for the password database. But we also discussed groups and group membership. For that, we have another database file, /etc/group. Much like /etc/passwd, /etc/group contains: a group name, a hashed password field, a numerical group ID and finally an array of usernames identifying the members of the group. But wait, why is there another field for a password hash? Of course it goes back to the olden days of Unix once more... It used to be the case that a user could only be a member of a single group at one time, so if they wanted to access a file owned by another group, they would have to change their group. To change your group, you'd run the 'newgrp(1)' command, and provide the group password. Nowadays, we have supplementary groups, but the newgrp(1) command still exists - and works! Here, let's demonstrate: Here we see a file owned by the group 'guest', of which user jschauma is not a member, so can't read the file. But the newgrp(1) command can change your primary group ID and start a new shell for you. If there is a password hash in the group file, then the user has to provide the password. Ok, let's give it a spin. Ok, password provided. And look at that, I'm now in the group 'guest'... ...and can read the file. But note that the /etc/group file does not show me as a member. And it shows the password hash, unlike /etc/passwd. Don't we have a separate file for the group password hash, maybe /etc/master.group? [pause] The answer is 'no' -- group passwords are hardly ever used, and supplementary group membership sufficiently solves the problem of granting the access without requiring managing group passwords. So you will likely never run into this, but hey, this class is also intended to give you sufficient geek creds to impress other unix nerds, even if that's not practically useful. You're welcome! Now let's see, the group 'games' doesn't seem to have a password, can I become a member of that group? newgrp says 'sorry' - it doesn't understand an empty password field. But note that even on failure the newgrp(1) command still started a new shell. Oh well, as I said, it's a rarely encountered tool.

getgrgid(2)/getgrnam(2)

Ok, how do we get group information from /etc/group? Just like getpwnam and getpwuid get you entries from /etc/passwd, we have getgrgid and getgrnam for groups. And just like for /etc/passwd, we can also iterate over the whole file sequentially.

get groups(2)

But groups are somewhat complicated: as we know, we have a _primary_ group ID, which is the group ID listed in /etc/passwd. And we can have a number of supplementary groups, as listed in /etc/group. POSIX provides us with the getgroups(2) syscall, but does not define whether or not that includes your primary gid, so if you want to get all groups you're a member of, you need to also look at /etc/passwd. The more comprehensive getgrouplist(3) function is non standard, so check if your Unix version supports it before relying on it. Here, let's implement the 'groups' command. You'll find that it is surprisingly more complex than handling user information. As before, we handle arguments based on username or userid. But since we don't know how many groups the user is a member of, we have to dynamically allocate space for all groups, and so we start out with a sysconf to determine a reasonable max. If we were given a username or uid, then we call getgrouplist, but it's possible that the number of groups the user is a member of doesn't fit into our initially allocated array of groups, so if that fails, we have to allocate more memory and then call it again. If we weren't passed an argument, we first get the primary gid, then call getgroups(2). Finally, we have a list of group IDs, but we want to print out the group names! So we have to call getgrgid for each and then print out the names. Ok, let's run it: This looks good. What groups is root a member of? Ok. Let's confirm. Here, we see that my username is not included in the /etc/group file entry for my primary gid, because my primary gid is included in /etc/passwd. Now let's change our group membership... ...and see if our program behaves. Yep. Good. We exit the newgrp shell and back to normal. Oh, and one more thing. The 'id' command includes similar functionality to the 'groups' command. If you look at the inodes, you should find that they _are_ the same command. That is, id(1) is a program that was written to behave differently based on how it's called -- it looks at argv[0] and behaves one way if it's called 'id' and another way if it's called 'groups'.

Other databases

Alright, we've seen two of the critical system databases, /etc/passwd and /etc/group. The functions used to access the data in them are similar, and so it should come as no surprise that we have other, similar functions to access other system resources. The general pattern here is: We have a certain resource that is described in plain text files under /etc, and for each we have library functions that retrieve the resource into a data structure. These functions generally follow the "getFOObyBAR" pattern to allow you to look up things. We'll see several of these functions in future lectures.

Recap

Ok, let's recap what we learned: handling the data from /etc/passwd is relatively straight forward. The password hash is kept in a separate file, that only euid 0 can access. On BSD systems, this file is called /etc/master.passwd and the getpw functions will automatically provide you with the hash, on other systems you may find the file /etc/shadow and need to use other functions. We noted that groups can have passwords. This is rarely seen, but an interesting thing to keep in mind. And we've learned that iterating over a user's groups is surprisingly more of a pain than iterating over their password data. The functions we saw here are illustrative of a consistent approach to system resources, and we'll see a few more, similar calls, in the future. Ok, this wraps up this video segment. In our next session, we'll try to make sense of time itself. But be warned, time is an illusion, and filetimes doubly so. Until then - cheers!