After working in a few pretty large environments I have some pretty strong opinions about hostnames. I've seen everything from cute theme based names to function/role based names to meaningless uuid based names.

Usually you see the theme based names in much smaller environments, though I've seen it up to hundreds of machines. One problem I have with this approach is that people begin to personify machines, excusing their behavior. "Oh that's just akira being akira." This is counter to actually understanding and diagnosing problems. You're also relying on potentially faulty/emotional memories rather than having an actual stored account of issues. Not to mention, once you have a sufficient number of machines you're going to need a decent service database to know what runs what anyways.

Function/Role based names are what I have the most historical experience with. CBS Interactive, CNET, YouTube, and Dropbox all went this route. This often seems like a great idea and can get you really far. Most configuration management allows you to define classes/nodes/etc based on regular expressions. Sudoers has built in support for hostname globbing. It's also easy to tell what a machine's role is when you get an alert. So why wouldn't you want to go this route?

This method has some pitfalls that become increasingly burdensome as you grow into a larger amount of machines. You might have many types of services that run on a single hardware class. This introduces overhead whenever you need to update a machines role. Imagine it's Saturday and your mobile-web pool is suddenly under provisioned. Well, we have plenty of capacity in our web pool. Just need to rename the host in the machine database, update DNS, update DHCP, etc. While this might not seem like a lot of effort it definitely adds up. When I worked on a fantasy sports product we were constantly renaming hosts as various seasons came around. Same thing with just keeping spare/provisioned hosts around. How do you know how to name the host so that someone can just grab it when they need it? You also eventually get the boxes that serve more roles than the box describes itself as. You'll end up with a box call misc or admin and no one remembers that one day someone set it up as your static origin.

Another fun problem is the inability to describe more in depth what a box does. mysql255 doesn't tell you if it's a master/slave, which data is on it, should it be backed up? I've actually seen places that encoded all of this in a hostname. This is the extreme but it does happen.

The interesting thing about both theme and function/role based names is that to use them effectively in a large environment you already need good tools for managing the roles because the hostname is not effective enough.

At Dropbox, they started with functional/role based names but we decided to move to positional hostnames. These encode the dc, rack, position, and chassis. One immediate benefit we saw with this was quickly identifying when a rack or quad goes down. It's very apparent in monitoring just at a glance. We also have a service database that maps hostnames to tags/roles and those tags appear along-side alerts so you can tell right away which service is affected. We get benefits in easy preprovisioning and reprovisioning. Our configuration has become more generic and easier to comprehend. We can run multiple roles easily on one machine and it's easily discoverable what machine serve what role. In puppet we use an External Node Classifier rather than regex. We only have to worry about base config when a rack is initially installed.

That's not to say this method is flawless. One of the biggest drawbacks to this approach is that machines become harder to talk about and typing them is more difficult. I'd argue that you shouldn't care about specific machines unless they're a problem and in that instance copy/paste should always be used to avoid typos. There's some cool things you can do with PowerDNS' Pipe Backend to get dynamic resolution to service names. Another problem with this approach is it just requires a lot more tooling to get started. Obviously not everyone has the time to build all of the infrastructure around this ideal.

Anyways, sorry for the rant, but everywhere I've worked we've started with role based names and regretted it in the long term. Now that I've finally been able to live what I've long dreamed of, I couldn't imagine choosing another solution.

Obviously hostname conventions are a contentious issue so I won't say anyone is wrong, but these are my experiences.