Scott Bilas/Loose Cannon Blog

***Note:*** The following was taken from an old defunct website containing Scott Bilas's blog. Sections may be incomplete due to not being saved via the Internet Archive.

## November 18, 2008

I’ve been trying to get the time to do a series of posts and I think I’m finally ready to get started. The high-level theme is “Loose Cannon’s workflow”. Like any mid-size team working on a multi-year title, we’ve built up a large set of processes and tools to support development. Some are very carefully planned and based on past experience, and that’s where I want to start first: talking about mature process and tools that I know work well. Well, at least for the types and sizes of teams I tend to work on.

Note that I’m only going to talk about the areas that I’ve either designed or been directly involved in designing. We have a large infrastructure for building and managing assets that I think works pretty well. Maya, exporting, plugins, that sort of thing. But I’m not too familiar with all that. At least not yet – looks like my task while in Peru will very much involve our content pipeline.

So anyway, within this theme, the topic for the next set of articles is how we use Perforce at LCS. Which means this series will be fly-over territory for most people Google sends here. I’ve only met a few people in my life that have actually been interested in things like depot design and the workflow around it. Well, I really love this stuff! Helping the team become more efficient and accurate is something that I enjoy more than any other kind of work. Whether it’s creating API’s or tools, or setting up bugbases, or building an ecosystem for code reviewing, I’m happy doing it all. So I can’t help but post about this.

This is the third company where I’ve set up a Perforce depot and been responsible for its design, maintenance and tool infrastructure. With input in particular from Matt Scott I think we’ve got the best source control setup I’ve ever worked with. We’ve solved a lot of nagging problems that have plagued me at previous gigs. There are still some issues, but overall things are very smooth. I plan to cover things like the design of the depot, standards and requirements, supporting tools, and server maintenance. Perforce, out of the box, is far from a complete system. I’m pretty annoyed with those people, actually. All that money we give them every year and they’re mostly standing still, too busy rolling around naked in cash to fix basic and ancient problems with their design. But that will have to wait for a future post.

## November 26, 2008

It would be an obvious understatement to say that modern game development has a lot of data.

Our hard drives fill up with all kinds of bits: tools, source code, code reviews, source and object assets, bug reports, concept art, docs, email, game builds and debug data, CD images, gameplay analysis data, SDK’s, error logs, and on and on. New types every day. Some you have initially, and some you add over the course of development. It all depends on the type and size of game.

In this next series of posts I am going to focus primarily on code and assets, and the tools and processes related to producing and managing them. That’s still a lot of data! During development, new classifications of data come up all the time, and we need to be able to answer the question of “ok great, now where does *that* stuff go?”

For example, let’s say that the powers-that-be decide that we’re going to add an hour of video cut-scenes to the game. This means a lot of new source assets need to be created, and then some truly huge output video files (made much worse if multiple platforms are involved).

Should everything go into revision control? Or get stored on a server file share? What about just keeping it on the video editing machine? The decision could have serious implications to team productivity, server space management, complaints from the IT department, automated builds and testing, and so on. So I want to provide some simple rules to help decide where data can and should go.

I originally was going to talk just about Perforce but in starting to write this article I realized I needed to back up a bit and talk about how we decide what even goes in revision control in the first place. In later posts I’ll talk about the particular environment we have at Loose Cannon. Specific types of data that flows around our network and how we organize it.

## December 9, 2008

In this upcoming series of posts, I’m going to catalog most of the types of data stores we use at Loose Cannon, along with features that would make you want to choose one versus another.

Sometimes the lines are blurred a bit, because data stores continue to add interesting features each year. For example, modern wikis are starting to get some decent revision control features for binary data. And some revision control systems such as Subversion have the ability to instrument the file system with metadata, like a database.

Sometimes tools can bridge the gap among these data stores as well. For example, a file system has no ability to notify via email of changes, but you can pretty easily write a file scanner that watches for changes and emails team members based on a spec. Overall, though, while tools like these are useful and can enhance the underlying data store, they can’t really change its basic nature. So I’ll be reviewing each with that in mind.

This overview of data stores is probably going to be very basic for most, and you could even question why I’m bothering with such obvious stuff. Well, that’s what Ctrl-Yawn-W is for. But this is how I think, and so I can’t help it. I always start with the basic foundation and work my way up to the top. It’s slow, but regularly revisiting old assumptions is critical. As I write these upcoming articles I’ll probably end up reconsidering some of the things I do now and will make notes for changes to the next version of the tool chain for late 2009.

Anyway, back to the post. What we’re trying to do is answer the question “where does this go?” We’ll start with the first, most obvious option.

### The Local Workstation Hard Drive

This is the fastest and easiest to use data store. Everyone stores at least *some* of their data on their local workstation’s hard drive.

For files that need to be “shared” with other people, you can copy or email files around to other people you work with. There are services that make it easier to do this for distributed groups (Groove comes to mind), but at that point we’re outside the scope of local storage and into the realm of services. Let’s stick with local.

### What Gets Stored Locally?

I see local storage used a lot with spreadsheets, concept art, in-progress design docs, audio samples, task lists, quickie test projects and scripts, and so on. Private and secure data is often stored locally as well: budgets with salary information, employee reviews, private emails, and so on. I myself keep loads of private notes, tasks, and research progress in OneNote and, occasionally, email (in recent years I’ve found email to be increasingly painful and hardly use it much for work).

Ownership and management of local data is clear and simple: you own the files and you organize them however you want! Nobody messes with them unless you explicitly share a location out and, even then, you control the permissions. You can send updates to whomever you want when you want. Operating system shells are pretty good at making this as simple and powerful as possible. You can add metadata and tags, create virtual search folders, organize using apps like Picasa and iTunes, and so on. It’s great!

Those same nice shells often store irritating hidden files like Thumbs.db, Picasa.ini, .DS\_Store, and so on – files that are not meant to be shared and often clutter up shared data stores. We have a special trigger in Perforce to prevent people checking in these files by accident (there’s also .user, .bak, .~, .svn\\* and a hundred other siblings in this temporary family).

### Local Storage Can Hurt

All of this lovely freedom can make things easy for you and a freaky mess for everyone else on the team.

Say someone leaves the company. Now you have a computer filled with files and nobody knows where anything is. Their desktop is invariably going to be chaotic, with files like “sword\_large\_02.psd”, “sword-marketing2008.psd”, “Copy of” files and “New Folder (2)” folders and so on. And more files stored in C:\ and My Documents and other seemingly random locations, maybe even burned to piled-up CD’s. I’ve known too many people that work like this. They run their hard drives down to 0 bytes free and then just delete random stuff to free up space. It’s Concentrated Crazy! Yet they still manage to kick out final PNG’s to the right spot in the Perforce depot, so nobody notices the mess until it’s too late.

At Sierra we had a policy of burning to CD the full hard drive of anyone who left the team. With the turnover we had on Gabriel Knight 3, of course, that gave us a mighty big stack of CD’s. Once or twice some poor intern had to go through it to (fail to) find this or that odd piece of marketing art we needed. I think all the CD’s ended up in an offsite vault eventually. All that effort archiving, storing, and searching that mess was such a ridiculous waste of resources.

Beyond the hopefully rare “developer leaves team” story, local files are simply the opposite of good communication. They’re by definition private. Even if you share a folder out, it’s still your machine. If you want to send an update of a file to someone you have to send an email or copy it to a share and IM them about it. And that’s private too, so you have to know who might need to know about these updates. This also assumes you even remember to do the update, and send the right file, and all that. I’ve lost track of the number of times I’ve gotten emails from remote workers, who attached the wrong version of the file. Oops! Sorry about that dude, let me re-send!

It’s a lot of seemingly optional responsibility to attach to a person who is probably too busy to keep it organized. At least, organized as much as the rest of the team might need.

### Don’t Mess With the Crazy

I used to strongly believe that if someone leaves the team you should be able to flatten their machine and give it to someone else without worrying about losing important data. Nobody should have to decode the Crazy. So of course I was often frustrated by people on my team who insisted on working this way.

Well, I still believe this, I’ve just given into reality. Apps are written with the assumption of either working locally, or working through some kind of custom sharing service (like Office + SharePoint and even that isn’t done very well). It’s an uphill battle to try to enforce a structure and process on what amounts to an inherently personalized and optimized development experience. So many things are just *easier* locally. Like working with folders! You can rename them, or move them wherever you like, instantly. In contrast, doing the same through source control is a tedious pain.

There’s nothing *necessarily* wrong with using local storage for ad-hoc, brainstorm-y, temporary, or private work. The trick is knowing when to move things off the local drive to another, more team-friendly data store. One where people can subscribe to changes made, get their own copy, get a history, and so on.

So how to know where it goes and when to start maintaining it there instead? In writing this I’m trying to figure out some basic rules for what can stay local and what should get promoted. But it depends so much on the team, discipline, type of data, and so on. Some things are easy and absolute: all source required to build the game and its content go into source control. Some things are more gray: where does that intermediate concept art go? Where do those little prototype gameplay modules go?

For the easy ones, look to the upcoming articles. For the more gray issues, I’m going to fall back on the “make sure it’s backed up” solution. Hope for the best, and if things go bad, go to backup for recovery. But we want to avoid the expensive problem of backing everything up and having a big disorganized mess of Crazy, right?

The solution to that I think is in a detour article I’m posting next on backup options. One great way to have your team members separate out the signal from the noise on their hard drive is through a *partial* backup solution. More on this next!

## January 6, 2009

I previously posted about using the local workstation as a data store. I concluded that local storage for some important data is an inevitability, and that’s ok as long as it’s backed up.

And I have yet to work at a studio where more than a few workstations had a formal local backup process in place. I believe this is mostly due to how rarely we lose work to catastrophic failures. There’s usually plenty of advance warning from a hard drive before it barfs, giving you time to move over to a new drive. If it ain’t broke often, no reason to fix it in advance. Even in the case of a total sudden failure, it only takes a day or two of re-imaging the OS and reinstalling apps to get back to being productive.

I tend to agree. I don’t think the occasional hard drive failure is worth a *general* studio-wide full workstation backup policy. It’s just too much work, storage, and time, and is going to cost far more than it will save. Instead, let’s have a more targeted approach. For most people in the studio, I prefer one or more of the options I’m going to enumerate below.

### Local-Based Backups

This is the method I use. Even though I just said it’s not a huge deal to have a total failure, some people (like me) are ultra paranoid about losing even the smallest amount of work or time. And for that, nothing beats a local full-drive USB backup.

### How?

Big fat USB drives are cheap. You can trade size and price for the performance you don’t need. I’d also get one around twice the size of the main workstation drive array if possible. That way two or three backup rotations plus incrementals can be kept going at once (assuming backup compression).

For software I’d get something that uses the shadow volume service to do copies so it can pick up the in-use files. I particularly like Ghost even though I loathe most things Symantec makes. It’s important to set up frequent (two to four times a day) incremental backups. Set them to low priority to avoid bothering you while you work. And set up automatic replacement of old backups when the drive is full. The system has to be fully automated and easy to use so you can forget about it until you need it when something goes wrong. Ghost even has a nice feature to let you use Google Desktop to search your various backups, a little reminiscent of Apple’s Time Machine.

I’d pick this over a mirrored RAID setup any day because you get incremental backups, and you get an easy way to restore data if your workstation dies and you need to move to a new machine quickly.

### But Why?

Why am I so paranoid about this? A couple reasons. The less important one actually is the total loss issue that I mentioned earlier. The *main* reason I have frequent incremental full machine backups going is that I frequently make incremental boneheaded mistakes in my work.

The most common scenario is where I’m working on some small task, and in the middle it explodes into a giant task. Before I know it I’m four days in with 50 files checked out and *crap* I just accidentally reverted a bunch because I thought I was working on my other client for that quick bug fix someone needed. Or a day later I confidently go down some new mental path and redo a bunch of work without realizing that this is the *wrong* way, and now I need to undo back to the previous day. This is especially useful when coding a little drunk after lunch. But if the incremental is only half a day old at worst, you’re not in too much trouble.

This problem is so common for me that I’ve started building some tools to fix it on top of the incremental backup. Right now I’m eyeing git for ideas because it has features like stash and bisect. I’m also considering adding on a continuous backup system (like Norton GoBack or perhaps I’ll mess with Previous Versions a bit more). Like I said, I’m paranoid.

Incidentally, this would be a lot less of a problem for me if P4 had a local repository concept like git and mercurial and other more more modern source control systems have. Then I could check in as often as I wanted for backups and revision history as I worked, and get all my favorite source control tools to help develop.

The P4 way to fake local repositories is to do private branches, but on a large project it’s surprisingly time-consuming to manage. Perforce does not provide any tools to help out with this so I’ll have to roll my own here as well. I’ll probably write about this in a future post. It’s getting increasingly frustrating working with P4. Especially considering its obnoxious price and how much time I’ve put into building my own tools for it.

### Avoid the Network

For full-drive backups, I’d stick with a nice simple local USB drive instead of a fancy server-based backup over the network. Doing it from a master server means you have to deal with the IT department every time you want a file back, and the network will get saturated at unpredictable times. This a lot of the benefit of using this system in the first place.

Now, the IT department should at least remotely monitor all of the workstations using this method to make sure that backups are working properly. Someone could easily have a USB hub failure or accidentally kick the connector out and not necessarily notice that backups were failing. People can’t be trusted to pay attention to balloon popups in the tray, they’ll just ignore them forever!

But like I said, this method is for the truly paranoid. Not many of those on the team, and so most can use the next method: server-based backups.

### Server-Based Backups

Ok so you’ve got a tape backup and you’re not afraid to use it. You want to roll some local data into the server based backup. But you don’t want to overwhelm the network and your tape storage with enormous full drive backups. Here are some other options. They aren’t mutually exclusive, either.

### User Folders on a Server Share

Give each user a public and/or private folder on the server (where the private folder has permissions only for them), and map it to a local drive or folder using a domain login script, perhaps P: for public and X: for private. Tell people to keep, copy, or sync files there that they want backed up. Simple, and easy to manage.

There are a couple big problems with this system. First is that it puts the burden on the user. They have to manage their files that are going to be backed up or not. So people may forget or not want to bother with this. Or once they start they may not want to keep it up and then you have outdated files sitting in the store (though this can have its own slight advantages). The other problem is that people tend to get lazy and copy large folder trees over (temporary .obj’s and so on included) and not worry about size or duplication much.

Ultimately certain types of users will use the system effectively for local workstation backup and others will not. It’s partly a matter of education but also a reason why (ideally) we want to provide multiple options. But overall this is the main (and usually only) method of local data backup I’ve seen in use at studios I’ve worked at.

### Remapped User Folders to the Server

This is a variation on the public/private folder idea. Windows lets you remap local folders (such as Documents, Images, Contacts, etc.) to a remote server share. You can just right-click on the folder, go to Properties, and change the folder’s Location to point wherever you want, like a private share on the server, set up per-user.

This is the method we use at home, actually. Ally mapped her Documents folder to our ReadyNAS in the closet. She doesn’t have to do anything special, and her stuff is mirrored, snapshotted every two hours, and backed up offsite. Works great. You can also manually set up NTFS junction points if the OS shell doesn’t permit remapping a particular folder (such as the desktop).

The down side of course is that *all these files are mapped directly over the network*. On our home network, which is very fast over the wire and only has two people using it, it’s not noticeable. But at a game studio with typical minimal IT investment and lots of people simultaneously working with large files, this could be a big problem. Another possible problem is that if the server goes down, people can’t work with these files locally because copies are not kept locally.

Offline files are a potential option here that I haven’t explored much. I haven’t heard good things about the sync performance, but it’s only anecdotal. Also, it sounds like you need Vista to make it work well, and our industry is really dragging its feet on moving to Vista from XP (sounds like we’re all just going to skip it and jump straight to Windows 7). Other options like rsync are available but we might as well use one of the other options below instead.

### Server Pull of Specific Folders

This option is where the backup server periodically reaches into each workstation and backs up files from a standard set of folders. Perhaps the Desktop and Documents folders, or maybe from a special “Backed Up” folder kept on the desktop of C:\ of each machine (or all of the above). With this option people continue to work with local files so performance for them is high.

As with the Remapped User Folders option, educate the team about which folders on their machine get backed up. Then they will know that anything they copy to the desktop or save to their documents folder or whatever (which is the default on so many programs) will be safe.

Now, the big downside with this option is that the server has to back up a lot of files through a relatively slow network. It’s more targeted than a full drive backup but still is going to be quite a lot of data. With all of the other options, where the primary data is kept on the server, the overall network cost is minimal because users are accessing files on demand. The server can back up things locally through ultra high speed links in the server room. But with server-pull from workstations, the server must access every file that is a candidate for backup over the studio’s more ordinary network, which takes a lot longer.

Also, it must deal with individual workstation problems. Backups are typically a serial process, so if one workstation is having issues (perhaps it’s running super slow due to an overnight render or batch process it’s doing) then the whole system is bottlenecked.

### Exclusions

With all of the above methods, we will have a big problem with users dumping things in their Documents folder that they really don’t need backing up such as movie trailers, music, game demos and so on. You can nag people but it’s easier to tell the server to exclude large junk file types like exe, pdb, mov, avi, mp3, m4a, iso, and so on from any of the local workstation data it backs up.

### Roaming profiles

A quick note on roaming profiles. *Don’t use them.*

Roaming profiles keep the user’s profile on the server. This includes their entire Users\Username folder minus local-only settings and temp and cache files. The profile is synced with the server on user login and logout. We used these at Loose Cannon before I pushed hard to get them killed.

Roaming profiles are interesting in theory, but in practice, they are a terrible, awful idea. I admit, it’s tempting to have a system that shares all your settings so that you can log into any machine on the domain and have the same setup. You can log into Bob’s machine and all your Visual Studio keyboard shortcuts come with you. Doesn’t that sound nice? But the reality is that sync is ultra slow and makes logging in/out or rebooting frustrating beyond belief. Why is it slow? Well, profiles tend to get really amazingly huge. *Everything* goes into the user’s profile. And if it hurts to log out/in then it will be even harder to get people to keep their systems patched, which is already a big problem.

Roaming profiles are bad enough with the latest versions of Vista and Windows Server. But they’re even worse on a Linux-managed “domain” (Samba), which is what we had way at Loose Cannon way back. Any bugs or compatibility problems on the way the sync happens get propagated down to your local machine, potentially damaging your profile forever. Our Linux-based server was apparently incompatible with Vista and I got my roaming profile horribly mangled a couple of times until I just forced it to go local. Since then, we’ve been on a nice, easy to admin Windows Server with local profiles only and haven’t looked back.

We may have been able to get roaming profiles working right after a lot of work and tuning, but is it really worth the trouble? Is it really all that useful to log into someone else’s machine and have all your settings be the same? Nope! Depends on some studio workflow policies (pair programming, standards on supported editors, etc.) but I still say no! Windows is simply not any good at this, and gets worse at it every year. Perhaps with a massive investment of custom tools, but…bah. Better things to work on.

Ok let’s move on. Next up: back to the series. Data Store 2: Server Shares.

## February 8, 2009

This has been on my mind a lot lately so I wanted to take a detour and write it up.

Ally and I decided to move to Peru to live for six months (her blog talks about who/what/when/where/why/how). I would have preferred a year but Ally’s got to go to school in the fall, so six months it is.

My boss Matt at Loose Cannon is super awesome and instead of firing my ass he decided to try to make it work. He gave me a very long term mostly research and prototyping project to create our next gen animation system. Intimidating as hell but cool. That means I don’t absolutely *need* to be online all the time.

Matt does want me to continue doing code reviews on our game if possible. This requires a fast turnaround to avoid blocking checkins for too long, especially as we get close to shipping. So it’s preferable if I can be online during normal work hours or at least a window of time during each weekday. Peru is 3 hours ahead of Seattle so that means I can go see pyramids in the morning and still get online and working by normal Seattle core office hours.

Anyway, I’ll be working remotely for six months. We’ve already passed through four hostels in three towns, and have finally gotten a real apartment in the comfy Yanahuara district of gorgeous Arequipa. We’ll do extended-weekend trips every couple weeks (which means working on buses to/from) and then move up to Trujillo in a few months. Lots of mobility required and it can’t interfere with my work. I really do have to keep a regular schedule.

This brings up several super important requirements, which I’ve spent a lot of time thinking about and working on.

1. Data and hardware must both be secure. If I can’t work because my laptop was stolen or I had a hard drive crash, then I lose a lot of time.
	* Our personal data must also be secure. As we’re managing all of our finances online now, we can’t afford passwords and credit card info and such to get out.
	* I also have to figure out a way to work with Ally’s, um, let’s say “lack of interest” in security.
2. I need at least periodic access to the internet so I can check in at work through the VPN. This means stealing wi-fi, buying cell-based broadband, plugging in direct, or using sketchy public computers at locutorios.
3. For longer term, I need a comfortable work environment. It’s still 40-50 hours a week, remote or not. Can’t work hunched over every day at the carpal tunnel festival. But I’m not bringing a whole office with me, it’s got to be small and light.

## November 15, 2009

Well obviously it’s been really quiet here lately. A lot has been going on in my personal life, but the end is in sight. One of the things I’ve been doing is working on a talk for Audiokinetic for their 2009 Wwise Tour. I presented it over at Microsoft last week as part of a tag-team with Robert Ridihalgh of OMNI Interactive (he was the principle audio engineer on Tornado Outbreak).

Oh, did I forget to post on here that our game shipped? Wow, I totally did! Well, we shipped in September on 360, PS3, and Wii. Become a fan on Facebook!

The reviews have generally been in the B range, though Metacritic’s average is a little depressing at ~70. Nearly every review uses the words “Katamari” and “clone” which makes me think they didn’t really give the game a chance. The similarities with Katamari begin and end at the concept of “grow bigger”. Might as well call Call of Duty a Doom clone because they both involve shooting things to get to the end of the level.

But whatever… I’m proud of our effort, and I think it’s a super fun game. Hell of a job for a small team with a tech base built from scratch, to ship on time and on budget on three platforms.

### Wwise Tour

One of the reasons we were able to ship on time was Audiokinetic’s Wwise. I’ve been evangelizing this excellent sound engine to everybody I meet. I just can’t say enough good things about our friends up in Montreal. I’d use Wwise on every future game if I could.

On Tornado Outbreak, I did most of the engineering and the initial audio rig design and prototyping. Robert and his team did the actual audio work, and took over management of the Wwise project. They probably did 95% of the audio related work on the project, which is awesome! As everybody knows, engineers are really slow and are pulled in 20 directions at once, so the more I could step out of the way, the better.

Audiokinetic asked me to put together a talk for the tour they’re doing right now to promote the product, particularly the new features they’ve been adding. I invited Robert to join me and we presented the two halves of our audio solution for Tornado Outbreak. We split the presentation roughly along the lines of our responsibilities.

The event was recorded, so at some point we may see video clips showing up online. That will be necessary to get Robert’s part of the talk, because he was exclusively walking through the Wwise project, using elements of it to tell his story. So no slides, you’ll have to get the video if it comes out. Anyone using or considering Wwise should try to get a hold of that – his talk was really interesting and includes some great tricks on saving memory without sacrificing variety.

Now, all I know how to do is present slides, so here they are! Tornado Outbreak + Wwise = Love.

### Our Project

I also got approval from the bosses to release the Tornado Outbreak project file for Wwise. This is really generous of them to agree! Here it is.
Note that this doesn’t include any content (wav files) but is only the project itself. That should be plenty though.

The point of releasing the project is to help out other studios who are integrating Wwise, in hopes that the favor will be returned. Everybody benefits from information sharing like this. With Wwise, in order to get a good rig set up you really need to have experience and good examples to draw from. As I say in my presentation, *Wwise is different*. Every other SDK out there is a “play samples + DSP” library. Getting the Wwise rig right is hard, and it’s not going to be right the first time. Just as if you were to build a Maya rig and had no experience with it before. You’ll screw it up for sure.

Audiokinetic provides some synthetic examples to help get started, but it’s not from a real, shipping game, and besides, every game is different. Tornado Outbreak has an enormous amount of unique objects that produce audio (over 400) and all those crashes and shakes and panics can become very difficult to manage. It would have saved a lot of time if, when I started building the initial rig, I had some examples to draw from. To that end, we’re releasing our particular solution for this situation. I hope that it inspires even better solutions and ideas on how to tackle these kinds of problems in the future.