I was 18 years old, and working at my first software development job. These were the waning days of dumb terminals, leased lines, and modems. We had no "test environment" - test was dev, test was production.

The system was an RS6000 minicomputer (actually, it was like an overgrown PC) running AIX, and I was remotely logged into it over a 9600 baud leased line, direct to the client. It was their production system for Arizona's indigent healthcare system - AHCCCS. My employer was a mom-n-pop shop that developed the system, in PICK (we used UniVerse on AIX).

All I recall today was that I was testing some kind of data conversion routine (or something of that nature) on the database for the transaction data; that is, the claims transactions. These were input via hand-entry and/or scanned document (usually a combination). My testing consisted of running the routine (which read from live data, but wrote to "test" database records), checking the output, clearing the test records, tweaking the code, then re-running the routine again - until it was right. You can see where this is going...

So there I was, doing a repetitious task, getting into a mindless rhythm - when disaster struck: I cleared the production tables. Oops.

I held my head, and let out a loud moan. People looked up from their desks. Before they could ask me what was wrong, all of the phones in the office lit up. Every single phone was ringing - because of me.

My supervisor and I were called into the owner's office, and asked what had happened. My head was hung, I explained the issue, he asked my supervisor why - I thought heads were about to roll, me being a kid, my super (thinking back to then) not being that much older (late 20s). The owner asked my supervisor what could be done; he thought quickly, realized that tape backups from the night before had been made, and that the table I blew away could be -mostly- reconstructed from two other tables I hadn't touched. A bit of fast coding, a restore from backup, and most of the data could be restored by morning, leaving their data entry people to re-enter a few hours worth from paper records.

Amazingly, we weren't fired on the spot, but told to get to work, and make it quick. We pair coded at that point, built and tested the conversion routines meticulously on our development box, then moved them to production to run. We backed up the remaining data, and ran the conversion on a test batch, then combed thru the output to make sure it looked right. We ended up staying thru the night to the next morning, but it got done, and our company continued on with the client. They were one of the bigger insurance companies in the state at the time; I have no idea how much money was lost due to my slip - but it wasn't insignificant, I'm certain.

I learned valuable lessons from that experience - the importance of backups (extremely recent ones!), the need for a testing area, and to never use production for testing, making sure to check twice before pressing the return key on a "delete everything" command, plus how to be fair and calm in the face of disaster, toward a very junior member of a team (that one from the owner).

Let's just say I haven't repeated that mistake in the 20+ years since; looking back on it now as an older and more experienced developer, it was definitely one of those "why would we fire you now? You've just been taught a very important lesson for a lot of money, which you'll never repeat again!" kind of moments.

That was my first software development job; the first that launched my career only 6 months out of high school with no degree. It was a great place to be to learn many lessons which have remained with me since.