I'm currently working on a project that hasn't died yet, but has gone through a lot of pain in the last couple years partly because of technical debt.

I ran the project for the first four years. It was my first large project and the first time I had run a team of any size. I made a few mistakes that might be worth learning from, but my mistakes weren't the only ones responsible for the debt.

The largest driver in the technical debt issue was the timelines. My boss was new to software development, and his expectations were not in line with reality. We frequently had to ship ad hoc features under tight deadlines to keep him happy. When I pushed back on the timelines he became unhappy. So we acquired debt in the form of 1) hastily designed sections of code that don't lend themselves to scalability or refactoring 2) a lot of small code smell issues that by themselves don't amount to much but in the aggregate form a kind of surface scum that makes future development and more importantly testing difficult and thus brittle.

The debt, and the accompanying bugs and delays it caused, eventually led to enough dissatisfaction that I was taken off the project (I am in a weird outside position now, sort of half on half off, supporting a tangent of the software but not working on the main branch or involved in architecture discussions, planning, or execution of the new replacement). A new manager was hired. We disagreed over strategy, so I was taken off the project.

He wanted to start over from scratch, which is what they are essentially doing now. The plan is to pattern the new solution off the old one by incorporating all the business rules (which they want me to document), but they have very different implementations in mind. They don't want to reuse existing libraries (some of which is driven by a lack of familiarity or understanding of those libraries, how they work, why they approached the problem the way they did).

They face some significant challenges:

1) their coding velocity is slow. more than 60% of the original team has left and been replaced, so a large part of the team hasn't been on the project for more than 2-3 months. this means that there is a significant dearth of institutional knowledge. i think a decent pace is good for development of software, but they aren't starting from scratch, and there's a lot of expectations to meet.

2) because of the dearth of institutional knowledge when they do start implementation, they are going to end up repeating many of the mistakes made by the old team. I have limited insight into what those are, but based on what exposure i do have, i can see planned missteps all ready.

3) my boss, the owner of the company, has learned some patience, but they've been at the rewrite for nearly 6 months and have little to show for it. in terms of feature parity with the old software, they are severely lacking. i don't expect he is going to be willing to wait another full year to get an app that does essentially what he already has only differently, even if that architecture is more flexible.

In all honesty, I hope they succeed. My boss is a good friend and this experience hasn't ruined that. I have and am dealing with some resentments, but I don't want him to fail. And because this project was my baby (so to speak) there is a part of me that wants it to live.

----------------------

In retrospect, I'm not entirely sure what I should have done differently. Had I ignored the requests for adhoc features, I likely wouldn't have made it as far as i did, because without those features he would have pulled the plug. The company grew at an exponential rate the first 2 to 3 years of that project, and part of what fueled that growth was the adhoc, fast turn around times of my dev team. We were incurring debt, but we were also making big gains.

If I was going to do this all over again here is what I would do:

1) I would push back more, in smaller increments, placing more emphasis on getting the details right before shipping.
2) I would push back more on requirement creep. My boss would frequently have these "great" ideas that he would insist we work on, which would get half done then discarded leaving the code base littered with dead ends that needed cleaning up later. Part of the debt we acquired stemmed from the fact that in order to keep a lot of those activities from impacting ongoing efforts, i built an architecture that was somewhat disjointed. the lots of little islands approach meant we had apps that were reinventing the wheel, taking different approaches, etc...
3) Place a greater emphasis on automated testing (fewer human testors, more test engineers).

The other aspect to this is the fact that when you are creating software to solve problems that don't currently have software solutions, you spend a fair amount of time going down trails that don't pan out. The R&D aspect left us with bit and pieces of code in the code base that were incomplete or incompatible, but that were relied upon by one app or another.

We lost time discovering that certain approaches didn't work.

I think thats it... I hope someone finds the story useful.