Performance is a feature, it doesn't get retrofitted. There is only one shot, specially in fixed budget projects.

So you’re saying it’s not possible to add indexes, increase the size of your database, increase the number of read replicas, increase the number of servers in your web farm, reconfigure your database to be multi-master, copy your static assets to a region closer to the customer or add a CDN after an implementation? I must be imagining things that I’ve been doing with AWS....

To calculate the initial set of VMs, database size, average users, network latency, you name it, it only requires reading the RFP requirements, having technical meetings with all partners about those requirements, and having a team that knows their stuff around CS.

So “knowing CS” would have helped us predict at one of my previous companies that our customer was going to more than double in size in less than a year through an acquisition? In fact this has happened at two separate companies. The other company we more than doubled in size and revenue literally overnight.

Will “good CS design” help us predict how successful our sales team will be in closing deals? We are a SAAS B2B company where one “customer” or new implementation from a current customer can increase our volume of transactions by enough to have to increase the number of app servers or with enough implementations increase the size of our database cluster.

If it already clear from deployment scenario that at very least 4 VMs will be needed, or that a DB node will need 100 GB on average, it would be very risky just to do on the go.

So now it’s “risky” to click on a button and increase the size of our web farm by increasing the desired number of servers in our autoscaling group or is it risky to click on another button and increase the size of the VMs in our database cluster? The number of app servers we have for one process goes from 1 to 20 automatically based on the number of messages in the queue. As far as storage space, if we need a terabyte as our client base grows instead of 100GB, I’m sure AWS has some spare hard drives laying around that they can give us. But transparently adding space to a SAN even on prem has been a solved problem for a long time. Even back at a previous company where we would boast to our client that we had a whole terabyte of storage space.

As for running at scale, that should already be obvious from RFP requirements, unless we are speaking about startups dreaming of being the next FANNG

Again, you don’t have to dream of being the “next FAANG”. Mergers and acquisitions happen. Getting new clients happen (hopefully). When you are a B2B company especially when you are a SAAS B2B company with a decent sales team, a sale to a couple of “whales” can mean adding more of everything.

Also the RFP is not going to tell you that your company that you are planning on implementing and hosting a solution for x users will need to be able to scale to handle 2x in a year after a merger closes. Should we have 5 or 10x the capacity now in anticipation for our sales team producing or should we scale up as needed?

MongoDB is a very good example of running at scale without doing the necessary engineering, but they do have a good marketing department to compensate for it.

I had no problem with the scalability of Mongo at a previous company. What type of scale do you think in your experience is too much for Mongo?