it's the "good problem". and the good problem can always be solved, though lack of reliability can seriously hurt your growth. but that doesn't mean that it's as easy as people make it out to be.

i wrote and ran a big php app (Zynga's Mafia Wars). this is just about the app server (apache + php side), single-master MySQL will give you far greater problems at that scale.

everyone is tearing away adding features, and the code just keeps getting bigger. everything includes everything else. so you can't just start fresh with every request, it takes way too long. so you use APC or whatever. now you have global shared state (and people start using it), but as long as no one really fucks it up that's ok.

php processes are always erring and dying in weird ways that no one can figure out, especially if anyone uses the half-baked OO stuff (circa 2008-9), and tracking down obscure php state errors ranges from difficult to impossible since you're never quite sure when or where something went wrong, but it works for most people most the time, as long as you remember to clear out the error logs so they don't get big enough to crash the box.

memory use still sucks at around 30M a process, meaning you can only have around 500 workers on at 16G RAM server, but it's workable. at least it's not rails.

because it takes a while to talk to even Memcache (go forbid you talk to MySQL) and do all your silly user updates (games are really write-heavy) you are taking a few hundred ms per request, so 16G mem servers (it's 2008) safely top out at around 500-600 req/sec, and i think would melt close to 1000 or so. remember, your throwing everything you have at growth and features so you're not really optimizing shit.

cool, just need more hardware. which is no problem. if you can get hardware. when you're co-located it takes forever to get new hardware online. not really an option. and even the F5 load balancers can only take so much. and there are issue with getting too many app servers talking to too many memcache servers. or something like that. after i left i think they saturated the data drops, which really puts you in a bind.

but that's not a problem now, with cloud services, right? except that those are not limitless either. i think FarmVille ran into a wall where AWS wouldn't / couldn't supply more instances (again, this was around 2009-10 so prob a lot better now). and those are still physical machines in physical datacenters somewhere that will have all the same problems and limitations and then some.

but you're going to have problems with any app runtime at that scale. and php's performance was an order-to-two or magnitude better than rails, and we never figured out how to get significantly complex and traffic-heavy java apps to stop leaking memory. though i'm sure you can. and you will! like i said, it's all solvable. hell, even MySpace didn't fail because of tech infrastructure.

what i'm saying is that php wasn't (and maybe still isn't?) particularly fun when throwing hardware at it doesn't work, and you end up losing a lot of the purity / simplicity of the model because it's too damn expensive. ffs, Facebook re-wrote the runtime. that's what it took for them. and my friends there tell me it's pretty much like writing java at this point. it's a fine choice, but i'd rather not use it again. had a much better time with python before javascript ate the world.