The real shift occurred in the 90s when computer-aided design techniques really took off and exotic lens designs (ultra-low dispersion glasses and aspheric surfaces) became affordable to use in mass production. It suddenly became viable to design lenses with many more elements and more exotic elements as well.

A necessary condition was the rise of multi-coating, as each air-to-glass interface increases the amount of reflection and degrades the image. Pentax developed their "SMC" (super-multi-coating) process in the 60s, and licensed it to Hasselblad as the T* coating (in exchange for a license to the Distagon design produced as the K28/2 and P67 55/4 designs). A few other high-end companies like Nikon independently developed their own multi-coatings, but most of the industry muddled along with single-coating until Pentax's patents expired, then immediately copied their process. So the explosion in lens complexity in the 90s was also partially driven by improvements in coatings that made it possible to minimize reflections and loss of contrast in lenses with high element counts (8-15 elements).

(this is why most older lenses were Triplets or Tessar formulas in the 30s and 40s... These lens formulas minimize the number of air-glass interfaces while providing a sufficient degree of sharpness. Modern designs like the Planar or Plasmat have been known for 100+ years (Planar was developed in 1896), but in those days the large number of air-glass interfaces would significantly degrade the contrast of these lenses. The development of single coatings prior to and during World War II drove an increase in lens elements during the 40s and 50s as well, to around 5-7 elements in high-end lenses, which is when you started seeing designs like the Plasmat really take off, like the Symmar or Sironar, and a shift to Planar in consumer cameras, eg Super Takumar or Nikkor-S 50/1.4 types.)

The other thing worth mentioning is that people got fond of zoom lenses in the 90s and 2000s. It is much harder to design a lens that is corrected across a whole range of focal lengths, meaning you need a larger number of elements (and more weight). Meanwhile, people who shoot prime lenses got fond of super-fast apertures, which also require more elements to make.

If you go back to an old-style f/3.5 or f/2.8 prime lens, there are some very lightweight modern designs available (eg Sigma DN 30/2.8 and 19/2.8). If you want a modern 24-105mm f/4 superzoom or a 35/1.4 superfast lens with 12 elements in it, you'll pay for it in weight. People want the faster apertures and wider zooms so lenses are getting heavier.

Those slower lenses are also much easier to correct, so they tend to be sharper than fast lenses are, unless the fast lens is much better corrected. In other words, you are better off shooting a f/2.8 prime at f/2.8 than an f/1.4 prime at f/2.8, in most cases (unless the f/1.4 lens has ~3x the number of elements). Older superfast lenses (eg Nikkor Ai-S 35/1.4) with spherical designs tend not to be that great.

Oh, yeah, I'm sure modern manufacturing also plays a role. Aspheric elements were first used around the 60s, the first I'm aware of being the Kilfitt Makro-Kilar 90/2.8 design, but were extremely expensive to manufacture. Typically you would produce them via single-point diamond turning, essentially using a diamond point on a lathe to produce a non-spherical curve on the glass. CAD/CAM manufacturing strikes again, these designs probably got a lot easier to produce in the 90s and 2000s, both from single-point turning on machine-assisted tools, as well as molded plastic lenses of useful composition.

So, really a convergence of various design and manufacturing improvements over the last 30 years, that made really high-end lenses viable to offer at prosumer-level price ranges.