If you weren't following the development of "new space" then SpaceX's achievements would seem to come out of nowhere for seemingly no reason, but that's not the case, there was a lot of buildup.

The main reason for "why now" is the end of the Cold War. During the Cold War there were geopolitical military implications to building launch vehicles, and there was a huge brain drain (or brain sink if you like) of aerospace talent into military projects.

In the 1990s there was a very substantial boom in "new space" companies (orbital launch startups). A lot of that growth was fueled by the growing venture capital from the dot-com boom, however in many ways this was a very flawed boom that did more to hurt new space than help it. For one, much of it was based on a then emerging launch market for low-orbit comsat constellations such as Iridium, Globalstar, and Teledesic (which, of course, were in turn vastly fueled by the tech boom).

More importantly though the heavy influence of VC firms in providing investment funding for many new space companies twisted the industry in weird ways. The way VC works in tech based startups is often to seek out highly disruptive companies that are both high risk but that have a high chance of an enormous return if they succeed. This doesn't work as well with companies founded on physical engineering and it didn't help much for launch vehicle development. A lot of the early new space companies were founded on highly innovative and unconventional launch vehicle designs (hybrid jet/rocket planes, rocket helicopters, RLV SSTOs, etc.) which were at best several generations ahead of the current state of the art (and at worst highly impractical or unworkable). When the dot-com bubble burst the comsat constellation companies did as well, bringing down many new space startups with them.

However, out of those ashes a new generation of launch startups grew. These new startups didn't have the luxury of fantastical potential revenue streams from a comsat boom, they had to build their businesses on hard-nosed accounting and either more proven or more likely revenue sources such as traditional launch business (still a very big market) or space tourism. This spurred companies to be more practical in their engineering choices, to use iterative development practices, and to spend more of their efforts in competing with traditional launch providers. In combination with the X-Prize (which had been created in 1996, in the midst of the first generation of the new space boom) this helped create the current crop of space startups we know and love today. Companies like SpaceX who built a very traditional expendable rocket but did so using a competitive spirit in terms of expense and complexity not seen with most of the launch vehicles built via multi-billion dollar cost-plus government contracts. Companies like XCOR and Virgin Galactic/Scaled who are paving the way in sub-orbital space tourism with very elegant and simplified reusable rocket ships designed for simplicity and robustness of operations. Companies like Blue Origin who took up an enormously promising reusable launch vehicle design that the government abandoned (Delta Clipper), or Bigelow who took up an enormously promising space station design also abandoned by the government (TransHab). Companies like Masten and Armadillo showing that you don't have to be a billion dollar aerospace megacorp to do good work or blaze new trails in rocketry.

Despite a late and rocky start, the startup/barnstormer phase of manned and unmanned rocketry has finally begun in earnest. This phase propelled the design of the airplane from experimental and impractical craft like the Wright flyer to advance within 2 decades to the industrial, commercial, and military workhorses made of metal, able to carry a dozen passengers, and fly across or between continents that were built by the dozens or hundreds in the 1910s and '20s. And then a decade after that to craft like the DC-3 and the Spitfire. Launch vehicle development may not advance at quite that pace but it is nevertheless an enormously exciting time filled with a great deal of promise for anyone interested in spaceflight.