>I don't know all the history of Japan, of course, but I assume there some chicken-egg situation. If there is better transit, it'll encourage people to choose greater density

No, the history is exactly the same as those walkable European cities. The cities were built and grew (usually organically) long before the automobile was invented, or even before trains existed, back when people got around by walking and horse. Eventually, trains and subways were invented and installed in places. Cars came later, but by then the city was already there and quite large so there was no easy way to retrofit tons of parking spaces for everyone to drive everywhere. Subways weren't even built in Tokyo until well after the war, for instance; they're relatively recent. It's probably the same for most European cities.

American cities mostly grew after WWII and the popularization of the car, so they were designed for cars (outside very small downtown areas that predate the car, when the city had a small fraction of its later population).

Spending billions to build more train transit in American cities now doesn't work that well, because people still need cars to get from the train station to wherever they're going, because the density is so low. There's no real way to fix this, other than to change the policies to push for higher density, build a bunch of transit at taxpayer expense, and then wait a few decades for it to not be an economic black hole, all while voters are mad that they're paying SO much money for a transit system that has low ridership and continues to cost them dearly.

>(2) proactively build transit to suburban towns, to provide alternative transit options for commutes.

That only works well if 1) you build lots of parking at the suburban train stations for everyone to park in, and 2) everyone is generally going to a very small number of destinations (e.g. downtown). These days, though, the workplaces are usually in more suburban areas away from the city center.

>When I visited Japan, I noticed that beyond the typical subways of Tokyo, there are lots of regional trains that get to smaller cities outside of the megalopolis.

Yes, but those smaller cities themselves are generally walkable and dense too. They're not like American suburbs.

>We have commuter rail in our biggest cities (eg. NYC)

Right, because so many jobs are actually in NYC, so people need to commute there, and driving into (central) NYC is infeasible because there's no parking and the roads aren't that large. This is totally abnormal for American cities. NYC is extremely unusual. It's a very old city built mostly long before the car was invented, just like those European cities. The only other places in America like that are now just small downtown districts, such as the French Quarter in New Orleans, or the boarded-up downtown districts of dying rust belt cities like Gary, Indiana.

>Maybe I should learn Japanese first :) But a tempting proposition.

The companies recruiting foreigners usually have English as their working language. Learning some Japanese certainly helps, and makes it much easier to live daily life outside work here, but for those companies it's not a requirement.