I'm kind of in a similar position but have the option to work remote, so I will probably do it (although I haven't pulled the trigger). My takes on these issues, FWIW

1. specialized jobs. True, and you will miss out on some career options, as this is the center of the world for tech by far. But nowadays there are remote work possibilities in many areas and the lower cost of living as well as much lower taxes outside CA make up for higher salaries in the bay area (remote work jobs tend to pay a bit less).

2. "Diversity". Meh, you can find all the diversity/culture you need in any decently sized city. My closest friends have been from Eastern Europe/balkans and I find those guys wherever I go. Whatever you are into, you'll find it, but you just have to seek it out. It's like people who rave about being in a town with so many art galleries and symphonies, but they don't actually attend said galleries or symphonies. And most people can't tell the difference between the Santa Fe symphony and the SF Symphony, TBH. Sure, it's a step down, but you will get all the culture you need. In terms of tolerance, the bay area is not what I would call tolerant -- it's a pretty authoritarian place. Try wearing a MAGA hat to work and see how tolerant they are. I'm not talking about getting dirty looks, odds are good you will get physically assaulted. Having lived here for a while, I've seen the transition from a quirky place with a strong libertarian bent and that likes to experiment to one that is more of a monoculture constantly looking for something to be outraged about, and the overton window for what is tolerated is shrinking every day. Even a lot of the old style hippies which I used to hang out with have moved out, many heading to Portland or TX. Oddly, places like Palo Alto and Cupertino are much more tolerant of other worldviews. A good rule of thumb is that if you can find a neighborhood with a thriving pentecostal church, a presbyterian church, a catholic church, and a synagogue all in reasonable proximity, then you have truly discovered a place where there is diversity of opinions and worldviews. Also, if the state isn't deep blue or deep red, then there is some diversity of worldviews. Most people, however, only pretend to want that. What they want is to be surrounded by those who share their values, which is completely understandable. You can find that and lower cost of living in many places.

3. Don't overlook walkability and nice, old fashioned architecture. This is huge. It's the best thing San Francisco has going for it -- human scale architecture combined with density. That's the one thing you will miss the most. There are so many benefits from not needing a car, not only health benefits, but mental health benefits. Despite all the problems with homelessness in the city, which have gotten much worse over the years, there many nice walkable neighborhoods, as well as nice staircase hikes, viewspots. It's lovely.

In terms of places to look at, there are lots of nice places that, in my opinion, would provide much higher standards of living.

1. Miami/beach places in Florida.

2. Small towns on the east coast, like Asheville, Charleston, Charlottesville.

3. Washington DC is underrated, but also expensive and there are crime issues. But not as pricey as SF.

4. Chicago has low cost of living if you can put up with the cold. Crime issues tend to be isolated to bad neighborhoods so the headlines aren't as bad. One thing to be aware of is the horrible state finances in Illinois -- high income people are going to be soaked, so caveat emptor.

5. Although NY is expensive, Long Island is relatively inexpensive still.

6. The entire desert southwest is awesome -- New Mexico, Arizona, Utah, are all gorgeous states with wonderful cities. Take a look at Flagstaff, Arizona.

7. New Hampshire is underrated, if you can take the cold.

For walkability, which is key for me, one option is to find a small town -- those tend to be walkable. It needs to be not dead and close to a big town. Access to healthcare is a real issue in small towns, so you will end up driving to big towns to see your doctor.