PhD in Quantum Optics and Atomic Physics from Oxford here (they call it 'DPhil in Atomic & Laser Physics' here).

I fully agree with what everyone else is saying here, it's really great advice. On the science aspect, their advice has you well-covered. From myself, I emphasise: the most important part of ingredient of your PhD, other than you, will be your prospective advisor. The choice of school is secondary to that.

My two cents: consider how well you will get on with your supervisor. Meet them, their students (past and present), their departmental colleagues and friends if possible. Get a feel for them, a vibe. Even the people who don't love them very much will give you an idea for what this person is like in just a few hours. Imagine: if this is what your supervisor will be like for one day, can you imagine being with them for several years? In this respect, it's like a long-term relationship, with one massive difference: if a (romantic) relationship goes south after a few years, you can break up, but for a PhD if you quit halfway you could end up without a piece of paper to show for it. You will then have to weigh up that possibility with the effort needed to continue, which drains from and adds a hefty cost to all parts of your life. I've seen this 'crossroads' occur for many people who left halfway.

To drive home the last point: my personal experience is such that I literally cannot think of anyone who's had a tougher PhD journey than myself and still managed to complete it---I'll concede that I'm biased, naturally. Backing up that statement: statistically speaking, across my department I don't think anyone in the whole Oxford Physics department fell in the same hole that I did for at least 20 years. I need not go into details, but if pressed, you could ask me directly (this is my first post on HN, I would have to figure out how to respond). Be cautious about the life aspect of the PhD, not just the science itself, is my point.

Also be aware of the life outside of your work. This is the point you highlight in your original question: it's great that you do. Specifically, the location matters since it defines your environment. For your PhD to work out, you need not just a support network but an environment in which you can succeed with as little effort as possible. Your hard work then compounds from there. You want to be able to get home/go out on the weekends and recover energy after your work, not drain it. If your PhD work takes your energy and you can't get back your energy or recover it, that's bad. Same goes for the work itself though, meaning your work should motivate you and give you energy too. Imagine, what will your evenings and weekends be like if you were in exactly the same place but didn't have any PhD work at all. Would you still enjoy it? Ultimately this decision is a personal one, and you have to use and trust your judgement. If you don't feel 100% confident to answer that, talk with others about this point before moving forward. It's difficult to get a feeling for this experience, so you need to be armed with quality advice.

Note that a PhD is a hard journey and a life-changing experience. Of course, it's about your life. There's lots of negative things to say about the experience, but I'll end positively: my PhD has completely opened my world to experiences that I wouldn't have been able to dream of otherwise. I simply wouldn't have known such things exist. It's broadened my horizons in a similar way to like someone moving from the island of Nauru in Oceania to Bay Area SF would experience. So, do what makes you happy :)

Good luck!

Footnote: Actually, it's late in Europe at the moment, so to save time I tried to dictate this post using ChatGPT Plus for the first time. It spoke out what I said for 7 minutes uninterrupted and absolutely flawlessly! Then ended with 'Transcript unavailable', and I lost the text. Perhaps this is like 'The house always wins", except there is no house, or winning. So, from now on, whenever I think of AI going forward, I will think: 'The always". Even if it doesn't make sense to anyone, I'm sticking with that thought, it has personality!