> Like I'd have to set up Colemak everywhere I go, if I sat at someone else's computer or a public computer it would be back to Qwerty

When I switched to Colemak (in Grad School) I bought a cheap laptop so I could make a clean break and never need to use a "public" computer again. (That was just before my first iPhone, and I actually imagine that should be even easier now that so much of what the need for borrowing "public" computers as a student in those days has moved to our phones. The rest was already mostly replaced at that point with tools like SSH and RDP remoting. One useful discovery at the time was that if I was RDPing into my desktop machine back in my dorm on a "public" computer in a school lab, I'd need to hunt and peck my address and password in in Qwerty, but as soon as I connected to my own machine, I could touch type in Colemak because it was my machine and set to use Colemak as default.)

I also made peace with looking like a slow idiot and hunting and pecking out QWERTY when trying to help someone on someone else's computer. When I need typing speed I use a computer I control. I learned to stop worrying about my typing speed on other people's computers.

> if you ever look down at your keys for reference (something you want to do a lot when you're learning a new layout)

This was a feature for me rather than a bug. Proper touch typing is "never glance at the keyboard" (and I was taught that over and over from childhood with Qwerty lessons, though I was bad at it with Qwerty; including teachers that insisted lessons needed to be done with a cover over your hands and the keyboard so that you couldn't look) and if you can't trust the key caps you have more reason to never look.

> Also the unlearning process can be slower than I'd like, and my typing speed plummets while I'm learning these things, so I'm less useful at work or school or working on personal projects.

For what it is worth, what worked well for me was taking a long holiday weekend where I didn't have work or school or personal project deadlines to do just about nothing but Colemak typing lessons. Going entirely cold turkey on not using anything but Colemak when touch typing from that point onward. (I also make a distinction between touch typing and "swipe typing" as entirely separate skills despite both involving "a keyboard" and don't have a problem touch typing in Colemak and swipe typing in Qwerty.)

That long holiday weekend was enough that I felt comfortably fast with Colemak for work/school/personal projects as I started to get back into them that I never felt like falling back on my old bad Qwerty habits, even if it was still maybe a small bit slower than where I had felt like I'd been before the switch, and as with anything once committed to it and using it every day speed only improved from there until I didn't feel any speed difference at all (and presumably am faster than I ever was with Qwerty though I never trusted speed metrics enough to say for sure).

I was highly motivated though to unlearn Qwerty, so obviously your mileage will vary. My Qwerty touch typing form was wrong from years of self-taught programming (my left hand was shifted off the home row left by a key to spend more time on the accelerator keys and my right hand had to move a lot more to compensate). Late in grad school with a mountain of assignments both school and otherwise I was dealing with an immense amount of hand pain that I was sure likely meant I was deep into the early stages of carpal tunnel and buying a cheap laptop and switching to Colemak seemed like a great cost savings over surgery. More than a decade later, almost closer to a decade and a half now, it still seems like a great deal and I'm very glad that I did it.