Wanted to do a top post, but since your situation is pretty much identical to mine...

I think the OP was wrong in avoiding pharmacological solutions out of hand. I 100% agree with his assertion that they come with side effects and addiction risks, but this only makes the game more complex. Throwing away a whole category of solutions is too wasteful.

First, CBT works. It worked for me as well, to a degree, although I suspect it's somewhat better when you have problems falling asleep than when you wake up too early (terminal or maintenance insomnia). Tip: most watches on the market (and my Oura ring) automatically calculate your sleep efficiency. So you don't need complex formulas - just limit your sleep every time sleep efficiency goes below a certain number (80%? varies from device to device). Another tip: at least for a while limit caffeine below what you think you should. Caffeine has an _average_ half life of 3 hours, but we're not all average - if in your case it's more like 6 hours, that by itself is enough to give you insomnia. And you'll never realize it because you only do what everybody else does. Third tip: relaxations exercises are rumored to work well, but only if you practice them for a couple of weeks.

Ok, pharmacological solutions. First, what worked for me. It turns out I have very weak anxiety, which, TBH, seems like a pretty common sense state to be in. The fix is to counter it with _very weak_ sedatives or anxiolytics, and to vary the dosage as needed, including day to day. For example, ~1mg of Mirtazapine (for reference, a pill is usually 30mg - I suggest grinding it with 30g of sugar and measuring grams). Side effects: slight afternoon somnolence and chocolate cravings if you take too much. Big advantage: no tolerance. In principle other mild sedatives would work as well, but I love its safety profile. It's an old school antihistamine, btw.

Ambien is ok to take when needed and I did take it for many years but... I don't like it. There are rumors of higher overall mortality, plus it hits your short term memory which makes it easy to overestimate how much it helped you sleep - you may just have forgotten you are awake. But it works, it's probably the safest benzo-like thing available and it goes out of your system in a few hours which makes it pretty good for morning insomnia. I still use it very occasionally. I found out that at least in my case it can be replaced with 0.5 mg of mirtazapine or Xanax. YMMV.

Melatonin had its own posts here. Short version: look for low dosage pills (1mg or less) and take them in two ways: either as a weaker but infinitely safer Ambien, or take it 3-5 hours before bed time to reset your circadian rhythm.

To stay awake: low dosage modafinil, aka 50mg or less. If you can feel the effects, you're taking too much. It's supposed to compensate for tiredness, not give you a high (which it will at 100mg and over). It goes well with a downer in the evening, like the 1mg of mirtazapine. Use your own wisdom on having a long term lifestyle of using pills to stay awake and fall asleep. In my case, I take it when I slept under 6 hours which tends to be about once a week, and it turns a bad work day into a very good day. I do not recommend it regularly or in higher dosages, just trust me on this. But it's awesome for the motivation: It helps a lot when I wake up groggy, check my Oura ring and instead of groaning I'm "yey, this is a low sleep day!".

CBT is supposed to replace drugs, but the version with very low dose modafinil and mirtazapine is a lot less painful.

Inositol is something I've just discovered and seems to click well with my brand of sleep issues. Of all the things I've tried it's absolutely the first which actually bumped the sleep efficiency over 90%. Good news: it's a supplement not a drug, and it's also supposed to be very safe. Bad news: dosage required for insomnia is pretty high, around 20g. Do your research - I've just started playing with it. But don't be surprised if lower dosages do nothing.

And a final word of advice: don't be afraid to experiment, within bounds. I'm testing solutions for almost 10 years now, but I've always had a current doctor to vet what I tried. My biggest input was playing with (usually much lower) dosages, and finding the mix which works for me.