"I am always anxious about them wobbling into my lane causing a horrible accident that I would be considered responsible for."

Cyclists don't generally "wobble", but if you suspect that there are issues with control, then you need to allow some space. And yes, you would be responsible if you did not allow that space. Such are the rights of people.

Your reference to a "use tax" is right on. If cyclists want additional infrastructure, then (IMHO) it only seems appropriate that they ante up.