There are a number of problems.

The least direct one that may still be fairly consequential is that if you own a controlling share in a company and are in the public health insurance, fluctuations in income can become an existential threat because while salaried employees have their rates adjusted on a monthly basis and Einzelkaufleute can use their income tax advances (which can be adjusted on a fairly short notice based on projected annual revenue) to adjust their rates, you don't pay an income tax advance (because your company instead pays corporate tax advances) but you also aren't considered salaried so public health insurers are legally required to use your most recent income tax return as the basis for your rate even if your salary changes. This can mean up to 2 years of delay between what your insurance rate is based on and what you actually make. Keep in mind that upon submitting a new income tax return, any difference is owed immediately. So if you decide to reduce your salary because you reduce you have to your number of hours or take time off for your kids, you continue having to pay the same monthly rate to your health insurance (although of course the difference will be refunded eventually) and if after doing this for a while you decide to go back to full-time employment you basically have to find out what your rate should be and set the difference aside so you can pay it back when the rate eventually catches up.

Another example with self-employment is that if you give birth, you are banned from working for a set period around the due date and this ban even applies if you're self-employed but because you are self-employed this also translates to zero income (or at least zero billable hours). Contrary to what some accountants might tell you, you can absolutely apply for Elternzeit though and the money you receive will be based on your salary, so there's that - but keep in mind what I said about public health insurance still applies even here.

Another fun fact about public health insurance is that as a mother giving birth, you receive compensation for the days around delivery via the health insurance. But this is tied to the "Krankentagegeld" and this is a "feature" you explicitly have to opt in to. So if at any point you switched health insurance companies you need to make sure you ticked the box - it's absurd that this is optional given that omitting it only saves you a few Euros per month but especially early on many solo entrpreneurs try to cut costs wherever they can. Keep in mind that there is not only a maximum insurance rate but also a minimum and many early entrpreneurs pay this disproportionate minimum while making barely any money - this is something private insurance companies prey on. Private health insurance is always a numbers game and not something that should be taken lightly even if public health insurance may seem extremely awkward an expensive. It's also much more difficult to go back to public once you've been in private (the usual hack is to register as unemployed for a couple of months to lose eligibility for private health insurance and automatically roll back into public insurance).

I don't recall the exact problem we ran into but the short version is that a lot of services Germans think of as "public" because they're publicly funded or paid by the government are actually tied to salaried employment or unemployment and "do you own at least 50% of a GmbH" appears on a surprising number of forms (as well as "does your GmbH employ more than one person including its owners").

Oh, and another thing worth mentioning: if your work is at all creative (somehow "building software products" doesn't tick that box, yet) or editorial, you may be subject to the Künstlersozialkasse. This is a social insurance that anyone hiring creative or editorial labor for commercial use has to pay into directly (i.e. it's not part of the invoice but the amount is based on the invoice). If you hire this labor via a GmbH, you don't have to pay this but the GmbH does. This means if you have a GmbH that provides creative or editorial labor (e.g. you do any design work), the amount owed to the Künstlersozialkasse will be based on the salary of the person that holds ultimate editorial control. If in doubt, this will be one of the owning managing directors, i.e. you. This can be a bit of a shock but if you are in the public health insurance this also means you can join the Künstlersozialkasse as a member, pay into public health insurance (via the KSK) the exact same way a salaried employee would (i.e. none of that "wait 2 years to adjust your rate after salary changes" nonsense) and not only does the money your company has to pay to the KSK go towards your own social security but through the magic of arcane accounting laws, they effectively contribute twice the amount you would if you paid directly (which for self-employed people is voluntary and largely pointless) - in other words you end up in a situation where you are legally self-employed but still benefit from public health insurance and the public retirement fund as if you were a regular salaried employee. I can't overstate how useful this is, especially if you are able to do this early on:

If you want to be self-employed in any creative field subject to the Künstlersozialkasse: JOIN THEM ASAP. You massively cut down on the headaches of self-employment and someone will have to pay into them for your work anyway so you might as well benefit from that.