One thing to keep in mind when discussing free speech is, the question speech free of what?

Free of unintended consequence?
Free of unintended consequence from state powers?

If you want to truly speak free, without consequence you shout into the empty woods — this however would be a little pointless: We want what we say to matter, to impact others, we want it to have consequences — it is just that there are certain consequences we want to avoid.

A society can decide to control these consequences either by law or by culture in collective decisions. E.g. publicly saluting Hitler in Germany is not only illegal, but can also have a ton of consequences socially (some of them good, if your ingroup is neo nazis, some of them bad if you are around regular people). This law however does not stifle discourse around the Nazi past of the country — in fact it is the opposite, there is probably no nation on earth that has an perspective on its past as well researched and discussed as the Germans.

So laws against certain kind of speech certainly don't have to harm discourse. Of course a German neo nazi would probably feel more free if Germany allowed them to hail the genocidal failed artist that lead their grandparents into an unwinnable attack war against the free world – but would it really improve the free discourse on ideas that allow any free democratic society to adapt to the world? I doubt that.

The question I am not clear about is: Where precisely to draw the border? Unintended consequences are certainly not something that can ever be ruled out as we live in social environments. If we say something the people that hear it don't like, they are free to react in a number of possible ways, most commonly deciding not to listen, to argue against, call you an idiot etc. What about the content of the speech? Is it a statement of fact that is shunned? Is it an opinion? Is it calculated polemic? Hate Speech? Harassment? Threats? A book? Fiction?

It is quite clear no free society could call itself free if statements of facts or books are banned or their publication lead to fines or imprisonment (e.g. calling an attack war "war", stating your findings on global warming). So much is clear. But is a society more free or less free when any minority is allowed to freely harass, threat and vilify others? I really don't think so either.

If you would have such a society, that by law not only allows, but protects harassment, threats and similar things, wouldn't it lead to a "weaponization" of the consequences people can let each others feel in the social space? From the European perspective one of the most valuable of the fundamental human rights states have to protect is the dignity of a person. If you critique a politician for their decisions, this is not attacking their dignity. If you harass them sexually, utter racist comments or threaten to kill them, very much so. People know that hurting others dignity is wrong – and if there is no law that protects human dignity, social conventions will have to do.

I'd rather have it codified in law tho, because otherwise speech will in fact get less free.

TL;DR: Some law controlling certain destructive kinds of speech will lead to more free speech, because there is less need for a society to come up with social consequences for destructive speech. No law however will lead to less free speech, because society will come up with their own, social consequences where a law is missing.