From a non-legal perspective, aren’t we all responsible for what people do in reaction to our speech, if we reasonably knew the outcome of our speech at the time?

Yep, I don't disagree with that in the slightest.

---

figures know statistically that if they keep denigrating a person long enough some unhinged person will take action

Guessing by "figures" you mean public figures, or otherwise someone with a platform. Yeah, absolutely there's a "who will rid me of this meddlesome priest" dynamic that's troublesome, and I believe anyone who uses their platform to do this is deeply unethical and irresponsible.

---

Anyone talking about a Final Solution in 1938 was complicit in the violence

The same with Rwanda and even now with Russia

"it wouldn’t be so bad if something maybe happened to this group"

These are all examples of speech that advocates for or suggests there is value in violence, genocidal or otherwise. If the people who claim speech is violence are referring exclusively to speech like this, I would have to agree with them.

But that's not the full scope of what some people mean. Some people include in their definition unpleasant speech, speech that offends, shocks or otherwise stands in opposition to their strongly held beliefs.

---

Here's a couple of representative examples from a set of op-eds[1] justifying violent protest against a speech by Milo Yiannopolous at Berkeley in 2017:

"the ideology they peddle perpetuates ideas that urgently endanger members of our community"

(Given the subject of the protest, the ideas here are presumably about the rights of trans people and undocumented immigrants)

How do ideas endanger people, let alone urgently? Obviously we are excepting the direct advocation of violence, which I addressed above.

Say someone publicly announces their opposition to (for instance) DACA on the grounds of their belief that it will cause more illegal immigration, which they oppose. Someone hears this statement and is inspired to commit a violent act against a person of color. What (non-legal) responsibility does the speaker have for this?

I posit, and am curious whether you agree that in the general case, advocating for or against a particular law or policy is not and cannot be violence.

"asking people to maintain peaceful dialogue with those who legitimately do not think their lives matter is a violent act"

I find this a particularly distasteful example of semantic gymnastics, and one that seeks to use the concept of speech-as-violence to justify actual violence against the speaker or related entities.

"[A]sking people to maintain peaceful dialogue" is an evasive way of saying "requiring that people are not violent". "[People] who legitimately do not think their lives matter" are presumably people that deserve violence for having the wrong opinion and sharing it.

Granted: Milo Yiannopolous has frequently skirted the line you and I agree on, naming undocumented immigrants, outing trans people, etc. This is unacceptable behavior and an obvious example of the "meddlesome priest" dynamic.

BUT - the quoted author cares little for the specific context of Yiannopolous' past actions, making a sweeping statement that is unquestionably intended to draw a rhetorical line in the sand on what generally constitutes violence. Not sure about you, but I find this hypocritical, counter-productive in the extreme, and yeah, kinda evil.

[1]: https://www.dailycal.org/2017/02/07/violence-self-defense/