A thought experiment:

Imagine you have a blob of seemingly random data. Nothing in the data contains anything recognizable as illegal or in violation of copyright.

Now imagine that the right input suddenly turns the data into illegal or infringing material, after a transformation operation. And not just a single unique input such as a password which clearly represents a mapping function between two sets of data.

But imagine if there were seemingly infinite possible inputs, each of which transformed the data into a different infringing blob of data. If these inputs exactly represented the novel, copyrightable or illegal aspects, but the blob itself was inert.

What should be illegal here? The blob, which by itself is free of any questionable bits of data, or the inputs which transform it into something tangible? Both? Neither?

Well, it has never been illegal to draw or paint something representing CSAM, for example. And it has never been illegal to draw or paint Mickey Mouse in your own home.

What's often illegal is publishing said data. Ignoring the free speech debate around artificially produced CSAM, publishing it is already illegal in many territories. It is also illegal to violate copyright in many countries when publishing information.

What's interesting is that it is not illegal to trace a drawing and hanging it up on your wall, instead of buying the the real drawing from its rights-holder. It's also not illegal to reproduce a tracing done by a friend. But the recording and film industries have been more successful in convincing us that it should be illegal to do the same for a song or film. That you should not be able to "trace" the data at home, and that you should not be able to share it with me, that I should not be able to trace over your tracing and bring home a copy for myself.

I can understand, and support a copyright system which regulates the publishing of copyrighted material. Even copyleft paradigms lean on regulation for enforcement. But the film and music industry actively try to restrict individual freedoms in the name of corporate profits, while still screwing over their clients and employees with respect to profit-sharing.

Back to the point: That blob should never be illegal. The activation functions should never be illegal. That is a basic extension of free speech. But publishing, that is a different story, and we already have laws offering such protections both with respect to illegally-produced or copyrighted content. Any attempt to regulate what kind of model I am allowed to run at home is a massive infringement on my rights as an individual, and is borne either out of gross ignorance of current copyright law from the same people crying, "But think of the copyrights!", or direct, insidious corporate greed.

You can adjust this thought experiment so that instead of dealing with a magic blob, we are dealing with a program that makes it really easy to produce illegal or copyrighted works after a bit of human interaction. Is there claim here now? Are we basing the law on how much human involvement was needed to create the output? We've faced similar arguments around technological leaps such as the printing press or mechanical loom. Did we, as a society, reject these advances in technology in order to protect loom workers and scribes?

Bottom line. You can pry my models out of my cold, dead or handcuffed hands. Times like these really shine a light on who is complicit in the system, and who suffers from it.

If you are in the creative industry, you need to understand how things are going to change. As an engineer with decades of investment into my craft, I also have to face the rude awakening that is ahead in my own industry as automation creates a gap between highly-skilled professionals and newcomers. Being a paid software engineer might become as hard of work as becoming a famous professional artist. Lots of connections, insane specialization and a lifetime devoted to the craft. A lot of people in school for engineering right now might struggle to find employment in 20 years or less if they cannot cross this gap in time. Artists aren't the only tribe experiencing a huge industry shake-up over a technology that will one day be so ubiquitous that it's inside of your toaster.