is that actually the case? [are model weights being published less frequently?]

It sure feels like it. But it could be survivorship bias: https://www.youtube.com/watch?v=_Qd3erAPI9w&t=52s&ab_channel... or regression to the mean: https://www.youtube.com/watch?v=1tSqSMOyNFE&ab_channel=Verit.... It's possible that StyleGAN and GPT-2 just happened to coincide, and both companies just happened to decide to release their models rather than hoard them, and that the usual case is for companies to rarely (if ever) release weights with their research.

But the more interesting question is, why wouldn't they? So onward to your second one:

are ethical concerns actually the reasons some models/weights are not released? Or is it about hoarding a potentially valuable commodity? Or some other reason, or a combination of several?

As with most things, the answer is likely the mundane, dull truth that big companies tend to have lots of inertia and friction preventing such things from happening. Sources of friction seem to include (but are not limited to):

- are we legally responsible for what other people do with the model? (no.)

- should we restrict people from using the model commercially? (why bother? but sure, nVidia did that for stylegan. It didn't stop Artbreeder from blatantly ignoring and monetizing it anyway. And while that sounds negative, Artbreeder is one of the coolest ML projects ever, as far as I'm concerned, so the world is better off for ignoring your stupid policy.)

- What if people start misusing the model? (So what? I feel like I should just get in the face of whoever is asking this question and repeat "So what?" until they have the police escort me off the premises. You can keep asking it to whatever they reply with, and eventually their logic never seems to go anywhere but one big ass-covering circle.)

- What if we might look bad because of it?

Now that last bullet point deserves some real treatment. In my opinion, the head of Facebook AI has done one of the most damaging things possible to the ML scene by getting everyone riled up about GPT generating "immoral" outputs:

https://twitter.com/an_open_mind/status/1285940858290933767

https://twitter.com/an_open_mind/status/1286034549345071105

https://twitter.com/an_open_mind/status/1285691693677850627

Good lord, AIs are generating racist outputs! They're saying that cis white men are intellectually superior to trans people! People are generating child porn in AI dungeon! Someone, do something!

... please. Every researcher I've talked to has rolled their eyes hard at the criticisms raised by Jerome. But since he's the head of Facebook AI, no one dares say so publicly. It may as well be an outsider like me: Jerome, I know your heart is in the right place, and that you believe very strongly that this is an important moral issue. But your moral concerns need to be balanced by the ethical considerations of the massive chilling effect you had by publicly shaming OpenAI so hard that they ended up completely losing their confidence, adding some ridiculous profanity filter to GPT-3 that flags pretty much anything mildly naughty, or the fact that AI dungeon is now in hot water because you've given the world an excuse to be pissed off about a mindless, memory-less AI (https://news.ycombinator.com/item?id=23346972) generating text that offends someone, somewhere, for some reason.

Well, people are going to be offended. Let them be. I am half Jewish. There is a very realistic chance that GPT-3 was trained on the full text of "Mein Kampf," and I could care less. Even if someone went out of their way to train the most offensive, most threatening language model the world has ever seen, what could they really do with that power? Are they going to hurt you right in the feelings? Is the language model going to argue vigorously for the extermination of all Jews? If it did, who would listen? No no, don't try to say that yes, there's a very real danger that people might generate propaganda and influence others. You know as well as I do that this is extraordinarily difficult in practice for multiple reasons, and that no one has ever seen an interactive, adaptive AI that can dynamically deliver the most persuasive propaganda to Reddit and fool everybody into thinking it's a genuine grassroots movement. And you know as well as I do that that would be roughly equivalent to inventing AGI, and that AGI is still nowhere in sight. It's not even living in the same country. Hell, it's not in our sector of the galaxy. See that star in the sky? It's possible that we're more likely to reach that star before we invent AGI, because nobody knows how to invent AGI yet.

Before I get too worked up, I should keep some focus and say constructive things. One. If you, as an ML researcher, find yourself wanting to release your work, but your corporation is putting the red tape around you, push back. They need you more than you need them, even if it doesn't feel like it. T

Two. Releasing work is generally beneficial for the world. I can't think of a single model release that has ever harmed the world. Let's wait until one actually does before we freak out about whether it might.

Three (https://www.youtube.com/watch?v=jpw2ebhTSKs&ab_channel=TheCh...). By releasing your work, you give countless people the opportunity to better themselves. Thanks to GPT-2 1.5B, we were able to vastly extend the capabilities of https://reddit.com/r/subsimulatorgpt2 to include over a hundred subreddits in a single model. This has roughly zero commercial value, yet has improved the lives of countless people who show up to giggle at all the (sometimes horribly offensive) things that robots say to each other. I was proud to be a part of that, and I want to be a part of more.

Please release your models. The most recent model release was CLIP, and it's already had a profound impact. Just look at how freaking awesome this is! https://twitter.com/l4rz/status/1367853921427984390 They're using CLIP to turn someone into Dracula! That's badass, and it inspires people (like me, at one time) to get into ML and become the researchers of tomorrow (as I try to be now).

I remember rooting for OpenAI's Dota 2 bot while they were facing off vs .. OG, I think? It's been several years. I spent five years playing dota-type games. It was one of the most exciting things I'd ever seen. And being able to beat rtz 1v1 mid SF? Not merely beat him, but annihilate him? Holy crap, give me that model. Why are you not releasing that model?! It's so cool!

And now nobody gets to play with it ever, and it's locked up in the real AI dungeon: OpenAI's. I hope they reconsider.