> factorizing this giant matrix to identify "neuron groups" ... is a brilliant idea.

As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. (In my experience, if Alex is excited about an idea, >50% odds you'll realize it's a super important idea 2 years later. :P )

I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it.

> I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks.

Absolutely! (I think that most of the ideas in our article are pretty general. :) )

> This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future.

Thanks! We're super excited as well!

> Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it... What is your view or opinionated guess on this?

Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt.

My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us. I suspect that if one could have an "interpretability curve" of how easy models are to understand vs model performance -- it's kind of hard to make this sensical, because both variables are actually these really nuance high-dimensional things, but imagine you could -- it would go up as you approach human performance and actually peak after it, before eventually declining again. This intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones.

But, of course, that one dimensional view is a gross simplification. I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions."

By a "refined abstraction" I mean something like ears in GoogLeNet. You see, GoogLeNet has dozens of detectors for different kinds of ear -- a much richer vocabulary of ear types than I can articulate -- and knows a lot about how those should influence class probabilities. Although I can't understand the detailed nuance of each ear detector, I can get the general gist and verify that it has reasonable consequences. Conversely, by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with.

Both "refined" and "alien" abstractions could give superhuman performance. We're in a much better state when refined ones dominate. For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence.

I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.