This is an interesting idea, and the paper is pretty well thought out. But I think one source of information not sufficiently explored is anatomy, which would help a great deal with the microprocessor, although it seems to help less with the brain. If you have the connectivity of the entire microprocessor (as the authors have determined using microscopy), then you can probably determine that there are recurring motifs. If you can figure out how a single transistor works, then you can figure out the truth tables for the recurring motifs. That takes care of most of Fig. 2. The only question that remains is if you could figure out the larger scale organization.

Anatomy also helps with the brain, but not nearly as much. People are still trying to figure out how the nematode C. elegans does its relatively simple thing even though the full connectome of its 302 neurons was published 30 years ago. But in larger brains, the fact that neurons are clustered according to what they do and projections are clustered according to organization in the places that they project from provides at least some level of knowledge. We are not merely applying Granger causality willy-nilly to random neural activity. We know what's connected to what (at a gross level) and in 6-layer cortex we even have an idea of the hierarchy based on the layers in which projections terminate (which is how Felleman and Van Essen got Fig. 13b in the paper).

OTOH, I think our failure to understand many neural networks at a conceptual level is quite disturbing, and perhaps a sign that the kind of conceptual understanding we seek will be forever beyond our reach. The authors mention this toward the end of the paper, although I think they overstate our understanding of image classification networks; I've never seen a satisfying high-level conceptual description of how ImageNet classification networks actually see. One possibility is that we simply don't have the right concepts or tools to form this kind of high-level description. Another possibility is that there simply is no satisfying high-level way to describe how these networks; there are only the unit activation functions, connectivity, weights, training data, and learning rule. We can find some insights, e.g., we can map units' receptive fields and determine the degree to which different layers of the network are invariant to various transformations, but something with the explanatory power of the CPU processor architecture diagram in Fig. 2a may very well not exist.

I hope that the brain's anatomical constraints provide some escape from this fate. Unlike most convolutional neural networks, the brain has no true fully connected layers, and this may serve to enforce more structure. We know that there is meaningful organization at many different scales well beyond early visual areas. At the highest levels of the visual system, we know that patches of cortex can be individuated by their preferences for different types of objects, and similar organization seems to be present even into the "cognitive" areas in the frontal lobe. It remains to be seen whether it's possible to put together a coherent description of the function of these different modules and how they work together to produce behavior, or whether these modules don't turn out to be modules at all.