Unit Testing for Concepts in Neural Networks (2022)

Abstract: Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models’ internal states.

## introduction understanding language requires having representations of the world to which language refers. prevailing theories in linguistics and cognitive science hold that these representations, or concepts, are structured in a compositional way-for example, the concept of ''car'' can be combined with other concepts (''gray'', ''new'')-and that the meanings of composite concepts (''gray car'') are inherited predictably from the meanings of the parts. state-of-the-art models for natural language processing (nlp) use neural networks (nns), in which internal representations are points in high-dimensional space. whether such representations can in principle reflect the abstract symbolic structure presupposed by theories of human language and cognition is an open debate. this paper maintains that the question of whether a model contains the desired type of symbolic conceptual representations is best answered at the computation level @xcite : that is, the diagnostics of ''symbolic concepts'' concern what a system does and why, rather than the details of how that behavior is achieved (e.g., whether it stores vectors vs. explicit symbols ''on disk''). even @xcite , in their vocal criticism of nns, assert that ''a connectionist neural network can perfectly well implement a classical architecture at the cognitive level'', foot_0 but do not say how to know if such an implementation has been realized. to this end, we propose an api-level specification based on criteria of ''what concepts have to be'' @xcite . our specification ( §3) defines the required behaviors and operations, but is agnostic about implementation. we then consider fully connectionist systems equipped with modern evaluation methods (e.g., counterfactual perturbations, probing classifiers) as candidate systems. we present evidence that the evaluated models learn conceptual representations that meet a number of the key criteria ( §5- §7) but fail on those related to causality ( §8- §9). we argue that more powerful tools for analyzing nns' internal states may be sufficient to close this gap ( §10). overall, our primary contribution is a framework for seeking converging evidence from multiple evaluation techniques in order to determine whether modern neural models are consistent with a specific theory of concepts. our experiments offer an updated perspective in the debate about whether neural networks can serve as the substrate of a linguistically competent system. ## ''what concepts have to be'' 2.1 criteria there is no single agreed-upon standard for what ''concepts'' are @xcite . we base our criteria on those put forth in @xcite as part of a theory which advocates symbolic representations and prioritizes explaining phenomena such as syntactic productivity and semantic compositionality. @xcite argues for five conditions required for a conceptual representation to be viable as a model of human-level cognition: c1: ''concepts are mental 2 particulars; specifically, they...function as mental causes and effects''; c2: ''concepts...apply to things in the world; things in the world ''fall under them''; c3: ''concepts are constituents of thoughts and... of one another. mental representations inherit their contents from the contents of their constituents''; c4: ''quite a lot of concepts [are] learned''; c5: ''concepts are public...to say that two people share a concept [means] they have tokens of literally the same concept type.'' ## system specification we translate key ideas from fodor's conditions into concrete unit tests for evaluating computational models. our mapping is not one-to-one: we combine c2 and c5 into a single test focused on whether a concept grounds consistently to perception; we split c3 into two tests and leave aspects to future work; we omit c4 since there is likely little controversy that modern nlp systems ''learn'' concepts. our tests apply to a system holistically, including implementations of diagnostic functions, not just the internal representations. thus, it is possible for one system to fail our tests, but for a different system with the same internal representation but different implementations of the functions to succeed. see discussions in §4.1.1 and §10. ## implementation our code, data, and results are available at: bit .ly/unit-concepts-drive. ## test 1: predictions are grounded is grounded requires that if, definitionally, the difference between ''dax'' and ''blick'' is roundness, then this visual attribute should dictate predictions. ## test 2: representations encode types is token of type requires that the system's representations of concepts can be mapped to discrete types in a reusable way. ## summary overall, our experiments suggest that models exhibit grounded behavior and possess conceptual representations that encode modular, context-independent types. however, we don't find evidence of a direct causal connection between the representations of constituent concepts and those of composite concepts, an essential feature of fodor's theory on which our specification is based. our discussions of each individual experiment ( §5.3, §6.3, §7.3, §8.3) together raise several general themes. first, success on our tests often depends on granting assumptions about how concepts are acquired: how should concepts be learned in the face of spurious correlations, how many training examples are necessary, and so forth? while fodor does not focus on acquisition in his criteria, other theories exist which make empirical predictions about how and when specific conceptual representations develop in humans @xcite . future work could translate such predictions into additional unit tests (measuring learning curves, processing times, etc.), in order to diagnose whether current models' errors should be interpreted as failures vs. expected signatures of conceptual learning. second, our proposed tests evaluate a system as a whole. thus, our ability to make claims about neural networks as an implementation of conceptual reasoning is dependent on the quality of the tools available for inspecting neural networks' internals. a particularly fruitful area for future work is finding alternative implementations of ablate. recent work by @xcite and @xcite could be promising places to start. finally, we observe interesting trends about the effect of pretraining on conceptual representations. the models we evaluate share the same architecture but have different pretraining regimes. only for is grounded, and possibly in our layerwise analysis, was there a clear benefit from pretraining. our results suggested that the pretrained models had an inductive bias for shape over color, and may show more promise in subsequent studies of causality. on other tests, pretraining did not translate to a clear improvement in conceptual structure.