A Biologically Plausible Parser (2021)

Abstract: We describe a parser of English effectuated by biologically plausible neurons and synapses, and implemented through the Assembly Calculus, a recently proposed computational framework for cognitive function. We demonstrate that this device is capable of correctly parsing reasonably nontrivial sentences.1 While our experiments entail rather simple sentences in English, our results suggest that the parser can be extended beyond what we have implemented, to several directions encompassing much of language. For example, we present a simple Russian version of the parser, and discuss how to handle recursion, embedding, and polysemy.

## introduction language is a distinguishing human function involving the creation, articulation, comprehension, and maintenance of hierarchically structured information about the world. it is beyond doubt that language is achieved through the activity of neurons and synapses-but how? there has been extensive previous work in cognitive experiments -psycholinguistics, computational psycholinguistics, and brain imaging-that has led to many insights into how the brain processes language (see section 2 for an overview). however, no concrete narrative emerges yet from these advances about the precise way in which the activity of individual neurons can result in language. in particular, we are not aware of an experiment in which a reasonably complex linguistic phenomenon is reproduced through simulated neurons and synapses. this is the direction pursued here. developing an overarching computational understanding of the way neurons in the human brain can make language is hindered by the state of neuroscience, which (a) predominantly studies sensory and motor brain functions of animals other than humans; and (b) with respect to computational modeling, focuses on the level of neurons and neuronal circuits, and lacks the kind of high-level computational model that seems to be needed for understanding how high-level cognitive functions can emerge from neuronal activity. very recently, a computational model of brain function, the assembly calculus (ac), has been proposed @xcite . the ac describes a dynamical system involving the following parts and properties, well attested in the neuroscience literature: (a) brain areas with random connections between neurons; (b) a simple linear model of neuronal inputs; (c) inhibition within each area such that the top k most activated neurons fire; (d) a simple model of hebbian plasticity, whereby the strength of synapses increase as neurons fire. an important object emerges from these properties: the assembly, a large set of highly interconnected excitatory neurons, all residing in the same brain area. assemblies were hypothesized by hebb seven decades ago @xcite , and were identified in the brain of experimental animals two decades ago @xcite . there is a growing consensus that assemblies play a central role in the way brains work @xcite , and were recently called ''the alphabet of the brain'' @xcite . assemblies can, through their near-simultaneous excitation, represent an object, episode, word, or idea. it is shown in @xcite that assemblies are an emergent property of the dynamical system under conditions (a)-(d) above, both in theory and in simulations. in the ac (reviewed in more detail in section 3), the dynamical system also makes it possible to create and manipulate assemblies through operations like projection, reciprocal projection, association, pattern completion, and merge. these operations are realistic in two orthogonal senses: first, they correspond to behaviors of assemblies that were either observed in experiments, or are helpful in explaining other experiments. second, they provably correspond (they ''compile down'') to the activity of individual neurons and synapses; in @xcite , this correspondence is proven both mathematically and through simulations. it is hypothesized in @xcite that the ac may underlie high-level cognitive functions. in particular, in the discussion section it is proposed that one particular operation of the ac called merge may play a role in the generation of sentences. note that, regarding the soundness of the ac, what has been established through mathematical proof is that ac operations work correctly with high probability, where the underlying probabilistic event is the creation of the random connectome. so far, the simulation experiments conducted with the ac have demonstrated that individual commands of the ac, or short sequences thereof, can be successfully and reliably implemented in a simulator @xcite . do such simulations scale? for example, can one implement in the ac a computationally demanding cognitive function, such as the parsing of sentences, and will the resulting dynamical system be stable and reliable? this is nature of the experiment we describe below. in this paper we present a parser powered by the ac. in other words, we design and simulate a biologically realistic dynamical system involving stylized neurons, synapses, and brain areas. we start by encoding each word in the language as a different assembly of neurons in the brain area we call lex. next, we feed this dynamical system with a sequence of words (signals that excite the corresponding sequence of word-coding assemblies in lex). the important questions are: will the dynamical system parse sentences correctly? and how will we know? to answer the last question first, our dynamical system has a readout procedure which, after the processing of the sequence, revisits all areas of the system and recovers a linked structure. we require that this structure be a parse of the input sentence. our experiments show that our parser can indeed parse correctly, in the above sense, reasonably nontrivial sentences, and in fact does so at a speed (i.e., number of cycles of neuron firings) commensurate with that of the language organ. our design of this device engages several brain areas, as well as fibers of synapses connecting these areas, and uses the operations of the ac, enhanced in small ways explained in section 3. it would in principle be possible to use the original set of ac operations in @xcite , but this would complicate its operation and entail the introduction of more brain areas. 2 the resulting device relies on powerful word representations, and is essentially a lexicalized parser, producing something akin to a dependency graph. while our experiments entail the parsing of rather simple sentences in english, in section 6 we argue that our parser can potentially be extended in many diverse directions such as error detection and recovery, polysemy and ambiguity, recursion, and languages beyond english. we also build a toy russian parser, as well as a universal device that takes as its input a description of the language in the form of word representations, syntactic actions, and connected brain areas. goals. this research seeks to explore the two questions already highlighted above: in particular, we are not claiming that our implementation of the parser necessarily resembles the way in which the brain actually implements language, or that it can predict experimental data. rather, we see the parser as an existence proof of a nontrivial linguistic device built entirely out of simulated neuron dynamics. ## related work computational psycholinguistics. there is a rich line of work in computational psycholinguistics on cognitively plausible models of human language processing. such work focuses chiefly on (a) understanding whether high-level parsing methods can be used to predict psycholinguistic data (such as reading-time or eye-tracking data) and neural data (e.g., fmri and ecog data from linguistic experiments); and (b) developing parsing methods that have specific, hallmark, experimentally established cognitive properties of human syntactic processing (most importantly incrementality of parsing, limited memory constraints, and connectedness of the syntactic structures maintained by the parser). see @xcite for a summary of the psycholinguistic desiderata of (b), as well as a discussion of evaluation standards for (a). exemplars of this line of work are jurafsky's use of probabilistic parsing to predict reading difficulty @xcite , the surprisalbased models of hale, levy, demberg, and keller @xcite @xcite , and the strictly incremental predictive models of @xcite @xcite . much work attempts to achieve the properties in (b) while maintaining (a), that is, neural and psycholinguistic predictiveness, for example, the pltag parser of @xcite , or the parser of @xcite , which is constructed in act-r, a high-level metamodel of cognitive processing. a related line of work takes modern neural (ann) parsing methods, and examines whether the internal states of these models at parse time can be used to model psycholinguistic and neurological data observed for the same sentences-see the work of @xcite , which uses an neural action-based parser, @xcite , and @xcite , which examine various rnn architectures and transformers; and @xcite , which compares a wide range of ann methods. a related direction is that of @xcite who build a reservoir network (a recurrent network that builds a representation of a sentence using fixed, random sparse matrices for weights) and build a classifier on top that predicts grammatical roles and that has also has some psycholinguistic predictivity (this has some similarities to aspects in assembly calculus, namely, the sparse, random connections). the present paper differs from these works in key ways. in all previous work, parsers are written in a high-level programming language, whereas we focus on whether a simple parser can be implemented by millions of individual neurons and synapses through the simulation of a realistic mathematical model of neuronal processing. in this framework, it is nontrivial to implement a single elementary step of syntactic processing, such as recording the fact that a particular input word is the sentence's subject, whereas in previous work such actions are built into the framework's primitives. in a survey paper of cognitive models of language that use anns, frank summarizes that ''in spite of the superficial similarities between artificial and biological neural networks . . . these cognitive models are not usually claimed to simulate processing at the level of biological neurons. rather . . . neural network models form a description at marr's algorithmic level'' @xcite . our work can be seen as largely orthogonal to the related work described above, as we attempt to bridge granular neuronal mechanics with the study of complex cognitive processes such as syntax. see section 6 for discussions of potential future work that connects the two areas. neuroscience. in building a neural parser, we begin with basic, established tenets of neuron biology: individual neurons fire when they receive sufficient excitatory input from other neurons; firing is an atomic operation; and some synapses can be inhibitory (firing one neuron decreases the total synaptic input of another neuron). we also assume a simplified narrative of synaptic plasticity: connections between neurons become stronger with repeated co-firing (hebbian plasticity); this is a well-known abstraction of the many kinds of plasticity in the brain. these tenets are covered in a number of textbooks (see for instance @xcite , chapters 7, 8, and 67). our parser is built on a mathematical formalization of these principles. how are higher-level cognitive processes computed by networks of individual neurons? highly interconnected sets of neurons, called assemblies, are an increasingly popular hypothesis for the main unit of higher-level cognition in modern neuroscience. first hypothesized decades ago by hebb, assemblies have been identified experimentally @xcite and their dynamics have been studied in animal brains (see, e.g., @xcite , displacing previously dominant theories of information encoding in the brain (see, e.g., @xcite . in a recent paper @xcite , a concrete mathematical formalization of assemblies is proposed. they demonstrate that this simplified model is capable of simulating assembly dynamics that are known experimentally. this model of neurons, assemblies, and their dynamics can be viewed as a computational system, called the assembly calculus (ac), bridging neuronal dynamics with cognition; this is the computational system in which we implement our parser. the ac is summarized in section 3. note that it has been long debated whether language is a highly specialized system or is based in general cognitive faculties (see, e.g., the summary of the debate in @xcite ). we are agnostic in this debate, because assemblies are the proposed unit of neural computation both specialized and generic, having been studied across a variety of systems and species @xcite . language in the brain. the ac model makes use of abstracted brain areas (defined in section 3); therefore the design of our parser starts with the identification of a set of such areas. here we discuss how our choices relate to what is known about language in the brain-a field in which, as of now, much is yet unknown or debated. it has been known for 150 years that wernicke's area in the superior temporal gyrus (stg) and broca's area in the frontal lobe are involved in language; it is now generally-but not universally, see the discussion below-accepted that broca's area is implicated in the processing of syntactic structures, while wernicke's area is involved in word use. language processing appears to start with access to a lexicon, a look-up table of word representations thought to reside in the left medial temporal lobe (mtl) opposite wernicke's area. major anatomical and cytological differences are known between humans and chimps at and near those areas of the left hemisphere, with evolutionary novel powerful fibers of synapses connecting these areas in humans @xcite . based on this, we believe the inclusion of a lexicon brain area (lex), containing assemblies representing words, is largely uncontroversial, as are its strong synaptic connections into other brain areas used by the parser. the book by @xcite provides an excellent overview of a major direction in the theory of the language organ. after word look-up, activity in the stg is thought to signify the identification of syntactic roles; for example, it is known that the same noun is represented at different points in stg when it is the subject vs. the object of a sentence @xcite , suggesting that there are specialized areas representing subject, object, presumably verb, and perhaps other syntactic categories. however, there is active discussion in the literature on whether brain areas are dedicated to specific linguistic functions such as syntactic and semantic processing, see, for example, @xcite @xcite . in our parser, we do make use of areas for different syntactic roles, but in doing so, we are not taking sides in the debate over the syntactic specialization of brain areas; we are not claiming that syntactic analysis is the exclusive function of these areas-even the lex area containing representations of words could be part of a larger memory area in the medial temporal lobe partaking in several aspects of language. at the highest level, the parser is generating a hierarchical dependency-based structure of a sentence that is processed incrementally. in the brain, creation of phrases or sentences seems to activate broca's area-what in @xcite is called ''merge in the brain.'' long sequences of rhythmic sentences each consisting of 4 monosyllabic words of the same syntactic structure as in ''bad cats eat fish,'' dictated at 4 hz, result in brain signals from the subject's brain with fourier peaks at 1, 2, and 4 hz, suggesting that the brain is indeed forming hierarchical structures @xcite . our parser represents a plausible hypothesis for the mechanism behind this process, in that it implements it within a realistic model of neural computation. ## the assembly calculus this section describes the version of the ac used in our simulator and experiments. this version of the ac is almost the same as that of the previous work of @xcite , but includes minor modifications described here. the ac is a computational system intended to model cognitive function in a stylized yet biologically plausible manner, by providing a high-level description and control of a dynamical system of firing neurons. there is a finite number a of brain areas a, b, ... each containing n excitatory neurons. the n neurons in each area are connected by a random weighted directed g n,p graph, meaning that every ordered pair of neurons has, independently, the same probability p of being connected. each synapse (i, j) has a synaptic weight w @xmath0, initialized to 1, which changes dynamically. for certain unordered pairs of areas (a, b), @xmath1, there is a random directed bipartite graph connecting neurons in a to neurons in b and back, again with probability p for each possible synapse. these connections between areas are called fibers. all said, the dynamical system is a large dynamically weighted directed graph @xmath2, e) with an nodes and random directed weighted edges. events happen in discrete time steps (think of each step as 20 ms). the state of the dynamical system at each time step t consists of (a) for each neuron i a bit f t @xmath3, 1} denoting whether or not i fires at time t, and (b) the synaptic weights w t ij of all synapses in e. given this state at time t, the state at time t + 1 is computed as follows: ## experiments we provide an implementation of the parser in python. 5 to achieve this, we have significantly extended the existing ac simulation library of @xcite (for instance adding inhibition mechanics, and making possible brain areas like lex with fixed, predefined assemblies). importantly, the entire algorithm, at its lowest level, is running by simulating individual neurons and synapses, and any higher-level concept or structure (such as the parse tree itself) is an abstraction that is encoded entirely in the individual firings and neuron-to-neuron synaptic weights of a giant graph and dynamical system representing the brain. we provide a set of 200 varied sentences that are parsed correctly by the parser. for each sentence we have a target correct dependency structure; we verify that the readout operation produces this dependency structure for each sentence. the sentences were designed by hand in order to demonstrate a variety of syntactic phenomena that our parser (and the ac) can handle; this includes prepositional phrases, transitivity, and intransitivity (including verbs that can be both transitive or intransitive), optional adverb placement pre-or post-verb, and several more. see section 8 for these example sentences and the syntactic pattern they represent. we remind the reader that to our knowledge, this is the first realistic parser implemented on the level of individual neurons and synapses. the 200 sentences parsed were sampled from 10 syntactic patterns the parser was designed to handle, from a vocabulary of 100 words. since the underlying dynamical system can in principle develop an instability with low probability, failure is still possible, but it did not happen in our experiments. there are syntactic structures that the parser does not currently handle, but which we believe are possible in our model in two senses: first, the grammar (word actions) can be expanded to handle these structures, and more importantly, the dynamical system will still correctly parse them with high probability. a prime example of such a structure is sentence embedding (''the man said that. . . ''). see section 6 for more discussion of such extensions. the speed of parsing in our simulations is commensurate with known bounds on neural language processing. in each step, the participating assemblies must fire enough times for the project * operation to stabilize; we find through simulation that for reasonable settings of the model's hyperparameters, convergence occurs within 10-20 firing epochs, even when multiple brain areas are involved. assuming 50-60 neuron spikes (ac time steps) per second, and allowing for the (parallel) execution of the inhibit/disinhibit operations in the word's action, which will take a small number of additional cycles, we arrive at a frequency range of 0.2-0.5 seconds/word. ## extensions and discussion the uparser and a parser for russian. the underlying meta-algorithm of the parser, which we call the uparser, takes as input a set of areas a, a set of fibers f, a set of words w , and a set of actions a w for each @xmath4, and parses with sac using algorithms 2 of section 3. the uparser is language-agnostic, and can be seen as modeling a universal neural basis for language: a specific language instantiates this basis by specifying a (roughly, its syntactic constituents), f, w , and the α i . this is in no way constrained to english; for instance, our model and algorithm are equally well equipped to handle a highly inflectional language with relatively free word order, where syntactic function is indicated morphologically. we demonstrate this with a parser for a toy subset of russian, a language with free word order in simple sentences. particularly, our parser works on a set of sentences that is closed under permutation; permutations of the same sentence are parsed correctly, and produce the same dependency tree (as verified with algorithm 3). see section 8 for more details on the russian parser. the big bad problem and recursion. the parser of section 3 is rather skeletal and needs to be extended to handle recursive constructions such as compounding and embedded clauses. consider the sentence ''the big bad problem is scary''. following the parser of section 3, after the word ''big'', an assembly a representing ''big'' has been created in adj which the system anticipates projecting into subj in a subsequent step (when a noun is encountered) to form an assembly in subj representing the subject np. however, the next word is not a noun but ''bad'', part of a chain of adjectives. now, if we project from lex into adj to form an assembly b representing ''bad'', we lose all reference to a, and there is no way to recover ''big''! there are several simple and plausible solutions to the chaining problem, which can also be used to parse adverb chains, compound nouns, and other compound structures. one solution is using two areas, comp 1 and comp 2 , in a. when we encounter the second adjective (or more generally a second word of the same part of speech which otherwise would fire into the same area, overwriting earlier assemblies), we project from lex into comp 1 instead of adj, but simultaneously project from adj into comp 1 to link the first adjective with the second. for a third adjective in the chain, we project lex into comp 2 but also comp 1 into comp 2 unidirectionally; generally, for adjective i in addition to lex we project unidirectionally from comp parity(i) (which contains the previous adjective assembly) into comp parity(i-1) . we demonstrate parsing chains of varying lengths with these two areas in our simulations. however, one limitation of this solution is that it requires unidirectional fibers; if the projection from comp 1 to comp 2 above is reciprocal, it won't be possible to link the assembly in comp 2 to another assembly in comp 1 . another approach, which may be more realistic and obviates the need for unidirectional fibers (though it cannot handle arbitrary-length chains) is to add more than 2 areas, comp 1 ,. . . ,comp m , for some small but reasonable m, perhaps 7. parsing proceeds as in the two-area solution but by chaining comp i to comp i+1 . the number of areas m could model well-studied cognitive limits in processing such complex structures (see @xcite . to model longer chains, in high-demand situations or with practice, the brain could recruit additional brain areas. such maneuvers can handle right recursion of the form @xmath5. center embedded recursion is more complicated. to construct a parse tree for a sentence with an embedded sentence or relative clause, for an execution of the parser's inner loop, the subjects and/or the verbs of the two sentences may need to coexist in the corresponding areas. this is not a problem, as brain areas can handle tens of thousands of assemblies, but the linking structure must be preserved. it must be possible to save the state of the embedding sentence while the embedded one is serviced. one possibility is to introduce currentverb areas, which can be chained like the compounding areas above, and will act as a stack of activation records for this more demanding form of recursion. the idea is that, by recovering the verb of the embedding sentence, we can also recover, through linking and after the end of the embedded sentence, the full state of the embedding sentence at the time the embedded sentence had begun, and continue parsing. this needs to be implemented and experimented with. disambiguation. in english, rock can be an intransitive verb, a transitive verb, or a noun; so far, we have ignored the difficult problems of polysemy and ambiguity. to handle polysemy, every word w may need to have many action sets α 1 w , α 2 w , . . ., and the parser must disambiguate between these. we believe that the parser can be extended to handle such ambiguity. the choice of the action could be computed by a classifier, taking as input a few words' worth of look-ahead and look-behind in the current sentence (or perhaps just their parts of speech), and selecting one of the action sets; the classifier can be trained on the corpus of previously seen sentences. this also needs implementation and experimentation. error detection. grammaticality judgment is the intuitive and intrinsic ability of native speakers of any language to judge well-formedness; this includes the ability to detect syntactic errors. neurolinguists are increasingly interested in the neural basis of this phenomenon; for instance, recent experiments have detected consistent signals when a sentence fragment is suddenly continued in a way that is illegal syntactically @xcite . built into the parser is the ability to detect some malformed sentences. there are at least two simple mechanisms for this. one is when a fragment is continued by a word that immediately makes it ungrammatical, such as following ''the dogs lived'' with ''cats''. the parser, having processed the intransitive verb ''lived'', has not disinhibited area obj, and all other noun-related areas are inhibited. upon encountering ''cats'', project* will not fire any assemblies; we call this an empty-project error. other kinds of syntactic violations can be detected by nonsense assembly errors. such an error occurs during readout when an area is projected into lex, but the resulting assembly in lex does not correspond to x w for any @xmath6; in other words, when the function getword() of the readout algorithm of section 3 fails, which indicates that the state of g must have resulted from an impossible sentence. we provide a list of illegal sentences for which our parser simulation detects empty-project or nonsense-assembly errors, indicating different kinds of syntactic violations. one kind of syntactic error our parser does not detect currently is number agreement: our simulator would not complain on input ''cats chases dogs;'' this is not hard to fix by having separate areas for the projection of singular and plural forms. other types of agreement, such as in gender, can be treated similarly. the role of broca's area. language processing, especially the formation of phrases and sentences, has been observed to activate broca's area in the left frontal lobe; so far, the parser only models processes believed to be in the left stg and mtl. we hypothesize that broca's area may be involved in the building of a concise syntax tree summarizing the sentence, consisting of only the subject, the verb, and the object if there is one (but with access to the rest of the parsed sentence through the other areas). this would involve new areas s (for sentence) and vp (for verb phrase), with fibers from verb and obj to vp, and from vp and subj to s. building the basic three-leaf syntax tree can be carried out passively and in the background while the rest of syntax processing is happening in our current model. ## conclusion few students of the brain think seriously about language, because (1) language is by far the most advanced achievement of any brain, and (2) despite torrential progress in experimental neuroscience, an overarching understanding of how the brain works is not emerging, and the study of language will require that. this is most unfortunate, because language has evolved rapidly over a few thousand generations, presumably to adapt to the capabilities of the human brain, and it therefore presents a great opportunity for neuroscience. this paper is a small first step towards establishing a framework for studying, computationally, linguistic phenomena from the neuroscience perspective, and we hope that it will be followed by bolder experiments and far-reaching advancements in our understanding of how our brain makes language and the mind.