TTR at the SPA : Relating type-theoretical semantics to neural semantic pointers (2023)

Abstract: This paper considers how the kind of formal semantic objects used in TTR (a theory of types with records, Cooper 2013) might be related to the vector representations used in Eliasmith (2013). An advantage of doing this is that it would immediately give us a neural representation for TTR objects as Eliasmith relates vectors to neural activity in his semantic pointer architecture (SPA). This would be an alternative using convolution to the suggestions made by Cooper (2019) based on the phasing of neural activity. The project seems potentially hopeful since all complex TTR objects are constructed from labelled sets (essentially sets of ordered pairs consisting of labels and values) which might be seen as corresponding to the representation of structured objects which Eliasmith achieves using superposition and circular convolution.

## introduction work on ttr, a theory of types with records, for example @xcite , claims that it can be used to model types learned by agents in order to classify objects and events in the world. if this is true, types must be represented in some way in brains. in this paper we will explore the possibility of using eliasmith's semantic pointer architecture (spa) @xcite for this purpose. the question of neural representations of types arises in connection with the theory of types proposed by ttr in a way that it does not in connection with more traditional type theories. the reason is that ttr aims to provide the kind of types that agents use in the perception of objects and events and which they use in interaction to communicate with each other. if it were to turn out that the kind of types used are in principle impossible to represent on arrays of neurons then this would call this project into question. we chose spa, since it is a model of a biological neural network. notwithstanding their practical and methodological success, artificial neural networks (ann) trained in deep learning leave open questions with respect to at least two areas of human cognition. firstly, being sub-symbolic, it is unclear how they relate to 'jackendoff's challenges' foot_0 @xcite and to higherorder, symbolic processing as observed, for instance, in sentence processing @xcite . secondly, despite being inspired by the human brain and potentially useful for neuro-scientific research (yang and wang, 2020), anns differ from biological neural networks. the first issue is addressed by vector symbolic architectures (vsa; @xcite , which define symbolic operations on high-dimensional numerical vectors. the second issue is addressed by biological architectures, where high-dimensional vectors receive a neural interpretation in terms of spiking patterns @xcite . formal semantics provides symbolic systems for analysing natural languages. however, as @xcite argue, it is questionable whether traditional, 'antirepresentationalist' formal semantics, which assigns truth conditions directly to sentences @xcite also lends itself to cognitive interpretations. this is different with a type theory with records (ttr; @xcite , which even has a neural interpretation @xcite . indeed there has been a wide range of work in this formalism, introduced in section 3, which includes the modelling of intensionality and mental attitudes @xcite , quantified nps @xcite @xcite , co-predication and dot types in lexical innovation, frame semantics for temporal reasoning, reasoning in hypothetical contexts @xcite , spatial reasoning @xcite , enthymematic reasoning @xcite , selfand other-repair @xcite , negation @xcite , nonsentential utterance resolution @xcite , iconic gesture (lücking, 2016), multimodality @xcite and symbol grounding @xcite . accordingly, this paper offers a first attempt to combine ttr with a biologically-based vsa, namely the semantic pointer architecture (spa) of @xcite . sections 2 and 3 provide a brief overview of semantic pointers and ttr, respectively. how to 'translate' ttr objects into spa is addressed in section 4. we conclude in section 5. ## spa (and nef) hypervector selection: when selecting vectors to represent basic entities one aims to create maximally different encodings. higher dimensional vector spaces offer sufficient space to maintain a large class of vectors distinct and moreover, they have the useful property that two random vectors are with very high probability quasi-orthogonal. a common strategy is to use a real range which is normally distributed with a mean of 0 and a variance of 1/d where d defines the number of dimensions. similarity measurement: vsas use similarity metrics to evaluate vector representations, in particular, to assess whether the represented symbols have a related meaning. the similarity metric plays the essential role of selecting the correct denoised vector from the database and to ensure a robust operation of vsas. the dot product of two vectors a, b is standardly computed as the sum of the product of their components, as in (1a). this is the basis for defining the cosine between two vectors as in (1b) in terms of the dot product and the vectors' lengths: following most vsa approaches, we use cosine as a measure of similarity. given (1b), this reduces to the dot product when the vectors are normalized (i.e., of length 1). if a • b ≈ 1, the vectors are (nearly) identical. for any vector a, (2) a • a ≈ 1 bundling: vsas use a bundling operator to superimpose (or overlay) given hypervectors. @xcite argues that a bundling operator must satisfy unstructured similarity preservation, namely a + b is similar to a and to b and to any bundle a + c that contains one of the vectors. bundling is typically handled using vector addition, but in the approach adopted here this requires a normalization step to a vector length of one. binding: binding × is used to connect two vectors, e.g., role-filler pairs. the output is again a vector from the same vector space. @xcite argues that binding needs to satisfy: in the current paper we generally follow the approach known as holographic reduced representations (hrr), first defined by @xcite , which is the approach utilized by eliasmith and implemented in nengo. however, as eliasmith notes, one could make different choices if clear motivation for these arises. specifically, with respect to binding we use circular convolution @xmath0: for @xmath1, . . . , d -1} circular convolution approximates the standard tensor outer product by summing over all of its (wrap-around) diagonals. this operator is commutative as well as associative. circular correlation provides an approximated inverse for circular convolution used for unbinding. the inverse is defined in (4a), exemplified in (4b), and its use for unbinding is given in (4c): foot_2 (4) inverse for circular convolution a. a -1 @xmath2, . . . , d -1} b. in other words: in what follows, we use b ′ for b -1 . ## ttr we give a brief sketch of those aspects of ttr which we will use in this paper. for more detailed accounts see @xcite . s : t represents a judgement that s is of type t . types may be either basic or complex (in the sense that they are structured objects which have types or other objects introduced in the theory as components). one basic type that we will use is ind, the type of individuals; another is real, the type of real numbers. among the complex types are ptypes which are constructed from a predicate and arguments of appropriate types as specified for the predicate. examples are 'man(a)', 'see(a,b)' where a, b : ind. the objects or witnesses of ptypes can be thought of as situations, states or events in the world which instantiate the type. thus s : man(a) can be glossed as "s is a situation which shows (or proves) that a is a man". another kind of complex type is record types. in ttr records are modelled as a labelled set consisting of a finite set of fields. each field is an ordered pair, ⟨ℓ, o⟩, where ℓ is a label (drawn from a countably infinite stock of labels) and o is an object which is a witness of some type. no two fields of a record can contain the same label. importantly, o can itself be a record. a record type is like a record except that the fields are of the form ⟨ℓ, t ⟩ where ℓ is a label as before and t is a type. the basic intuition is that a record, r is a witness for a record type, t , just in case for each field, ⟨ℓ i , t i ⟩, in t there is a field, ⟨ℓ i , o i ⟩, in r where o i : t i . (note that this allows for the record to have additional fields with labels not included in the fields of the record type.) the types within fields in record types may depend on objects which can be found in the record which is being tested as a witness for the record type. we use a graphical display to represent both records and record types where each line represents a field. example (5) represents the type of records which can be used to model situations where a man runs. it is possible to combine record types. suppose that we have two record types c 1 and c 2 : in this case, c 1 ∧ c 2 is a type; more specifically, a meet type. in general if t 1 and t 2 are types then t 1 ∧ t 2 is a type and a : t 1 ∧ t 2 iff a : t 1 and a : t 2 . a meet type t 1 ∧ t 2 of two record types can be simplified to a new record type by a process similar to unification in feature-based systems. if t 1 and t 2 are record types then there will be a type t 1 ∧ . t 2 equivalent to t 1 ∧ t 2 (in the sense that something will be of the first type if and only if it is of the second type). the operation ∧ . is referred to as merge. 4 relating spa and ttr 4.1 the basic idea we define a mapping, σ, from types in ttr to patterns (types) of neural activity represented as vectors in spa foot_4 . on the basis of this we define neural judgement conditions of the form "agent a judges s to be of type t if a particular neural condition involving σ(t ) holds. the connective here is a conditional rather than a biconditional because we allow more than one pattern of neural activity to correspond to the same ttr judgement. for example, a may judge s to be of t because of, say visual perception, or because s has been stored in memory corresponding to the witness cache discussed in @xcite . this is in contrast to the proposal in @xcite which defines a function from types to patterns (types) of neural activity but does not take the additional step of giving neural judgement conditions. the move from representing types to representing judgements, which belong to the theory of action defined on the theory of types, appears to us to be a conceptual improvement. essentially, the correspondence we define characterizes the brain activity of an agent when engaged in an act of making a type judgement, rather than simply giving a neural representation of a type. this seems promising for building a theory of how an embodied agent perceives its environment rather than creating a neural representation of a type without specifying how it would link to the world. another way in which the approach taken here differs from that of @xcite is that the approach to representing the structure of complex types relies on the vector operations used in spa, such as circular convolution, rather than the phasing of neural activity as in @xcite following in a tradition of neural modelling stemming from @xcite . this raises a question of whether the modelling in terms of vector operations reveals enough structure which we will leave open in this paper. our aim in this paper is to begin mapping out a possible correspondence between ttr and spa. we do not yet have a complete definition and there are a number of questions about what we have so far. nevertheless, we hope that what we have represents a promising beginning. below, we often use t ∼ t to mean σ(t ) = t. we will also often use t to represent σ(t ) we will frequently let equality or near similarity between two patterns of neural activation in spa terms characterize ttr neural judgement conditions. in doing this we will exploit the fact the the dot product of two (nearly) identical vectors a and b, a • b is approximately equal to 1 (see @xcite . ## summary and conclusions in this paper, we took initial steps towards relating ttr to spa, with mostly encouraging results. we accounted for basic types, perceptual and cachebased judgements, singleton types, record types, meet types and merging of record types, ptypes, and subtyping. as indicated above, more work is needed to account for subtyping and judgements involving ptypes. work is ongoing to cover more aspects of ttr in spa, including records and functions. in addition to these, several ttr elements remain to be covered, including join types, asymmetric merge, and type stratification to name but a few. the benefit of succeeding with this effort would be a true hybrid between formal and neural semantics that could potentially have the benefits of both but the drawbacks of neither. we also hope that this work may throw light on many puzzling issues regarding the relation between formal and neural semantics.