B ayesian Classification and Inference in a Probabilistic Type Theory with Records (2021)

Abstract: We propose a probabilistic account of semantic inference and classification formulated in terms of probabilistic type theory with records, building on Cooper et. al. (2014) and Cooper et. al. (2015). We suggest probabilistic type theoretic formulations of Naive Bayes Classifiers and Bayesian Networks. A central element of these constructions is a type-theoretic version of a random variable. We illustrate this account with a simple language game combining probabilistic classification of perceptual input with probabilistic (semantic) inference.

## introduction a probabilistic type theory was presented in @xcite and @xcite , which extends cooper's type theory with records (ttr, @xcite @xcite . non-probabilistic ttr (in common with other type theories) works with judgements of the form a : t ("a is of type t ") and assumes that such judgements are categorical. in probabilistic ttr (probttr) we associate probabilities with judgements: p(a : t ) ("the probability that a is of type t "). ttr has been used previously for natural language semantics (see, for example, @xcite , and to analyze semantic coordination and learning (for example, @xcite . it has also been applied to the analysis of interaction in dialogue (for example, @xcite , and in modelling robotic states and spatial cognition (for example, @xcite . two main considerations motivated recasting ttr in probabilistic terms. first, a probabilistic type theory offers a natural framework for capturing the gradience of semantic judgements. this allows it to serve as the basis for an account of vagueness in interpretation, as shown by @xcite . second, such a theory lends itself to developing a model of semantic learning that can be straightforwardly integrated into more general probabilistic explanations of learning and inference. furthermore, we believe this will provide the foundation for a unified probabilistic account of natural language semantics that accounts for reasoning (logical as well as non-logical/enthymematic as in @xcite , learning (semantic and factual) and interaction, and that integrates low-level, subsymbolic real-valued perceptual information and high-level symbolic information @xcite . in this paper we suggest a way of incorporating a probabilistic inference and classification into probttr. we do this because we believe that vagueness, learning, inference and classification are central rather than peripheral notions in semantics, and that probabilistic reasoning is central to all of them. also, in contrast to an approach where e.g. classifiers are implemented outside the semantic theory, we want the reasoning underlying an agent's behaviour to be as transparent as possible to the agent itself (and thereby potentially also to its interlocutors). to incorporate a probabilistic inference and classification into probttr, we will need to introduce a probttr version of a random variable, not discussed in @xcite . we will also show how probabilistic classification of perceptual evidence can be combined with probabilistic reasoning. we first provide a brief overview of ttr and probabilistic ttr. section 4 provides some background on probabilistic inference and classification. section 5 introduces conditional probabilities and defines a type theoretic version of a random variable. we use these variables to characterise a naive bayes classifier in section 6. we illustrate naive bayes classification with an example of semantic classification. in section 7 we show how probabilistic perception and reasoning can be combined in probttr. we then introduce a probttr characterisation of bayesian networks, and briefly discuss semantic learning. in section 10 we present our conclusions and discuss directions for future work. ## ttr: a brief introduction we give a brief sketch of those aspects of ttr which we will use in this paper. for more detailed accounts see @xcite ; cooper (in prep). another kind of complex type is record types. in ttr records are modelled as finite sets of fields. each field is an ordered pair, ℓ, o , where ℓ is a label (drawn from a countably infinite stock of labels) and o is an object which is a witness of some type. no two fields of a record can contain the same label. importantly, o can itself be a record. a record type is like a record except that the fields are of the form ℓ, t where ℓ is a label as before and t is a type. the basic intuition is that a record, r is a witness for a record type, t , just in case for each field, ℓ i , t i , in t there is a field, ℓ i , o i , in r where o i : t i . (note that this allows for the record to have additional fields with labels not included in the fields of the record type.) the types within fields in record types may depend on objects which can be found in the record which is being tested as a witness for the record type. we use a graphical display to represent both records and record types where each line represents a field. example (1) represents the type of records which can be used to model situations where a man runs. we will introduce further details of ttr as we need them in subsequent sections. ## probabilistic ttr fundamentals the core of probttr is the notion of probabilistic judgement. there are two kinds of judgement corresponding to the two kinds of judgement in non-probabilistic ttr. the first is a judgement that a situation, s, is of type, t , with some probability. p(s : t ) is the probability that s is a witness for t . the second is a judgement that there is some witness of type t . p(t ) is the probability that there is some witness for t . this introduces a distinction that is not normally made explicit in the notation used in probability theory. it is useful to have type theoretic objects corresponding to judgements that a situation is of a type. following terminology first introduced in @xcite , we call these austinian propositions. a probabilistic austinian proposition is an object (a record) that corresponds to, or encodes, a probabilistic judgement. probabilistic austinian propositions are records of the type in (3). ## probabilistic inference and classification a bayesian network is a directed acyclic graph (dag) foot_0 . the nodes of the dag are random variables, each of whose values is the probability of one of the set of possible states that the variable denotes. its directed edges express dependency relations among the variables. when the values of all the variables are specified, the graph describes a complete joint probability distribution (jpd) for its random variables @xcite . @xcite give the example bayesian network in figure 1 . the only directly observable evidence is whether it is cloudy or not, and the queried variable is whether the grass is wet or not. we do not know if it is raining, or whether the sprinkler is on. both of these factors depend on whether it is cloudy, and both affect the grass being wet. from this bayesian network we can compute the marginal probability of the grass being wet (@xmath0. here, s, r and l can be either t(rue) or f(alse). the bayesian network in figure 1 allows us to simplify the computation of this jpd by encoding independence relations between variables, so that: a standard naive bayes model is a bayesian network with a single class variable c that influences a set of evidence variables e 1 , . . . , e n (the evidence), which do not depend on each other. figure 2 illustrates the relation between evidence variables and a class variable in a naive bayes classifier. a naive bayes classifier computes the marginal probability of a class, given the evidence: ## type theoretic probabilistic inference and classification we now turn to an account of probabilistic classification in probttr. we first show how probabilistic inference can be modelled in probttr. we then provide a naive bayes classifier with a detailed example. finally, we generalise this account to bayesian networks. ## semantic classification: example we will now illustrate classification in probttr using a naive bayes classifier for fruits. we can imagine this classification taking place in the setting of an apple recognition game. in this game a teacher shows a learning agent fruits (for simplicity, we assume there are only apples and pears in this instance of the game). the agent makes a guess, the teacher provides the correct answer, and the agent learns from these observations. (this paper describes only the classification step, leaving the learning step for future work.) we will use shorthand for the types corresponding to an object being an apple vs. a pear (17) a. @xmath1: ind c apple : apple(x) b. @xmath2: ind c pear : pear(x) we take it that the probability of judgements that something is of type ind is always 1.0, and that (18) p(s : so that e.g. if (19) @xmath3, then (20) p(s : apple) = p(prf : apple(a)) . furthermore, we will assume that the objects in the apple recognition game have one of two shapes (a-shape or p-shape) and one of two colours (green or red). we define shorthands for the record types involved. the class variable type is fruit, with value types r(fruit) = {apple, pear}. the evidence variable types are (i) col(our), with value types r(col) = {green, red}, and (ii) shape, with value types r(shape) = {ashape, pshape}. figure 3 shows the evidence and class types of the apple recognition game in a simple bayesian network. ## conclusions here, we have proposed a bayesian account of semantic classification and inference formulated in terms of probabilistic type theory. we have suggested probabilistic type theoretic formulations of naive bayes classifiers and bayesian networks. a central element of these constructions is a probttr version of a random variable. future work includes applying bayesian inference and classification in probttr to a variety of problems in natural language semantics, including vagueness (where some initial steps are taken in @xcite ), probabilistic reasoning in dialogue, and learning grounded meanings from interaction (along the lines of @xcite ). we will also implement this integrated system in order to demonstrate its viability as a computational model of natural language learning, reasoning and interaction.