Semantic Classification and Learning Using a Linear Tranformation Model in a Probabilistic Type Theory with Records (2021)

Abstract: Starting from an existing account of semantic classification and learning from interaction formulated in a Probabilistic Type Theory with Records, encompassing Bayesian inference and learning with a frequentist flavour, we observe some problems with this account and provide an alternative account of classification learning that addresses the observed problems. The proposed account is also broadly Bayesian in nature but instead uses a linear transformation model for classification and learning.

## introduction a probabilistic type theory was presented in @xcite and @xcite , which extends cooper's type theory with records (ttr, @xcite ; @xcite ). this theory, probabilistic type theory with records (probttr) assigns probability values, rather than boolean truth-values, to type judgements. ttr has been used previously for natural language semantics (see, for example, @xcite and @xcite ), and to analyse semantic coordination and learning (for example, @xcite ). it has also been applied to the analysis of interaction in dialogue (for example, @xcite and breitholtz (2020)), in modelling robotic states and spatial cognition (for example, @xcite ), and to the problem of learning perceptual meaning from interaction @xcite . we believe that a probabilistic version of ttr could be useful in all these domains. two main considerations motivated recasting ttr in probabilistic terms. first, a probabilistic type theory offers a natural framework for capturing the gradience of semantic judgements. this allows it to serve as the basis for an account of vagueness in interpretation, as shown by @xcite . second, and this is the focus of the present paper, such a theory lends itself to developing a model of semantic classification and learning that can be straightforwardly integrated into more general probabilistic explanations of learning and inference. this paper presents an account of probabilistic classification (inference) and learning in probttr based on a linear transformation model. recent work @xcite @xcite has developed and used a bayesian account of classification and a learning theory with a frequentist flavour. below in section 2, we first introduce ttr and probttr, and explain briefly how a naive bayes classifier can be formulated in probttr. we then review earlier work on semantic classification and learning using probttr, and introduce a simple language game (the fruit recognition game) that has been used as an example there. in section 3, we note some drawbacks of the frequentist account of classification and learning, motivating the exploration of alternative accounts. the main contribution of this paper is the account of semantic classification and learning using a linear transformation model presented in section 4. we show how classification (section 4.2) and learning (section 4.3) is handled in this account, again taking the fruit recognition game as our example. in section 4, we provide conclusions and point towards future work. ## background this section reviews the background needed to follow the rest of the paper: ttr, probabilistic ttr fundamentals, and bayes nets and naive bayes classifiers. ## semantic classification and learning using a linear transformation model in this section we present a model where the probabilities given in j are used to compute a linear transformation model θ that generalises over j and which is used to compute the conditional probabilities used in classification. such a model can be made more computationally efficient than the frequentist model, and is also compatible with the way probabilistic inference and learning is encoded in neural network models. ## classification using a transformation model when we use a transformation model for classification, the idea is to evaluate the likelihood of a distribution dc (s) which according to the model θ accounts for the observed evidence d e i (s) 4 . this means we need to represent meta-level probabilities of a probability distribution given another probability distribution. when classifying fruits in the apple game, we want to estimate the probability of the class value types given the observed distribution over the evidence value types. the probability for a particular distribution d c (s) is estimated using bayesian marginalisation: if we want to find the distribution d c(s) that maximises the observed evidence in light of the model, for a single evidence variable type e i we want to find and for evidence variable types e κ 1 , . . . e κ n we want to find (30 to classify a situation s with respect to each c @xmath0, (31) p(s : c j ) = dc (s) j in the fruit game, for each c @xmath1, (32) p(s : c j ) = dc (s) @xmath2,1] 2 p(d col (s)|z)p(d shp (s)|z)prior(z) conditional probabilities instead of estimating the conditional probability of an evidence value type given a class value type, as in the frequentist model, we here estimate the conditional probability of a distribution over evidence value types given a distribution over class value types belonging to the class variable type. the probability of an observed probability distribution d e i (s) over evidence value types e @xmath3: probability density functions in reality, d c (s) is a continuous variable (since it is a probability distribution), so p(d c (s)) = 0. basically, since there are uncountably many possible probability distributions, the probability of any one of them is zero. however, the same kind formula works for probability density functions (pdfs) which give probability distributions over a continuous variable. writing f for pdf, we have: as before, when classifying we want the the distribution dc (s) that maximises the probability that the model θ accounts for the evidence. for a single evidence variable, this is for n evidence variables: priors for d c there are many ways to give a prior for d c . we know that (1) it must be a function of θ and (2) must be a probability distribution. one way to satisfy these requirements is to follow the same recipe as for evidence (but with no dependency). according to this recipe, we have the formula: dc (s) = softmax(θ κ c ) this way, there is a functional dependency from θ to dc (s), and therefore any prior density function on θ yields another density function on dc (s), called hereafter f prior ( dc (s)) foot_2 . note that here, θ κ c is a vector, not a matrix. the priors of each element in θ can be an independent uniform distribution over reals. ## conclusions and future work previous work proposed a frequentist bayesian account of semantic classification and learning formulated in terms of a probabilistic type theory with records. we observed some problems with this approach, including accounting for the effect of definitions as opposed to examples in learning meanings from interaction, and proposed an alternative account of learning that keeps the broadly bayesian model of classification, but where classification is based on a linear transformation model. we argued that the account proposed here can address some of the problems of the frequentist account. in future work, we wish to implement both the frequentist model (including some amendments to address observed problems) and the linear transformation model, and evaluate and compare them practically with respect to the problems p1-p5 noted above.