Discrete and Probabilistic Classifier-based Semantics (2020)

Abstract: We present a formal semantics (a version of Type Theory with Records) which places classifiers of perceptual information at the core of semantics. Using this framework, we present an account of the interpretation and classification of utterances referring to perceptually available situations (such as visual scenes). The account improves on previous work by clarifying the role of classifiers in a hybrid semantics combining statistical/neural classifiers with logical/inferential aspects of meaning. The account covers both discrete and probabilistic classification, thereby enabling learning, vagueness and other non-discrete linguistic phenomena.

## background we are developing a formal judgement-based semantics where notions such as perception, classification, judgement, learning and dialogue coordination play a central role @xcite @xcite @xcite @xcite . a key idea introduced in @xcite and @xcite is the modelling of referential meanings as classifiers of real-valued (perceptual) data, and training these classifiers in interaction with the world and other agents. there is a growing body of work in computational and formal semantics which is in line with the approach taken here @xcite @xcite @xcite . we propose a way of connecting this line of work to formal semantics, to enable combining it with the successes of formal semantics @xcite . using a type theory with records @xcite , @xcite presents a formal semantics for perception, using classifiers to model the relation between perception and linguistic utterances. this paper substantially improves on the formal machinery used in @xcite and incorporates insights from the implemented version of ttr @xcite as well as related work on visual question answering @xcite . ## ttr: a brief introduction we will be formulating our account in a type theory with records (ttr). we can here only give a brief and partial introduction to ttr; see also @xcite and @xcite . to begin with, s : t is a judgment that some s is of type t . to make explicit who is making this judgment, the oftype relation may be subscripted with an agent a, as in : a t . one basic type in ttr is ind, the type of an individual; another basic type is r, the type of real numbers. given that t 1 and t 2 are types, t 1 → t 2 is a functional type whose domain is objects of type t 1 and whose range is objects of type t 2 . next, we introduce records and record types. if a 1 : t 1 , a 2 : t 2 (a 1 ), . . . , a n : t n (a 1 , a 2 , . . . , a n-1 ), where t (a 1 , . . . , a n ) represents a type t which depends on the objects a 1 , . . . , a n , the record to the left in figure 1 is of the record type to the right. in figure 1 , ℓ 1 , . . . ℓ n are labels which can be used elsewhere to refer to the values associated with them. a sample record and record type is shown in figure 2 . types constructed with predicates may be dependent. this is represented by the fact that arguments to the predicate may be represented by labels used on the left of the ':' elsewhere in the record type. in figure 2 , the type of c man is dependent on ref (as is c run ). if r is a record and ℓ is a label in r, we can use a path r.ℓ to refer to the value of ℓ in r. similarly, if t is a record type and ℓ is a label in t , t .ℓ refers to the type of ℓ in t . records (and record types) can be nested, so that the value of a label is itself a record (or record type). as can be seen in figure 2 , types can be constructed from predicates, e.g., "run" or "man". such types are called ptypes and correspond roughly to propositions in first order logic. given a set of predicates and a set of possible arguments, the set of possible ptypes is ptype, thus allowing for polymorphic predicates. the arity of a ptype p is a set of tuple of types arity(p ). for example arity(run) = { ind }. a fundamental type-theoretical intuition is that something of a ptype t is whatever it is that counts as a proof of t . one way of putting this is that "propositions are types of proofs". in figure 2 , we simply use prf(t ) as a placeholder for proofs of t ; below, we will show how low-level perceptual input can be included in proofs. foot_0 ## the left-or-right game as an illustration, we follow @xcite in using a simple dialogue game called the left-or-right (lor) game. in this game, one agent places objects on a square surface, and the other agent classifies these objects as being to the right or not. in first language acquisition, training of perceptual meanings typically takes place in situations where the referent is in the shared focus of attention and thus perceivable to the dialogue participants. we assume that our dps (dialogue participants) are able to establish a shared focus of attention. a (simple) sensor collects some information (sensor input) from the environment and emits a real-valued vector. the sensor is assumed to be oriented towards the object in shared focus of attention. ## putting the model to work in this section, we show an illustrative example of how the framework above might be put to work in the context of the lor game, when contextually interpreting utterances and when deciding whether they describe the situation correctly.