S PARSE COMMUNICATION VIA MIXED DISTRIBUTIONS
A BSTRACT
Neural networks and other machine learning models compute continuous representations, while humans communicate mostly through discrete symbols.
Reconciling these two forms of communication is desirable for generating human-readable interpretations or learning discrete latent variable models, while maintaining endto-end differentiability.
Some existing approaches (such as the Gumbel-Softmax transformation) build continuous relaxations that are discrete approximations in the zero-temperature limit, while others (such as sparsemax transformations and the Hard Concrete distribution) produce discrete/continuous hybrids.
In this paper, we build rigorous theoretical foundations for these hybrids, which we call “mixed random variables.
Our starting point is a new “direct sum” base measure deﬁned on the face lattice of the probability simplex.
From this measure, we introduce new entropy and Kullback-Leibler divergence functions that subsume the discrete and differential cases and have interpretations in terms of code optimality.
Our framework suggests two strategies for representing and sampling mixed random variables, an extrinsic (“sample-and-project”) and an intrinsic one (based on face stratiﬁcation).
We experiment with both approaches on an emergent communication benchmark and on modeling MNIST and Fashion-MNIST data with variational auto-encoders with mixed latent variables.
Our code is publicly available.
I NTRODUCTION
Historically, discrete and continuous domains have been considered separately in machine learning, information theory, and engineering applications: random variables (r.v.) and information sources are chosen to be either discrete or continuous, but not both (Shannon, 1948).
In signal processing, one needs to opt between discrete (digital) and continuous (analog) communication, whereas analog signals can be converted into digital ones by means of sampling and quantization.
Discrete latent variable models are appealing to facilitate learning with less supervision, to leverage prior knowledge, and to build more compact and interpretable models.
However, training such models is challenging due to the need to evaluate a large or combinatorial expectation.
Existing strategies include the score function estimator (Williams, 1992; Mnih & Gregor, 2014), pathwise gradients (Kingma & Welling, 2014) combined with a continuous relaxation of the latent variables (such as the
Concrete distribution, Maddison et al.
; Jang et al.
(2017)), and sparse parametrizations (Correia et al., 2020).
Pathwise gradients, in particular, require continuous approximations of quantities that are inherently discrete, sometimes requiring proxy gradients (Jang et al., 2017; Maddison et al., 2017), sometimes giving the r.v. different treatment in different terms of the same objective (Jang et al., 2017), sometimes creating a discrete-continuous hybrid (Louizos et al., 2018).
Since discrete variables and their continuous relaxations are so prevalent, they deserve a rigorous mathematical study.
Throughout, we will use the name mixed variable to denote a hybrid variable that takes on both discrete and continuous outcomes.
This work takes a ﬁrst step into a rigorous study of mixed variables and their properties.
We will call communication through mixed variables sparse communication : its goal is to retain the advantages of differentiable computation but still be able to represent and approximate discrete symbols.
Our main contributions are:
• We provide a direct sum measure as an alternative to the Lebesgue and counting measures used for continuous and discrete variables, respectively (Halmos, 2013).
The direct sum measure hinges
[Table]
[Table]
on a face lattice stratiﬁcation of polytopes, including the probability simplex, avoiding the need for
Dirac densities when expressing densities with point masses in the boundary of the simplex (
[Table] [Table]
• We use the direct sum measure to formally deﬁne K th
-dimensional
mixed random variables .
We provide extrinsic (“sample-and-project”) and intrinsic (based on face stratiﬁcation) characterizations of these variables, leading to several new distributions: the K-D Hard Concrete, the Gaussian
Sparsemax
[Table]
and the Mixed Dirichlet (summarized in Table 1).
See Figure 1 for an illustration
• We propose a new direct sum entropy and Kullback-Leibler divergence , which decompose as a sum of discrete and continuous (differential) entropies/divergences.
We provide an interpretation in terms of optimal code length, and we derive an expression for the maximum entropy (§4).
• We illustrate the usefulness of our framework by learning mixed latent variable models in an
emergent communication task and with VAEs to model Fashion-MNIST and MNIST data (§5).
B ACKGROUND
We assume throughout an alphabet with K ≥ 2 symbols, denoted [ K ] = { 1 , . . .
, K } .
Symbols can be encoded as one-hot vectors e k .
R K denotes the K -dimensional Euclidean space, R K> 0 its strictly positive orthant, and (cid:52) K − 1 ⊆
R K the probability simplex , (cid:52) K − 1 :=
{ y ∈ R K | y ≥ 0 , 1 (cid:62) y
= 1 } , with vertices { e 1 , . .
, e K } .
Each y ∈ (cid:52) K − 1 can be seen as a vector of probabilities for the K symbols, parametrizing a categorical distribution over [ K ] .
The support of y ∈ (cid:52) K − 1 is the set of nonzero-probability symbols supp( y ) :
= { k ∈
[ K ] | y k > 0 } .
The set of full-support categoricals corresponds to the relative interior of the simplex, ri( (cid:52) K − 1 ) :
= { y ∈ (cid:52) K − 1 | supp( y ) =
[ K ] } .
2. T RANSFORMATIONS FROM R K TO (cid:52) K −
In many situations, there is a need to convert a vector of real numbers z ∈ R K (scores for the several symbols, often called logits ) into a probability vector y ∈ (cid:52) K − 1 .
The most common choice is the softmax transformation (Bridle, 1990): y = softmax( z ) ∝ exp( z ) .
Since the exponential function is strictly positive, softmax reaches only the relative interior ri
( (cid:52) K − 1 ) , that is, it never returns a sparse probability vector.
To encourage more peaked distributions (but never sparse) it is common to use a temperature parameter β > 0 , by deﬁning softmax β ( z ) :
= softmax( β − 1 z ) .
The limit case β → 0 + corresponds to the indicator vector for the argmax , which returns a one-hot distribution indicating the symbol with the largest score.
While the softmax transformation is differentiable (hence permitting end-to-end training with the gradient backpropagation algorithm), the argmax function has zero gradients almost everywhere.
With small temperatures, numerical issues are common.
A direct sparse probability mapping is sparsemax (Martins & Astudillo, 2016), the Euclidean projection onto the simplex: sparsemax( z ) :=
arg min y ∈(cid:52)
K − 1 (cid:107) y − z
(cid:107) .
Unlike softmax, sparsemax reaches the full simplex (cid:52) K − 1 , including the boundary, often returning a sparse vector y , without sacriﬁcing differentiability almost everywhere.
With K = 2 and parametrizing z =
( z, 1 − z ) , sparsemax becomes a “hard sigmoid,” (cid:2) sparsemax (cid:0) ( z, 1 − z )
(cid:1)(cid:3)
1 = max { 0 , min { 1 , z }} .
We will come back to this point in §3.3.
Other sparse transformations include α -entmax (Peters et al., 2019;
Blondel et al., 2020), top- k softmax (Fan et al., 2018; Radford et al., 2019), and others (Laha et al., 2018; Sensoy et al., 2018; Kong et al., 2020; Itkina et al., 2020).
D ENSITIES OVER THE SIMPLEX
Let us now switch from deterministic to stochastic maps.
Denote by Y a r.v. taking on values in the simplex (cid:52) K − 1 with probability density function p Y ( y ) .
The density of a Dirichlet r.v.
Y ∼ Dir( α ) , with α ∈ R K> 0 is p Y ( y ; α )
∝ (cid:81)
Kk =
y α k − 1 k .
Sampling from a Dirichlet produces a point in ri( (cid:52) K − 1 ) , and, although a Dirichlet can assign high density to y close to the boundary of the simplex when α < 1 , a Dirichlet sample can never be sparse.
A Logistic-Normal r.v. (Atchison & Shen, 1980), also known as Gaussian-Softmax by analogy to other distributions to be presented, is given by the softmax-projection of a multivariate Gaussian r.v.
with mean z and covariance Σ : Y = softmax( z + Σ 12 N ) with N k ∼ N (0 , 1) .
Since the softmax is strictly positive, the Logistic-Normal places no probability mass to points in the boundary of (cid:52) K − 1 .
A Concrete (Maddison et al., 2017), or Gumbel-Softmax (Jang et al., 2017), r.v. is given by the softmax-projection of K independent Gumbel r.vs.
, each with mean z k : Y = softmax β ( z + G ) with
G k ∼ Gumbel(0 , 1) .
Like in the previous cases, a Concrete draw is a point in ri( (cid:52) K − 1 ) .
When the temperature β approaches zero, the softmax approaches the indicator for argmax and Y becomes closer to a categorical r.v.
(Luce, 1959; Papandreou & Yuille, 2011).
Thus, a Concrete r.v. can be seen as a continuous relaxation of a categorical.
2. T RUNCATED UNIVARIATE DENSITIES
Binary Hard Concrete.
For K = 2 , a point in the simplex can be represented as y =
( y, 1 − y ) and the simplex is isomorphic to the unit interval, (cid:52) 1 (cid:39) [0 , 1] .
For this binary case, Louizos et al.
(2018) proposed a Hard Concrete distribution which stretches the Concrete and applies a hard sigmoid transformation (which equals the sparsemax with K = 2 , per §2.1) as a way of placing point masses at 0 and 1 .
These “stretch-and-rectify” techniques enable assigning probability mass to the boundary of (cid:52) 1 and are similar in spirit to the spike-and-slab feature selection method (Mitchell & Beauchamp, 1988; Ishwaran et al., 2005) and for sparse codes in variational auto-encoders (Rolfe, 2017; Vahdat et al., 2018).
We propose in §3.3 a more general extension to K ≥ 2 .
Rectiﬁed Gaussian.
Rectiﬁcation can be applied to other continuous distributions.
A simple choice is the Gaussian distribution, to which one-sided (Hinton & Ghahramani, 1997) and two-sided rectiﬁcations (Palmer et al., 2017) have been proposed.
Two-sided rectiﬁcation yields a mixed r.v.
in [0 , 1] .
Writing y
=
( y, 1 − y ) and
z =
( z, 1 − z ) , this distribution has the following density:
[Equation]
where δ s ( y ) is a Dirac delta density.
Extending such distributions to the multivariate case is nontrivial.
For K > 2 , a density expression with Diracs would be cumbersome, since it would require a combinatorial number of Diracs of several “orders,” depending on whether they are placed at a vertex, edge, face, etc.
Another annoyance is that Dirac deltas have −∞ differential entropy, which prevents information-theoretic treatment.
The next section shows how we can obtain densities that assign mass to the full simplex while avoiding Diracs, by making use of the face lattice and a new base measure.
F ACE S TRATIFICATION AND M IXED R ANDOM V ARIABLES 3. T HE F ACE L ATTICE
Let P be a convex polytope whose vertices are bit vectors ( i.e. , elements of { 0 , 1 } K ).
Examples are the probability simplex (cid:52) K − 1 , the hypercube [0 , 1] K , and marginal polytopes of structured variables (Wainwright & Jordan, 2008).
The combinatorial structure of a polytope is determined by its face lattice (Ziegler, 1995, §2.2), which we now describe.
A face of P is any intersection of P with a closed halfspace such that none of the interior points of P lie on the boundary of the halfspace; we denote by F ( P ) the set of all faces of P and by ¯ F ( P ) :
= F ( P ) \ { ∅ } the set of proper faces.
We denote by dim( f ) the dimension of a face f ∈ ¯ F ( P ) .
Thus, the vertices of P are 0 -dimensional faces, and P itself is a face of the same dimension as P , called the “maximal face”.
Any other face of P can be regarded as a lower-dimensional polytope.
The set F ( P ) has a partial order induced by set inclusion, that is, it is a partially ordered set (poset), and more speciﬁcally a lattice .
The full polytope P can be decomposed uniquely as the disjoint union of the relative interior of its faces, which we call face stratiﬁcation : P = (cid:70)
f ∈ ¯ F ( P ) ri( f ) .
For example, the simplex (cid:52) 2 is composed of its face ri( (cid:52) 2 ) ( i.e. , excluding the boundary), three edges (excluding the vertices in the corners), and three vertices (the corners).
This is represented schematically in
Figure 2.
Likewise, the square [0 , 1] 2 is composed of its maximal face (0 , 1) 2 , four edges (excluding the corners) and four vertices (the corners).
The partition above implies that any subset A ⊆ P can be represented as a tuple A
=
( A f ) f ∈ ¯ F ( P ) , where A f = A ∩ ri( f ) ; and the sets
A f are all disjoint.
where points represent faces, and a directed path from a point to another represents face inclusion.
Simplex and hypercube.
If P is the simplex (cid:52) K − 1 , each face corresponds to an index set I ⊆
[ K ] , i.e. , it can be expressed as f I = { p ∈ (cid:52) K − 1 | supp( p ) ⊆
I} , with dimension dim( f I ) =
|I| − 1 : the set of distributions assigning zero probability mass outside I .
The set ¯ F ( (cid:52) K − 1 ) has 2 K −elements.
Since (cid:52) 1 (cid:39) [0 , 1] , the hypercube [0 , 1] K can be regarded as a product of K binary probability simplices.
It has 3 K nonempty faces – for each dimension we choose between { 0 } , { 1 } , and [0 , 1] .
We experiment with P ∈ {(cid:52) K − 1 , [0 , 1] K } in §5.
3. M IXED R ANDOM V ARIABLES
Categorical distributions assign probability only to the vertices of (cid:52) K − 1 .
In the opposite extreme, the densities listed in §2.2 assign probability mass to the maximal face only, that is, Pr { y ∈ f I } = (cid:82) f I p Y
( y )
=
for any I (cid:54)
=
[ K ] .
Any proper density (without Diracs) has this limitation, since nonmaximal faces have zero Lebesgue measure in R K − 1 .
While for K =
it is possible to circumvent this by deﬁning densities that contain Dirac functions (as in §2.3), this becomes cumbersome for
K > 2 .
Fortunately, there is a more elegant construction that does not require generalized functions.
The key is to replace the Lebesgue measure by a measure inspired by face stratiﬁcation.
Deﬁnition 1 (Direct sum measure) .
The direct sum measure on a polytope P is
[Equation]
where µ f is
the dim( f ) -dimensional Lebesgue measure for dim( f ) > 0 , and the counting measure for dim( f ) = 0 .
We show in App.
A that µ ⊕ is a valid measure on P under the product σ -algebra of its faces.
We can then deﬁne probability densities p ⊕ Y ( y ) w.r.t.
this base measure and use them to compute probabilities of measurable subsets of P .
Such distributions can equivalently be deﬁned as follows: (i) deﬁne a probability mass function P F ( f ) on ¯ F ( P ) , and (ii) for each face f ∈ ¯ F ( P ) , deﬁne a probability density p Y
| F ( y | f ) over ri( f ) .
Random variables with a distribution of this form have a discrete part and a continuous part, thus we call them mixed random variables .
This is formalized as follows.
Deﬁnition 2 (Mixed random variable) .
A mixed r.v. is a r.v.
Y over a polytope P , including the boundary.
Let F be the corresponding discrete r.v. over ¯ F ( P ) , with P F ( f ) =
Pr { y ∈ ri( f ) } .
Since the mapping Y → F is deterministic, we have p ⊕ Y ( y )
=
P F ( f ) p Y | F ( y | f ) for y ∈ ri( f ) .
The probability of a set A ⊆ P is given by:
[Equation]
Equation (3) may be regarded as a manifestation of the law of total probability mixing discrete and continuous variables.
Using (3), we can write expectations over a mixed r.v. as
[Equation]
where g : P → R .
[Equation] [Equation]
Both discrete and continuous distributions are recovered with our deﬁnition: If P F ( f ) = 0 for dim( f ) > 0 , we have a categorical distribution, which only assigns probability to the 0-faces.
In the other extreme, if P F ( P ) = 1 , we have a continuous distribution conﬁned to ri( P ) .
That is, mixed random variables include purely discrete and purely continuous r.vs.
as particular cases.
To parametrize distributions of high-dimensional mixed r.vs.
, it is not efﬁcient to consider all degrees of freedom suggested in Deﬁnition 2, since there can be exponentially many faces.
Instead, we need to derive parametrizations that exploit the lattice structure of the faces.
We next build upon this idea.
3. E XTRINSIC AND INTRINSIC CHARACTERIZATIONS
There are two possible characterizations of mixed random variables: an extrinsic one, where one starts with a distribution over the ambient space ( e.g. R K ) and then applies a deterministic, noninvertible, transformation that projects it to P ; and an intrinsic one, where one speciﬁes a mixture of distributions directly over the faces of P , by specifying P F and p Y | F for each f ∈ ¯ F ( P ) .
We next provide constructions for both cases: We extend the Hard Concrete and Rectiﬁed Gaussian distributions reviewed in §2.3, which are instances of the extrinsic characterization, to K ≥ 2 ; and we present a new Mixed Dirichlet distribution which is an instance of the intrinsic characterization.
K -D Hard Concrete.
We deﬁne the K -D Hard Concrete as the following generative story:
[Equation]
When K = 2 , sparsemax becomes a hard sigmoid and we recover the binary Hard Concrete (§2.3).
For K ≥ 2 this is a projection of a “stretched” Concrete r.v. onto the simplex – the larger λ , the higher the tendency of this projection to hit a non-maximal face of the simplex and induce sparsity.
Gaussian-Sparsemax.
A similar idea (but without any stretching required) can be used to obtain a sparsemax counterpart of the Logistic-Normal in §2.2, which we call “Gaussian-Sparsemax”:
[Equation]
Unlike the Logistic-Normal, the Gaussian-Sparsemax can assign nonzero probability mass to the boundary of the simplex.
When K = 2 , we recover the double-sided rectiﬁed Gaussian described in §2.3.
In that case, using Dirac deltas, the density with respect to the Lebesgue measure in R has the form in (1).
With θ 0 = 1 − erf( z/ ( √ 2 σ )),
[Equation]
and θ c = 1 − θ 0 − θ 1 , the same distribution can be expressed intrinsically via the density p ⊕ Y ( y ) =
P F ( f ) p Y | F ( y | f ) as
P F ( { 0 } ) =
[Equation]
, P F ( { 1 } ) = θ 1 , P F ([
[Equation]
[Equation]
[Equation]
[Equation]
For K > 2 , expressions for P F and p Y | F ( i.e. , an intrinsic representation) are less direct; we express those distributions as a function of the orthant probability of multivariate Gaussians in App.
B.
Mixed Dirichlet.
We now propose an intrinsic mixed distribution over (cid:52) K − 1 , the Mixed Dirichlet, whose generative story is as follows.
First, a face F = f I is sampled with probability
[Equation]
where w ∈ R K is the natural parameter (a.k.a. log-potentials ), φ ( f I ) ∈ {− 1 , 1 } K is the sufﬁcient statistic, and log Z ( w ) is the log-normalizer.
We then parametrize a Dirichlet distribution over the relative interior of f I , that is, Y | F
= f I ∼ Dir( α ( I )) ,
where α ( I ) ∈
R |I| > 0 .
For a compact parametrization, we have a single K -dimensional vector α of concentration parameters, one parameter per vertex, and α ( I ) gathers the coordinates of α associated with the vertices in f .
The normalizer of (8) can be evaluated in time O ( K ) via the forward algorithm (Baum & Eagon, 1967) on a directed acyclic graph (DAG) that encodes each non-empty corner I ⊆
[ K ] of the face lattice as a path.
Similarly, we can draw independent samples by stochastic traversals through this DAG.
The graph needed for this construct and the associated algorithms are detailed in App.
C.
I NFORMATION T HEORY FOR M IXED R ANDOM V ARIABLES
Now that we have the theoretical pillars for mixed random variables, we proceed to deﬁning information theoretic quantities for them: their entropy and Kullback-Leibler divergence.
Direct sum entropy and KL divergence.
The entropy of a r.v. X with respect to a measure µ is:
[Equation]
(9) where p X ( x ) is a probability density satisfying (cid:82)
X p X
( x )d µ ( x ) = 1 .
When X is ﬁnite and µ is the counting measure, the integral becomes a sum and we recover Shannon’s discrete entropy , which is non-negative and upper bounded by log |X| .
When X ⊆
R k is continuous and µ is the Lebesgue measure, we recover the differential entropy , which can be negative and, for compact X , is upper bounded by the logarithm of the volume of X .
When relaxing a discrete r.v. to a continuous one in a variational model ( e.g. , using the Concrete distribution), correct variational lower bounds require switching to differential entropy (Maddison et al., 2017)
(although discrete entropy is sometimes used (Jang et al., 2017)).
This is problematic since the differential entropy is not a limit case of the discrete entropy (Cover & Thomas, 2012).
Our direct sum entropy, deﬁned below, obviates this.
The key idea is to plug in (9) the direct sum measure (2).
Since Y → F is deterministic, we have
H ( Y, F ) =
H ( Y ) .
This leads to:
Deﬁnition 3 (Direct sum entropy and KL divergence) .
The direct sum entropy of a mixed r.v.
Y is
[Equation]
The KL divergence between distributions p ⊕ Y ≡ ( P F , p Y | F ) and q ⊕ Y ≡ ( Q F , q Y | F ) is:
[Equation]
As shown in Deﬁnition 3, the direct sum entropy and the KL divergence have two components: a discrete one over faces and an expectation of a continuous one over each face.
The KL divergence is always non-negative and it becomes + ∞ if supp( P F ) (cid:42) supp( Q F ) or if there is some face where supp( p Y | F = f ) (cid:42) supp( q Y | F = f ) .
1 App.
D provides more information theoretic extensions.
Relation to optimal codes.
The direct sum entropy and KL divergence have an interpretation in terms of optimal coding, described in Proposition 1 for the case P
= (cid:52) K − 1 and proven in App.
D.
In words, the direct sum entropy is the average length of the optimal code where the sparsity pattern of y ∈ (cid:52) K − 1 must be encoded losslessly and where there is a predeﬁned bit precision for the fractional entries of y .
On the other hand, the KL divergence between p ⊕ Y and q ⊕ Y expresses the additional average code length if we encode variable Y ∼ p ⊕ Y ( y ) with a code that is optimal for distribution q ⊕ Y ( y ) , and it is independent of the required bit precision.
Proposition 1.
Let Y be a mixed r.v. in (cid:52) K − 1 .
In order to encode the face of Y losslessly and to ensure an N -bit precise encoding of Y in that face we need the following bits on average:
[Equation]
Entropy of Gaussian-Sparsemax.
Revisiting the Gaussian-Sparsemax with K = 2 (§3.3) and using the intrinsic representation (7), the direct sum entropy becomes
[Equation]
This leads to simple expressions for the entropy and KL divergence, detailed in App.
E. We use these expressions in our experiments with mixed bit-vector VAEs in §5.
Entropy of Mixed Dirichlet.
The intrinsic representation of the Mixed Dirichlet (§3.3) allows for O ( K ) computation of H ( F ) and
D KL ( P F || Q F ) via dynamic programming, necessary for H ⊕ ( Y ) and D ⊕ KL ( p ⊕ Y || q ⊕ Y ) (see App.
C for details).
The continuous parts H ( Y | F ) and E
P F [ D KL ( p Y | F = f
|| q Y
| F = f )] require computing an expectation with an exponential number of terms (one per proper face) and can be approximated with an MC estimate by sampling from P F and assessing closed-form the differential entropy of Dirichlet distributions over the sampled faces.
Maximum entropy density in the full simplex.
An important question is to characterize maximum entropy mixed distributions.
If we consider only continuous distributions, conﬁned to the maximal face ri( (cid:52) K − 1 ) , the answer is the ﬂat distribution, with entropy − log (cid:0) ( K − 1)!
(cid:1) , corresponding to a deterministic F which puts all probability mass in this maximal face.
But constraining ourselves to a single face is quite a limitation, and in particular knowing this constraint provides valuable information that intuitively should reduce entropy.
2 What if we consider densities that assign probability to the boundaries?
This is answered by the next proposition, proved in App.
F.
Proposition 2 (Maxent mixed distribution on simplex) .
Let Y be a mixed r.v. on (cid:52) K − 1 with
P F ( f ) ∝ 2 N (dim( f )) dim( f )!
and p Y | F ( y | f ) uniform for each f ∈ ¯ F ( (cid:52) K − 1 ) .
Then, Y has maximal direct sum entropy H ⊕ N, max ( Y ) .
The value of the entropy is:
[Equation]
) where L ( α ) n ( x ) denotes the generalized Laguerre polynomial (Sonine, 1880).
For example, for K = 2 , the maximal entropy distribution is P F ( { 0 } ) =
P F ( { 1 } )
=
1 / (2 + 2 N ) ,
P F ([0 , 1])
= 2 N / (2 + 2 N ) , with H ⊕ N, max ( Y ) = log(2 + 2 N ) .
Therefore, in the worst case, we need log 2 (2 + 2 N ) bits on average to encode Y ∈ (cid:52) 1 with bit precision N .
For K = 3 , H ⊕ N, max ( Y ) = log(3 + 3 · 2 N + 2 2 N − 1 ) .
A plot of the entropy as a function of K is shown in Figure 4, App.
F.
E XPERIMENTS
We report experiments in three representation learning tasks: 3 an emergent communication game, where we assess the ability of mixed latent variables over (cid:52) K − 1 to induce sparse communication
[Table]
between two agents; a bit-vector VAE modeling Fashion-MNIST images (Xiao et al., 2017), where we compare several mixed distributions over [0 , 1] K and study the impact of the direct sum entropy (§4); and a mixed-latent VAE , where we experiment with the Mixed Dirichlet over (cid:52) K − 1 to model
MNIST images (LeCun et al., 2010).
Full details about each task and the experimental setup are reported in App.
G. Throughout, we report our three mixed distributions described in §3.3 alongside the following distributions: Concrete (Maddison et al.
(2017); a purely continuous density),
GumbelSoftmax (Jang et al.
like above, but using a discrete KL term in a VAE, which introduces inconsistency), Gumbel-Softmax ST (Jang et al.
(2017); categorical latent variable, but using Concrete straight-through gradients), and Dirichlet and Gaussian, when applicable.
Emergent communication.
This is a cooperative game between two agents: a sender sees an image and emits a single-symbol message from a ﬁxed vocabulary [ K ] ; a receiver reads the symbol and tries to identify the correct image out of a set.
We follow the architecture in Lazaridou & Baroni (2020);
Havrylov & Titov (2017), choosing a vocabulary size of 256 and 16 candidate images.
For the mixed variable models, the message is a “mixed symbol”, i.e. , a (sparse) point in (cid:52) K − 1 .
Table 2 reports communication success (accuracy of the receiver), along with the average number of nonzero entries in the samples at test time.
Gaussian-Sparsemax learns to become very sparse, attaining the best overall communication success, exhibiting in general a better trade-off than the K -D Hard Concrete.
Bit-Vector VAE.
We model Fashion-MNIST following Correia et al.
(2020), with 128 binary latent bits, maximizing the ELBO.
Each latent bit has a uniform prior for purely discrete and continuous models, and the maxent prior (Prop. 2), which assigns probability 1 / 3 to each face of (cid:52) 1 , in the mixed case.
For Hard Concrete (Louizos et al., 2018)
and Gaussian-Sparsemax, we use mixed latent variables and our direct sum entropy, yielding a coherent objective and unbiased gradients.
For the Hard Concrete, we estimate H ( Y | X = x ) via MC; for Gaussian-Sparsemax, we consider both cases: using an MC estimate and computing the entropy exactly using the expression derived in §3.3.
In Table 2 (right) we report an importance sampling estimate (1024 samples) of negative log-likelihood (NLL) on test data, normalized by the number of pixels.
For Gaussian-Sparsemax, exact entropy computation improves performance, matching Gumbel-Softmax while being sparse.
Categorical is marginalized exactly.
[Table]
Mixed-Latent VAE on MNIST.
We model the binarized MNIST dataset using a mixed-latent VAE, with y ∈ ∆ 10 − 1 , a maximum-entropy prior on Y , and a factorized decoder.
We use a feed-forward encoder with a single-hidden-layer to predict the variational distributions ( e.g. , log-potentials w and concentrations α , for Mixed Dirichlet).
For Mixed Dirichlet, gradients are estimated via a combination of implicit reparametrization (Figurnov et al., 2018) and the score function estimator (Mnih & Gregor, 2014), see App.
C for details.
We report a single-sample MC estimate of distortion (D) and rate (R) as well as a 1000-samples importance sampling estimate of
NLL.
The Gaussian prior has access to all of the R 10 , whereas all other priors are constrained to the simplex ∆ 10 − 1 , namely, its relative interior (Dirichlet), its vertices (Gumbel-Softmax ST, and
Categorical), or all of it (Mixed Dirichlet).
The Dirichlet model clusters digits as well as a Gaussian
VAE does, while purely discrete models struggle to discover structure.
App.
G lists qualitative evidence.
Compared to a Dirichlet, the Mixed Dirichlet makes some plausible confusions (Fig. 3, left), but, crucially, it solves part of the problem by allocating digits to speciﬁc faces (Fig. 3, right).
Even though we employed a sampled self-critic, Mixed Dirichlet seems to suffer from variance of
SFE, which may explain why Mixed Dirichlet under-performs in terms of NLL.
R ELATED W ORK
Existing strategies to learn discrete latent variable models include the score function estimator (Williams, 1992; Mnih & Gregor, 2014) and pathwise gradients combined with a Concrete relaxation (Maddison et al., 2017; Jang et al., 2017).
The latter is often used with straightthrough gradients and by combining a continuous latent variable with a discrete entropy, which is theoretically not sound.
Besides the Concrete, continuous versions of the Bernoulli and Categorical have also been proposed (Loaiza-Ganem & Cunningham, 2019; Gordon-Rodriguez et al., 2020), but they are also purely continuous densities, assigning zero probability mass to the boundary of the simplex.
Our approach is inspired by discrete-continuous hybrids based on truncation and rectiﬁcation (Hinton & Ghahramani, 1997; Palmer et al., 2017; Louizos et al., 2018), which have been proposed for univariate distributions.
We generalize this idea to arbitrary dimensions replacing truncation by sparse projections to the simplex.
The direct sum measure and our proposed intrinsic sampling strategies are related to the concept of “manifold stratiﬁcation” proposed in the statistical physics literature (Holmes-Cerfon, 2020).
Other mixed r.vs. have also been recently considered (for a few special cases) in (Bastings et al., 2019; Murady, 2020; van der Wel, 2020).
For instance, Burkhardt &
Kramer (2019) induce sparsity in hierarchical models by deterministically masking the concentration parameters of a Dirichlet distribution yielding a special case of Mixed Dirichlet with degenerate P F , trained with the straight-through estimator.
Our paper generalizes these attempts, providing solid theoretical support for manipulating mixed distributions in higher dimensions.
Discrete and continuous representations in the context of emergent communication have been discussed in Foerster et al.
; Lazaridou & Baroni (2020).
Discrete communication is computationally more challenging because it prevents direct gradient backpropagation (Foerster et al., 2016;
Havrylov & Titov, 2017), but it is hypothesized that this “discrete bottleneck” forces the emergence of symbolic protocols.
Our mixed variables bring a new perspective into this problem, leading to sparse communication , which lies in between discrete and continuous communication and supports gradient backpropagation without the need for straight-through gradients.
C ONCLUSIONS
We presented a mathematical framework for handling mixed random variables, which are discrete/continuous hybrids.
Key to our framework is the use of a direct sum measure as an alternative to the Lebesgue-Borel and the counting measures, which considers all faces of the simplex.
We developed generalizations of information theoretic concepts for mixed symbols, and we experimented on emergent communication and on variational modeling of MNIST and Fashion-MNIST images.
We believe the framework described here is only scratching the surface.
For example, the Mixed
Dirichlet is just one example of an intrinsic mixed distribution; more effective intrinsic parametrizations may exist and are a promising avenue.
While our main focus was on the probability simplex and hypercube, mixed structured variables are another promising direction, enabled by our theoretical characterization of direct sum measures, which can be deﬁned for any polytope via their face lattice (Ziegler, 1995; Grünbaum, 2003).
Current methods for structured variables include perturbations (continuous, Corro & Titov (2019); Berthet et al.
Paulus et al.
(2020)), and sparsemax extensions (discrete, Niculae et al.
; Correia et al.
(2020)), lacking tractable densities.
Ethics statement.
We highlight indirect impact of our work through applications such as generation and explainable AI, where improved performance must be carefully scrutinized.
Our proof-of-concept emergent communication, like previous work, uses ImageNet, whose construction exhibits societal biases, including racism and sexism (Crawford & Paglen, 2019) that communicating agents may learn.
Even in unsupervised settings, biases may be learned by communicating agents.
Reproducibility statement.
We now discuss the efforts that have been made to ensure reproducibility of our work.
We state the full set of assumptions of our theoretical results and include complete proofs in App.
A, App. B, App.
C, App.
D, App.
E, and App.
F. Additionally, code and instructions to reproduce our experiments are available at https://github.com/deep-spin/ sparse-communication .
We report the standard error over 10 runs for the emergent communication experiment due to the high variance of results across seeds.
We include the type of computing resources used in our experiments in App.
G.5.
A CKNOWLEDGMENTS
We would like to thank Mário Figueiredo, Gonçalo Correia, and the DeepSPIN team for helpful discussions, Tim Vieira, who answered several questions about order statistics, Sam Power, who pointed out to manifold stratiﬁcation, and Juan Bello-Rivas, who suggested the name “mixed random variables.”
This work was built on open-source software; we acknowledge Van Rossum & Drake (2009); Oliphant (2006); Virtanen et al.
Walt et al.
(2011); Pedregosa et al.
(2011), and Paszke et al.
AF and AM are supported by the P2020 program MAIA (LISBOA-01-0247- FEDER045909), the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para
a
Ciência e Tecnologia through contract UIDB/50008/2020.
WA received funding from the European
Union’s Horizon 2020 research and innovation programme under grant agreement No(GoURMET).
VN is partially supported by the Hybrid Intelligence Centre, a 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientiﬁc Research ( https://hybrid-intelligence-centre.nl ).
A P ROOF OF W ELL -D EFINEDNESS OF D IRECT S UM M EASURE
We start by recalling the deﬁnitions of σ -algebras, measures, and measure spaces.
A σ -algebra on a set X is a collection of subsets, Ω ⊆ 2 X , which is closed under complements and under countable unions.
A measure µ on ( X, Ω) is a function from Ω to R ∪ {±∞} satisfying (i) µ ( A ) ≥ 0 for all
A ∈ Ω , (ii) µ ( ∅ ) = 0 , and (iii) the σ -additivity property: µ ( (cid:116) j ∈ N A j ) =
(cid:80) j ∈ N µ ( A j ) for every countable collections { A j } j ∈ N ⊆ Ω of pairwise disjoint sets in Ω .
A measure space is a triple ( X, Ω , µ ) where X is a set, A is a σ -algebra on X and µ is a measure on ( X, A ) .
An example is the Euclidean space X
= R K endowed with the Lebesgue measure, where Ω is the Borel algebra generated by the open sets ( i.e. the set Ω which contains these open sets and countably many Boolean operations over them).
The well-deﬁnedness of the direct sum measure µ ⊕ comes from the following more general result, which appears (without proof) as exercise I.6 in Conway (2019).
Lemma 1.
Let ( X k , Ω k , µ k ) be measure spaces for k = 1 , . . .
, K .
Then, ( X, Ω , µ ) is also a measure space, with X = (cid:76)
Kk =1 X k = (cid:81)
Kk =1 X k (the direct sum or Cartesian product of sets
X k ), Ω =
{ A ⊆ X |
A ∩ X k ∈ Ω k , ∀ k ∈ [ K ] } , and µ ( A ) = (cid:80)
Kk =
1 µ k
( A ∩ X k ) .
Proof.
First, we show that Ω is a σ -algebra.
We need to show that (i) if A ∈ Ω , then ¯ A ∈ Ω , and (ii) if A i ∈ Ω for each i ∈ N then (cid:83) i ∈ N A i ∈ Ω .
For (i), we have that, if A ∈ Ω , then we must have
A ∩ X k ∈ Ω k for every k , and therefore ¯ A ∩ X k = X
k \ A = X k \ ( A ∩ X k ) ∈
Ω k , since Ω k is a σ -algebra on X k .
This implies that ¯ A ∈ Ω .
For (ii), we have that, if A i ∈ Ω , then we must have
A i ∩ X k ∈ Ω k for every i ∈ N and k ∈ [ K ] , and therefore (cid:0)(cid:83) i ∈
N A
i (cid:1) ∩
X k = (cid:83)
i ∈ N ( A i ∩
X k ) ∈
Ω k , since Ω k is closed under countable unions.
This implies that (cid:83) i ∈ N A i ∈ Ω .
Second, we show that µ is a measure.
We clearly have µ ( A ) =
(cid:80)
Kk =
1 µ k
( A ∩ X k ) ≥ 0 , since each µ k is a measure itself, and hence it is non-negative.
We also have µ ( ∅ ) =
(cid:80) Kk =
1 µ k ( ∅ ∩ X k ) =
(cid:80)
Kk =
1 µ k ( ∅ )
= 0 .
Finally, if { A j } j ∈ N ⊆ Ω is a countable collection of disjoint sets, we have µ ( (cid:116) j ∈ N A j ) =
(cid:80)
Kk =
1 µ k ( (cid:116) j ∈ N
( A j ∩
X k ))
= (cid:80)
Kk =1 (cid:80) j ∈ N µ k
( A j ∩ X k ) =
(cid:80) j ∈ N
cid:80)
Kk =
1 µ k ( A j ∩ X k ) =
(cid:80) j ∈ N µ ( A j ) .
We have seen in §3 that the simplex (cid:52) K − 1 can be decomposed as a disjoint union of the relative interior of its faces.
Each of these relative interiors is an open subset of an afﬁne subspace isomorphic to R k − 1 , for k ∈ [ K ] , which is equipped with the Lebesgue measure for k > 1 and the counting measure for k = 1 .
Lemma 1 then guarantees that we can take the direct sum of all these afﬁne spaces as a measure space with the direct sum measure µ = µ ⊕ of Deﬁnition 1.
B K -D IMENSIONAL G AUSSIAN -S PARSEMAX
We derive expressions for the density p Y,F ( y, f ) for the sample-and-project case (stochastic sparsemax).
We assume without loss of generality that f = { 1 , 2 , . . .
, s } and that we want to compute the density p Y 2:
K ( y 2 , . .
, y s , 0 , . .
. , 0) for given y 2 , . .
, y s > 0 such that y 1 :
= 1 − (cid:80) sj =2 y j > 0 .
The process that generates the data is as follows: ﬁrst, z i ∼ p
Z i ( z i ) independently for i ∈
[ K ] .
Then, y is obtained deterministically from z 1: K as y = sparsemax( z 1: K ) .
We assume here that each p Z i ( z i ) is a univariate Gaussian distribution with mean µ i and variance σ 2 i .
We make use of the following well-known properties of multivariate Gaussians:
Lemma 2.
Let Z ∼ N ( µ, Σ) .
Then, for any matrix A , not necessarily square we have
AZ ∼ N
( Aµ, A Σ A (cid:62) ) .
Furthermore, splitting
Z
[Equation]
Y
[Equation]
(15) we have the following expression for the marginal distribution P X ( x ) :
[Equation]
[Equation]
and the following expression for the conditional distribution P X | Y ( x | Y = y ) :
[Equation]
[Equation]
We start by picking one index in the support of y – we assume without loss of generality that this pivot index is 1 and that the support set is { 1 , . . .
, s } as stated above – and introducing new random variables
U
i :
= Z i − Z 1 for i ∈ { 2 , . .
. , K } .
Note that these new random variables are not independent since they all depend on Z 1 .
Then, from the change of variable formula for non-invertible transformations, we have, for given y 2 , . .
, y s > 0 such that y 1 :
= 1 − (cid:80) sj
[Equation]
: p
[Equation]
( y
[Equation]
, y s , 0 , . .
=
p
[Equation]
[Equation] [Equation]
× |
[Equation]
|
[Equation] [Equation]
where u i =
y i − y 1 for i ∈ 2 , . . .
, s , which can written as u 2: s =
By 2: s + c with B =
I s − 1 − 1 s − 1 1 (cid:62) s −and c
= − 1 s − 1 .
The determinant of B is simply | B | = s .
Note that U 2: K = AZ 1: K , where A =
[ − 1 K − 1 , I K − 1 ] ∈ R ( K − 1) × K , that is
A =   − 1 1 0 · · ·− 1 0 1 · · ·...
− 1 0 0 ·
[Equation] [Equation]
From Lemma 2, we have that
[Equation]
and the marginal distribution p U 2: s ( u 2: s ) is
[Equation]
Note that this is a multivariate distribution whose covariance matrix is the sum of a diagonal matrix with a constant matrix.
We now calculate the conditional distribution of the variables which are not in the support conditioned on the ones which are in the support.
We have according to the notation in Lemma 2:
[Equation]
Using the Sherman-Morrison formula
[Equation]
we obtain
[Equation]
from which we get
[Equation]
and
[Equation]
Finally
[Equation]
also from Lemma
[Equation]
we get
[Equation] [Equation]
Note that this is again a multivariate Gaussian distribution with a covariance matrix ˜Σ which is the sum of a diagonal and a constant matrix.
Putting everything together, we get
[Equation]
where F ( v ; ˜Σ)
= (cid:82)
−∞ · · · (cid:82)
−∞ N (0; v, ˜Σ) is the negative orthant cumulative distribution of a multivariate Gaussian with mean v and covariance ˜Σ .
Efﬁcient Monte Carlo approximations of this integral have been proposed by Genz (1992).
Using again Lemma 2 but now in the reverse direction, the function F ( v ; ˜Σ) , setting v = µ ( s +1): K + (cid:80) si =1 σ
[Equation]
2 i ( y i − µ i ) (cid:80) si =1 σ − 2 i 1 K − s , can be reduced to a unidimensional integral:
[Equation]
[Equation]
where φ ( x ) and Φ( x ) are the p.d.f. and c.d.f.
of a standard Gaussian distribution, respectively, and we applied the change of variables formula in the last line, with u = Φ (cid:18) (cid:80) si =1 σ −
i ( t + y i − µ i ) / (cid:113)(cid:80) si =
[Equation]
σ
[Equation]
i (cid:19) , whose inverse is t =
[Equation]
the Jacobian of this transformation cancels with the φ -term.
We can compute the quantities above via the erf function and its inverse erf − 1 , as Φ( x ) = 12 (cid:16) 1 + erf (cid:16) x √ 2 (cid:17)(cid:17) and Φ − 1 ( u ) = √ 2erf − 1 (2 u − 1) .
When the variance is constant, σ i = σ ∀ i ∈
[ K ] , and since (cid:80) si =
y i = 1 , the expressions above simplify to
[Equation]
C S AMPLING FROM THE M IXED D IRICHLET D ISTRIBUTION
PyTorch code for a batched implementation of the procedures discussed in this section will be made available online.
Distribution over Face Lattice.
The distribution P F is a discrete exponential family whose support is the set of all 2 K − 1 proper faces of the simplex.
The computation of the natural parameter in
Equation (8) factorizes over vertices, but, because the empty face is not in the support of P F , the computation of the log-normalizer requires special attention.
The obvious strategy is to enumerate all assignments to ( F I , w (cid:62) φ ( F I )) , one per proper face.
This strategy is not feasible for even moderately large K .
We can, however, exploit a compact representation of the set of proper faces by encoding the face lattice in a directed acyclic graph (DAG) where each proper face is associated with a complete path from the DAG’s source to the DAG’s sink.
This DAG G =
( Q , A ) is a collection of states u ∈ Q ⊂ { 0 , . .
, K + 1 } × { 0 , 1 } × { 0 , 1 } and arcs ( u, v )
∈
A ⊆ Q × Q .
A path that ends in ( k, b, s ) ∈ Q corresponds to a non-empty subset of vertices if, and only if, s = 1 .
A path that contains ( k, b, s ) ∈ Q corresponds to a face that includes the k th vertex if, and only if, b = 1 .
If u = ( k 1 , b 1 , s 1 ) ∈ Q and v = ( k 2 , b 2 , s 2 ) ∈ Q , then u ≤ v iff k 1 ≤ k 2 .
The state (0 , 0 , 0) is the DAG’s unique source, and the state ( K + 1 , 0 , 1) is the DAG’s unique sink.
Because = 1 for the sink, no complete path ( i.e. , from source to sink) will correspond to an empty face.
An arc is a pair of states ( u, v ) , where u is the origin and v is the destination.
For every state ( k, b, s ) such that k < K , we have arcs (( k, b, s ) , ( k + 1 , 1 , 1)) and (( k, b, s ) , ( k + 1 , 0 , s )) .
For every state ( K, b, 1) , we have an arc to the ﬁnal state ( K + 1 , 0 , 1) .
By construction, the number of states (and arcs) in the DAG is proportional to K .
A complete path (from source to sink) has length K + 1 , and it uniquely identiﬁes a proper face.
To compute the log-normalizer of Equation (8), we run the forward algorithm through G , with an arc’s weight given by ( − 1 1 − b ) w k if the arc’s destination is the state ( k, b, s ) or 0 if the arc’s destination is the DAG’s sink.
Running the backward algorithm through G (that is, running the forward algorithm from sink to source) evaluates the marginal probabilities required for ancestral sampling.
Entropy of Mixed Dirichlet.
The intrinsic representation of a Mixed Dirichlet distribution allows for efﬁcient computation of H ( F ) and D KL ( P F || Q F ) necessary for H ⊕ ( Y ) and D ⊕ KL ( p ⊕ Y
|| q ⊕ Y ) .
The continuous parts H ( Y | F ) and E P F [ D KL ( p Y | F = f || q Y | F = f )] require solving an expectation with an exponential number of terms (one per proper face).
The distribution P F is a discrete exponential family indexed by the natural parameter w ∈ R K , its discrete entropy is H ( F ) =
log Z ( w ) − (cid:104) w , ∇ w log Z ( w ) (cid:105) , where ∇ w log Z ( w ) =
E [ φ ( F )] .
KL divergence from a member
Q F of the same family but with parameter v is given by
[Equation]
The forward-backward algorithm computes both the log-normalizer and its gradient (the expected sufﬁcient statistic) in a single pass through a DAG of size O ( K ) .
For results concerning entropy, cross-entropy, and relative entropy of exponential families see for example Nielsen & Nock (2010).
The continuous part H ( Y | F ) is the expectation of the differential entropy of Y | F = f , each a
Dirichlet distribution over the face f , under P F .
For small K we can enumerate the 2 K − 1 terms in this expectation, since the entropy of each Dirichlet is known in closed-form.
In general, for large K , we can obtain an MC estimate by sampling independently from P F and assessing the differential entropy of Y | F = f only for sampled faces.
For E P F [ D KL ( p Y | F = f
|| q Y
| F = f )] the situation is similar, since KL for two Dirichlet distributions on the same face f is known in closed-form.
Gradient Estimation.
Parametrizing a Mixed Dirichlet variable takes two parameter vectors, namely, w and α .
In a VAE, those are predicted from a given data point ( e.g. , an MNIST digit) by an inference network.
To update the parameters of the inference network we need a Monte Carlo estimate of the gradient ∇ w , α E F | w [ E Y | F = f, α [ (cid:96) ( y )]] with respect to w and α of a loss function (cid:96) computed in expectation under the Mixed Dirichlet distribution.
By chain rule, we can rewrite the gradient as follows:
[Equation]
Given a sampled face F = f , we can MC estimate both contributions to the gradient, the ﬁrst takes
MC estimation of E Y | F = f (cid:48) , α [ (cid:96) ( y )] , which is straightforward, the second can be done via implicit reprametrization (Figurnov et al., 2018).
To reduce the variance of the score function estimator (ﬁrst term), we employ a sampled self-critic baseline, i.e. , an MC estimate of E Y | F = f (cid:48) , α [ (cid:96) ( y )] given an independently sampled face f (cid:48) .
D I NFORMATION T HEORY FOR M IXED R ANDOM V ARIABLES D. M UTUAL I NFORMATION FOR M IXED R ANDOM V ARIABLES
Besides the direct sum entropy and the Kullback-Leibler divergence for mixed distributions, we can also deﬁne a mutual information for mixed random variables as follows:
Deﬁnition 4 (Mutual information) .
For mixed random variables Y and Z , the mutual information between Y and Z is
[Equation]
With these ingredients it is possible to provide counterparts for channel coding theorems by combining
Shannon’s discrete and continuous channel theorems (Shannon, 1948).
D. P ROOF OF P ROPOSITION 1 (C ODE O PTIMALITY )
Proposition 1 is a consequence of the following facts (Shannon, 1948):
The discrete entropy of a random variable representing an alphabet symbol corresponds to the average length of the optimal code for the symbols in the alphabet, in a lossless compression setting.
Besides, it is known (Cover & Thomas, 2012) that the optimal number of bits to encode a D -dimensional continuous random variable with N bits of precision equals its differential entropy (in bits) plus ND .
Therefore, the direct sum entropy (10) is the average length of the optimal code where the sparsity pattern of y ∈ (cid:52) K − 1 must be encoded losslessly and where there is a predeﬁned bit precision for the fractional entries of y .
Eq.
12 follows from these facts and the deﬁnition of direct sum entropy (Deﬁnition 3).
E D IRECT S UM E NTROPY AND K ULLBACK -L EIBLER D IVERGENCE OF 2D G AUSSIAN -S PARSEMAX
From (1) and Deﬁnition
[Equation]
the direct sum entropy of the 2D Gaussian sparsemax is:
[Equation]
33) where we have P 0 = 1 − erf( z/ ( √ 2 σ )), P 1 = 1+erf(( z − 1) / ( √ 2 σ ))
[Equation]
and − (cid:90)
[Equation]
which leads to a closed form for the entropy.
As for the KL divergence
[Equation]
we have:
[Equation]
where we have P
=
[Equation]
erf(
[Equation] [Equation]
, and
[Equation] [Equation]
F M AXIMUM D IRECT S UM E NTROPY OF M IXED D ISTRIBUTIONS
Start by noting that the differential entropy of a Dirichlet random variable Y ∼ Dir( α ) is
[Equation]
where α 0 =
(cid:80)
Kk α k and ψ is the digamma function.
When α = 1 , this becomes a ﬂat (uniform) density and the entropy attains its maximum value:
[Equation]
This value is negative for K > 2 ; it follows that the differential entropy of any distribution in the simplex is negative.
We next determine the distribution p ⊕ Y ( y ) with the largest direct sum entropy.
Considering only the maximal face, which corresponds to ri( (cid:52) K − 1 ) , the distribution with the largest entropy is the ﬂat distribution, whose entropy is given in (38).
In our deﬁnition of entropy in (10) this corresponds to a deterministic F which puts all probability mass in this maximal face.
At the opposite extreme, if we only assign probability to pure vertices, i.e. , if we constrain to minimal faces, a uniform choice leads to a (Shannon) entropy of log K .
We will show that looking at all faces further increases entropy.
Looking at (10), we see that the differential entropy term H ( Y | F = f ) can be maximized separately for each f , the solution being the ﬂat distribution on face f (cid:39) (cid:52) k − 1 , which has entropy − log( k − 1)!
, where 1 ≤ k ≤ K .
By symmetry, all faces of the same dimension k − 1 look the same, and there are (cid:0) Kk (cid:1) of them.
Therefore the maximal entropy distribution is attained with P F of the form P F ( f ) =
g ( k ) / (cid:0) Kk (cid:1) where g : [ K ] → R + is a function satisfying (cid:80)
Kk =1 g ( k ) =
1 (which can be regarded as a categorical probability mass function).
If we choose a precision of N bits, this
[Figure]
leads to:
[Equation]
Note that the ﬁrst term is the entropy of g ( · ) and the second term is a linear function of g ( · ) , that is, (40) is a entropy-regularized argmax problem, hence the g ( · ) that maximizes this objective is the softmax transformation of the vector with components log (
[Equation]
, that is:
[Equation]
and the maximum entropy value is
[Equation]
where L ( α ) n ( x ) denotes the generalized Laguerre polynomial (Sonine, 1880), as stated in Proposition 2.
Figure 4 shows how the largest direct sum entropy varies with K , compared to the discrete and continuous entropies.
For example, for K = 2 , we obtain g (1) =
2 2+2 N , g (2) =
2 N 2+2 N , and
H ⊕ N ( Y ) = log(2 + 2 N ) , therefore, in the worst case, we need at most log 2 (2 + 2 N ) bits to encode
Y ∈ (cid:52) 1 with bit precision N .
This is intuitive: the faces of (cid:52) 1 are the two vertices { (0 , 1) } and { (1 , 0) } and the line segment [0 , 1] .
The ﬁrst two faces have a probability of 1 2+2 N and the last one have a probability 2 N 2+2 N .
To encode a point in the simplex we ﬁrst need to indicate which of these three faces it belongs to (which requires
[Equation]
bits), and with 2 N 2+2 N probability we need to encode a point uniformly distributed in the segment [0 , 1] with N bit precision, which requires extra N 2 N 2+2 N bits on average.
Putting this all together, the total number of bits is log 2 (2 + 2 N ) , as expected.
G E XPERIMENTAL DETAILS
In this section, we give further details on the architectures and hyperparameters used in our main experiments (App. G.1, App.
G.2, and App.
G.3), which are focused on learning sparse representations, in App.
G.4 we experiment with the Mixed Dirichlet as a likelihood function to model simplex-valued data.
Finally, in App G.5, we describe our computing infrastructure.
G. E MERGENT COMMUNICATION GAME
Let V = { v 1 , . .
. , v |V| } be the collection of images from which the sender sees a single image v j .
We consider a latent variable model with observed variables x
= ( V , j ) ∈ X and latent stochastic variables y chosen from a vocabulary set Y .
We let the sender correspond to the probability model π
( y | x, θ ) =
p ( y | v j , θ π ) of the latent variable and the receiver be modeled as p ( j | V , y, θ (cid:96) ) .
The overall ﬁt to the dataset D is (cid:80) x ∈D L x ( θ ) , where we marginalize the latent variable y to compute the loss of each observation, that is
[Equation]
where the receiver is used to deﬁne the downstream loss, (cid:96) ( x, y ; θ ) :
= − log p ( j | V , y, θ (cid:96) ) .
Notably, we do not add a (discrete) entropy regularization term of π ( y | x, θ ) to (43) with a coefﬁcient as an hyperparameter as in Correia et al.
Data.
The dataset consists of a subset of ImageNet (Deng et al., 2009) 5 containing 463,000 images that are then passed through a pretrained VGG (Simonyan & Zisserman, 2015), from which the representations at the second-to-last fully connected layers are saved and used as input to the sender and the receiver.
To get the dataset visit https://github.com/DianeBouchacourt/
SignalingGame (Bouchacourt & Baroni, 2018).
Architecture and hyperparameters.
We follow the experimental procedure described in Correia et al.
(2020): the architecture of both the sender and the receiver are identical to theirs, we set the size of the collection of images |V| to 16 , the size of the vocabulary of the sender to 256 , the hidden size to 512 , and the embedding size to 256 .
We choose the best hyperparameter conﬁguration by doing a grid search on the learning rate (0.01, 0.005, 0.001) and, for each conﬁguration, evaluating the communication success on the validation set.
For the Gumbel-Softmax models, the temperature is annealed using the schedule τ = max(0 .
5 , exp − rt ) , where r = 1 e − 5 and t is updated every
N = 1000 steps.
For the K -D Hard Concrete we use a scaling constant λ = 1 . 1 and for GaussianSparsemax we set Σ
= I .
All models were trained for 500 epochs using the Adam optimizer with a batch size of 64.
G. B IT -V ECTOR VAE
In this experiment, the training objective is to minimize the negative ELBO: we rewrite (43) with (cid:96) ( x, y ; θ (cid:96) )
= − log p ( x,y | φ ) q ( y | x,λ ) , where we decompose the approximate posterior as q ( y | x, λ ) =
(cid:81)
Di =1 q ( y i | x, λ ) , with D = 128 being the number of binary latent variables
y i .
For each bit, we consider a uniform prior p
( y ) when the approximate posterior is purely discrete or continuous, and the maxent mixed distribution (Prop. 2) for the mixed cases.
Data.
We use Fashion-MNIST (Xiao et al., 2017) 6 , comprising of 28 × 28 grayscale images of fashion products from different categories.
Architecture and hyperparameters.
We follow the architecture and experimental procedure described in Correia et al.
(2020), where the inference and generative network consist of one 128-node hidden layer with ReLU activation functions.
We choose the best hyperparameter conﬁguration by doing a grid search on the learning rate (0.0005, 0.001, 0.002) and choosing the best model based on the value of the negative ELBO on the validation set.
For the Gumbel-Softmax models, the temperature is annealed using the schedule τ = max(0 .
5 , exp − rt ) , where we search r ∈ { 1 e − 5 , 1 e − 4 } and t is updated every N = 1000 steps; also, for these models, we relax the sample into the continuous space but assume a discrete distribution when computing the entropy of π ( y | x, θ ) , leading to an incoherent evaluation for Gumbel-Softmax without ST.
For the Hard Concrete distribution we follow
Louizos et al.
and stretch the concrete distribution to the ( − 0 .
1 , 1 . 1) interval and then apply a hard-sigmoid on its random samples.
For the Gaussian-Sparsemax, we use σ 2 = 1 .
All models were trained for 100 epochs using the Adam optimizer with a batch size of 64.
G. M IXED -L ATENT VAE ON MNIST
Data.
We use stochastically binarized MNIST (LeCun et al., 2010).
The ﬁrst 55,000 instances are used for training, the next 5,000 instances for development and the remaining 10,000 for test.
Architecture and hyperparameters.
The model is a VAE with a K -dimensional latent code (with
K = 10 throughout), both the decoder (which parametrizes the observation model) and the encoder (which parametrizes the inference model) are based on feed-forward neural networks.
The decoder maps from a sampled latent code z to a collection of D = 28 × 28 Bernoulli distributions.
We use a feed-forward decoder with two ReLU-activated hidden layers, each with 500 units.
The encoder maps from a D -dimensional data point x to the parameters of a variational approximation to the model’s posterior distribution.
For Gaussian models we predict K locations and K scales (with softplus activation).
For Dirichlet models we predict K concentrations using softplus clamped to [10 − 3 , 10 3 ] .
For Mixed Dirichlet models we predict K log-potentials clamped to [ − 10 , 10] , and K concentrations using softplus clamped to [10 − 3 , 10 3 ] .
For regularization we employ dropout and L penalty.
We use two Adam optimizers, one for the parameters of the decoder and another for the parameters of the encoder, each with its own learning rate.
We search for the best conﬁguration of hyperparameters using importance-sampled estimates of negative log-likelihood of the model given a development set.
The parameters we consider are: learning rates (in { 10 − 5 , 10 − 4 , 10 − 3 } , and their halves), regularization strength (in { 0 , 10 − 7 , 10 − 6 , 10 − 5 } ), and dropout rate (in { 0 , 0 . 1 , 0 . 2 , 0 . 3 } ).
All models are trained for 500 epochs with stochastic mini batches of size 100.
During training, for gradient estimation, we sample each latent variable exactly once.
For importance sampling, we use 100 samples for model selection, and 1000 samples for evaluation on the test set.
Gradient estimation.
We use single-sample reparametrized gradient estimates for most models:
Gaussian (unbiased), Dirichlet (unbiased), Gumbel-Softmax ST (biased due to straight-trough), and
Mixed Dirichlet (unbiased).
A gradient estimate for Mixed Dirichlet models, in particular, has two sources of stochasticity, one from the implicit reparametrization for Dirichlet samples, another from score function estimation for Gibbs samples.
To reduce variance due to score function estimation, the Mixed Dirichlet model we report employs an additional sample, used to compute a self-critic baseline.
We have also looked into a simpler variant, which dispenses with this self-critic baseline, and instead employs a simple running average baseline, this model performed very close to that with a self-critic.
The purely discrete latent variable model (Categorical) uses the exact gradient of the
ELBO, given a mini batch, which we obtain by exactly marginalizing the K = 10 assignments of the latent variable.
Prior, posterior, and KL term in the ELBO.
The Gaussian model employs a standard Gaussian prior and a parametrized Gaussian posterior approximation, the KL term in the ELBO is computed in closed-form.
The Dirichlet model employs a uniform Dirichlet prior (concentration 1 ) and a parametrized Dirichlet posterior approximation, the KL term in the ELBO is also computed in closed-form.
Both the Categorical and the Gumbel-Softmax ST models employ a uniform Categorical prior and a Categorical approximate posterior (albeit parametrized via Gumbel-Softmax ST for the latter model), the KL term in the ELBO is between two Categorical distributions and thus computed in closed form.
The Mixed Dirichlet model employs a maximum entropy prior with bit-precision parameter N = 0 and a parametrized Mixed Dirichlet approximate posterior, the KL term in the
ELBO is partly exact and partly estimated, in particular, the contribution of the Gibbs distributions over faces is computed in closed form.
Additional plots.
Figure 5 shows tSNE plots where each validation digit is encoded by a sample from the approximate posterior obtained by conditioning on that digit.
The continuous (Gaussian and
Dirichlet) and mixed (Mixed Dirichlet) models learn to represent the digit rather well, with some plausible confusions by the Mixed Dirichlet model ( e.g. , 4, 7, and 9).
The models that can only encode data using the vertices of the simplex (Categorical and Gumbel-Softmax ST) typically map multiple digits to the same vertex of the simplex.
For models whose priors support the vertices of the simplex (Mixed Dirichlet, Categorical, and Gumbel-Softmax ST), we can inspect what the vertices of the simplex typically map to (in data space).
Figure 6 shows 100 such samples per vertex.
We also include samples from the Dirichlet model, though note that the Dirichlet prior does not support the vertices of the simplex, thus we sample from points very close to the vertices (but in the relative interior of the simplex).
We also report conditional and unconditional generations from each model in Figures 7 and 8.
G. S IMPLEX -V ALUED R EGRESSION
Simplex-valued data are observations in the form of probability vectors, they appear in statistics in contexts such as time series ( e.g. , modeling polling data) and in machine learning in contexts such as knowledge distillation ( e.g. , teaching a compact network to predict the categorical distributions predicted by a larger network).
[Figure]
Data and task.
We experiment with the UK election data setup by Gordon-Rodriguez et al.
Section 5.2).
The UK electorate is partitioned into 650 constituencies, each electing one member of parliament in a winner-takes-all vote.
Hence the data are 650 observed vectors of proportions over the four major parties plus a ‘remainder’ category ( i.e. , each observation is a point in the ∆ 5 − 1 simplex, including its faces).
Modeling simplex-valued data with the Dirichlet distribution is tricky for the
Dirichlet does not support sparse outcomes.
While pre-processing the data into the relative interior of the simplex is a simple strategy ( e.g. , add small positive noise to coordinates and renormalize), it is ineffective for the Dirichlet pdf either diverges or vanishes at extrema (neighbourhoods of the lower-dimensional faces of the simplex).
Gordon-Rodriguez et al.
(2020) document this and other difﬁculties in modeling with the Dirichlet likelihood function.
To address these limitations they develop the continuous categorical (CC) distribution, an exponential family that supports the entire simplex ( i.e. , it assigns non-zero density to any point in the simplex), and does not diverge at the extrema.
The CC enjoys various analytical properties, but it still cannot assign non-zero mass to the lower-dimensional faces of the simplex, thus while it is a better choice of likelihood function than the
Dirichlet, CC samples are never truly sparse (thus test-time predictions are always dense).
Architecture and hyperparameters.
For Dirichlet and CC, we use a linear layer to map from the input predictors to 5 log-concentration parameters.
For Mixed Dirichlet we use 2 linear layers: one maps from the input predictors to 5 scores (clamped to [ − 10 , 10] ) which parametrize P F , the other maps to 5 strictly positive concentrations (we use softplus activations, with pre activations constrained to [ − 10 , 10] ) which parametrize P Y | F = f .
We train all models using Adam with learning rate 0 .
1 and no weight decay for exactly 400 steps without mini-batching with 20% of the available data used for training.
Following Gordon-Rodriguez et al.
(2020), we pre-process the data (add 10 − 3 to each coordinate and renormalize) for Dirichlet and CC, but this is not done for Mixed Dirichlet.
Results.
Figure 9 compares the three choices of likelihood function.
We report two prediction rules for our Mixed Dirichlet model.
Sample mean : we predict stochastically by drawing 100 samples and outputting the sample mean.
Most probable mean : we predict deterministically by outputting the mean of the Dirichlet on the face that is assigned highest probability by the model (ﬁnding this most probable face is an operation that takes time O ( K ) , and recall K = 5 in this task).
We can see that
Mixed Dirichlet does not suffer from the pathologies of the Dirichlet, due to face stratiﬁcation, and lowers test error a bit more than CC (see Table 9), likely due to the ability to sample actual zeros.
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
The Mixed Dirichlet uses twice more parameters (we need to parametrize two components), but training time is barely affected (sampling and density assessments are all linear in K ), the training loss and its gradients are stable, and the algorithm converges just as early as CC’s.
As the Mixed Dirichlet produces sparse samples, it is interesting to inspect how often it succeeds to predict whether an output coordinate is zero or not ( i e. , whether y k > 0 , which is true for 77.3% of the targets in the text set).
The sample mean predicts whether y k > 0 with macro F1 0.92, whereas the most probable mean achieves macro F1 0.94.
G. C OMPUTING INFRASTRUCTURE
Our infrastructure consists of 5 machines with the speciﬁcations shown in Table 5.
The machines were used interchangeably, and all experiments were executed in a single GPU.
Despite having machines with different speciﬁcations, we did not observe large differences in the execution time of our models across different machines.
[Table]