Concrete Score Matching: Generalized Score Matching for Discrete Data
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Representing probability distributions by the gradient of their density functions hasproven effective in modeling a wide range of continuous data modalities.
However,this representation is not applicable in discrete domains where the gradientis undeﬁned.
To this end, we propose an analogous score function called the“Concrete score”, a generalization of the (Stein) score for discrete settings.
Givena predeﬁned neighborhood structure, the Concrete score of any input is deﬁnedby the rate of change of the probabilities with respect to local directional changesof the input.
This formulation allows us to recover the (Stein) score in continuousdomains when measuring such changes by the Euclidean distance, while using the
Manhattan distance leads to our novel score function in discrete domains.
Finally,we introduce a new framework to learn such scores from samples called Concrete
Score Matching (CSM), and propose an efﬁcient training objective to scale ourapproach to high dimensions.
Empirically, we demonstrate the efﬁcacy of CSM ondensity estimation tasks on a mixture of synthetic, tabular, and high-dimensionalimage datasets, and demonstrate that it performs favorably relative to existingbaselines for modeling discrete data.
Introduction
When estimating a statistical model, the representation of the underlying probability distributionhas profound implications on the downstream modeling task.
Likelihood-based model families suchas Variational Autoencoders (VAEs) [1, 2, 3, 4], normalizing ﬂows [5, 6, 7, 8, 9, 10] and diffusionmodels [11, 12, 13] are forced to either approximate the intractable normalizing constant or utilizerestrictive model architectures; implicit models [14, 15, 16, 17] try to capture the underlying samplingprocess by relying on unstable adversarial training procedures.
On the other hand, representingthe distribution as the gradient of the log probability density function—also known as the (Stein)score—allows us to circumvent such issues.
This is key to score matching’s success on a widerange of continuous data modalities [18, 19, 20].
In fact, its recent resurgence has led to signiﬁcantadvances in machine learning applications of density estimation, such as image generation [21, 13]and audio synthesis [22, 23], among others.
However, score matching approaches that rely on modeling the gradient of the data distribution areinherently designed for continuous data; such methods hinge on the existence of the gradient, whichis undeﬁned for discrete domains [19].
Given the ubiquitous nature of structured, discrete data inour world today such as graphs, text, genomic sequences, images, we desire an approach that wouldallow us to expand the successes of score-based generative models into discrete domains
To address this challenge, we ﬁrst introduce the Concrete score: a generalization of the (Stein) scorethat is amenable to both continuous and discrete data types 1 .
Given a predeﬁned neighborhoodstructure, the Concrete score leverages structural information in the data to construct surrogategradient information about the discrete space by considering the similarity between two neighboringexamples.
More precisely, the Concrete score of any input is deﬁned by the rate of change of theprobabilities with respect to directional changes of the input.
In direct analogy to the continuous(Stein) score, the intuition is that the Concrete score should remain small when two examples are“close” to one another; when the two examples are very different, the Concrete score will be large.
Weﬁnd that measuring such changes by the Euclidean distance recovers the (Stein) score in continuousdomains, while using the Manhattan distance leads to our novel score function in discrete domains.
Using this deﬁnition of the Concrete score, we then introduce a framework to learn such scores fromsamples called Concrete Score Matching (CSM).
To successfully scale our method to high dimensions,we also propose an efﬁcient training objective as well as several variations of our approach thatimprove performance in practice.
Empirically, we demonstrate the efﬁcacy of CSM on a wide rangeof density estimation tasks on a mixture of synthetic, tabular, and high-dimensional image datasets,and demonstrate that it performs favorably relative to existing baselines for discrete data.
The contributions of our work can be summarized as follows:
We propose the Concrete score, a generalization of the (Stein) score to discrete domains.
We construct the Concrete score such that it leverages structural information in the data to
capture surrogate “gradient” information over discrete spaces.
We introduce a novel score matching framework for estimating such scores from samples
called Concrete Score Matching (CSM).
We then outline several connections between CSM
and existing (continuous) score matching techniques, such denoising score matching (DSM).
We propose efﬁcient learning objectives that allow our method to scale gracefully to high-
dimensional datasets, and show how to port over the recent successes in continuous score
matching approaches to discrete domains.
Preliminaries
Let p data ( x ) be the unknown data distribution over X 2 R D , for which we have access to i.i.d.samples { x i } Ni =1 ⇠ p data ( x ) .
The (Stein) score of p data ( x ) is the ﬁrst order derivative of the log datadensity function s ( x ) =
r x log p ( x ) .
The goal of score matching is to learn an unnormalized densityq ✓ ( x ) indexed by ✓ 2 ⇥ such that the estimated score function s ✓ ( x ) =
r x log q ✓ ( x ) is close to s ( x ) .
In practice, we leverage a score network s ✓ : R D !
R D that is trained to minimize the Fisherdivergence between q ✓ ( x ) and p data ( x ) [20, 21].
Although the original objective is intractable tocompute due to its dependence on the ground truth data scores s ( x ) , we can leverage integration byparts [18] to simplify the objective as follows:
[Equation]
where tr ( · ) denotes the matrix trace, and the optimal score model satisﬁes s ✓ ⇤ ( x ) ⇡ s ( x ) .
Whilethe trace term in Eq.
1 remains problematic—it requires an expensive evaluation of the Hessian ofthe log-density function—recent works [20, 13] leverage the Skilling-Hutchinson trace estimator[25, 26] or directional derivatives [27] to efﬁciently approximate the training objective.
Despite their key role in successfully scaling score-based generative models to high-dimensionaldatasets, we note two critical limitations of existing score matching techniques: (1) X must becontinuous; and (2) the density function p data ( x ) must be differentiable.
This poses a signiﬁcantchallenge in discrete domains, where both requirements break down.
The Concrete Score
The above limitations prevent the direct application of score matching techniques to discrete data.
We thus propose the Concrete score, a generalization of the (Stein) score for discrete settings.
The key intuition is that although the gradient is undeﬁned in discrete spaces, we can still construct asurrogate for the gradient by leveraging local directional changes to the input.
We do so by exploitingspecial neighborhood structures in the data, and elaborate upon the necessary conditions on thesestructures to guarantee that our surrogate gradient indeed recovers a valid score function that (1)completely characterizes the distribution, and (2) is amenable for parameter estimation.
3.1 Constructing Surrogate Gradients for Discrete Data
To be more precise, let p data ( x ) be the data distribution over X .
We denote N : X !
X
K as thefunction mapping each example x 2 X to a set of neighbors, such that N ( x ) = { x n 1 , ..., x n k } 87 and K =
|N ( x ) | 2 .
This neighborhood induces a particular graphical structure onto the support of 88 p data ( x ) , which we call the "neighborhood-induced graph", that will play a key role in constructingthe surrogate gradient.
We provide a formal deﬁnition below.
Deﬁnition 1 (Neighborhood-induced graph) .
Let p data ( x ) be the data distribution over X and N bethe function mapping each node x 2 X to its set of neighbors.
The neighborhood-induced graph G is 92 the directed graph which results from adding a directed edge from x to each node in its neighborhoodset x n 2 N ( x ) , for all x 2 supp(p data ( x )) .
An important point is that the neighborhood structure can be asymmetric, since the neighborhoodinduced graph is a directed graph.
This implies that there may exist cases where N ( x ) = { x 1 } doesnot necessarily imply that N ( x 1 ) = { x } .
We provide an intuitive example below.
Example 1.
When X = { x 0 , x 1 , . . .
, x 5 } , and the neighborhood structure N is deﬁned as: N ( x 0 ) =
{ ; } and N ( x i ) = { x 0 } 8 i = 1 , . . .
, 5 , the neighborhood-induced graph is the star graph in Figure 1.
There exist a wide range of neighborhood structures, all of which yield different neighborhoodinduced graphs.
We visualize ﬁve common structures in Figure 1.
[Figure]
# ! $ !
# ! $ ! ' !
# ! $ ! # ! $ ! ' !
[Figure]
Such graphs provide insight into the kinds of neighborhood structures that are amenable to our
[Figure]
framework.
One necessary characteristic of such graphs is connectedness .
In fact, the ﬁve common
[Figure]
graphical structures (among others) shown in Figure 1 are all weakly connected graphs.
We emphasize
[Figure]
that connectedness does not take the directionality of the underlying graph
G ’s edges into account—it
[Figure]
does not matter whether a node x has an incoming or outgoing edge.
[Figure]
With the above deﬁnitions in place, we can now deﬁne the surrogate gradient as the local differencesbetween a set of examples (similar to directional derivative in the continuous case).
Using this notionof the “gradient”, we construct the Concrete score as the rate of change of the probabilities withrespect to these local directional changes in the input x as deﬁned below.
Deﬁnition 2 (Concrete score) .
Let N be a function mapping each point x to its set of neighbors
N ( x ) = { x n 1 , ..., x n k } .
Then, the Concrete score c p data ( x ; N ) : X !
R |N
( x ) | for a givendistribution p data ( x ) evaluated at x is: [Equation] data
[Equation]
x
[Equation]
T . [Equation] Although we have deﬁned the Concrete score, we have yet to justify whether it is indeed a suitablescore function for estimating p data ( x ) .
The necessary condition, which we call completeness [28], ensures that the Concrete score preserves enough information about p data ( x ) such that we cansuccessfully learn p data ( x ) from data samples using score matching.
We make this statement moreprecise in the following theorem.
Theorem 1 (Completeness) .
Let p data ( x ) be a (discrete) data distribution.
Denote c p data ( x ; N ) asthe Concrete score of p data ( x ) with neighboring structure N , and c ✓ ( x ; N ) the Concrete score for adistribution p ✓ ( x ) parameterized by ✓ 2 ⇥ .
When the graph induced by the neighborhood structure
N is connected, c ✓ ( x ; N )
=
c p data ( x ; N ) implies that p ✓ ( x ) =
p data ( x ) 8 x 2 X .
Proof sketch.
If two nodes x and x 0 in a neighborhood-induced graph G share an edge, then theirdensity ratio p data ( x 0 )
/p data ( x ) can be uniquely identiﬁed using c p data ( x ; N ) .
Thus when G isconnected, the density ratio between any node pairs in G is uniquely identiﬁed given c p data ( x ; N ) .
Therefore, knowing the density ratio between any two points uniquely identiﬁes p data ( x ) .
We provide the full proof in Appendix A.
We emphasize here that the connectedness of theneighborhood-induced graph (a kind of regularity condition on N and p data ( x ) ) was crucial fordemonstrating the completness of the Concrete score.
Note that Theorem 1 only requires the neighborhood-induced graph to be connected.
This implies thatgiven a connected graph, we can augment it with additional edges—the new graph will still remaincomplete, but contain additional information in the augmented neighborhood structure that may helpparameter estimation in practice.
This is a phenomenon that we observe empirically in Section 5.3.
3.2 Connection to Stein Scores and Existing Score Matching Techniques
The form of the Concrete score in Eq.
2 suggests a natural connection with the Stein score incontinuous domains.
In the following proposition, we illustrate the connection between the twowhen we deﬁne the neighborhood structure to be similar to the grid in Figure 1.
In particular, asthe distance between neighboring nodes shrinks to zero, our Concrete score converges to the Steinscore up to a multiplicative constant.
Proposition 1.
For x 2 R D , p ( x ) 2 P ( R D ) , and  > 0 , let the neighborhood structure N  ( x )
=
{ x +  e i } Di =1 .
Then, we have:
[Equation]
We note that while from this perspective the Concrete score can be seen as a ﬁnite-differenceapproximation to the continuous (Stein) score, CSM is different from ﬁnite difference score matching(FD-SM) [27].
We introduce a new family of score functions generalizable to discrete domains in theform of ( p ( x + v )  p ( x )) /p
( x ) , where v is the direction from x pointing to its neighbor with length .
On the other hand, FD-SM still attempts to match the Stein score, and approximates directionalderivatives with ﬁnite difference using two neighbors in the form of log p ( x + v )  log p ( x  v ) .
Nevertheless, the differences between the two forms are o (  ) as  ! 0 .
3.3 Inference with Concrete Scores
To perform inference, we note that there exist several ways to deﬁne Markov Chain Monte Carlo(MCMC)
samplers that only rely on the Concrete score—we highlight the simplest setting of
Metropolis-Hastings [29, 30, 31] for clarity of exposition.
We ﬁrst note that, by deﬁnition:
[Equation]
This implies that we can obtain the density ratios between the neighboring pairs p data ( x n 1 ) /p data ( x )and p data ( x ) /p data ( x n 1 ) given the Concrete score.
Given a proposal distribution q , our samplerwill accept the proposed update x 0 with acceptance probability A ( x 0 | x ) =
min
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
⌘ .
Speciﬁcally, we can choose q to sample uniformly among N ( x ) given a particular x .
When the neighborhood structure is symmetric, this selection leads to a simpliﬁed acceptance probability A ( x 0 | x ) =
min ⇣
[Equation]
⌘ .
We note that when the underlying graph is connected, the chain is aperiodic (due to the presence of a rejection step), irreducible, and positive recurrent (since there is a ﬁnite number ofdiscrete states), so the Markov chain is guaranteed to converge in the limit to the model distribution.
This means that we can also compute tighter bounds on log-likelihood estimates obtained by unnormalized probability models trained via CSM.
In particular, we can compute both a lower boundestimated via Annealed Importance Sampling (AIS) [32] and a conservative upper bound estimatedvia the Reverse Annealed Importance Sampling Estimator (RAISE) [33].
This is because both
AIS and RAISE allow us to approximate the intractable normalizing constant by constructing asequence of intermediate distributions between our estimated target distribution and another proposaldistribution, and we know that the Concrete score can be repurposed to obtain the necessary densityratios between neighboring pairs of distributions along this sequence.
Learning Concrete Scores with Concrete Score Matching
4.1 The Concrete Score Matching Objective
Next, under the assumption that N induces a weakly connected graph over the support of p data ( x ) ,we propose a new score matching framework called Concrete Score Matching (CSM) for estimating
Concrete scores from data samples.
To do so, we estimate c p data ( x ; N ) with a score model c ✓ ( · ; N ) :
R D !
R |N
( x ) | , where |N ( x ) | denotes the size of the neighborhood of x .
Our proposed trainingobjective is quite natural, as it measures the average ` 2 difference between our score model c ✓ ( · ; N )and the true score c p data ( · ; N ) similar to (continuous) score matching:
[Equation]
In the following theorem, we demonstrate that Eq.
5 is indeed a principled learning objective: theestimated ✓ ⇤ is a consistent estimator for the true underlying distribution p data ( x ) .
Theorem 2 (Consistency) .
In the limit of inﬁnite data and inﬁnite model capacity, the optimal ✓ ⇤that minimizes Eq.
5 recovers the true Concrete score, or satisﬁes c ✓ ⇤ ( x ; N ) =
c p data ( x ; N ) .
Theorem 1 implies that the estimated (underlying) p ✓ ⇤ ( x ) from c ✓ ⇤ ( x ; N ) satisﬁes p ✓ ⇤ ( x ) = 181 p data ( x ) 8 x 2 X .
Next, we note that Eq.
5 can be simpliﬁed to an objective that we can tractably optimize by removing its dependence on the unknown c p data ( x ; N ) , similar in spirit to Eq.
Althoughthe added ﬂexibility of the score model may potentially lead to situations where it does not correspondto a valid probability distribution—similar to how continuous score models do not necessarily form aconservative vector ﬁeld—this does not appear to hurt CSM’s performance empirically.
Theorem 3.
Optimizing Equation (5) is equivalent to optimizing:
[Equation]
where N ( x ) = { x n 1 , ..., x n k } is the set of neighbors of x .
We note that in Eq. 6, the objective J CSM ( ✓ ) can be approximated using Monte Carlo samples.
Weelaborate on the empirical training details in Section 4.2.
4.2 Efﬁcient Training via Monte Carlo
In practice, Eq. 6 is still inefﬁcient; it involves a summation over all |N ( x ) | neighbors for x , whichcan be expensive for high-dimensional datasets.
Therefore, we propose several modiﬁcations to theoriginal training objective that we found to be crucial for scaling up and improving our approach.
We begin with the leftmost term J 1 .
First, we approximate the outer expectation w.r.t.
p data ( x )with Monte Carlo samples from the empirical data distribution.
Next, rather than evaluating theobjective for all |N ( x ) | neighbors, we sample a neighbor x n i uniformly at random among N ( x i ) for 197 every x i in a mini-batch.
We then upweight the output from the model using this sample by |N
( x ) | .
We provide the unbiased training algorithm for the leftmost term in Algorithm 1.
We note that issimilar in spirit to sliced score matching [20], an approximation technique to make continuous scorematching scalable to higher dimensions.
Next, we turn to the rightmost term J 2 .
Similar to J 1 , we can estimate J 2 using an unbiased estimator 202 based on samples as demonstrated in Algorithm 2.
We deﬁne
N  1 ( x 0 ) = { ( x , i ) |N
( x ) i = x 0 } 203 as the “reverse neighborhood” set, where an element ( x , i ) 2 N  1 ( x 0 ) indicates that x 0 is the i -thneighbor of x . There could be multiple x with the same i in N  1 ( x 0 ) as in the case of the star graph.
Computing and storing N  1 as a hash table mapping x to the set of N  1 ( x ) has a time and spacecomplexity of at most O (
P x |N ( x ) | ) ( i.e. , the number of edges).
For special structures ( e.g. , grid),we can design speciﬁc algorithms that bypass this process.
We provide more details in Appendix C.
Sample x ⇠ p data ( x ) .
Sample i ⇠ Uniform { 1 , . . .
, |N x ) Output
[Equation]
. We leverage the grid structure in our experiments, where the neighbors of x are deﬁned to be
N ( x ) = { x ± e d } Dd =1 .
Algorithm 1 and Algorithm 2 allow us to efﬁciently train the model by 211 sampling a dimension d 2 [ D ] and ﬂipping the bit for the d th entry of x for each x n .
4.3 Denoising Concrete Score Matching
Finally, we propose another training objective to approximate Eq. 6, with a focus on computationalefﬁciency.
Similar to denoising score matching (DSM) for (Stein) score estimation [19], we derive adenoising counterpart for Concrete score estimation called “Denoising Concrete Score Matching”(D-CSM).
Speciﬁcally, given a discrete data distribution p data ( x ) and a discrete noise distribution˜ q (˜ x | x ) , we deﬁne the perturbed data distribution ˜ p (˜ x ) =
[Equation]
( x )˜ q (˜ x | x ) , and the posteriorq
[Equation]
Then, we can show that the Concrete score of the perturbed data distribution˜ p (˜ x ) can be obtained via c ˜ p
(˜ x ) (˜ x ; N ) =
[Equation] (˜ x ; N ) q ( x | ˜ x ) .
This property allows us to obtain 220 the D-CSM objective, as shown in the following theorem:
Theorem 4 (Denoising Concrete Score Matching) .
The objective:
[Equation]
is minimized when c ✓ (˜ x ; N ) =
c p (˜ x ) (˜ x ; N ) .
We draw a direct analogy to DSM in order to provide additional insight into D-CSM.
Recall that in
DSM, there exists a noise distribution q (˜ x | x ) ( i.e . N (˜ x | x ,  2 I ) )
such that the magnitude of Steinscores captures the distance between the perturbed ˜ x and its clean counterpart x .
D-CSM enjoys a similar interpretation.
In particular, consider an input space x 2 X ✓ Z D with the neighborhood structure
N ( x ) =
{ x + v | v 2 {  1 , 0 , 1 } D } .
If we deﬁne q (˜ x | x ) = 1 2 D + k where k is the Manhattan distancebetween ˜ x and x , then the Concrete score c ˜ p (˜ x ) (˜ x ; N ) captures how close x is to ˜ x as measured bythe Manhattan distance.
Therefore, D-CSM can also be understood through the lens of a denoiser.
We provide additional experimental results exploring the performance of D-CSM in Appendix B.3.
Experimental Results
In this section, we evaluate the performance of CSM on synthetic, tabular, and discrete image datasetson a variety of sampling and density estimation tasks.
We provide additional details on speciﬁcexperimental settings and hyperparameter conﬁgurations in Appendix C.
5.1 Baselines
In our experiments, we compare CSM to two relevant baselines for modeling discrete data: Ratio
Matching and Discrete Score Matching with Marginalization ( Discrete Marginalization ).
Forconciseness, we provide additional details and derivations for their training objectives in Appendix D.
Ratio Matching.
Similar to score matching, Ratio Matching [28] leverages the fact that ratios ofprobabilities are independent of the intractable normalizing constant (due to cancellation).
Themethod then seeks to match the ground truth density ratios p data ( x ) p data ( x  d ) =
q ✓ ( x ) q ✓ ( x  d ) where x  d denotesthe vector x with the d th entry bit-ﬂipped (e.g. from 0 to 1).
Discrete Score Matching with Marginalization (Discrete Marginalization).
The Discrete Marginalization baseline is another way of estimating discrete probability distributions with score matching asin [28].
However, we note that this approach is difﬁcult to scale because it requires us to marginalizeover dimensions in the data, which may also cause instabilities during training.
5.2 1-D Discrete Data
In this experiment, we consider a 16-category 1-D data distribution as shown in Figure 2.
Weparameterize our Concrete score model c ✓ ( x , N ) using a shallow MLP model with Tanh activations,and use a Cycle neighborhood structure during training.
We demonstrate that the learned Concretescore model can faithfully capture the true data distribution when trained with CSM.
As shown onthe left side of Figure 2, we ﬁnd that CSM generates samples (blue) using MH that almost perfectlymatches the samples drawn from p data ( x ) (green).
We also observe in Appendix A.3 that the Concrete score on p data ( x ) can recover the Stein score of adata distribution perturbed with triangular noise.
Even though the Concrete score model is trained ondiscrete data (green), this connection allows us to sample with Langevin dynamics using the recovered
Stein score.
On the right side of Figure 2, we show how the samples generated using Langevindynamics (smoothed orange) closely match the triangular noise-perturbed data distribution (gray).
Then, we leverage a closed-form denoising formula for the perturbed distribution (Appendix A.3) torecover the clean data distribution (green) from samples obtained via Langevin dynamics (smoothedorange).
The resulting denoised samples (orange) are labeled as Denoised in Figure 2.
We providemore details and discussion in Appendix A.3.
[Figure]
5.3 Sampling with Toy 2-D Multi-Class Datasets
Next, we consider three 2-D toy benchmark datasets with multiple modes and discontinuities ascommonly used in the density estimation literature [34, 35].
We quantize the data into 91 ⇥ 91 binsto obtain the discrete training data for the experiment (see Figure 3).
We compare our method againstthe two baselines: (1) ratio matching [36]; and (2) discrete marginalization [28].
In particular, wefound that the original equations in [28] were incorrect, and provide results using the correct trainingobjectives (denoted as Ratio-fixed and Marginal-fixed ) in Figure 3.
We provide a step-by-stepderivation addressing this issue with the correct expressions for the training objectives in Appendix D.
For a fair comparison, we use the same model architecture and training conﬁgurations across allmethods.
We use the grid neighborhood structure for training CSM.
As shown in Figure 3, thesamples from CSM best capture the shape of the underlying data distributions across all three datasets(leftmost column).
Baselines implemented using the original equations in [28] indeed demonstratepoor performance on the density estimation task ( Ratio and Marginal columns) in Figure 3.
5.4 Likelihood Evaluation on Discrete Tabular Data
We then evaluate the performance of CSM on density estimation tasks for a wide range of tabular(discrete) datasets drawn from both the Twenty Datasets [37] and the Amazon Baby Registriesbenchmarks [38]. In order to evaluate likelihoods, we directly parameterize the density with a [Figure] Ratio
Ratio
[Table] Datasets [Table] discrete autoregressive model (MADE [39]), but train the model using the baseline approaches and
CSM.
For a fair comparison, we use the same model architecture and experimental conﬁgurationsacross all methods.
Similar to our previous experiments, we use the grid neighborhood structure fortraining CSM.
As shown in Table 1, our approach ( CSM ) demonstrates favorable performance relativeto the Ratio Matching and Discrete Marginalization baselines.
We provide additionalexperimental results in Appendix B.1 and more experimental details in Appendix C.
5.5 Generative Modeling with High-dimensional Images
For our ﬁnal experiment, we demonstrate that we can scale CSM to achieve good performance oncomplex, high-dimensional binarized image datsets.
We experiment with the MNIST [40] dataset,which has 784 dimensions.
We directly parameterize the Concrete score function with a U-Net [41],and train the model using the CSM based on a grid neighborhood structure.
Similar to [21], weperturb the image with different levels of discrete (Categorical) noise, and train the models at differentnoise levels with annealing.
After the Concrete score model is trained, we sample from the modelusing Metropolis-Hastings as discussed in Section 3.3 and follow a similar sample initializationprocess as [21].
We provide more details in Appendix C. We present the samples from our model in Figure 4b.
We observe that our CSM approach with Metropolis-Hastings is able to generate samplesthat look similar to the digit examples in the training set (Figure 4a).
Related Work
Score Matching.
Our work builds on the body of literature on score matching [18, 19] andscore-based generative modeling [20, 21, 13].
Notably, we build upon the generalization of score [Figure] matching to discrete data [36, 42, 28].
Our score function can be viewed through the lens ofgeneralized score matching as in [28], where the Concrete score serves as a particular instantiationof the linear operator in their framework.
However, we provide a novel perspective in terms ofconstructing the Concrete score via surrogate gradient information over a structured discrete space,and provide efﬁcient training objectives for CSM.
Our work is also closely related to score matchingwith ﬁnite difference [27].
However, rather than approximating the gradient with ﬁnite differencesfor continuous data, we leverage the ﬁrst-order forward difference to construct our Concrete scorefunction in discrete domains.
In addition, CSM bears similarities to a concurrent work on scalingup Ratio Matching [28] for discrete energy-based models (EBMs) [43] called RMwGGIS—ourapproach can be viewed as a different way to train discrete EBMs (via the Concrete score).
Generative Modeling for Discrete Data.
CSM also provides another way to train generative modelsfor discrete data.
A related work is [44], which trains a score-based model on distributions overgraphs.
However, they perturb the adjacency matrices with Gaussian noise and use the continuousvariant of DSM.
Although there exist several model families for learning binary and Categoricalprobability distributions such as normalizing ﬂows [45, 46, 47], Sum-Product Networks (SPNs)[48, 49], denoising diffusion probabilistic models [50], discrete (latent) [51, 52] EBMs [43], and Generative Flow Networks (GFNs) [53], there does not yet exist a discrete score-based model thatcan scale to high-dimensional discrete datasets.
Finally, our approach bears similarities with Gibbswith Gradients (GWG) [54].
In particular, GWG augments existing MCMC samplers with gradientinformation by leveraging local structure to estimate likelihood ratios for transitioning to the nextstate.
However, GWG utilizes gradients from the probability mass functions of the underlying discretedistributions, in contrast to the way in which we construct the Concrete score function.
Conclusion
We introduced Concrete Score Matching (CSM), a novel framework for learning discrete probabilitydistributions via score matching.
We proposed to leverage particular kinds of structural information inthe data to construct surrogate gradient information about the discrete space, and used this to deﬁnea valid score function.
We also introduced several modiﬁcations to the original training objective thatallowed CSM to scale gracefully to high-dimensional datasets, and demonstrated that CSM performswell on a variety of sampling and density estimation tasks.
However, this work is not without limitations.
Since CSM depends on the neighborhood structure,certain types of graphs may work better for score matching in practice than others.
Additionally,our efﬁcient training objectives may suffer from high variance when scaling to high dimensions forparticular kinds of neighborhood-induced graphs, such as the Star graph.
For future work, althoughwe ﬁxed the number of neighbors for each x in our experiments, it would be interesting to adaptivelydetermine the optimal number of neighbors to use for each x during training.
We also believe that
CSM can be generalized to leverage more complex neighborhood structures than those we exploredin the paper.
Additionally, empirically investigating the performance of D-CSM with Langevindynamics relative to CSM would be interesting future work.