Original: M AXIMUM
Revised: M AXIMUM

Original: E NTROPY
Revised: E NTROPY

Original: COMPETES
Revised: COMPETES

Original: WITH
Revised: WITH

Original: M AXIMUM
Revised: M AXIMUM

Original: LIKELIHOOD
Revised: L

Revised: IKELIHOOD

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Maximum entropy (MAXENT) method has a large number of applications in theoretical and applied machine learning, since it provides a convenient nonparametric tool for estimating unknown probabilities.
Revised: Maximum entropy (MAXENT) method has a large number of applications in theoretical and applied machine learning, since it provides a convenient nonparametric tool for estimating unknown probabilities.

Original: The method is a major contribution of statistical physics to probabilistic inference.
Revised: The method is a major contribution of statistical physics to probabilistic inference.

Original: However, a systematic approach towards its validity limits is currently missing.
Revised: However, a systematic approach towards its validity limits is currently missing.

Original: Here we study MAXENT in a Bayesian  set-up, i.e. assuming that there exists a well-deﬁned prior Dirichlet density for unknown probabilities.
Revised: Here we study MAXENT in a Bayesian decision theory set-up, i.e. assuming that there exists a welldeﬁned prior Dirichlet density for unknown probabilities, and that the average

Original: This allows to employ the average Kullback-Leibler (KL) distance in evaluating the relevance of various MAXENT constraints, checking its general applicability, and comparing MAXENT with estimators having various degrees of dependence on the prior, viz.
Revised: Kullback-Leibler (KL) distance can be employed for deciding on the quality and applicability of various estimators. These allow to evaluate the relevance of various MAXENT constraints, check its general applicability, and compare MAXENT with estimators having various degrees of dependence on the prior, viz.

Original: the regularized maximum likelihood (ML) and the Bayesian estimators.
Revised: the regularized maximum likelihood (ML) and the Bayesian estimators.

Original: We show that MAXENT applies in sparse data regimes and caneven outperform the optimally regularized
Revised: We show that MAXENT applies in sparse data regimes, but needs speciﬁc types of prior information. In particular, MAXENT can outperform the optimally regularized

Original: ML provided that there are prior rank correlations between the estimated random quantity and its probabilities.
Revised: ML provided that there are prior rank correlations between the estimated random quantity and its probabilities.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The maximum entropy (MAXENT) method was proposed within statistical physics (Jaynes, 1957;1978; Balian, 2007; Press ´ e et al., 2013), and later on got a wide range of inter-disciplinary applications in data science, probabilistic inference, biological data modeling etc ; see e.g. (Erickson &Smith, 2013).
Revised: The maximum entropy (MAXENT) method was proposed within statistical physics (Jaynes, 1957; Balian, 2007; Press ´ e et al., 2013), and later on got a wide range of inter-disciplinary applications in data science, probabilistic inference, biological data modeling etc ; see e.g. (Erickson &

Original:  MAXENT estimates unknown probabilities (that generated data) via maximizing the Boltzmann-Gibbs-Shannon entropy under certain constraints which can be derived from the observed data (Erickson & Smith, 2013).
Revised: Smith, 2013). MAXENT estimates unknown probabilities (that generated data) via maximizing the Boltzmann-Gibbs-Shannon entropy under certain constraints which can be derived from the observed data (Erickson & Smith, 2013).

Original: MAXENT leads to non-parametric estimators whose form does not depend on the underlying mechanism that generated data (i.e. prior assumptions).
Revised: MAXENT leads to non-parametric estimators whose form does not depend on the underlying mechanism that generated data (i.e. prior assumptions).

Original: Also, MAXENT avoids the zero-probability problem, i.e. when operating on a sparse data—so that certain values of the involved random quantity may not appear due to a small, but non-zero probability—MAXENT still provides a controllable non-zero estimate for this small probability.
Revised: Also, MAXENT avoids the zero-probability problem, i.e. when operating on a sparse data—so that certain values of the involved random quantity may not appear due to a small, but non-zero probability—MAXENT still provides a controllable non-zero estimate for this small probability.

Original: Though for MAXENT has a rich background, some of its problems stay open. These are not academic problems of providing yet another formal justiﬁcationof MAXENT; cf. (Jaynes, 1957; 1978;
Revised: MAXENT has has several formal justiﬁcation (Jaynes, 1957; Chakrabarti & Chakrabarty, 2005;

Original: Chakrabarti & Chakrabarty, 2005; Baez et al., 2011; Van Campenhout & Cover, 1981; Topsøe, 1979; Shore & Johnson, 1980; Paris & Vencovsk´a, 1997).
Revised: Baez et al., 2011; Van Campenhout & Cover, 1981; Topsøe, 1979; Shore & Johnson, 1980; Paris & Vencovsk´a, 1997).

Original: The following open problems are basic for MAXENT, because their insufﬁcient understanding prevents its valid applications.
Revised: But the following open problems are basic for MAXENT, because their insufﬁcient understanding prevents its valid applications.

Original: (i) Which constraints of entropy maximization are to be extracted from data?
Revised: (i) Which constraints of entropy maximization are to be extracted from data, which is necessarily ﬁnite and noisy?

Original: (ii) When and how these constraints can lead to overﬁtting, where —due to a noisy data—involving more constraints leads to poorer results?
Revised: (ii) When and how these constraints can lead to overﬁtting, where —due to a noisy data—involving more constraints leads to poorer results?

Original: (iii) How predictions of MAXENT compare with those of  the (regularized) maximum likelihood?
Revised: (iii) How predictions of MAXENT compare with those of other estimators, e.g. the (regularized) maximum likelihood?

Original: Here we approach these open problems via tools of Bayesian decision theory (Cox & Hinkley, 1979).
Revised: Here we approach these open problems via tools of Bayesian decision theory (Cox & Hinkley, 1979).

Original: We assume that the data is given as an i.i.d.
Revised: We assume that the data is given as an i.i.d.

Original: sample of a ﬁnite length M from a random quantity with n outcomes and unknown probabilities.
Revised: sample of a ﬁnite length M from a random quantity with n outcomes and unknown probabilities that are instanced from a non-informative prior

Original: Following the tenets of the Bayesian approach we shall assume that those probabilities are instanced from a non-informative prior Dirichlet density, or a mixture of such densities.
Revised: Dirichlet density, or a mixture of such densities.

Original: Focusing on the sparse data regime M < n we calculate average distances between real probabilities and their estimates, decide on the quality of MAXENT under various constraints, and compare it with the (regularized) maximum-likelihood (ML) estimator.
Revised: Focusing on the sparse data regime M < n we calculate average KL-distances between real probabilities and their estimates, decide on the quality of MAXENT under various constraints, and compare it with the (regularized) maximum-likelihood (ML) estimator.

Original: Our main results is that MAXENT doesnot apply to sparse data generated in the fully non-informative way (i.e. via a single homogeneous Dirichlet prior).
Revised: Our main results are that MAXENT does apply to sparse data, but does demand speciﬁc prior information. We explored two different scenarios of such information. First, the unknown probabilities are most probably deterministic.

Original: However, such an application is secured under prior rank correlations between the inferred random quantity and its probabilities.
Revised: Second, there are prior rank correlations between the inferred random quantity and its probabilities.

Original: Moreover, in the latter case the non-parametric MAXENT estimator is better in terms of the average KL-distance than the optimally regularized ML (parametric) estimator.
Revised: Moreover, in the latter case the nonparametric MAXENT estimator is better in terms of the average KL-distance than the optimally regularized ML (parametric) estimator.

Original: Some of above questions were already studied in literature.
Revised: Some of above questions were already studied in literature.

Original: (Good, 1970;  Zhu et al., 1997; Pandey & Dukkipati, 2013) applied formal principles of statistics (related to or directly based on the Minimum Description Length) to the selection of constraints (question (i) ).
Revised: (Good, 1970; Christensen, 1985; Zhu et al., 1997; Pandey & Dukkipati, 2013) applied formal principles of statistics (e.g. the Minimum Description Length) to the selection of constraints (question (i) ).

Original: Our approach to studying this question will be more direct, since the Bayesian statistics allows to state the prior assumptions unambiguously, and compare all predictions with the  Bayesian estimator.
Revised: Our approach to studying this question will be direct and unambiguous, since, as shown below, the Bayesian decision theory leads to clear criteria for the validity of MAXENT estimators. We can also compare all predictions with the optimal Bayesian estimator.

Original: The latter is normally not available in practice due to insufﬁcient knowledge of prior details, but it still does provide an important theoretical benchmark.
Revised: The latter is normally not available in practice due to insufﬁcient knowledge of prior details, but it still does provide an important theoretical benchmark.

Original: Note that (Thomas, 1979; Lebanon & Lafferty, 2002; Kazama & Tsujii, 2005; Altun & Smola, 2006; Dudik, 2007.
Revised: Note that (Thomas, 1979; Lebanon & Lafferty, 2002; Kazama & Tsujii, 2005; Altun & Smola, 2006; Dudik, 2007.

Original: ; Rau, 2011; Campbell, 1999; Friedlander & Gupta, 2005) studied soft constraints that allow incorporation of prior assumptions into the MAXENT estimator making it effectively parametric.
Revised: ; Rau, 2011; Campbell, 1999; Friedlander & Gupta, 2005) studied soft constraints that allow incorporation of prior assumptions into the MAXENT estimator making it effectively parametric.

Original: Here MAXENT will be taken in its original meaning as providing non-parametric estimators.
Revised: Here MAXENT will be taken in its original meaning as providing non-parametric estimators.

Original: The rest of this paper is organized as follows.
Revised: This paper is organized as follows.

Original: Section 2  describes the data-generation set-up and introduces prior densities.
Revised: Section 2 recall the tenets of the Bayesian decision theory and describes the data-generation set-up.

Original: Section 3 introduces and motivates the Bayesian estimator and the regularized ML estimator.
Revised: Section 3 introduces and motivates the Bayesian estimator and the regularized ML estimator.

Original: Section 4 recalls the basic formulas of MAXENT,and applies them to the studied set-up, and discusses their symmetry features.
Revised: Section 4 recalls the basic formulas of MAXENT, applies them to the studied set-up, and discusses their symmetry features.

Original: Section 5 compares predictions of MAXENT with the regularized ML.
Revised: Section 5 compares predictions of MAXENT with the regularized ML.

Original: We close in the last section with discussing open problems.
Revised: We close in the last section with discussing open problems.

Revised: Appendix A shows how to apply MAXENT to categorical data.

Revised: Appendix B presents our preliminary results on the afﬁne symmetry of MAXENT estimators, and establishes relations with the minimum entropy principle proposed in (Good, 1970; Christensen, 1985; Zhu et al., 1997; Pandey & Dukkipati, 2013).

Original:  D ATA GENERATION
Revised:  B AYESIAN DECISION THEORY

Original: Consider a random quantity Z with values ( z 1 , ..., z n ) and respective probabilities q = ( q 1 , ..., q n )
Revised: Consider a random quantity Z with values ( z 1 , ..., z n ) and respective probabilities q = ( q 1 , ..., q n )

Original: =
Revised: =

Original: ( q ( z 1 ) , ..., q ( z n )) .
Revised: ( q ( z 1 ) , ..., q ( z n )) .

Original: We look at an i.i.d.
Revised: We look at an i.i.d.

Original: sample of length M :(
Revised: sample of length M :

Original: [Equation]
Revised: [Equation]

Original: where  m k is the number of appearances of z k in (1).
Revised: where Z u ∈ ( z 1 , ..., z n ) ( u = 1 , ..., M ), and m k is the number of appearances of z k in (1).

Original: This sample will be an instance of our data, e.g. constraints of MAXENT will be determined from it.
Revised: This sample will be an instance of our data, e.g. constraints of MAXENT will be determined from it.

Original: The conditional probability of m reads
Revised: The conditional probability of data D reads

Original: [Equation]
Revised: [Equation]

Original: To check the performance of various inference methods, theinferred probabilities ˆ q ( m ) 
Revised: To check the performance of various inference methods, the probabilities ˆ q ( D ) =

Original: = { ˆ qz k ; m ) } nk =1 (whose forms are given below and can depend on both ( z 1 , ..., z n ) and ( m 1 , ..., m n ) ) are compared with true probabilities q = { q ( z k ) } nk =1 via the(non-symmetric) KL-distance
Revised: { ˆ q k ( D ) } nk = inferred from (1) are compared with true probabilities q = { q ( z k ) } nk =1 via the KL-distance

Original: [Equation]
Revised: [Equation]

Original:  The choice of distance (3) is motivated below, where we recall that it implies the global optimality of the standard (posterior-mean)
Revised: where concrete forms of ˆ q ( D ) are given below. The choice of distance (3) is motivated below, where we recall that it implies the global optimality of the standard (posterior-mean)

Original: Bayesian estimator.
Revised: Bayesian estimator.

Original: Another possible choice of distance is the squared (symmetric) Hellinger distance: dist H [ q, ˆ q ] ≡ 1 − (cid:80) nk =1 √ q k ˆ q k .
Revised: Another possible choice of distance is the squared (symmetric) Hellinger distance: dist H [ q, ˆ q ] ≡ 1 − (cid:80) nk =1 √ q k ˆ q k .

Original: In our situation, it frequently leads to the same qualitative results as (3).
Revised: In our situation, it frequently leads to the same qualitative results as (3).

Original: Given an inference method ˆ q ( m ) , we shall average (3) over samples (1) with ﬁxed length M : [Equation]
Revised: How to compare various estimators with each other, and decide on the quality of a given estimator?

Original: Following the Bayesian approach (Cox & Hinkley, 1979), we shall assume that the probabilities of ( z 1 , ..., z n ) are generated from a known probability density D ( q 1 , ..., q n ) that encapsulates the prior information about the situation.
Revised: Bayesian decision theory comes to answer this question; see chapter 11 of (Cox & Hinkley, 1979). The theory assumes that the probabilities of ( z 1 , ..., z n ) are generated from a known probability density P ( q 1 , ..., q n ) that encapsulates the prior information about the situation.

Original: Since K does depend on q = { q (z k ) } nk =1 , we average it over the prior density D( q 1 , ..., q n ) :
Revised: Next it decides on the quality of an estimator ˆ q ( D ) via the average distance

Original: [Equation]
Revised: [Equation]

Original: Hence (cid:104) K (cid:105) eventually determines the averageperformance (quality) of  the inference method
Revised: where K is the average of (3) over samples (1) with ﬁxed length M . Sometimes the Bayesian decision theory replaces the distance by the utility, loss etc (Cox & Hinkley, 1979). Note the difference between the proper Bayesian approach and the Bayesian decision theory; cf.

Original: ˆ q ( m ) .
Revised: chapters 10 and 11 in (Cox & Hinkley, 1979). The former employs the data for moving from the prior (5) to the posterior (7). It averages over the prior, e.g. when calculating the posterior mean. The latter advises on choosing estimators, whose form may or not may not depend on the prior; see below for examples.

Original:  For the prior density of q = { q k } nk =1 we choose the Dirichlet density (or a mixture of such densities as seen below) (Frigyik et al., 2010; Schafer, 1997):[Equation]
Revised: The decision theory averages both over the data and over the prior, as seen in (4). For the prior density of q = { q k } nk =1 we choose the Dirichlet density (or a mixture of such densities as seen below) (Frigyik et al., 2010; Schafer, 1997):

Original: [Equation]
Revised: [Equation]

Original: where Γ[ x ] = (cid:82) ∞ 

Revised: where Γ[ x ] = (cid:82) ∞

Original: d y y x − 1 e − y is Euler’s Γ -function and delta-function δ ( (cid:80) nk =1 q k − 1) ensures the normalization of probabilities.
Revised: d y y x − 1 e − y is Euler’s Γ -function and delta-function δ ( (cid:80) nk =1 q k − 1) ensures the normalization of probabilities.

Original: Parameters α k > 0 determine the prior weight of q k (Frigyik et al., 2010; Schafer, 1997):
Revised: Parameters α k > 0 determine the prior weight of q k (Frigyik et al., 2010; Schafer, 1997):

Original: [Equation]
Revised: [Equation]

Original:  Dirichlet density (6) is unique in holding several desired features of non-informative prior density over unknown probabilities; see (Frigyik et al., 2010; Schafer, 1997) for reviews.
Revised: 6) where the integration range goes over the simplex 0 ≤ q k ≤ 1 , ∀ k , and (cid:80) nk = 1 q k = 1 . Dirichlet density (5) is unique in holding several desired features of non-informative prior density over unknown probabilities; see (Frigyik et al., 2010; Schafer, 1997) for reviews.

Original: An important feature of density (6) is that it is conjugate to the multinomial conditional probability (2)
Revised: An important feature of density (5) is that it is conjugate to the multinomial conditional probability (2)

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (8) is convenient when studying i.i.d.
Revised: (7) is convenient when studying i.i.d.

Original: samples (1) of discrete random quantities.
Revised: samples (1) of discrete random quantities.

Revised: Here we assume that the prior density is known exactly [see however (32)].

Revised: In practice, such a knowledge need not be available.

Revised: For example, it may be known that the prior density belongs to the Dirichlet family, but its hyper-parameters { α k } nk =1 are unknown and should be determined from the data, e.g. via empirical Bayes procedures; see (Frigyik et al., 2010; Schafer, 1997; Claesen & De Moor, 2015;

Revised: Ran & Hu, 2017; Bergstra & Bengio, 2012) for reviews on hyper-parameter estimation.

Original:  B AYESIAN AND REGULARIZED MAXIMUM LIKELIHOOD (ML) ESTIMATORS
Revised:  B AYESIAN AND REGULARIZED MAXIMUM LIKELIHOOD (ML) ESTIMATORS

Original: We minimize (5) over inferred probabilities ˆ q ( m ) to ﬁnd the best estimator:
Revised: Starting from (4), we ﬁnd the best estimator in terms of the minimal, average KL-distance: min[ (cid:104) K (cid:105) ] = (cid:88) D P ( D )

Original: [Equation]
Revised: [Equation]

Original: where P ( m ) P ( q | m ) =
Revised: (cid:34) (cid:90) n (cid:89) k [Equation] [Equation] (cid:35)

Original:  P ( m | q )  D  (  q )with P ( q| m ) read from (2, 6).
Revised: [Equation] (8) where the minimization goes over inferred probabilities { ˆ q ( D ) } , and where P ( q |D ) is recovered from P ( D| q ) : P ( D ) P ( q |D ) = P ( D| q ) P ( q ) ; cf. The equality in (8) follows from the fact that if ˆ q ( D ) minimizes (cid:82) (cid:81) nk =1 d q k P ( q |D ) K [ q, ˆ q ( D )] , then it will minimize each term of the sum for every D , and thus will minimize the whole sum.

Original: Implementing the constraint (cid:80) nk
Revised: Then implementing the constraint (cid:80) nk

Original: =1 ˆ q k
Revised: =1 ˆ q k

Original: ( m ) = 1 via a Lagrange multiplier, we get from (9):
Revised: ( D ) =via a Lagrange multiplier, we get from (8):

Original: [Equation]
Revised: [Equation]

Original: We emphasize that the posterior average in (10) emerged as the best Bayesian estimator only for the KL distance K [ q, ˆ q ( m )] . 
Revised: We got in (9) the posterior average, because we employed the KL distance K [ q, ˆ q ( D )] . The optimal estimator will be different upon using another distance, e.g. KL distance K [ˆ q ( D ) , q ] of ˆ q ( D ) from q , or the Hellinger distance. Note that in the proper Bayesian approach the posterior mean is simply postulated to be an estimator, since it is just a characteristics of the posterior distribution.

Original: The best estimator will be different upon using another distance in (9), e.g.
Revised: In the present Bayesian decision approach the posterior emerges from minimizing a speciﬁc ( viz.

Original: K [ˆ q ( m ) , q ] , or the Hellinger distance. 
Revised: KL) distance. If another distance is used, the posterior mean is not anymore optimal.

Original: If the prior is a single Dirichlet density (6) we get from (8, 10) for the Bayesian estimator:
Revised: If the prior is a single Dirichlet density (5) we get from (7, 9) for the Bayesian estimator:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: The average KL-distance (5) for the estimator (11) reads from (8, 4, 2) (denoting ψ [ x ] ≡ dd x ln Γ[ x ] ): (cid:104)
Revised: The average KL-distance (4) for the estimator (10) reads from (7, 2) (denoting ψ [ x ] ≡ [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: If the prior density is given by mixture of Dirichlet densities with weights {π a } La =1 :
Revised: If the prior density is given by mixture of Dirichlet densities with weights {

Original: [Equation]
Revised: [Equation]

Original: then instead of (7) and (11) we have from (10)
Revised: then instead of (6) and (10) we have from (9)

Original: [Equation]
Revised: [Equation]

Original:  For a mixture prior density, the Bayesian estimator (15) depends on all numbers { m k ; α [1] k , ..., α [1] k } not just on m k .
Revised: = [Equation] [Equation] For a mixture prior density, the Bayesian estimator (14) depends on all numbers { m k ; α [1] k , ..., α [ L ] k } not just on m k .

Original: Below we show on concrete examples that not knowing precisely details of the prior mixture can lead to serious losses when applying Bayesian estimators.
Revised: Below we illustrate that not knowing precisely details of the prior mixture can lead to serious losses when applying Bayesian estimators.

Original: It is interesting (both conceptually and practically) to have a simple estimator, where the dependence on the prior is reduced to a single parameter.
Revised: It is interesting (both conceptually and practically) to have a simple estimator, where the dependence on the prior is reduced to a single parameter.

Original: A good candidate is the regularized maximum likelihood (ML) estimator (see (Hausser & Strimmer, 2009) for a review):
Revised: A good candidate is the regularized maximum likelihood (ML) estimator (see (Hausser & Strimmer, 2009) for a review):

Original: [Equation]
Revised: [Equation]

Original: where the regularizer b (or λ ) takes care of the fact that for a ﬁnite sample (1) not all values z k had a chance to appear (i.e. m k = 0 for them).
Revised: where the regularizer b (or λ ) takes care of the fact that for a ﬁnite sample (1) not all values z k had a chance to appear (i.e. m k = 0 for them).

Original: Then (16) avoids to claim a zero probability due to b > 0 .
Revised: Then (15) avoids to claim a zero probability due to b > 0 .

Original: Eq.
Revised: Eq.

Original: (16) is a shrinkage estimator, where the proper ML estimator m k M is shrunk towards uniform distribution 1 n by the shrinkage factor λ .
Revised: (15) is a shrinkage estimator, where the proper ML estimator m k M is shrunk towards uniform distribution 1 n by the shrinkage factor λ .

Original: The proper ML estimator p ML ( z k ) | b = will be shown to be a meaningless estimator for not very long samples (1) producing results that are worse than { q ( z k ) = 1 n } nk =1 .
Revised: The proper ML estimator p ML ( z k ) | b =0 will be shown to be a meaningless estimator for not very long samples (1) producing results that are worse than { q ( z k ) = 1 n } nk =1 .

Original: Moreover, for such samples the correct choice of b (based on the prior information) is crucial, i.e. (16) is  a parametric estimator (Hausser & Strimmer, 2009).
Revised: Moreover, for such samples the correct choice of b (based on the prior information) is crucial, i.e. (15) is generally a parametric estimator.

Original: The estimator (16) recovers true probabilities for M → ∞ (Cox & Hinkley, 1979), where n and b are ﬁxed, hence λ
Revised: The estimator (15) recovers true probabilities for M → ∞ (Cox & Hinkley, 1979), where n and b are ﬁxed, hence λ

Original: → 1 in (16).
Revised: → 1 in (15).

Original: For the optimal estimator (16), the value of b is found by minimizing the average KL-distance (5).
Revised: For the optimal estimator (15), the value of b is found by minimizing the average KL-distance (4).

Original: When the prior is given by a Dirichlet density (6), the average KL-distance amounts to (12), where we need to replace ln( M + A ) → ln( M + nb ) and ln( m + α k )
Revised: When the prior is given by a Dirichlet density (5), the average KL-distance amounts to (11), where we need to replace ln( M + A ) → ln( M + nb ) and ln( m + α k )

Original: → ln( m + b ) .
Revised: → ln( m + b ) .

Original: Now (10, 11) imply that for a homogeneous Dirichlet prior, i.e. for (6) with α k =
Revised: Now (9, 10) imply that for a homogeneous Dirichlet prior, i.e. for (5) with α k =

Original: α , we have b opt =
Revised: α , we have b opt =

Original: α for the optimal value of b , i.e. the regularized ML estimator coincides with the Bayesian estimator: p ML ( z k ) =
Revised: α for the optimal value of b , i.e. the regularized ML estimator coincides with the Bayesian estimator: p ML ( z k ) =

Original: p ( z k ) .
Revised: p ( z k ) .

Original: This does not anymore hold for the mixture of Dirichlet prior densities; see below.
Revised: This does not anymore hold for the mixture of Dirichlet prior densities.

Original: Then b is to be found numerically.

Original: For M (cid:29) n the precise value of b is not essential.

Original:  T HE MAXIMUM ENTROPY (MAXENT) METHOD
Revised:  T HE MAXIMUM ENTROPY (MAXENT) METHOD

Original: MAXENT infers probabilities from maximizing the Boltzmann-Gibbs-Shannon entropy
Revised: MAXENT infers probabilities from maximizing the Boltzmann-Gibbs-Shannon entropy

Original: [Equation] 
Revised: [Equation] [Equation]

Original: (17) under constraints taken from the sample (1).
Revised: under constraints taken from the sample (1).

Original: The rationale of maximizing (17) is that a larger S means a smaller bias (or information) according to several axiomatic schemes (Jaynes, 1957;Press´e et al., 2013; Chakrabarti & Chakrabarty, 2005;
Revised: The rationale of maximizing (16) is that a larger S means a smaller bias (or information) according to several axiomatic schemes (Jaynes, 1957;

Original:  Baez et al., 2011; Van Campenhout & Cover, 1981; 
Revised: Chakrabarti & Chakrabarty, 2005; Baez et al., 2011; Van Campenhout & Cover, 1981; Topsøe,

Original: Topsøe, 1979; Shore & Johnson, 1980; Paris & Vencovsk´a, 1997).
Revised: 1979; Shore & Johnson, 1980; Paris & Vencovsk´a, 1997; Balian, 2007; Press´e et al., 2013).

Revised: Note that physical applications of MAXENT operate with constraints that are known precisely, e.g. the mean energy constraint is deduced from the corresponding conservation law (Jaynes, 1957; Balian, 2007; Press´e et al., 2013).

Revised: Such situations are rare in statistics and machine learning.

Revised: Hence we need to understand which constraints are to be taken from the noisy data.

Original: First we can apply no constraint and maximize the entropy:
Revised: First we can apply no constraint and maximize the entropy:

Original: [Equation]
Revised: [Equation]

Original: The calculation of the average distance is straightforward from (12, 18) both for a single Dirichlet prior and a mixture of such priors.
Revised: The calculation of the average distance is straightforward from (4, 11, 17) both for a single Dirichlet prior and a mixture of such priors.

Original: We examplify the single Dirichlet case (6):
Revised: We examplify the single Dirichlet case (5):

Original: [Equation]
Revised: [Equation]

Original: Now (cid:104) K [ q, q [0] ] (cid:105) plays an important role: once (18) is completely data-independent and simply reproduced the prior expectation on the unbiased probabilities, estimators that provide the average
Revised: Now (cid:104) K [ q, q [0] ] (cid:105) plays an important role: once (17) is completely data-independent and simply reproduced the prior expectation on the unbiased probabilities, estimators that provide the average

Original: KL-distance larger than (19) are meaningless; see below for examples.
Revised: KL-distance larger than (18) are meaningless; see below for examples.

Original: Next, we employ the empiric mean of (1) as a constraint for the expected value of Z :
Revised: Next, we employ the empiric mean of (1) as a constraint for the expected value of Z :

Original: [Equation]
Revised: [Equation]

Original: Maximizing (17) leads to the famous Gibbs formula:
Revised: Maximizing (16) under constraint (19) via the Lagrange method leads to the famous Gibbs formula (Jaynes, 1957; Balian, 2007; Press ´ e et al., 2013):

Original: [Equation]
Revised: [Equation]

Original: 21) where the Lagrange multiplier β is found from  (20) (Jaynes, 1957).
Revised: where the Lagrange multiplier β is found from (19). Appendix presents an example of applying (20) to real data.

Original: We order the values of Z as z 1 < ... <
Revised: We order the values of Z as z 1 < ... <

Original: z n , and note a speciﬁc feature of (21): depending on the sign of β , we either getq [1] ( z
Revised: z n , and note a speciﬁc feature of (20): depending on the sign of β , we either get

Original: [Equation]
Revised: [Equation]

Original: ) ≤ ... ≤ q [1] ( z n ) or q [1] ( z 1 ) ≥ ... ≥ q [1] ( z n ) .

Original: One can try to acquire further information from  (1) by ﬁxing also the second  moment:
Revised: One can try to acquire further information from sample (1) by looking at the second empiric moment:

Original: [Equation]
Revised: [Equation]

Original: and maximize (17) under two constraints (20) and (23):
Revised: Now we maximize (16) under two constraints (19) and (22):

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where  β 1 and β 2 are determined from solving (20) and (23) simultaneously.
Revised: where Lagrange multipliers β 1 and β 2 are found from solving both (19) and (22).

Original: Eqs.
Revised: Eqs.

Original: (21, 24) make obvious the result of ﬁxing other (e.g. fractional) moments.
Revised: (20, 23) make obvious how to involve other (fractional) moments.

Original: Such maximization results are unique, because (17) is a convex function of { p k } nk =1 , while the moment constraints are linear.
Revised: The maximizations of (16) lead to unique results, because (16) is a concave function of { p k } nk =1 , while the moment constraints are linear.

Original: Let the values ( z 1 , ..., z n ) of Z be subject to afﬁne transformation
Revised: Let the values ( z 1 , ..., z n ) of Z be subject to afﬁne transformation

Original: [Equation]
Revised: [Equation]

Original: Hence as a result of transformation (25): µ 1 → (cid:101) µ 1 = gµ [-
-]
Revised: Hence as a result of transformation (24): µ 1 → (cid:101) µ 1 = gµ [-
-]1 + [Equation] and µ [Equation]

Original: + h and µ 2 → (cid:101) µ 2 =
Revised: → [Equation]

Original: g 2 µ 2 [-+ h 
-]
Revised: µ 2 [-+ h 
-]= g [Equation] µ

Original: + 2 ghµ 1 ; see (20, 23).
Revised: [Equation] 1 ; see (19, 22).

Original: These relations show that (25) leaves the inferred probabilities (21, 24) invariant, because the resulting set of equations for the unknowns in (21, 24) are identical for both the original ( z 1 , ..., z n ) and transformed values (25).
Revised: These relations show that (24) leaves the inferred probabilities (20, 23) invariant, because the resulting set of equations for the unknowns in (20, 23) are identical for both the original ( z 1 , ..., z n ) and transformed values (24).

Original: Likewise, ﬁxing ﬁrst p moments µ 1 , ..., µ p produces afﬁneinvariant probabilities.
Revised: Likewise, involving ﬁrst p moments µ 1 , ..., µ p produces afﬁne-invariant probabilities.

Original: Note that ﬁxing only (23) [without ﬁxing (20)] will lead to the invariance of the probabilities with respect to a limited afﬁne-symmetry, where h = 0 in (25).
Revised: Note that involving only (22) [without involving (19)] will lead to the invariance of the probabilities with respect to a limited afﬁne-symmetry, where h = 0 in (24).

Original: Another example of limited afﬁne symmetry is ﬁxing the fractional moment (cid:80) nk =1 q k √ z k (for z k ≥ 0 and instead of (20, 23)).
Revised: Another example of limited afﬁne symmetry is involving the fractional moment (cid:80) nk =1 q k √ z k (for z k ≥ 0 and instead of (19, 22)).

Original: Then the probabilities q [1 / 2] ( z k ) ∝
Revised: Then the probabilities q [1 / 2] ( z k ) ∝

Original: e − β 1 / 2 √ z k will stay intact only under h = 0 and g > 0 in (25).
Revised: e − β 1 / 2 √ z k will stay intact only under h = 0 and g > 0 in (24).

Original: Note in this context that the ML estimator (16) is invariant with respect (25) with an arbitrary bijective F , which keeps the values of (cid:101)
Revised: Note in this context that the ML estimator (15) is invariant with respect (24) with an arbitrary bijective F , which keeps the values of (cid:101)

Original: z k different.
Revised: z k different.

Original: The symmetry features of various estimators are clearly important, though we so far have no analytical results that would relate them to the estimation quality quantiﬁed by (5).
Revised: The symmetry features of various estimators are clearly important, though we so far have no analytical results that would relate them to the estimation quality quantiﬁed by (4).

Original: But we noted from numerical comparison of MAXENT estimators based on various constraints, that estimators with the largest afﬁne symmetry—i.e. (25) with arbitrary g and h —tend to be better in terms of the average
Revised: But we noted from numerical comparison of MAXENT estimators based on various constraints, that estimators with the largest afﬁne symmetry—i.e. (24) with arbitrary g and h —tend to be better in terms of the average

Original: KL-distance (5).
Revised: KL-distance (4).

Original: Intuitively, higher (afﬁne) symmetry should be related to higher susceptibility with respect to noises.
Revised: Intuitively, higher (afﬁne) symmetry should be related to higher susceptibility with respect to noises; see Appendix B for further results.

Original: Thus based on numerical evidence, we below never employ MAXENT estimators with fractional ﬁxed moments or with the second moment alone.

Original:  N UMERICAL RESULTS 
Revised:  N UMERICAL RESULTS 5. A SINGLE D IRICHLET DENSITY

Original: Recall that maximization of entropy (17) can be applied if there is no prior information that distinguishes one probability from another.
Revised: Recall that maximization of entropy (16) can be applied if there is no prior information that distinguishes one probability from another.

Original: If such information is present, MAXENT is generalized to the minimum relative entropy method (Shore & Johnson, 1980).
Revised: If such information is present, MAXENT is generalized to the minimum relative entropy method (Shore & Johnson, 1980).

Original: We shall not study this generalization here.
Revised: We shall not study this generalization here.

Original: Hence to ensure applicability of MAXENT, we always choose prior densities such that (cid:104) q k
Revised: Hence to ensure applicability of MAXENT, we always choose prior densities such that (cid:104) q k

Original: (cid:105)
Revised: (cid:105)

Original: = 1 n ; i.e. all n values are equally likely to be generated, on average.
Revised: = 1 n ; i.e. all n values are equally likely to be generated, on average.

Original: As seen from (7), for a single prior Dirichlet density (6) condition (cid:104) q k (cid:105)
Revised: As seen from (6), for a single prior Dirichlet density (5) condition (cid:104) q k (cid:105)

Original: =
Revised: =

Original: 1 n implies:
Revised: 1 n implies:

Original: [Equation]
Revised: [Equation]

Original: [Table] Now recall from (16, 11) that under (26) the Bayesian and the regularized ML coincide.
Revised: Now recall from (15, 10) that under (25) the Bayesian and the regularized ML coincide.

Original: I e. we conclude that the regularized ML is a better estimator than MAXENT (under any constraint)
Revised: I.e. we conclude that the regularized ML is a better estimator than MAXENT (under any constraint). Though (cid:104) q k (cid:105) =

Original: How meaningful are predictions of MAXENT under (26)?{+q k do depend on the magnitude of α . Finding (cid:101) q k from (5, 25) amounts to maximizing L ( q ) = ( α − 1) (cid:80) nk =
 ln q k + γ (cid:80) nk+}
Revised: 1 n does not depend on α , the most probable values (cid:101) q k of {+q k do depend on the magnitude of α . Finding (cid:101) q k from (5, 25) amounts to maximizing L ( q ) = ( α − 1) (cid:80) nk =
 ln q k + γ (cid:80) nk+}

Original: Table 1 compares predictions of (16) with those of MAXENT solutions (21) and (24) for the Dirichlet prior (6) holding (26).
Revised: =1 q k , where the Lagrange multiplier γ ensures (cid:80) nk =1 q k = 1 . For α > 1 , L ( q ) is a concave function of q , and its global maximum is found after differentiating it. Hence (cid:101) q k holds [Equation]

Revised: For α < 1 , L ( q ) is a convex function, it does not have local maxima with q k > 0 ( k = 1 , ..., n ).

Revised: Its maxima are located at points, where q k = 0 for certain k .

Original: It is seen that the very applicability of MAXENT demands a sufﬁciently large sample length M .
Revised: Repeating this argument, we see that the maxima of L ( q ) are at those points, where a possibly large number of q k are zero: [Equation] which means deterministic probabilities. Eq.

Original: Otherwise,
Revised: (27) is consistent with (cid:104) q k (cid:105) = 1 /n , because there are n equivalent most probable values. Let us start with the regime α > 1 ; cf.

Original:  MAXENT is meaningless, because the trivial estimator (18) provides a smaller average KL-distance; cf.For parameters of Table 1 predictions of MAXENT are meaningless for M ≤ 35 with n = 60 .
Revised: Table 1 compares predictions of (15) with those of MAXENT solutions (20) and (23) for the Dirichlet prior (5) holding (25) with α = 2 . It is seen that MAXENT is meaningless, because the trivial estimator (17) provides a smaller average KLdistance; cf.

Original: In contrast, for the Bayesian estimator even M = 1 leads to a meaningfull prediction; e.g. for parameters of Table 1 we have: (cid:104) K Bayes (cid:105)| M =1 = 0 .
Revised: For the Bayesian estimator even M = 1 leads to a meaningfull prediction; e.g. for parameters of Table 1 we have: (cid:104) K Bayes (cid:105)| M =1 = 0 .

Original: 224 < 0 . 225 .
Revised: 224 < 0 . 225 .

Original: [Table]
Revised: The above conclusion holds more generally (as we checked numerically): for the homogeneous Dirichlet prior (25) with α ≥ 1 , MAXENT estimators (20, 19) and (23, 19, 22) are meaningless at least in the sparse data regime M < n . This puts a serious limitation on the validity of MAXENT. The situation changes for sufﬁciently small values of α in the regime (27); see Table 2 for α = 0 .

Original: Thus the applicability range of MAXENT  is to be restricted to having some prior knowledge about the unknown probabilities { q k } nk =1 .
Revised: Here the MAXENT estimators are meaningful provided that the sample length M is sufﬁciently large (but still in the sparse data regime M < n ): (20, 19) is meaningful for M ≥ 9 ( M < n = 60 ), while the estimator (23, 19, 22) is meaningful for M ≥ 25 ; see Table 2. Though predictions of MAXENT are still far from those of the Bayesian estimator, we should recall that the latter estimator is parametric, i.e. it depends on the prior (via the parameter α ) in contrast to MAXENT estimators. Table 2 demonstrates the overﬁtting phenomenon: for 9 ≤ M ≤ 15 the MAXENT estimator (20, 19) is meaningful, but adding the second constraint makes the MAXENT estimator (23, 19, 22) not meaningful.

Original: We still assume that prior expectations are equal: (cid:104) q k (cid:105) = 1 n . As a prior density we shall now apply a mixture of Dirichlet densities such that there are deﬁnite rank correlations between the random variable ( z 1 , ..., z n ) , and probabilities ( q 1 , ..., q n ) .
Revised: The situation is worsened since (22) is again estimated from the noisy data and gathers more noise than information. This overﬁtting disappears for larger values of M , i.e. M ≥ 25 , as

Original: I.e. if
Revised: Table 2 demonstrates. Now adding the second constraint (22) is beneﬁcial.

Original: [Table]
Revised: [Table]

Original: 2 < K 1 <

Original: [Table]
Revised: [Table]

Original:  K 
Revised: (cid:104) K 1 (cid:105) for 35 ≥ M ≥ 1 . We do not show (cid:104) K 1 (cid:105)| M =1 , since it is larger than the average KL-distance for all other estimators.

Original: [Equation] [Table]
Revised: [Table] [Table] [Table]

Original: [Equation]
Revised: For modeling more complex types of prior information about the unknown probabilities { q k } nk =1 , we shall assume that the prior density is a mixture of two Dirichlet densities; see (12).

Original: [Table]
Revised: Relations (cid:104) q k (cid:105) = 1 n ( k = 1 , ..., n ) will be still kept, since they are necessary for applying MAXENT. Now we assume that that there are (prior) conditional rank correlation between the values ( z 1 , ..., z n ) of

Original: we assume the ordering ( z 1 < ... <
Revised: Z —ordered as ( z 1 < ... <

Original: z n )  , then ( q 1 , ..., q n ) prefer to be ordered  inthe same way ( q 1 < ... < q n ) or in the opposite way: ( q 1 > ... > q n ) ; cf.
Revised: z n ) —and its probabilities ( q 1 , ..., q n ) . For one component of the mixture, the probabilities ( q 1 , ..., q n ) prefer to be ordered as in ( q 1 < ... < q n ) . For another component they tend to be ordered in the opposite way ( q 1 > ... > q n ) .

Original: Such a prior knowledge can be modeled via a mixture (13) of two Dirichlet priors with L = 2 , π 1 = π 2 = 12 , and α[ [Equation]
Revised: This type of prior knowledge can be modeled via a mixture (12) of two Dirichlet priors with L = 2 , π 1 = π 2 = 12 , and α

Original: [Equation]
Revised: [Equation]

Original: for any
Revised: any

Original: [Equation]
Revised: [Equation]

Original: (28) where (28) ensures the needed (cid:104) q k (cid:105) = 1 n , as seen from (14).
Revised: 29) where (29) ensures the needed (cid:104) q k (cid:105) = 1 n , as seen from (13).

Original: A simple case that leads to (27, 28) is
Revised: A simple case that leads to (28, 29) is

Original: L = 2 ,
Revised: L = 2 ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where A [1] =
Revised: where A [1] =

Original: A [2] =
Revised: A [2] =

Original: α 0 n + (cid:15)n ( n − 1).
Revised: α 0 n + (cid:15)n ( n − 1).

Original: Recall that for a mixture of Dirichlet densities the Bayes estimator (15) and the optimally regularized ML estimator (16) are different.
Revised: Recall that for a mixture of Dirichlet densities the Bayes estimator (14) and the optimally regularized ML estimator (15) are different.

Original: For numerical illustration we choose { z k = k } nk =1 .
Revised: For numerical illustration we choose { z k = k } nk =1 .

Original: Prior probability densities generated via (29) will be employed with z 1 < ... <
Revised: Prior probability densities generated via (30) will be employed with z 1 < ... <

Original: z n .
Revised: z n .

Original: Now Tables2 and 3  show that for M ≥ 5 the MAXENT estimator (21, 20) is clearly better than the optimally regularized ML estimator (16) (where the optimal value of b is found from minimizing the averaged KL-distance (5)):
Revised: Now Tables 3 and 4 show that for M ≥ 5 the MAXENT estimator (20, 19) is clearly better than the optimally regularized ML estimator (15):

Original: [Equation]
Revised: [Equation]

Original:  Moreover, for M ≥ 7 , we see that (cid:104) K 1 (cid:105) is closer to the optimal (cid:104) K Bayes (cid:105) than to (cid:104) K ML (cid:105) b
Revised: where the optimal value of b is found from minimizing the averaged KL-distance (4). Moreover, for M ≥ 7 , we see that (cid:104) K 1 (cid:105) is closer to the optimal (cid:104) K Bayes (cid:105) than to (cid:104) K ML (cid:105) b

Original: = b opt .
Revised: = b opt .

Original: Note that such threshold values for M do depend on the assumed prior density and on n .
Revised: Note that such threshold values for M do depend on the assumed prior density and on n .

Original: For M → ∞ the performance of the optimally regularized ML estimator (16) (for a ﬁxed b ∼ 1 ) will be better than MAXENT with any ﬁnite number of constraints, since the regularized ML converges to the true probabilities for M → ∞ (Cox & Hinkley, 1979), while MAXENTunder a few constraints does not.
Revised: For M → ∞ the performance of the optimally regularized ML estimator (15) (for a ﬁxed b ∼ 1 ) will be better than MAXENT with any ﬁnite number of constraints, since the regularized ML converges to the true probabilities for M → ∞ (Cox & Hinkley, 1979), while MAXENT does not.

Original: But as Table 3 shows, MAXENT with constraints (20) or (20)+(23) still performs better than the optimally regularized ML even for M as large as 241 (for n = 60 ).
Revised: But as Table 4 shows, MAXENT with constraints (19) or (19)+(22) still performs better than the optimally regularized ML even for M as large as 241 (for n = 60 ).

Original: Table 2 shows that MAXENT with two constraints (20)+(23) performs worse than the method under the single constraint (20) although the afﬁne invariance (25) of probabilities holds.
Revised: Table 3 shows that MAXENT with two constraints (19, 22) performs worse than the method under the single constraint (19) although the afﬁne invariance (24) of probabilities holds.

Original: The situation is worsened since the second constraint (23) is again estimated from the data, and with too little data the estimated constraints may be be far from the true values, and overﬁtting these constraints leads to worse results.

Original: Naturally, this situation changes for larger values of M , i.e. M ≥ 45 , as Tabledemonstrates.
Revised: This overﬁtting situation changes for larger values of M , i.e. M ≥ 45 , as Table 4 demonstrates.

Original: We emphasize that the advantage (30) of MAXENT closely relates to the agreement between (27) and the ordering ( z 1 < ... <
Revised: To stress the relevance of rank correlations, we note that the advantage (31) of MAXENT closely relates to the agreement between (28) and the ordering ( z 1 < ... <

Original: z n ) of Z .
Revised: z n ) of Z .

Original: If the vector ( z 1 < ... < z n ) is randomly permuted, and employed for values of Z , predictions of MAXENT become devastating even for rather large values of M .

Original: To illustrate this point, let us take parameters of Table 2, but apply a random permutation to values { z k = k } n =60 k =1 .

Original: Taking this new vector  for values of Z ,we noted that even for M = 85 > n = 60 predictions of MAXENT under constraint (20) is meaningless, since the average KL distance amounts to 0 . 215 , which is larger than the prediction 0 . 212 of(19). For M = 105 this prediction 0 .
Revised: If the vector ( z 1 < ... < z n ) is randomly permuted, and employed for values of Z , predictions of MAXENT become meaningless even for rather large values of M > n .

Original: 210 is not meaningless in this sense, but rather poor by itself.

Original: It does not substantially improve even for M = 500 (for n = 60 ), where it amounts to 0 . 207 .

Original: Though such predictions are not meaningless, they are practically useless.

Original: Recall that both the Bayesian (15) and the regularized ML estimator (16) are parametric, i.e. the very their form depends on the prior, which is frequently not available in practice.
Revised: Recall that both the Bayesian (14) and the regularized ML estimator (15) are parametric, i.e. the very their form depends on the prior, which is frequently not available in practice.

Original: Hence we need to understand how strong is dependence.
Revised: Hence we need to understand how strong is dependence.

Original: Let us assume that one has to built a Bayesian estimator without knowing the full form of the equal-weight mixture (29).
Revised: Let us assume that one has to employ a Bayesian estimator without knowing the full form of the equal-weight mixture (30).

Original: Instead one knows the average values of α k = 

Revised: Instead one knows the average values of α k =

Original: [ α 0 + (cid:15) ( k − 1)]+ 12 [ α 0 + (cid:15) ( n − k )] from (29), prescribes them to a single Dirichlet prior (6, 26) and builts up from (11) an estimator
Revised: [ α 0 + (cid:15) ( k − 1)]+ 12 [ α 0 + (cid:15) ( n − k )] from (30), prescribes them to a single Dirichlet prior (5, 25) and builds up from (10) an estimator

Original: [Equation]
Revised: [Equation]

Original: The performance of this perturbed Bayesian estimator deteriorates and gets worse than that of the
Revised: The performance of this perturbed Bayesian estimator deteriorates and gets worse than that of the

Original: MAXENT solution: (cid:104) K 1 (cid:105) <
Revised: MAXENT solution: (cid:104) K 1 (cid:105) <

Original: (cid:104) K Bayes (cid:105) ; see Table 2.
Revised: (cid:104) K Bayes (cid:105) ; see Table 3.

Original: Likewise, the choice of b in the regularized ML estimator (16) is important.
Revised: Likewise, the choice of b in the regularized ML estimator (15) is important.

Original: If just some reasonable value is chosen instead of the optimal one—e.g. b = 1 instead of b ∼ 2 .
Revised: If just some reasonable value is chosen instead of the optimal one—e.g. b = 1 instead of b ∼ 2 .

Original: in Table 2—then the ML estimator can turn meaningless; see Tablefor M ≤ 15 .
Revised: in Table 3—then the ML estimator can turn meaningless; see Tablefor M ≤ 15 .

Original: In this context, we emphasize that the Bayes estimator and the optimally regularized
Revised: In this context, we emphasize that the Bayes estimator and the optimally regularized

Original: ML estimator are never meaningless even for M = 1 .
Revised: ML estimator are never meaningless even for M = 1 .

Original: For parameters of Table 2, the MAXENT estimator (21) becomes meaningless for M ≤ 3 .
Revised: For parameters of Table 3, the MAXENT estimator (20) becomes meaningless for M ≤ 3 .

Original:  S UMMARY AND D ISCUSSION
Revised:  S UMMARY AND D ISCUSSION

Original: The maximum entropy (MAXENT) method provides non-parametric estimators for inferring unknown probabilities (Erickson & Smith, 2013).
Revised: The maximum entropy (MAXENT) method provides non-parametric estimators for inferring unknown probabilities (Erickson & Smith, 2013).

Original: MAXENT is widely applied both in statistical physics and probabilistic inference.
Revised: MAXENT is widely applied both in statistical physics and probabilistic inference.

Original: However, its physical applications are mostly data-free and are based on additional principles (e.g. conservation laws (Balian, 2007; Press ´ e et al., 2013)) that are  absent in inference.
Revised: However, its physical applications are mostly data-free and are based on additional principles (e.g. conservation laws (Balian, 2007; Press ´ e et al., 2013)) that are normally absent in statistics and machine learning.

Original: This fact is/was not sufﬁciently emphasized by creators and practitioners of MAXENT.

Original: Hence we needed a systematic approach towards understanding the validity limits of MAXENT as an inference tool.
Revised: Hence we needed a systematic approach towards understanding the validity limits of MAXENT as an inference tool.

Original: Here we presented a Bayesian  approach that allows to decide on whether MAXENT is applicable at all, i.e. whether it is better than a random guess.
Revised: Here we presented a Bayesian decision theory approach that allows to determine on whether MAXENT is applicable at all, i.e. whether it is better than a random guess.

Original: It also allows to compare different estimators with each other (e.g. to compare MAXENT with the regularized maximum likelihood) and determine the relevance of various constraints employed in MAXENT.
Revised: It also allows to compare different estimators with each other (e.g. to compare MAXENT with the regularized maximum likelihood), and study the relevance of various constraints employed in MAXENT.

Original: In particular, the approach allows to discuss the phenomenon of overﬁtting, where adding more constraints leads to poorer inference.

Original: Our results can be brieﬂy summarized as follows.
Revised: Our results are summarized as follows.

Original: MAXENT doesnot apply to a sparse data generated by homogeneous (i.e. maximally non-informative) Dirichlet density (26).
Revised: MAXENT does apply to a sparse data, but demands speciﬁc prior information.

Original: Here sparse means M < n , i.e. the sample length M is smaller than the number of probabilities n to be inferred. 
Revised: Here sparse means M < n , i.e. the sample length M is smaller than the number of probabilities n to be inferred. We explored two different scenarios of such prior information.

Original: We explored the simplest prior information that makes MAXENT applicable and superior over the optimally regularized maximum-likelihood estimator: prior rank correlations between the random quantity and its probabilities.
Revised: First, the unknown probabilities generated by homogeneous Dirichlet density (25) are most probably deterministic. Second, there are prior rank correlations between the random quantity and its probabilities. This seems to be the simplest prior information that makes MAXENT applicable and superior over the optimally regularized maximum-likelihood estimator.

Original: We currently explore more general types of prior information that can make MAXENT applicable.
Revised: Our approach is capable of describing several phenomena that are relevant for applying and understanding estimators: overﬁtting (i.e. adding more noisy constraints leads to poorer inference), instability of optimal Bayesian parametric estimators with respect to variation of prior details, inapplicability of non-parametric MAXENT estimators to very short samples etc .

Original: Several important problems were uncovered by this study and should be addressed in future together with more detailed understanding of the presented results.
Revised: Several important problems were uncovered by this study and should be addressed in future.

Original: First of all, this concerns the applicability of MAXENT to a categorical data, where the values ( z 1 , ..., z n ) of the random variable Z in sample (1) are not numerical, but instead refer to certain distinguishable categories.
Revised: First of all, this concerns the applicability of MAXENT to a categorical data, where the values ( z 1 , ..., z n ) of the random variable Z in sample (1) are not numerical, but instead refer to certain distinguishable categories.

Original: The major difference between maximum likelihood and MAXENT estimator is that the former freely applies to categorical data.
Revised: The major difference between maximum likelihood and MAXENT estimator is that the former freely applies to categorical data.

Original: In contrast, MAXENT does depend on the concrete numerical implementation (i.e. encoding ) of data, though this dependence is somewhat weakened by the afﬁne symmetry (25).
Revised: In contrast, MAXENT does depend on the concrete numerical implementation (i.e. encoding ) of data, though this dependence is somewhat weakened by the afﬁne symmetry (24).

Original: Thus an open problem demands considering various encoding schemes in view of their applicability to MAXENT estimators.
Revised: Thus an open problem demands considering various encoding schemes in view of their applicability to MAXENT estimators.

Original: (In this paper we in fact assumed the simplest encoding via natural numbers; see Tables.) 
Revised: (In this paper we in fact assumed the simplest encoding via natural numbers; see Tables.) Appendix A reports preliminary results in this direction along with a real data example.

Original: The second open problem relates to the inﬂuence of afﬁne symmetries on the performance of various MAXENT estimators.
Revised: The second open problem relates to the inﬂuence of afﬁne symmetries on the performance of various MAXENT estimators.

Original: We observed numerically that the contraints which produce afﬁne-invariant probabilities produce better estimators; see after (25).
Revised: We observed numerically that the constraints which produce afﬁne-invariant probabilities produce better estimators; see after (24).

Revised: Preliminary results along this direction are given in Appendix B, where we also show relations of our results with the minimum entropy principle proposed in (Good, 1970; Christensen, 1985; Zhu et al., 1997; Pandey & Dukkipati, 2013) for contraint selection.

Revised: A A PPENDIX : MAXENT APPLIES TO CATEGORICAL DATA

Revised: The MAXENT method can be applied to any multinomial data (1), provided that numeric values ( z 1 , ..., z n ) of the random quantity Z are given.

Revised: MAXENT estimators depend on ( z 1 , ..., z n ) modulo the afﬁne symmetry (24).

Original: We were not able to prove ordisprove this statement in general, and hence it is left as an open problem.
Revised: This creates a problem in applying MAXENT to categorical data, since for MAXENT one now needs a speciﬁc encoding of categorical Z into a numeric representation ( z 1 , ..., z n ) of categories. Recall that there is a degree of arbitrariness in choosing the regularizer in maximum likelihood (ML) estimator, or in choosing prior parameters for Bayesian inference. Here the arbitrariness lies in different encodings. In practice, the proper encoding of categories arises in any problem dealing with categorical data.

Revised: If categories are ordinal (e.g. military ranks, education levels), then one can use z 1 < z 2 < ... <

Revised: z n encoding.

Revised: However, for nominal categories (e.g. ethnicity, preference, disease) there is no such ordering.

Original: A A PPENDIX{+Let us illustrate the MAXENT method with the simple data from pre-election presidential polling conducted in 1988, where out of M = 
 voters m 1 = 727 preferred Bush, m 2 = 583 preferred Dukakis, and m 3 = 137 preferred other candidates or had no preference. The data together with its+}
Revised: {+Let us illustrate the MAXENT method with the simple data from pre-election presidential polling conducted in 1988, where out of M = 
 voters m 1 = 727 preferred Bush, m 2 = 583 preferred Dukakis, and m 3 = 137 preferred other candidates or had no preference. The data together with its+}

Revised: Bayesian analysis is taken from (Gelman et al., 2013).

Revised: Here our random variable Z is voter’s preference with three outcomes ( z 1 , z 2 , z 3 ) =

Revised: (’Bush’, ’Dukakis’, ’Other’) and unknowns ( q 1 , q 2 , q 3 ) , which just represent the fractions of the population with each preference.

Revised: The goal here is to estimate q 1 − q 2 , i.e. whether Bush has more support than Dukakis.

Revised: One can assume the Dirichlet noninformative prior distribution for ( q 1 , q 2 , q 3 ) with parameters α 1 = α 2 = α 3 = 1 , compute the posterior means of q 1 and q 2 (ˆ q 1 , ˆ q 2 ) and take the difference (Gelman et al., 2013).

Revised: The results show that Bush has more support: ˆ q 1 > ˆ q 2 .

Revised: Since the data is purely categorical, we shall apply the frequency encoding for MAXENT: each category is represented with its frequency in the data set, e.g. in this example ( z 1 , z 2 , z 3 ) =

Revised: Now empiric mean is equal to 0 .

Revised: 42 and the maximizing solutions of (16) with (cid:80) 3 k =1 q k z k = 0 .

Revised: 42 are ( q [1] ( z 1 ) , q [1] ( z 2 ) , q [1] ( z 3 ))

Revised: = (0 . 535 , 0 . 36 , 0 . 105) .

Revised: Thus, also the

Revised: MAXENT result shows more support for Bush.

Original: {+To see if the prediction of MAXENT is reliable (on average) here, the same Bayesian decision model for these samples is set up, where ﬁrst a sample of ( q 1 , q 2 , q 3 ) is drawn from the Dirichlet distribution with α 
+}
Revised: {+To see if the prediction of MAXENT is reliable (on average) here, the same Bayesian decision model for these samples is set up, where ﬁrst a sample of ( q 1 , q 2 , q 3 ) is drawn from the Dirichlet distribution with α 
+}

Revised: = α 2 = α 3 = 1 , and then using this sample as category probabilities, categorical data sets of size M are generated with categories replaced by its frequency encodings.

Revised: The process is repeated and the average (cid:104) K 1 (cid:105) from (4) is computed via generating 10 3 instances of { q k } 3 k = and 10 3 categorical samples.

Original: You may include other additional sections here.
Revised: For the present case α 1 = α 2 = α 3 = 1 and n = 3 , we have (cid:104) K [ q, q [0] ] (cid:105) = 0 . 265 from (18).

Revised: Now for M > 15 we get that (cid:104) K 1 (cid:105) <

Revised: (cid:104) K [ q, q [0] ] (cid:105) , i.e. the MAXENT solution is reliable.

Revised: For example, at M = 17 we have (cid:104) K Bayes (cid:105) = 0 .

Revised: 046 < (cid:104) K 1 (cid:105)

Revised: = 0 .

Revised: 116 < (cid:104) K [ q, q [0] ] (cid:105) = 0 .

Revised: 265 , where (cid:104) K Bayes (cid:105) refers to the Bayesian (posterior mean) estimator (10).

Revised: Already for M = 47 predictions of

Revised: MAXENT are close to those of the optimal Bayesian estimator (cid:104) K Bayes (cid:105) = 0 .

Revised: 019 < (cid:104) K 1 (cid:105) = 0 .

Revised: For the actual sample size M = 1447 , we get even closer results (cid:104) K Bayes (cid:105) = 0 .

Revised: 00087 <

Revised: (cid:104) K 1 (cid:105)

Revised: = 0 .

Revised: To summarize the present real categorical data example, we saw that the frequency encoding of the categorical variable allows to apply MAXENT.

Revised: The MAXENT estimator (19) agrees with Bayesian estimator, and is going to be reliable already for modest sample sizes M > 15 .

Revised: For a sufﬁciently large M the average KL distance of the MAXENT estimator gets close to that of the (optimal) Bayes estimator.

Revised: B A PPENDIX : A FFINE SYMMETRY AND THE MINIMUM ENTROPY PRINCIPLE

Revised: Above we focused on MAXENT estimators (20, 19) (the ﬁrst empiric moment is ﬁxed) or (23, 19, 22) (the ﬁrst and second empiric moments are ﬁxed).

Revised: As discussed around (24), both (19) and (22) lead to afﬁne-invariant probabilities.

Revised: We studied several alternative constraints that do not have the full afﬁne symmetry, i.e. this symmetry is partial and relates to restriction on the parameters in (24).

Revised: An example of this is constraining the square-root (fractional) moment

Revised: [cf.

Revised: [Equation]

Revised: where β 1 / 2 is determined from (34), and where we assumed z k > 0 .

Revised: For estimator (33) the symmetry (24) is kept under g > 0 and h = 0 .

Revised: We denote the corresponding average KL distances by (cid:104) K 1 / 2 (cid:105) .

Revised: Let us now compare two different MAXENT estimators each one employing its own constraint; e.g. we compare (20) with (33).

Revised: We saw from extensive numeric simulations that whenever these constraints have different degrees of the afﬁne symmetry, then the estimator having the largest symmetry provides a smaller average KL distance.

Revised: A particular example of this general relation is:

Revised: [Equation]

Revised: 35) which was veriﬁed on parameters of Tables 1–4.

Revised: Recall that Refs.

Revised: (Good, 1970; Christensen, 1985; Zhu et al., 1997) proposed the minimum entropy principle: when comparing two possible contraints to be employed in the maximum entropy method, then it is preferable to use the one that provides the smaller (maximized) entropy.

Revised: The heuristic motivation of the principle is that it avoids overﬁtting by not insisting too much on the entropy maximization.

Revised: This principle was motivated via the minimum description length in (Pandey &

Revised: Dukkipati, 2013).

Revised: We ask whether in cases similar to (35) we can compare the average entropies, i.e. for (35) we compare (cid:104) S

Revised: [ q [1] ( z k )] (cid:105) and (cid:104) S

Revised: [ q [1 / 2] ( z k )] (cid:105) , where the averages are deﬁned as in (4).

Revised: In all cases we were able to check, relations similar to (35) are accompanied by the result that the constraint which provide a smaller average KL distance (i.e. a better costraint) also has a smaller average entropy, e.g.

Revised: [Equation]

Revised: The theoretical origin of this relation between the average KL distance and the average (maximized) entropy is not yet clear.

Revised: Here is a concrete numerical example that illustrates (35, 36).

Revised: For parameters of Table 2 we noted for M = 55 :

Revised: [Table]

Revised: K

Revised: [Table]

Revised: and

Revised: [Table]

Revised: <

Revised: [Table]