Paper:
On Learning Markov Chains
The problem of estimating an unknown discrete distribution from its samples is a fundamental tenet of statistical learning. Over the past decade, it attracted significant research effort and has been solved for a variety of divergence measures. Surprisingly, an equally important problem, estimating an unknown Markov chain from its samples, is still far from understood. We consider two problems related to the min-max risk (expected loss) of estimating an unknown k-state Markov chain from its n sequential samples: predicting the conditional distribution of the next sample with respect to the KL-divergence, and estimating the transition matrix with respect to a natural loss induced by KL or a more general f -divergence measure. For the first measure, we determine the min-max prediction risk to within a linear factor in the alphabet size, showing it is Ω(k log log n/n) and O(k 2 log log n/n). For the second, if the transition probabilities can be arbitrarily small, then only trivial uniform risk upper bounds can be derived. We therefore consider transition probabilities that are bounded away from zero, and resolve the problem for essentially all sufficiently smooth f -divergences, including KL-, L 2 -, Chi-squared, Hellinger, and Alpha-divergences.

Review:
Summary: The paper’s goal is to study the minimax rates for learning problems on Markovian data. The author(s) consider an interesting setting where the sequence of data observed follow a Markovian dependency pattern. They consider discrete state Markov chains with a state space [k] and study the minimax error rates for the following two tasks: Prediction: Given a trajectory X_1 -> X_2  … -> X_n from an unknown chain M, predict the probability distribution of the next state X_n+1, i.e., P(. | X_n). Measured by  Prediction risk  = E[ Loss(P(. | X_n) , P_hat(. | X_1…n)) ] where the expectation is over the trajectory X_1…X_n. The loss function the paper focuses on is KL-divergence and presents a conjecture for how the L_1 loss should scale with respect to k and n. Estimation:  Given a trajectory X_1 -> X_2 … -> X_n from the unknown chain M, estimate the transition matrix M (call it M_hat). Goal is do well under the following loss: Estimation risk = max_{i \in [k]} E[ Loss( M( i , . ) , M_hat( i , . )) ]         If the Markov chain is such that some states are observed infrequently, this affects the estimation problem but not so much the prediction problem as a trajectory ending in such a state will be rarely observed and we only want to do well on the average trajectory hence we can afford to do badly on such ‘rare’ states. Summary of results: The paper has upper and lower bounds for the minimax rates for the prediction task which are off by a factor of k. The authors conjecture that the right dependence is probably that of the upper bound. The paper gives tight results for the estimation problem for loss functions which are ordinary f-divergences and also has additional results for learning under l_2 loss.  Overall the paper is well-written and presents the results well. It uses some interesting tools giving bounds on surprise probabilities and concentration inequalities for number of visits to a state in a Markov chain. Adding some intuition as to why the stated minimax rates are achieved would be nice.  Comments: It would be nice to have a more comprehensive related work section (at least in the supplementary material). The asymptotic analysis of the minimax rates for the prediction problem seem to be ignoring an important term which decays as O_k(1/n) in the interest of the marginally dominating term O_k(loglog n/ n). Ignoring the 1/n decay term and quantifying the exact dependence on k with respect to the loglogn/n seems to not really be capturing the whole picture with respect to the dependence on k. For instance if the 1/n decaying term has a k^3 dependence for instance, then this term would dominate the sample complexity until n > 2^2^k which can be huge. Since the paper’s main contribution for this problem appears to be identifying the range of dependencies on k, it feels like identifying the dependence on k for the O_k(1/n) term is also important. The estimation problem as formulated by the authors seems to be somewhat harsh in the notion of the risk they use. A more natural risk function to study would seem to be  \sum_{i \in [k]} \pi_i * E[ Loss( M( i , . ) , M_hat( i , . )) ] as it gives less importance to states we rarely visit and hence are hard to estimate accurate transitions from anyway. Any justification for why the chosen risk function is the natural one? Moreover, the harsh risk function used intuitively makes the lower bound somewhat straightforward. The lower bound construction essentially hides differences in one node of the chain and ensures that it is the least frequently visited node then the total sample complexity . Also the upper bound doesn’t seem very surprising given that once we wait a mixing time period (the paper state ignore the first sqrt(n) samples but treats n as increasing with the mixing time and stationary probabilities remain fixed, hence this can be interpreted as waiting for a mixing time steps?) concentration results on the number of visits to a state kick in and the problem seems to reduce to solving k IID distribution estimation problems?  Experiments: For the prediction task, it feels the right experiment to perform would have been one which scales up k the number of states in the chain as the main contribution of the paper compared to previous work is identifying the dependence on k. Also for the chosen small values of n (between 250 and 1000) it is not clear if the theoretical results for the prediction task are really being verified since  a) the theory is developed in an asymptotic manner and 1000 is too small a sample size to see a difference of loglog n factors,  and  b) since the tailing term is O_k(1/n) which yields a similar scaling down behavior to that of O_k(loglog n/n) for the chosen k value and range of n.  A discussion of the proof intuitions would be nice to have in the main body. It was hard to get intuition from reading the proofs in the supplementary material.  Also some intuition on what might be the rates for ordinary f-divergences beyond KL for the prediction tasks would be nice to have. If the problem becomes much harder with other f-divergences, some intuition as to why that is the case might be helpful.  Overall, I feel that this paper explores an interesting direction but the results seem a bit incremental/incomplete (for the prediction task) and somewhat straightforward for the estimation task given the choice of the risk function (bypasses having to deal with the Markovian nature of the data)