Paper:
Learning Shuffle Ideals Under Restricted Distributions
The class of shuffle ideals is a fundamental sub-family of regular languages. The shuffle ideal generated by a string set U is the collection of all strings containing some string u ∈ U as a (not necessarily contiguous) subsequence. In spite of its apparent simplicity, the problem of learning a shuffle ideal from given data is known to be computationally intractable. In this paper, we study the PAC learnability of shuffle ideals and present positive results on this learning problem under element-wise independent and identical distributions and Markovian distributions in the statistical query model. A constrained generalization to learning shuffle ideals under product distributions is also provided. In the empirical direction, we propose a heuristic algorithm for learning shuffle ideals from given labeled strings under general unrestricted distributions. Experiments demonstrate the advantage for both efficiency and accuracy of our algorithm.

Review:
This paper studies the problem of learning shffle ideals in the PAC-learning model under some simple distributions such as product distribution, or distributions generated by Markov chains. A shuffle ideal is a particularly simple type of regular language. Given alphabet \Sigma, and a string u =(u_1, …, u_L) the prime shuffle ideal generated by u is the set of strings obtained by inserting any strings s_0, …, s_L in u to get s_0 u_1 s_1 … u_L s_L.  The shuffle ideal is a more general class of regular languages where instead of a string u we consider strings in a product set U = U_1 \times … \times U_L where U_i \subseteq \Sigma and take the union of the prime shuffle ideals generated by strings in U. The problem of learning regular languages has many applications. The theoretical study of this problem is in the PAC model. This however seems to be a hard problem, and so people try to study special cases and shuffle ideals are one such case. I think there’s a disconnect here between theory and practice as I don’t see any bridge between them mentioned in the paper under review or in [3]. However, I do find the problem being studied interesting in its own right; it seems natural and basic to me and thus could be of use in other contexts. The results are (seemingly) substantial generalization of previous work in that several of the results here are for learning shuffle ideals (as opposed to prime shuffle ideals)  and hold in situations where the distribution is generated by a Markov chain, or distribution over \Sigma^n where each position from 1 to n has its own distribution and one independently samples a symbol from \Sigma for each position. The algorithms work in the statistical query model and thus also in the PAC-learning model (for specific distributions). The paper also gives a simple greedy heuristic for learning prime shuffle ideals in the general case without proving any guarantees. Experimental results show it to be superior to the previous work. Thus the results seem to be interesting and non-trivial and represent progress. However, I have serious concerns regarding the technical parts as I now explain:[The following is based on the first 9 pages.]The writing is rather dry with little by way of proof outlines or intuitive explanation of the main ideas behind the proofs. The only explanation is after Theorem 3, which I didn't find very helpful. There should be a comparison with [3] in terms of new techniques and ideas. A cursory glance at [3] reveals high-level similarities in the notation and algorithms. So I suppose the present paper is building upon [3] and that makes it important to include a comparison with [3].The writing is not very clear; in most cases, where something is not clear I can guess what is meant but at some places I am not so sure. A few examples: The algorithm in Figure 3 seems to be missing some initialization. In particular, what’s V? What does it mean to approximately learn a shuffle ideal? I couldn’t find it being defined anywhere. Presumably it means that the symmetric difference between the learned concept and the actual concept is small? And when one is learning a prime shuffle ideal what does it mean? Is it that the Hamming distance or edit distance between the learned string and the actual string generating the ideal is small? But then that would require some quantification of approximation and the Theorems 1, 2, 3 do not have it.  What is \sigma in the statements of these theorems? Is it a parameter that one can choose or is it somehow related to approximation? Is L (as in Def. 2) a constant or logarithmic in n? I guess some small upper bound on L is needed to justify the use of conditioning like V \sqsubseteq x[1:n-1] in statistical query model. Also I’d expect L to show up in the statements of theorems, but it doesn’t.Mention that the problem is information-theoretically solvable by VC-dimension arguments (i.e. polynomial number of samples suffice though the algorithm may not be efficient). Thus I have to say that even the theorem statements are not completely clear to me. The paper would have been much better if the authors had devoted more space to one of the proofs (say Theorem 3), and given sufficient explanation to be convincing about its correctness. Other results could be stated in less detail to meet the page limit. This also makes me unable to comment on the correctness and novelty of the proofs, although I have no reason to have doubt.