Paper:
Completing State Representations using Spectral Learning
A central problem in dynamical system modeling is state discovery-that is, finding a compact summary of the past that captures the information needed to predict the future. Predictive State Representations (PSRs) enable clever spectral methods for state discovery; however, while consistent in the limit of infinite data, these methods often suffer from poor performance in the low data regime. In this paper we develop a novel algorithm for incorporating domain knowledge, in the form of an imperfect state representation, as side information to speed spectral learning for PSRs. We prove theoretical results characterizing the relevance of a user-provided state representation, and design spectral algorithms that can take advantage of a relevant representation. Our algorithm utilizes principal angles to extract the relevant components of the representation, and is robust to misspecification. Empirical evaluation on synthetic HMMs, an aircraft identification domain, and a gene splice dataset shows that, even with weak domain knowledge, the algorithm can significantly outperform standard PSR learning.

Review:
SUMMARY:  This paper proposes a method to incorporate prior knowledge into the spectral learning algorithm for predictive state representations (PSR). The prior knowledge consists of an imperfect/incomplete state representation which is 'refined' and 'completed' by the learning algorithm. This contribution addresses one of the main caveats of spectral methods: while these methods are fast and consistent, they tend to perform worse than local methods (e.g. EM) in the low data regime. By leveraging domain specific knowledge, the proposed algorithm overcomes this issue.  The proposed extension, PSR-f, is relatively straightforward: the belief vector at each time step is the concatenation of the user-specified state representation f with a learned state representation b; the parameters of b are learned in the same fashion as for the classical method by solving linear regression problems constrained to the row space of the concatenation of some Hankel/system matrices (e.g. now mapping [P(T | h) ; P(h)f(h)] to P(oT | h) for each B_o). The authors then give sufficient conditions for the algorithm to be consistent and formally analyze to which extent providing prior information to the algorithm can be beneficial or detrimental; indeed, the number of states of a minimal PSR-f can be greater than the one of the corresponding minimal PSR when f is 'irrelevant'! To circumvent this issue, the authors proposed a method to extract the relevant information from f before performing the spectral learning algorithm by analyzing the principal angles between the classical Hankel matrix and some feature matrix induced by the user-specified state representation f.  The proposed method is evaluated on both synthetic and real data experiments where PSR-f is compared with the basic PSR spectral learning algorithm (i.e. not using the prior knowledge of f) and with only using the user-specified state representation f.   OPINION:  I liked the paper, it is well written and relatively easy to follow for someone familiar with the spectral learning literature. As mentioned above, I think that the problem addressed in the paper is relevant and the solution proposed by the author is natural and sound (I did not check the maths thoroughly but the results stated in the Theorems seems accurate and expected), and that this contribution is a nice addition to the spectral learning literature. The paper could maybe benefit from a more high-level / abstract exposition of the technical material, e.g. reasoning also in terms of subspaces rather than only with the Hankel sub-blocks in Defs 2 to 5 (see my comment below), but this may be a matter of personal taste.     REMARKS/QUESTIONS:  - I find it strange to introduce the \beta_o as parameters, is it standard notation? I am more familiar with having a normalizer  beta_\infty as a parameter of the model instead and defining the update equation with \beta_\infty B_o b(h) at the denominator (see e.g. [4]). How are the two related?  - Is the requirement that the core tests and histories are minimal necessary for the consistency results?   - In definitions 2 and 3, I believe that H (and even T) can be replaced directly with O^* instead of having a universal quantifier and a sup respectively. Then, in my opinion, defining rank(f;M) as the dimension of the intersection of the row spaces of P_{O^*,O^*} and P_{f,O^*} is simpler and more intuitive (this is actually what is indirectly done in Prop. 7 in the supplementary). Indeed, this intersection is the smallest subspace of P_{f,O^*} that is contained in P_{O^*,O^*}, i.e. that is linearly relevant to M.  Then the matrix U_f in definition 4 can be understood as some projection onto this intersection (i.e. the rows of U_f^T U_f P_{f,H} belongs both to the row spaces of P_{O^*,H} and P_{f,H}).  Finally, Theorem 2 simply follows from the formulas for the dimension of the complement of P_{O^*,H} \cap P_{f,H} in P_{O^*,H} (for |T|), and for the dimension of the sum P_{O^*,H} + P_{f,H} (for |H|). Now, similarly to classical PSR, the minimal sizes for T and H are equal to rank of the concatenation of P_{O^*,H} and P_{f,H}, which can be as low as rank(M) but as high as rank(M) + n (when the intersection of  P_{O^*,H} and P_{f,H} is {0}).  - I find the phrasing "solving by matrix inverse" a bit odd.    MINOR COMMENTS/TYPOS:  - line 52: since there are no restriction on Z line 48, setting Z = R^n and assuming that P(o|h) is linear in the belief is actually a restriction rather than a generalization.  - line 76: R -> R^m - line 113-114: use (resp. H) ... (reps. T). - line 136: spurious ) after H - line 158: paramter -> parameter