Paper:
A Representation Theory for Ranking Functions
This paper presents a representation theory for permutation-valued functions, which in their general form can also be called listwise ranking functions. Pointwise ranking functions assign a score to each object independently, without taking into account the other objects under consideration; whereas listwise loss functions evaluate the set of scores assigned to all objects as a whole. In many supervised learning to rank tasks, it might be of interest to use listwise ranking functions instead; in particular, the Bayes Optimal ranking functions might themselves be listwise, especially if the loss function is listwise. A key caveat to using listwise ranking functions has been the lack of an appropriate representation theory for such functions. We show that a natural symmetricity assumption that we call exchangeability allows us to explicitly characterize the set of such exchangeable listwise ranking functions. Our analysis draws from the theories of tensor analysis, functional analysis and De Finetti theorems. We also present experiments using a novel reranking method motivated by our representation theory.

Review:
SUMMARYThe paper derives a representation theorem for the class of "exchangeable" listwise scoring functions employed in learning to rank problems. The results show that for finite instance spaces, such functions are expressible as a particular combination of similarity functions over pairs of instances. For compact spaces, a similar result holds for the case of lists of size two; for general lists, the result holds under additional assumptions by appeal to deFinetti's representation theorem.SIGNIFICANCEThe issue of designing listwise loss functions has been well studied, but the design of listwise scoring functions less so. As noted in the paper, some works focus (e.g. [4]) involve scoring each instance within the list with a single function, albeit one that is optimised in a listwise sense. The results in the paper are interesting, and while largely intuitive  not a-priori obvious. The stated goal is to give a recipe with which to guide the design of listwise ranking functions. I think the observation that this can be done by designing similarity functions on pairs of instances, and suitably combine them, is a good first step towards this goal.All results rely on a symmetry assumption on the listwise ranking function. This seems reasonable, and minimally holds e.g. for the pointwise case. The results for the compact case in 3.2.2 additionally rely on the scoring function being an unnormalised probability density over the instances. It is less clear to me how reasonable this is. The requirement of nonnegativity of the scoring function could be handled by an appropriate link function (e.g. exp(.) as done in the experiments). But this would require suitable interplay with the surrogate loss being optimised to ensure convexity, something that may deserve some comment.The experiments show that it is possible to post-process the outputs of a pointwise ranker (even one trained to optimise a listwise loss) using the representation theorem. The experiments are the weakest part of the paper, and likely more could be done to make a convincing case for the value of the recipe (e.g. try a wider class of pairwise functions, have a comparison of pairwise "as-is" versus pointwise to check the perils of overfitting, et cetera). But good performance in this regime is still interesting, as (to my knowledge) improving upon methods trained to optimise listwise losses is non-trivial. So I think they are indicative of the promise of the representation theorem in guiding the design of scoring functions.QUESTIONS/COMMENTS- The statement of Theorem C.1 apparently holds for m > 2 instances in the list. This seems to not agree with the text; perhaps the point is that existence of a multilinear symmetric decomposition would imply the result? If so the appeal to Theorem 3.8 in the proof seems misleading. It does seem like you could have such a corollary to 3.11, so perhaps that should be stated.- Even if a simple consequence of 3.10, I think a proof of Theorem 3.11 should be included in the appendix. The statement of the theorem alone leaves some questions:	* There is an implicit "for every m" needed? In which case, f should actually be defined for every possible m in X^m?		* In 3.10 the "g" function is (I presume) the marginal, derived appropriately from the joint p. Does a similar restriction exist here in terms of the function g?	* The requirement of an infinite sequence in deFinetti must of course carry over to the theorem, but it is a little hard to understand what exactly it is requiring of the ranking function and/or the input space. The comment about the result requiring "infinitely many objects for which every leading finite subsequence is exchangeable" could be expanded. Isn't this implicit when one assumes that f is symmetric in its arguments, and hence exchangeable regardless of the provided inputs (which could be one of infinitely many possible values, since X is infinite)?	* What does it mean for theta to be a "random distribution" in the prelude to the theorem?- The extension of 3.11 to the partially symmetric case seems non-obvious, and minimally requires a careful statement of the infinite sequence assumption, and the appropriate boundedness of the integral of f. I would recommend making this explicit.OTHER COMMENTS- I like that the paper builds up from the simple case of finite instance spaces to a more general theory.- Line 211, for three documents f_i : X^3 \to R, so f_i is a tensor? Or do you group the last two elements into one?- Remark II pg 5, tensor completion seems reasonable for finite spaces and suitably large training sets, but in practice it seems likely that one would face cold-start problems (no observations for a particular "row") due to limited coverage of the instance space.- Line 268, is the corresponding decomposition theorem referred to Theorem C.1?- It might be better to include Theorem C.1 in the body and defer 3.8 to the appendix if space is an issue.- Appendix F is unclear. What are r, s? How exactly is the surrogate loss used for optimisation?- The paper is generally well written. There were some small typos worth addressing:	* Def 2.1, \to instead of \mapsto in definition of f_j.	* Missing period line 169.	* Period instead of comma line 186.	* Theorem 3.8 proof, missing boldface in x, y.	* Theorem 3.11, we need f : X^m \to R_+?	* Fix capitalisation in ref [3].- Other comments:	* Why the use of braces in { x_{\i} }?	* Include summation indices in Prop 3.5, 3.6, Thm 3.8?	* Def 3.4, use \pi rather than \sigma for consistency?	* Theorem C.1 proof should come after 3.8.	* Notation clash in (14), S for training set and S_k for similarity function?