From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper1167/Area_Chair_bpPV
metareview: This is a strong paper with interesting theoretical results and important practical contributions (much faster parameter learning than prior methods with little performance drop-off). All reviewers agreed it was above the bar for acceptance to NeurIPS.
recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_LyYF
title: Thank you for your response
comment: Thank you for your response. My concerns are addressed.

I encourage the authors to describe the similarities and differences in the proof technique as compare to the Rank Centrality paper. This will make the paper stronger, not weaker.

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_evfB
title: acknowledgment of response
comment: Thank you very much for the targeted and professional response. I think the discussion of modeling with MIRT is a great addition and will be welcomed by practitioners and the new Bayesian results are also a good addition since that is a highly used baseline for practitioners.  Thank yo ualso for the clarification to prior work above - I thin that paragraph would definitely help situate the paper.

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_L9zk
title: Read the author comments
comment: I'm happy with the authors' response - especially with the new results on the full Bayesian method. Although the AUC dropoff on a large dataset, ML-100K, is not minimal, the overall efficiency improvement is significant. If other reviewers are happy with the response I can up my score.

From: NeurIPS.cc/2022/Conference/Paper1167/Authors
title: Addressing some of your specific comments
comment: We hope that the additional experiments in our main reply above address your question on how our method would compare to a Bayesian method. The Bayesian algorithm that we compare the spectral method to is based on a recent and well-cited paper [2] so we believe that it is a strong baseline. The two methods are comparable in terms of accuracy. However, the Spectral method is orders of magnitude faster, and the difference is especially pronounced on large datasets with sparse responses like ML-100K.

> at least discuss what Bayesian estimation methods would do ... where you know the real item parameter prior 

MMLE is a semi-Bayesian method where one first assumes a prior distribution over the user parameters. The algorithm then maximizes the marginal likelihood in which the user parameters have been integrated out, to obtain the item parameters. Figure 1 in our main paper shows how such an algorithm might do when the prior is correctly specified vs incorrectly specified. There we see that the accuracy of the prior significantly impacts the overall accuracy of the algorithm. 

Your comments on the discussions of our experiment results are well appreciated. If accepted, we will certainly discuss the experiment results in greater depth as the final version allows more space. 

An important question that you posed and can be a really good discussion point is
> Why AUC, and log-likelihood are similar while top-K accuracy differs significantly between the methods

For smaller-scale education datasets (UCI, 3 Grades, LSAT) the number of items (tests) is small and there are no missing responses. There is much data relative to the number of parameters. As the algorithms, after all, operate on the same statistical model, they output very similar estimates and the performance is very close. The difference is more apparent in large-scale datasets and in top-K accuracy (which is not defined for the education datasets) where the responses are sparse and there is a large number of items.

To transform rating data to binary response data, for each user, we convert all ratings higher than the average to 0 and 1 otherwise (an item with a higher parameter value is better) like in [3]. The reference top-K set is the set of items with the highest average ratings. However, we have also removed items with very high but few ratings (<= 10) from this reference set as we treat these ratings as noisy. 

Note that the $K \in [10, 25, 50]$ are relatively small compared to the number of items in the large-scale datasets that we use. This is applicable in settings such as RecSys where we care about identifying the top few items. We observe that the other methods tend to "overvalue" items with noisy responses whereas our method, which uses pairwise differentials, is less susceptible to noisy ratings. This could explain why in some datasets, some of the baseline algorithms fail to identify any of the top items. Perhaps, this highlights the limitations of these classical IRT methods in applications like RecSys where there is much noisy data. 

On the other hand, AUC and log-likelihood are evaluated on a held-out dataset that contains a large number of items and users. The difference among the methods, when averaged over the users and items, is quite small. Therefore, the performance difference becomes less significant.

As for the results on the BX dataset, we would like to point out that the difference between the spectral method and the best method (MMLE) is relatively small. So the spectral method is not significantly worse than the competitors. 

> Some minor points about the presentation

These are all good points and we'll certainly address these in a final version.

[2] Natesan et al., Bayesian Prior Choice in IRT Estimation Using MCMC and Variational Bayes

[3] Lan et al., An Estimation and Analysis Framework for the Rasch Model (ICML  18')

From: NeurIPS.cc/2022/Conference/Paper1167/Authors
title: Addressing some of your specific comments
comment: We hope that our main reply above addresses both of your comments about potential extensions of the spectral method as well as joint parameter estimation. 

For joint parameter estimation, another approach is to run the spectral algorithm twice, one for the item parameter and the other one for the student parameter. Algorithm 1 estimates the item parameter $\hat\beta$ using the data matrix $X$. Using $X^T$ instead,  Algorithm 1 gives an estimation of the student parameter $\hat \theta$. However, since the spectral algorithm essentially learns the 'difference' in the parameters and outputs a normalized version of the user and item parameters, one needs to "align" the two sets of parameters. Fortunately, this alignment only comes in as a single scalar $a$. We estimate $\hat a$ by solving the following problem (keeping $\hat \beta$ and $\hat \theta$ fixed).

$\hat a = \arg\max_a \sum_{l\in [n], i\in[m]} X_{li} \log\frac{1}{1+\exp(-(\hat\theta_l + a -  \hat\beta_i))} + (1-X_{li})  \log\frac{1}{1+\exp(-(\hat\beta_i - \hat\theta_l - a))}$

This is a concave maximization problem in a single scalar variable and can be solved efficiently.

From: NeurIPS.cc/2022/Conference/Paper1167/Authors
title: Addressing some of your specific comments 
comment: We hope that our main reply above shows how the spectral algorithm can be extended to more expressive IRT models such as the mixed Rasch model.

As for your specific comments about the 2PL and 3PL model. The spectral algorithm is derived based on certain properties that are characteristic of the 1PL model. Therefore, we are unsure what an extension to the 2PL and 3PL model would look like. The idea of a factored Markov chain sounds interesting but we're not familiar with the topic so we can't say much here.

> Potential negative social impacts ...

This is a very good point. We will certainly include a more thoughtful discussion of this in the final version of the paper.

> Minor points on presentation ...

Thanks for pointing these out, especially our claims on item parameter estimation. Perhaps a better way of phrasing this would be "one-sided parameter estimation", whether it is estimating the students' abilities or the items' parameters in the context of recommendation systems.

> The authors cite several other spectral methods but do not make clear if any of the theoretical results come from those papers. Is Proposition 2.1 specific to the current paper or was it proved in one of the earlier works?

This proposition and its proof are new. This is because the pairwise transition probabilities of the Markov chain in our algorithm are different from those in Rank Centrality.

> For Theorem 3.3 (where m grows) the authors need to describe to this audience why JMLE fails in this case and what is different about their algorithm that makes it succeed.

We're actually not claiming that JMLE fails in this regime and this theorem only refers to the spectral method (the punchline is that when m grows, the spectral algorithm enjoys better, in fact, optimal error rate). However, it has been shown that JMLE fails when the number of items m is a constant whereas our algorithm is still consistent under that regime (Theorem 3.1).

> related works and how our algorithm differs

You raised very good points about the related work section and we will incorporate the suggested changes in a final version. We also hope to clarify the connection between our algorithm and Rank Centrality, the closest spectral algorithm in the literature. Superficially, they seem similar as both construct a Markov chain and estimate the parameters from the stationary distribution. However, the construction of the Markov chains, the pairwise transition probabilities are different. The resulting analysis is also different for our algorithm as we have a different sampling model from the Erdos-Reyni graph sampling model in the Rank Centrality paper. Furthermore, the BTL model where Rank Centrality is applied to has 1 set of parameters whereas the Rasch model has two. This difference plays out in Theorem 3.1 and Theorem 3.3 where we have two different guarantees for two regimes of m. Here the message is that the relative size of m and n affects the estimation error. Similarly, the theorems and propositions in our paper are new and not simple extensions of the results for Rank Centrality.

From: NeurIPS.cc/2022/Conference/Paper1167/Authors
title: Addressing some of your specific comments
comment: In addition to our main reply above, we hope to address some of your specific comments here.

> "Theorems 3.5 and 3.6 appear to be missing something ... "

This is a Cramer-Rao lower bound so the estimator $T$ is assumed to be an unbiased estimator. This estimator has to output the true parameter given infinite data for any $\beta^*$. If it always outputs $T(X) = 0$ then it's no longer an unbiased estimator. The expectation is over $X$.

> "existence and uniqueness of the estimator, i.e., of the stationary distribution of (3)"

This is a good point. The existence and uniqueness of the stationary distribution can be guaranteed by making sure that the Markov chain is ergodic. Roughly speaking, this means that from every state, one can land at another state (after some time) with a positive probability. This can be guaranteed using our regularization scheme which ensures that the pairwise transition probabilities are always positive and that the graph underlying the Markov chain is connected. If accepted, we will emphasize this point in the final version of the paper.

> "The similarities and differences to existing pairwise & spectral approaches"

This is a good suggestion as well. The main reason we don't discuss this in greater depth is because of the lack of space in the paper. If accepted, we should be able to discuss the connection to other pairwise methods in the final version which allows for more space.

> Typos and references
 
Thank you for pointing out these details. We'll surely address them in the final version.

> Why is the MMLE so bad in the Top-K case, despite achieving good AUC and log-likelihood? (I don't understand the caption of Table 1).

This seems to be a point of confusion for reviewer 4 (id: L9zk) as well. Please take a look at our reply to reviewer 4 where we clarify our setup and provide more explanations as to why the spectral method tends to outperform other methods in terms of top-K accuracy while AUC and log-likelihood tend to be similar among the methods.

> Can you comment to what extent the proofs of Thms 3.1 and 3.2. mirror the proofs in Rank Centrality paper for the BTL model?

It is reasonable to see a lot of similarities between our algorithm and Rank Centrality as both construct a Markov chain and computes its stationary distribution to recover parameter estimate. As a starting point, the analyses of the two algorithms use similar tools (such as Lemma A.3 in our paper) to bound the error of the stationary distribution of a perturbed Markov chain. However, the pairwise transition probabilities in the two algorithms are defined differently. Our sampling model is also different from the Erdos-Reyni sampling model for the comparison graph in the analysis of Rank Centrality. Furthermore, the Rasch model has two sets of parameters while the BTL model has one. Here, we see this difference plays out as we have two separate results for two regimes of m -- when m is a constant or small relatively to n vs when m also grows with n.

From: NeurIPS.cc/2022/Conference/Paper1167/Authors
title: Potential extensions of the spectral algorithm + additional experiments with a Bayesian method
comment: We thank the reviewers for all of their comments. We’re glad to see that the reviewers appreciate the novel application of the spectral algorithm to the Rasch model and the theoretical guarantees obtained in our work. We will address here the main comments from the reviewers, mostly pertaining to our experiments and extensions of the spectral method. We will also address more specific comments from each reviewer individually.

## Extensions of the spectral algorithm

As our work proposes a new approach to IRT based on spectral methods, extensions of the algorithm to more expressive IRT models are certainly promising research directions.

Reviewer 3 asked how the spectral algorithm can be extended to perform joint estimation of both user and item parameters. We can apply the spectral algorithm to obtain the item parameter $\hat\beta$. Each individual user parameter can be obtained by maximizing the log-likelihood function
$\hat \theta_l = \arg\max_{\theta} \sum_{i} X_{li}\log\frac{1}{1+\exp(-(\theta_l - \hat\beta_i))} + (1-X_{li})\log\frac{1}{1+\exp(-(\hat\beta_i-\theta_l))}.$
This is a concave optimization problem in $\theta_l$ and can be solved efficiently.

Reviewers 2 and 3 asked how the spectral algorithm can be extended to model expressive IRT models. As an example, we can extend the spectral algorithm to account for population heterogeneity under the mixed Rasch model [1]. The mixed Rasch model is similar to the Multivariate IRT model in that it aims to capture the idea pointed out by reviewer 2: a student might be good at one subject (arithmetic) while being bad at another (literature). In terms of modeling, each of these subjects corresponds to a mixture component. Both the student abilities and the test difficulties differ from one mixture component to another.

A mixture learning algorithm can proceed as follows:
* Perform spectral clustering on the rows of the response matrix $X$ (such as via sparse PCA) to produce $K$ clusters of rows.
* Run the spectral algorithm on these K clusters to obtain the K set of parameters.
* The value of $K$ can be chosen using cross-validation on a held-out validation set.

Given that the spectral algorithm runs significantly faster than other estimation algorithms, we can see its utility in learning mixtures of Rasch models where $K$ is large. The learned parameter estimates can be used to study the different subpopulations of students and how the test difficulties vary among these subpopulations.

[1] J Rost, C Carstensen, M Von Davier. Applying the Mixed Rasch Model to Personality Questionnaires (1997).

## Comparisons to a Bayesian method

We have performed some additional experiments with a 1PL Bayesian estimation method. Its implementation can be found at https://github.com/nd-ball/py-irt, based on a recent paper [2]. The Bayesian algorithm uses a hierarchical prior and variational inference for parameter estimation. Given that the paper [2] is well cited, we believe that this is a strong baseline. 

The message is similar to the experiment results in our main paper. The spectral algorithm is competitive in terms of accuracy. However, it is significantly more efficient than the Bayesian method. Furthermore, like other methods shown in our experiments, the Bayesian method doesn’t perform as well as the spectral method in terms of top-K ranking. The results imply that the spectral method will enjoy more practical advantages in applications where accurate top-K ranking is desired such as recommendation systems, approval voting systems, crowdsourcing, etc.

```
|  Dataset |   AUC (Bayesian)  |   AUC (Spectral)  |
|:--------:|:-----------------:|:-----------------:|
|   LSAT   |       0.706       |       0.707       |
| 3 Grades |       0.532       |       0.532       |
|    UCI   |       0.565       |       0.565       |
|  ML-100K |       0.681       |       0.662       |
|:--------:|:-----------------:|:-----------------:|
|          | LogLik (Bayesian) | Loglik (Spectral) |
|   LSAT   |       -0.487      |       -0.487      |
| 3 Grades |       -0.681      |       -0.687      |
|    UCI   |       -0.693      |       -0.706      |
|  ML-100K |       -0.635      |       -0.646      |
|:--------:|:-----------------:|:-----------------:|
|          |  Top-K (Bayesian) |  Top-K (Spectral) |
|  ML-100K |    0; 0; 0.04;    |   0.4; 0.6; 0.54  |
|:--------:|:-----------------:|:-----------------:|
|          |  Time (Bayesian)  |  Time (Spectral)  |
|   LSAT   |         63        |       0.028       |
| 3 Grades |         27        |       0.015       |
|    UCI   |         26        |       0.021       |
|  ML-100K |        2.8k       |         2         |
```

[2] Natesan et al., Bayesian Prior Choice in IRT Estimation Using MCMC and Variational Bayes

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_LyYF
rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper develops a spectral algorithm for estimating the item parameters in the Rasch item response model.

The authors show that the proposed estimator is consistent, and they provide finite-sample error bounds that are (near-)optimal.

Empirically, the authors show that their approach performs favorably in comparison to existing estimators: Predictive performance is on par or better, and the algorithms typically runs faster than competing approaches.
strengths_and_weaknesses: Strengths:

- The Rasch model is widely used, and the problem addressed by this paper is very relevant to the NeurIPS community.
- The theoretical contributions are comprehensive and address a number of important questions that remain mostly open for other estimators for the Rasch model: consistency, finite-sample error, optimality.
- In practice, the algorithm is simple to understand and to implement, and performs favorably. As such, the paper's contributions also relevant to practitioners.
- Beyond the specific results presented by the author, this paper also provides a bridge between the literature on spectral algorithms for the Bradley-Terry model and the Rasch model. I expect this paper to unlock further research on the topic.
- The experimental evaluation is excellent. The illustration of failure cases (Fig. 1) for various estimators is helpful in understanding the trade-offs that different methods have. The real-data experiments are thorough, and also include datasets and metrics where there is not necessarily a clear advantage for their proposal, giving a balanced picture of the empirical benefits.
- The paper is well-written and easy to follow. Throughout the paper, the authors strike a good balance between building intuition and formalizing their claims.

Altogether, the combination of these strengths make for a significant contribution that will have impact on ML researchers and practioners.

Weaknesses

- Theorems 3.5 and 3.6 appear to be missing something. Clearly setting $\beta^* = 0$ and $T(X) = 0$ achieves zero error (and satisfies the unbiasedness assumption). Perhaps $\beta^*$ needs to be bound, e.g., by taking the supremum over a certain range. Similarly, it is not clear what the expectation is over (I read it to be over $X$ but this could be made explicit).
- There is no discussion about the existence and uniqueness of the estimator, i.e., of the stationary distribution of (3). This needs to be formalized more rigorously. Under what conditions on `X` does the spectral estimate exist? How do Algorithms 1 & 2 handle cases where this condition is not satisfied?
- The similarities and differences to existing pairwise & spectral approaches for the Rasch model should be emphasized in the main text. Reading through Appendix D, it is apparent that the differences are major, but this was not clear in the main text (beyond the brief comment in the introduction about dense vs. sparse matrices). I strongly suggest making use of any additional space to move parts of Appendix D to the main text. Finally, statements like that of line 135, "our algorithm instantiates the general spectral approach [...]" are slightly misleading, given prior work.

These weaknesses are relatively minor and I believe they can be addressed in a camera-ready version.

Small comments:

- I am surprised that Agarwal et al. [2] is not cited in Section 4, "accelerating the spectral algorithm".
- Paragraph starting on line 50: an explicit expression for the CMLE would be helpful; it is hard to understand what the "likelihood conditioned on $\{ s_l \} $" looks like.
- l. 60: later -> latter
- l.116 and 204: practicioners -> practitioners
questions: - Why is the MMLE so bad in the Top-K case, depsite achieving good AUC and log-likelihood? (I don't understand the caption of Table 1).
- Can you comment to what extent the proofs of Thms 3.1 and 3.2. mirror the proofs in Rank Centrality paper for the BTL model?
limitations: Limitations are adequately addressed.
ethics_flag: No
soundness: 3 good
presentation: 4 excellent
contribution: 4 excellent

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_evfB
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: UPDATE: I increased my score based on the authors' response for noted in my counter-response.

The paper presents a novel representation of the statistical correlation between items in a 1PL (Rasch) IRT model that allows faster and (empirically) more accurate estimation of the underlying item difficulties (beta).  By embedding the items in a Markov Chain and calculating its stationary distribution, the authors are able to prove finite sample complexity bounds, which I believe is a first for such a method.  They provide extensions that speed up learning by using different normalization constants for each item and also add a regularization term to handle sparsity in real datasets.  Their empirical results show either parity or improvement against existing methods with an order-of-magnitude speedup against the most-accurate competitor.
strengths_and_weaknesses: The paper provides a very thorough analysis of the spectral approach in the Rasch model case.  I have some questions below about the differentiation to other spectral methods and some terms and definitions but overall those are minor and I think the analysis in the Rasch case is first rate.  The theorems look correct to my reading and exploring the connection between a MC's  stationary distribution and the overall item difficulty parameters is a clever choice.  I also liked the real-world practitioner extensions with variable normalization and regularization which make the algorithm truly applicable to the always-way-to-sparse item response data.

 However, I was disappointed that the authors did not mention or attempt to extend their approach to more modern IRT models including 3PL (where items have 3 axes of rating, not just difficulty) or multivariate IRT (where there are multiple thetas, for instance a student’s ability adding fractions and their ability to calculate percentages).  These more modern models are often calibrated using machine learning techniques (see below) and have far more intense computational burdens than Rasch so it is disappointing to not see at least a proposed extension to those cases.

More specifically, if we move to just 2-PL, how would the edges in the spectral embedding (the probability of transitions) represent both item parameters?  Could this be done using a factored Markov chain or some other more complicated graphical model?  Or is the spectral approach limited to only a single item-difficulty parameter? 

In the case of MIRT, we have seen recent successes of machine learning methods that use pairwise item comparisons to advance the very difficult problem of calibrating IRT models, see:
“Multidimensional Item Response Theory in the Style of Collaborative Filtering” (https://link.springer.com/article/10.1007/s11336-021-09788-9) as an example.  Since computational burdens are much larger with MIRT I would like to know if the computational advances in the Rasch have repercussions in that space?

Returning to the Rasch case and the results in the paper, I found the spectral embedding clever and a unique statistical approach for solving this problem.  I very much appreciate the finite-sample complexity bounds in the paper.  There are, however, are several places where I would like to see clarifications:

The authors cite several other spectral methods but do not make clear if any of the theoretical results come from those papers. Is Proposition 2.1 specific to the current paper or was it proved in one of the earlier works?

For Theorem 3.3 (where m grows) the authors need to describe to this audience why JMLE fails in this case and what is different about their algorithm that makes it succeed.

Table 1 needs to be reformatted to fit the margins.  Also, the authors should make a bigger deal about the speedup compared to MMLE.  It’s not just faster, the new method is an order of magnitude quicker.

I found Section 6 unhelpful and it seemed to be trying to justify why this paper fits at NeurIps by describing tangentially related problems.  I know there are connections to general logistic regression problems here but this section would be better used describing more complex IRT models like MIRT and how the approach could be used there.  In terms of fit at Neurips, I think the paper is ok as a statistical learning theory approach that models a psychological phenomenon although ultimately this result belongs in a Psychometrics journal to be really picked up by practitioners.
 
Minor points:

Lines 29 and 30: citations needed for these claims

Line 31: “Traditionally, the goal of estimation under the Rasch model is to recover the item parameters” … No, the goal is to assess the mastery parameters theta – at the end of the day we want to know whether students have certain skills or not.  Calculating beta values are a means to that end but they are not the goal.

questions: From the review, how does the new approach expand our capabilities in the 2PL/3PL or MIRT case?  

The authors cite several other spectral methods but do not make clear if any of the theoretical results come from those papers. Is Proposition 2.1 specific to the current paper or was it proved in one of the earlier works?

For Theorem 3.3 (where m grows) the authors need to describe to this audience why JMLE fails in this case and what is different about their algorithm that makes it succeed.

In accordance with Neurips guidelines on AI ethics, the authors should provide in their response a more detailed description of potential biases in this method compared to existing IRT approaches and clearly spell out possible negative societal impacts.
limitations: Beyond the missing extensions above I do think a discussion of ethics is necessary for this paper to be published, given Neurips's dedication to the topic.  I understand the work here is purely theoretical so I didn't flag it for ethics review below, but the application domain for this IRT method is evaluating children.  So any systematic errors or biases in the new approach need to be spelled out much more clearly.  The authors should provide in their response a more detailed description of potential biases in this method compared to existing IRT approaches and clearly spell out possible negative societal impacts.
ethics_flag: No
soundness: 4 excellent
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_d3YA
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: The paper proposes an inference algorithm based on a spectral estimation approach for estimation of the item parameter in the IRT model with finite-sample guarantees. The authors provide detailed convergence analyses and demonstrate the utility of the proposed estimator via both simulated and real-world datasets, on which the proposed approach achieves performance comparable to other estimators (e.g., MLE) at a fraction of the computational cost.
strengths_and_weaknesses: The paper makes interesting theoretical contributions to the inference algorithm for IRT model. The algorithm is simple to implement and has finite-sample guarantees. The analyses are sound and the experiments corroborate with the analytical results. The proposed method is also competitive with other inference algorithms. I do not find major weaknesses; some suggestions are outlined in the "limitations" part.


questions: Will and how does the algorithm generalize to settings in which student/user parameters need be estimated instead of sampled from a prior?
limitations: It seems that the proposed approach is applicable to only 1PL IRT model (Rasch model). Although still widely used, there are other, potentially more power, variants of the IRT models (e.g., 2 or 3 PL IRT models) and it is not quite clear whether or how the proposed algorithm generalizes to other IRT variants. This is a potential limitation, although not a main concern.

The proposed approach relies on prior distribution on the student parameter, i.e., the student parameters are not estimated. For applications where only item parameter estimations are needed, the proposed approach might be sufficient. In other, also common, settings where the student parameters also need be estimated, it is unclear how the proposed approach would be applicable.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper1167/Reviewer_L9zk
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper proposes a spectral estimation algorithm for item parameters in the 1PL item response (aka Rasch) model. The authors conducted theoretical analyses of the proposed algorithm in several scaling regimes, proposes a variant of their algorithm under practical constraints, and conducted experiments on real data to show that their method is on-par with existing algorithms in terms of estimation accuracy while being more computationally efficient. 
strengths_and_weaknesses: Strengths: 
- Although spectral estimation has been thoroughly studied, its application to the Rasch model is novel
- I liked the fact that the authors provided a version of their algorithm in practice where one can use a different normalization factor d for every item. This is important and likely helps a lot in terms of Markov chain convergence - a uniform d can be really bad for some items on real world datasets
- I have not checked the proofs but the theoretical results and discussions seem sound

Weaknesses:
- ML is only one family of estimation methods but there are others, say posterior mean etc. I get that the proposed methods may be better than M/J/CMLE, which have their own problems for sure, but how would its accuracy/runtime compares to Bayesian methods is not clear. See below. 
- The experimental settings and results are not clear. See below. 
- The presentation could be improved in several places. See below. 

questions: 1. Main concern: experiments. The experimental settings and results (on real data) are really underdiscussed, even in the supplementary material. The overall takeaway is a solid one - the proposed spectral algorithm is on-par with existing algorithms in terms of accuracy and is much faster, which is perfectly fine. However, some details give me a sense that some baselines are not properly treated. Some things that the authors can do to help are:
- at least discuss what Bayesian estimation methods would do; more accurate when the prior is right? Too inefficient on large real world datasets? A simulation study where you know the real item parameter prior or experiments on one real world dataset would certainly help. 
- many datasets are not discussed (LSAT, UCI, 3GRADES) where the number of items is very very small. Here, AUC is the same for all methods. The experimental results are not thoroughly discussed; I'm a little unsure about the huge variation in terms of ranking performance since after all, these are different algorithms for the same underlying statistical model. The 0.0 results for top-K accuracy for some methods are a bit suspicious; it's not totally surprising, but more discussion is definitely needed. Otherwise, it's just really hard to properly evaluate how beneficial the proposed algorithm is in practice. 
- AUC scores for most methods are pretty close on most datasets, which makes perfect sense. However, the proposed method is obviously worse on BX; why?

2. Some minor aspects of the presentation could be improved:
- in the abstract, it's probably more accurate to call the proposed algorithm "item parameter estimation" instead of "item estimation"
- line 32 "educational testing"
- line 37JMLE does not really have to use an alternating maximization algorithm, right? The problem is jointly convex in the item and ability parameters
- I would like to see Theorem 3.1 and 3.3 written with ||beta - beta*||_2/sqrt(m) on the left hand side - this quantity makes more sense in practice
- Need to zoom into the small n regions in Figure 1. Lines are overalpping at times and it's hard to tell which is which. 
limitations: This field is not really applicable to this paper
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good