A Spectral Approach to Item Response Theory
Abstract
The Rasch model is one of the most fundamental models in item response theory and has wide-ranging applications from education testing to recommendation systems.
In a universe with n users and m items, the Rasch model assumes that the binary response X li ∈ { 0 , 1 } of a user l with parameter θ ∗ l to an item i with parameter β ∗ i (e.g., a user likes a movie, a student correctly solves a problem) is distributed as P ( X li = 1)
= 1 / (1 + exp ( − ( θ ∗ l − β ∗ i ))) .
In this paper, we propose a new item estimation algorithm for this celebrated model (i.e., to estimate β ∗ ).
The core of our algorithm is the computation of the stationary distribution of a
Markov chain deﬁned on an item-item graph.
We complement our algorithmic contributions with ﬁnite-sample error guarantees, the ﬁrst of their kind in the literature, showing that our algorithm is consistent and enjoys favorable optimality properties.
We discuss practical modiﬁcations to accelerate and robustify the algorithm that practitioners can adopt.
Experiments on synthetic and real-life datasets, ranging from small education testing datasets to large recommendation systems datasets show that our algorithm is scalable, accurate, and competitive with the most commonly used methods in the literature.
Introduction
Item response theory (IRT) is the study of the relationship between latent characteristics (a student’s ability versus a test’s difﬁculty or a user’s taste versus a movie’s features) and the manifestations of these characteristics (a student’s performance on a test or a user’s rating of a movie).
Originally developed by the psychometric community [45, 51], item response theory has been applied to diverse settings such as education testing [37], crowdsourcing [54], recommendation systems [12], ﬁnance [50] and marketing research [10].
One of the most fundamental models in IRT is the Rasch model [45].
It models the binary response
X li ∈ { 0 , 1 } of user l with latent parameter θ ∗ l ∈ R to item i with latent parameter β ∗ i ∈ R by
[Equation]
For example, in education testing, θ ∗ l corresponds to the ability of student l , β ∗ i the difﬁculty of problem i and X li = 1 if the student correctly solves the problem.
Binary response data has grown abundantly in modern domains: Netﬂix famously switched from a 5-star rating system to a binary like/dislike feedback system, data on students’ engagement and performance grows signiﬁcantly as education moves online during the pandemic.
Traditionally, the goal of estimation under the Rasch model is to recover the item parameters β ∗ .
In education testing, an estimate of the item parameters can be used to calibrate scores across different 36th Conference on Neural Information Processing Systems (NeurIPS 2022). versions of a test.
In recommendation systems, the item parameters can be used to produce a ranking over the items.
In general, estimation is challenging under the Rasch model because for each user and item pair, we only get a single observation or none in the case of missing data.
Joint maximum likelihood estimate (JMLE) is one of the earliest methods developed for the estimation problem [3, 22, 25, 27].
It estimates both the user and item parameters by maximizing the joint likelihood function using an alternating maximization algorithm.
While efﬁcient, JMLE is known to be inconsistent (that is, even as n → ∞ , JMLE does not recover β ∗ ) when the number of items is ﬁnite [3, 24] (e.g., Figure 1a).
Intuitively, this is because there are many nuisance user parameters to a ﬁnite number of item parameters.
As a result, JMLE is mostly used for prelimary parameter estimation and researchers have developed other solutions to address the inconsistency problem, broadly consisting of 3 approaches as follows.
The ﬁrst approach is marginal maximum likelihood estimate (MMLE) [7].
The statistician ﬁrst speciﬁes a prior distribution over the user parameters.
The objective of MMLE is to maximize the marignal likelihood function which integrates out the user parameters.
In pratice, MMLE runs quite fast, handles missing data well and is reasonably accurate.
However, its performance depends on the accuracy of the prior distribution .
If misspeciﬁed, MMLE may produce inaccurate estimates (e.g.,
Figure 1b).
Model selection is thus a crucial procedure when applying MMLE to real data.
The second approach is conditional maximum likelihood estimate (CMLE)
CMLE builds on the fact that under the Rasch model the total number of positive responses s l for each user l is a sufﬁcient statistic for the user parameter θ ∗ l .
Instead of the joint likelihood function, CMLE maximizes the likelihood conditioned on { s l } nl =1 .
Unlike JMLE, CMLE is statistically consistent without requiring any distribution assumptions about θ ∗ .
For small datasets with no missing data,
CMLE is quite accurate.
However, it may incur high computational cost and numerical issues on large datasets with many items and missing entries.
Practioners have observed that CMLE often produces inaccurate estimates [35, 36] in this regime (e.g., Figure 1c).
The third approach, which our algorithm follows, uses pairwise information that can be extracted from binary responses.
Intuitively, if a user responds to two items, one negatively and one positively, we learn that the later is ‘better’.
Following this intuition, previous authors [23, 17, 48] have designed spectral algorithms that ﬁrst construct an item-item matrix and then compute its leading eigenvector.
One common limitation of these methods is that the item-item matrix is assumed to be dense .
Therefore, these methods aren’t directly extendable to large scale datasets in applications such as recommendation systems where the item-item observation is sparse.
Furthermore, most theoretical guarantees for the above methods are asymptotic ( n → ∞ ) .
However, having ﬁnite sample error guarantees is useful in real-life applications.
For example, when we only observe a handful of responses to a new item, it is important to have an accurate estimate of the error over the item parameter.
Asymptotic guarantees, on the other hand, are accurate mostly under a large sample size regime, and can be inaccurate in the data-poor regime.
z Our Contributions: Motivated by known limitations of the existing methods, we propose a new, theoretically grounded algorithm that addresses these limitations and performs competitively with the most commonly used methods in the literature.
More speciﬁcally:
• In Sections 2 and 4, we describe the spectral algorithm and practical modiﬁcations – an accelerated version of the original algorithm and a regularization strategy – that allow the algorithm to scale to large real-life datasets with sparse observation patterns and alleviate numerical issues.
• In Section 3, we present non-asymptotic error guarantees for the spectral method – the ﬁrst of their kind in the literature – in Theorems 3.1 and 3.3.
Notably, under the regime where m grows, the spectral algorithm has optimal (up to a constant factor) estimation error achievable by any unbiased estimator (Theorem 3.4).
Under the challenging regime where m is a constant or grows very slowly we show that the spectral algorithm is, unlike JMLE, consistent (Corollary 3.2).
• In Section 5, we present experiment results on a wide range of datasets, both synthetic and real, to show that our spectral algorithm is competitive with the most commonly used methods in the literature, works off-the-shelf with minimal tuning and is scalable on large datasets.
1.1 Notations and Problem Formulation
As brieﬂy described before, in a universe of n users and m items, each user l has a latent parameter θ ∗ l ∈ R and each item i has latent parameter β ∗ i ∈ R .
The reader may recognize that there is a fundamental identiﬁability issue associated with the Rasch model pertaining to translation.
That is, { θ ∗ , β ∗ } and { θ ∗ + α 1 n , β ∗ + α 1 m } describe the same model for any α ∈ R .
For this reason, we impose a normalization constraint on the item parameters β ∗(cid:62) 1 m = 0 .
We consider the ﬁxed range setting where β ∗ i ∈
[ β ∗ min , β ∗ max ] ∀ i ∈ [ m ] for some constants β ∗ min , β ∗ max .
Similarly, we assume that θ ∗ l ∈ [ θ ∗ min , θ ∗ max ] for some constants θ ∗ min , θ ∗ max 1 .
The observed data is X ∈ { 0 , 1 , ∗} n × m where ∗ denotes missing data and for entries where X li (cid:54) = ∗ , X li is independently distributed per Equation (1).
Let A ∈ { 0 , 1 } n × m denote the assignment matrix where A li = 1 if user l responds (either negatively or positively) to item i and 0 if user l does not respond to item i (i.e., X li = ∗ ).
Deﬁne B =
A (cid:62) A , i.e., B ij is the number of users who respond to both items i, j .
The goal of item estimation is to obtain an estimate β from the observed data X and the metric of interest is the (cid:96) 2 error, (cid:107) β − β ∗ (cid:107) 2 .
The Spectral Estimator
In this section we describe our spectral algorithm which is summarized in Algorithm 1.
At a high level, the algorithm constructs a Markov chain deﬁned on a graph whose vertices are the items and its transition probabilities are estimated using the observed user-item response data.
The algorithm then computes the stationary distribution of this Markov chain and the estimate β is obtained following a simple transformation.
We ﬁrst deﬁne, for each item pair i, j and a ﬁxed assignment A , a quantity which we term pairwise differential measurement :
[Equation]
Intuitively, Y ij is the number of users who respond 1 to i and 0 to j .
Given the pairwise differential measurements, consider a Markov chain P ∈ [0 , 1] m × m whose transition probabilities are deﬁned as follows:
[Equation]
where d is a sufﬁciently large normalization factor chosen such that the resulting pairwise transition probability matrix does not contain any negative entries.
Typically, d = O (max i ∈ [ m ] (cid:80) k
(cid:54) =
i B ik ) .
The algorithm then computes the stationary distribution π of the Markov chain (e.g., using power iteration) and recover β using a post-processing step.
In real-life datasets, the constructed Markov chain is often sparse (not every pair of items has non-zero pairwise differential measurements).
Practicioners could take advantage of this sparsity to speed up the computation of the stationary distribution such as by using sparse matrix-vector multiplication subroutines.
To understand the intuition behind our spectral algorithm, let us consider the following idealized
Markov chain where the state transition probabilities are exact:
[Equation]
where Y ∗ ij = (cid:80) nl =1 A li A lj E
[ X li (1 − X lj )] .
For every pair i, j , given a sufﬁciently large number of users who respond to both items, Y ij will concentrate around Y ∗ ij .
Then, under an appropriately large scaling factor d , P ij ≈ P ∗ ij and the two Markov chains are ‘close’.
This means that the stationary distribution of P is also close to that of P ∗ .
At the same time, the true item parameter β ∗ is directly related to the stationary distribution of P ∗ .
This relation is summarized by Proposition 2.1.
Proposition 2.1.
Consider the idealized Markov chain described in Equation (4).
The stationary distribution π ∗ of P ∗ satisﬁes π ∗ i = e β ∗ i / ( (cid:80) mk =1 e β ∗ k ) for i ∈ [ m ] .
Essentially Proposition 2.1 states that π ∗ is proportional to e β ∗ .
Thus β ∗ can be recovered from π ∗ up to a global normalization.
Now, given a sufﬁciently large number of users, the empirical stationary distribution π will be close to π ∗ and naturally the obtained estimate β is also close to β ∗ .
Readers who are familiar with the ranking from pairwise comparison literature might recognize the similarity between the spectral algorithm and Rank Centrality [43] for parameter estimation under the Bradley-Terry-Luce model [38].
Similarly to Rank Centrality, our algorithm constructs a Markov chain on the item-item graph and recovers parameter estimate from its stationary distirbution.
In both cases, the Markov chain interpretation is motivated by the unique characteristics of the BTL and Rasch likelihood function. However, the speciﬁc construction differs between our algorithm and Rank Centrality and so does the resulting analysis.
Theoretical Analysis
In this section, we present the main theoretical contributions of the paper.
Speciﬁcally, we obtain in
Section 3.1 two ﬁnite sample error bounds for two different regimes of m : where m is a constant or grows very slowly and where m grows at least logarithmically relative to n .
In addition to our upper bounds, we show in Section 3.2 a Cramer-Rao lower bound for the mean squared error of any unbiased estimator, establishing the optimality of the spectral algorithm under the the second regime.
For the special case m = 2 , we show that the error rate obtained by the spectral algorithm is optimal up to a log factor.
3.1 Finite Sample Error Guarantees
Sampling Model: Let us consider a random sampling model where for each user l ∈ [ n ] , each item i ∈ [ m ] is independently shown to that user with probability p (i.e., P ( A li = 1) = p ).
Once shown an item l , the user i responds with X li distributed according to Equation (1).
Under this sampling model and the regime where m is a constant or grows very slowly , we obtain the following upper bound on the estimation error of the spectral algorithm which is, to the best of our knowledge, the ﬁrst ﬁnite sample error guarantee for any consistent estimator under the Rasch model in the literature.
Theorem 3.1.
Consider the sampling model described above.
Suppose that np 2 ≥ C (cid:48) log m for a sufﬁciently large constant C (cid:48)
then the output of the spectral algorithm statisﬁes
[Equation]
with probability at least 1 − min { e − 12 m ,
[Equation]
where C, C 1 are constants
[Equation]
As alluded to before in our algorithm description, the proof of Theorem 3.1 uses Markov chain analysis and a central object is the idealized Markov chain P ∗ with its stationary distribution π ∗ .
The proof is rather long and involved so we defer the details to the supplementary materials and describe here the main idea.
The starting point is a Markov chain eigen-perturbation bound (see Lemma A.3):
[Equation]
where µ ∗ ( P ∗ ) is the spectral gap of the idealized Markov chain.
We then bound the numerator and the denominator separately.
We will show under the setting of Theorem 3.1 that
[Equation]
Combining these bounds with the following relation gives us the desired error bound:
[Equation]
As an immediate consequence of Theorem 3.1, we can also prove the consistency of the spectral algorithm under the constant m regime.
As mentioned previously, JMLE, one of the most well known methods in the Rasch modeling literature, is inconsistent in this regime.
Corollary 3.2.
Consider the setting of Theorem 3.1.
For a ﬁxed m and p = 1 , the spectral algorithm is a consistent estimator of β ∗ .
That is, its output β satisﬁes lim n →∞ [Equation] [Equation] β
[Equation] β ∗ [Equation] 2 < (cid:15) )
= 1 , ∀ (cid:15) > 0 .
Under the regime where m grows , we could sharpen the results of Theorem 3.1.
Speciﬁcally, when the number of items shown to each user is sufﬁciently large, we improve by a √ p factor which can be signiﬁcant when p is small.
This is summarized by the following theorem.
Theorem 3.3.
Consider the setting of Theorem 3.1.
Assume further that mp ≥ C
(cid:48)(cid:48) log n for a sufﬁciently large constant C (cid:48)(cid:48)
then the output of the spectral algorithm statisﬁes
[Equation]
with probability at least 1 − exp (cid:0) − C 2 np 2 (cid:1) − 2 n − 9 , where C ∗ , C 2 are constants.
The reader may wonder why there would be a difference between the two regimes.
Intuitively, when m is a small constant, the distribution of the number of items shown to the users are not tightly concentrated.
Some users are shown all of the items while some are shown only one.
By design, our spectral algorithm uses pairwise differential measurements.
This means that when a user responds to only one item, that information is not fully used.
On the other hand, when mp =
O (log n ) , the number of items shown to the users is concentrated (all users are shown approximately the same number of items) and more pairwise differential measurements are available.
There is less information being under-utilized by the algorithm and it enjoys a tighter (in fact optimal) error rate.
3.2 Cramer-Rao Lower Bound
In this section, we present complementary results to our ﬁnite error guarantees obtained in the previous section.
Notably, under the regime where m is allowed to grow with n , we show that the minimum mean squared error achievable by any unbiased estimator is no more than a constant factor smaller than the upper bound for the spectral algorithm established in Theorem 3.3.
This optimality result is summarized by the following theorem.
Theorem 3.4.
Consider the sampling model described in in Section 3.1.
Let T be any unbiased estimator for the item parameters.
Then the mean squared error of such estimator is lower bounded as
[Equation]
where T ( X ) is the output of the estimator T when given data X and c is a constant.
Now note that under the settings of Theorem 3.3, the output of the spectral algorithm also statisﬁes (cid:107) β − β ∗ (cid:107)
= O ( mnp ) .
To the best of our knowledge, this is the ﬁrst non-asymptotic optimality result for any item estimation method under the Rasch model.
As noted before, when the number of items is constant, our error bound in Theorem 3.1 incurs an additional 1 / √ p factor.
We now argue that the upper bound obtained there may already be optimal in this challenging regime.
Consider the special case when m = 2 .
Essentially, the goal is to estimate the difference of the two item parameters .
For a particular user l , suppose that we have no information about her parameter θ l other than the bounded condition.
If the user’s responses consist of a single response (the response to one item is not observed) or that her responses to both items are identical (either both 0 or 1 ), then we learn little about the difference between the two items.
We refer to these responses as ‘bad’ responses.
The relative difference between the items is only revealed if the user responds differently to the items.
As noted in the description of our spectral algorithm, we refer to such information as pairwise differential measurements .
For the special case m = 2 , both CMLE and JMLE actually ignore ‘bad’ responses.
This is because in both algorithms, it has been shown that including bad responses in the respective objective likelihood function leads to parameter unidentiﬁability [26].
As mentioned earlier, MMLE, requires an accurate prior distribution in order to obtain good estimate accuracy.
This is not possible when we have no information about the user parameters.
With the exception of MMLE, all estimation methods that we are aware of in the literature only use pairwise differential measurements.
With these points considered, if we restrict our attention to the class of algorithms that use pairwise differential measurements, then the spectral algorithm indeed achieves the best possible (up to a log factor) estimation error.
Theorem 3.5.
Fix m = 2 and consider the sampling model described in Section 3.1.
Let T be any unbiased estimator for the item parameters that only uses pairwise differential measurements .
Then the mean squared error of such estimator is lower bounded as
[Equation]
where T ( X ) is the output of the estimator T when given data X and c (cid:48) is a constant.
As seen from Theorem 3.1, the estimate produced by the spectral algorithm satisﬁes (cid:107) β − β ∗ (cid:107)
= ˜
O
( 1 np 2 ) , establishing its near optimality.
Practical Implementation Aspects
In this section, we discuss two important practical aspects that practicioners may consider when applying the spectral algorithm real-life datasets of which observation pattern may not correspond exactly to the sampling model described in Section 3.1.
Firstly, we identify slow convergence as a problem encountered by the spectral algorithm when the data is skewed in the sense that some items are highly responded to by users while some items are rarely responded to.
To address this issue, we propose an accelerated spectral algorithm that enjoys the same error guarantees as the original spectral algorithm but runs signiﬁcantly faster in practice and suffers from fewer numerical issues.
Secondly, we discuss regularization strategy when the spectral algorithm is applied to datasets with sparse observation patterns.
Accelerating the Spectral Algorithm: Recall that in the original spectral Algorithm 1, we use a common normalization constant d that generally scales as O (max i (cid:80) k (cid:54) =
i B ik ) .
In practice, the distribution of the user-item assignment could be skewed such that some items are rarely responded to while some elicit many user responses.
In such cases, the items with few responses will have few pairwise differential measurements (cid:28) d .
The induced Markov chain will contain large self-loops for these items.
We observe that these large self-loops lead to a slower convergence when computing the stationary distribution and more numerical issues.
This observation was also noted in [2] and we propose a similar solution to eliminate large self-loops that is to use a different normalizing factor for each vertex .
Consider the following modiﬁed Markov chain:
[Equation]
where Y ij is deﬁned in Equation (2) and { d i } mi =1 are appropriately chosen normalization factors such that the resulting transition probability matrix does not contain any negative entries.
In our experiments, we choose d i
= O ( (cid:80)
k (cid:54) =
i B ik ) .
The accelerated spectral algorithm computes the stationary distribution of the above modiﬁed Markov chain, and recovers the item parameters via a post-processing step.
The algorithm is summarized in Algorithm 2.
Interestingly, the accelerated algorithm produces essentially the same estimate as the original algorithm.
Under some regularity conditions, there is a direct one-to-one relation between the stationary distribution π obtained using the original Markov chain in Algorithm 1 and the stationary distribution ¯ π of the Markov chain parametrized by ¯ P .
This result is summarized in Theorem 4.1.
Theorem 4.1.
Consider the modiﬁed Markov chain ¯ P constructed per Equation (5) and the original
Markov chain P constructed per Equation (3).
Suppose that both ¯ P and P admit unique stationary distributions ¯ π and π , respectively.
Then
[Equation]
where d i are the normalization factors in the construction of the modiﬁed Markov chain ¯ P .
Assuming perfect numerical precision , the two versions of the spectral algorithm output the same estimates.
Therefore the guarantees in Theorems 3.1 and 3.3 also apply to the accelerated spectral algorithm.
However, in our experiments, we observe that the accelerated spectral algorithm converges much faster and suffers from fewer numerical issues than the original version on real-life datasets, leading to better performance overall.
We thus use the accelerated version in our experiments.
Regularization for Sparse Datasets: In some real-life datasets, we observe that certain pairs of items have few pairwise differential measurements.
Furthermore, the pairwise differential data is one-sided (e.g., users who respond to the two items always respond positively to one but negatively to the other).
This could happen to pairs that have been shown to only few users.
The existence of many such pairs may also introduce numerical issues and parameters unidentiﬁability.
For example, when there is an item i such that Y ji = 0 ∀ j (cid:54) =
i , the stationary probability correpsonding to this item will be 0 and the item parameter estimate will be −∞ .
As a solution, we propose adding regularization in the construction of the Markov chain.
Speciﬁcally, for every pair of items i, j such that B ij >redeﬁne
[Equation]
where ν is a small constant.
In all of our experiments on real-life datasets, we use ν = 1 and ﬁnd that the regularization parameter requires little tuning.
Regularization also ensures the uniqueness of the stationary distribution.
So long as the graph underlying the Markov chain is connected, adding regularization ensures that no pairwise transition probability is 0.
This makes the constructed Markov chain ergodic and there is a unique stationary distribution [44].
Experiments
In this section, we present empirical ﬁndings which support the practical value of our spectral algorithm.
Our baselines are the most commonly used estimation algorithms in the literature: conditional maximum likelihood estimate (CMLE), marginal maximum likelihood estimate (MMLE) and joint maximum marginal likelihood (JMLE).
Their open source implementation can be found online [49].
We include the python implementation of our spectral algorithm in the supplementary materials.
[Figure] Synthetic Data: We generate the item parameters β ∗ from a standard normal distribution and user parameters θ ∗ from N (0 , σ 2 ) where σ 2 varies for different model settings.
Recall that MMLE requires the statistician to specify the prior distribution over the user parameters and for synthetic experiments, we specify this prior distribution to be the standard normal distribution.
Subﬁgures (a)-(c) of Figure 1 show (cid:107) β − β ∗ (cid:107) 2 against n for all 4 algorithms under 3 different model settings while
Subﬁgure (d) shows inference time.
When there is no missing data and MMLE’s prior distribution is correctly speciﬁed ( σ = 1 ), CMLE, MMLE and the spectral algorithm perform equally well.
However, when MMLE’s prior distribution is misspeciﬁed ( σ = 2 ), it produces inconsistent estimates.
As mentioned before, JMLE is known to produce inconsistent estimate when m is small relative to n .
On the other hand, CMLE’s performance degrades under moderately sized dataset with missing data.
The spectral algorithm, however, consistently performs well across all of these settings and is comparatively scalable.
Real Data: We perform experiments on a wide range of real-life datasets from education testing datasets to book and movie ratings datasets for recommendation systems.
In order to transform ratings data to binary response data, we follow the procedures in previous works [33, 19].
Speciﬁcally, for each user, we convert all ratings higher than the average to 0 and 1 otherwise (so that items with a higher parameter value is ‘better’).
Since real-life datasets do not come with true β ∗ parameters, we compare the algorithms on four metrics: area under the ROC curve (denoted AUC) on heldout test data; log-likelihood on heldout test data; inference time and top- K accuracy where the reference top-
K set is determined by average ratings. We have also followed the standard procedure in the recommendation systems literature to remove items that have few ( ≤ 10 ) ratings from the reference top- K set.
In our experiments, we consider K ∈ { 10 , 25 , 50 } .
We defer extra experiment results where we include additional algorithms such as Bayesian estimation [42] and pairwise likelihood estimation as well as datasets descriptions to the supplementary materials but mention a few notable datasets: RIIID [1] ( m = 6 k, n = 23 k , education testing dataset), ML-20M [28] ( m = 27 k, n = 138 k ), Book-Genome [31] ( m = 10 k, n = 350 k ). [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
Table 1 summarizes the performance of the four methods in our experiments.
Note that the ﬁrst four datasets are education testing datasets and thus there are no top- K ranking metrics being measured.
For tuning the prior distribution for MMLE, we select the prior distribution that admits the highest log-likelihood on a validation set.
On the other hand, CMLE, JMLE and the spectral method requires minimal model tuning.
For small scale education datasets (LSAT, UCI, 3GRADES), there are no missing responses and all methods perform very similarly to one another. For large scale ratings datasets, the spectral method remains competitive with the baselines in terms of AUC and log-likelihood but tends to outperforms in top- K accuracy and is signiﬁcantly more efﬁcient. In large scale ratings datasets, we observe that the competitor methods tend to assign large parameter value to items that receive only a few but very high ratings (so after data processing, all of the responses to such items are 0 ) and these items are not included in the reference top-
K set.
The spectral method, because it operates on pairwise differentials, is less susceptible to these noisy responses and thus more accurately recovers the items in the reference top-
K set.
Related Works
The Rasch modeling literature is quite broad and we refer the interested reader to recent surveys [6, 46].
Since its original formulation to model psychological tests outcome [45], the model has been extended to account for more complicated response patterns such as numerical ratings and ordinal responses [4, 5, 55, 39].
Higher-order models that incorporates bias variables such as the 2PL and 3PL model [9] and multivariate models [21] remain active areas of research where machine learning techniques have recently been applied with substantial success [8].
The Rasch inference problem is also closely connected to the 1-bit matrix completion problem in machine learning where we observe a sparse n × m binary matrix with underlying entrywise probability f
( M ) where f is a mapping function (e.g., logistic) and M is a real-valued matrix.
There, the goal is obtain an estimate of M .
A commonly proposed approaches for 1-bit matrix completion based on alternating optimization is exactly joint maximum likelihood estimate [15, 14].  Ethical Considerations
Our work proposes an algorithm of which real-life applications very often involve actual human data with sensitive information. For example, the Rasch model is often studied in the context of education testing and psychological testing where the subjects of studies are students and patients.
Therefore, deploying our algorithm (or any algorithms in this context) needs to be accompanied by thoughtful and thorough ethical considerations.
In this work, we provide the algorithmic tool that lays the foundation for our algorithm and its theoretical guarantee. We believe that a socially constructive application of our algorithm should always be accompanied by detailed explanation of its fundamental limitations, assumptions and decision makers need to take into account these aspects when interpreting the results returned by the algorithm.
Conclusion
We propose a new spectral algorithm for the item estimation problem under the celebrated Rasch model.
Our algorithm is theoretically well-founded, practically performant and should be added to the statistician’s quiver of estimation methods when analyzing binary response data.
Extending our algorithm to more expressive IRT models such as 2PL or 3PL and response types is an open avenue.
Another important aspect inline with the ethical considerations that need to go into IRT models is privacy.
Given the human centric nature of IRT models and how IRT algorithms are often deployed on datasets containing sensitive information (e.g., a student passing or failing a test, a voter supporting a legislation), it is quite concerning that there has not been any proposed privacy-preserving estimation method in the literature.
Acknowledgement
The authors would like to thank William Zhang for proofreading and leaving helpful comments on earlier drafts of this paper.
We would also like to thank the anonymous reviewers for their thoughtful suggestions that have been incorporated to improve this paper.
D.N. and A.Z. are supported by NSF
Grant DMS-2112988.
A.Z. acknowledges ﬁnancial support from the Alfred H. Williams Faculty
Scholar award.
Any opinions expressed in this paper are those of the author and do not necessarily reﬂect the views of the National Science Foundation.