  In this paper, we study limiting laws and consistent estimation criteria for
the extreme eigenvalues in a spiked covariance model of dimension $p$. Firstly,
for fixed $p$, we propose a generalized estimation criterion that can
consistently estimate, $k$, the number of spiked eigenvalues. Compared with the
existing literature, we show that consistency can be achieved under weaker
conditions on the penalty term. Next, allowing both $p$ and $k$ to diverge, we
derive limiting distributions of the spiked sample eigenvalues using random
matrix theory techniques. Notably, our results do not require the spiked
eigenvalues to be uniformly bounded from above or tending to infinity, as have
been assumed in the existing literature. Based on the above derived results, we
formulate a generalized estimation criterion and show that it can consistently
estimate $k$, while $k$ can be fixed or grow at an order of $k=o(n^{1/3})$. We
further show that the results in our work continue to hold under a general
population distribution without assuming normality. The efficacy of the
proposed estimation criteria is illustrated through comparative simulation
studies.
