tttjjj/elt_bench huggingface dataset:
If you order through amaxon I hope you looked at used books often they have very good prices for books in good condition.
After the burn in period.
Certainly, if you're likely to do anything related to say allometry\\*, it could certainly have *some* value, but my experience is that such cases don't seem to crop up a lot (I don't recall any such questions here for example). \\* that's not the only kind of analysis where one might be interested in shape of course. It's a pity, really, I find the stuff to do with shape fascinating, but it doesn't seem to generating all that many questions.
You should not do linear regression.
http://stat.ethz.ch/R-manual/R-patched/library/base/html/rank.html
@conjectures $\\theta$ is the estimated coefficient. The calcualted values with these coefficient are all in-sample fits. This is what the fitted method does. As already stated, this is not about model checking or something like this, it is just to understand how to forecast a MA(1). Normally, at this time point you would not know this value (you would get a different estimate, since you do not have the following values if you do the estimation).
it is unlikely to be linear, you'll likely have to do one of these to heart rate: (1) pre-specify categories (2) quintiles (3) restricted cubic splines. For face-validity you'll want some measure of blood pressure (MAP) and there may be an interaction between these two variables that could complicate analysis.
@Lan I think Wolfgang's answer nicely illustrates how to make the test you are proposing.  It is an example of what meant by testing a hypothesis "in the usual way."
@muratoa thanks for feedback. I am not sure I follow. If $2^k$ is the max block size the scheme is not like permuting pairs within blocks. For instance, for $k=2$, you can obtain the order `4:3:2:1` with probability 1/8, which is not a pair permutation. As for $k_{\\min}$ and $k_{\\max}$, this is what I refer to in point 3. This is the way to shuffle scales from $k_{\\min}$ and $k_{\\max}$.
It is indeed evident that the answer can be anywhere between $0$ and $N/2$, but that does not imply the non-existence of a general result.  What it does imply is we should consider answers where the fraction depends on some properties of the underlying distribution such as its mean and SD.  Those are enough, along with the CLT, to provide specific and quantitative information about how the $x[i]$ are distributed compared to their sum, so it's reasonable to hope for such a result.
You need to explain more, with a full example; `mvpart()` is a complex function doing all sorts of things. What do you mean by "tree output"? Use the example in `?mvpart` and edit your Q to include a worked example of what you are trying to do.
I just ran into this precise problem. Thanks for the excellent explanation!
Although this recommendation might work for some datasets, it is exquisitely sensitive to any outliers that might exist, and so is not a good general procedure.
@ChuckKillerDoll I saw your answer, but thought it would be useful to point out that this seems similar to a more general situation in Machine Learning.
@Noah,  have you considered various stochastic volatility models, such as GARCH?
The question is answered in the book [Elements of statistical learning](http://www-stat.stanford.edu/~tibs/ElemStatLearn/) page 222. The validation set is used for model selection, the test set for final model (the model which was selected by selection process) prediction error.
Nice to know someone is reading what I write! I hope i've fixed the errors now... :)
Is it a question or a pool? If the latter, it should be community wiki. If the first, give more details about what you want to achieve? For instance, is it all-relevant or rather  minimal-optimal selection? How much is many? How   hard is the classification problem?
My problem is that I am unsure what is the function I should feed into optim() to have beta1 and beta2 as result. Please take a look at updated question.
When you used the functions you pointed to involving $\\log(1+\\dotsc)$, did you use the `log1p` function in your language? This uses the Taylor expansion around 1.
Thank you very much for such a great answer Steffen. I think I will try to limit to first 5 actor/actress first and see how the classification go.
and you can get the exercises and solutions book too: http://books.google.ca/books?id=kEqtNJOxaCoC&printsec=frontcover&dq=jun+shao+mathematical+statistics&hl=en&ei=P2oITo-WDMG20AGS-t32Cg&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC4Q6AEwAA#v=onepage&q&f=false
I have no explanation for this, but I was completely fool. It´s rather obvious to try updating the installation. I did what you said, the command ran and I obtained the same results that those that appear in Rabe-Hesketh & Skrondal´s book. Thank you very much and see you soon!
And please change $p$ to $\\rho$. $p$ usualy denotes price.
That might be a bad sign. See [here](http://yudkowsky.net/rational/technical): `Occasionally an AI researcher discovers that if they add noise to one of their algorithms, the algorithm works better. This result is reported with great enthusiasm, followed by much fulsome praise of the creative powers of chaos, unpredictability, spontaneity... But how sad is an algorithm if you can increase its performance by injecting entropy into intermediate processing stages?... If injecting randomness results in a reliable improvement, then some aspect of the algorithm must do reliably worse than random.`
Thanks Juampa for the papers, very useful.
Thank you Ben for clarification and the link.
If you know the correct answer, why do the analysis at all?
Welcome to the site.

There are several similar questions already on this site:
e.g.
[Sample size for unequal groups in logistic regression](http://stats.stackexchange.com/questions/22328/sample-size-for-unequal-groups-in-logistic-regression)

and

[Does an unbalanced sample matter when doing logistic regression?](http://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression)

Do these answer your question? If not, please say what other information you need.
Sorry for the my presentation. The predictor in my problem is $w*$. I want to know if $w*$ can be a matrix with the size $K \\times 3$ where $K$ is fixed. In the function defined above the value of $K$ was 1.
@ Gavin  Because mvpart has a more flexible, nicer-looking output. Thank you for your comments and suggestions....
Thank you Jonathan for your response. I have added additional details.
I am not sure factor analysis would be useful here: I guess OP wanted to use the classifier to classify the disease given a single spirometry measurement of a new patient, without waiting to collect three measurements first (so you would not be able to apply FA on the real test data).
I would say the same providing another link http://www.google.gr/url?sa=t&source=web&cd=1&ved=0CB8QFjAA&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.98.4088%26rep%3Drep1%26type%3Dpdf&ei=nlx3TIq-LJCk4Abn_Li3Bg&usg=AFQjCNHO-_yjWAJrRVnJms7MbcqaJkd8eg&sig2=sqERq2v68UvOhJDMviOklg
If you have enough data to do sufficient cross-validation, I'm not sure it trying to use more parameters will hurt you or lead to over-fitting.
There is a difference between a moment *existing* and being *finite*. In particular, a moment can exist, but be infinite. The terminology you're being introduced to is a bit imprecise. In any event, this is a standard result about $L_p$ spaces; it is not true that "no exact proof exists". :)
If the authors of your literature are honest, they will eat their own words when they encounter linear regression.
Proof is in Appendix B of this paper.
In what sense are the changes that occur when re-running this code "small"?  The size of a change is not an absolute but must be understood within some relevant context.  You also seem to assume the changes in output are "large," but the same criterion applies: large with respect to what?  Here, the basis for comparison should be the covariance matrix of the estimates.  What does it say about the sizes of the changes in these coefficients?  (BTW, your sample data seem to describe the Brobdingnags, certainly not humans: these people include some 25 footers!)
+1. Many authors emphasize that stationarity typically is not a fact about the world but rather is a *modeling decision.* As such, questions about how to "test" for stationary need to be nuanced: what alternative model is being considered by any such test and what effects would the choice of model have on one's investigation objectives?
Sparese PC is for example L1 (lasso)-penalized PCA.  In ordinary PCA we can usually enter terms in order of variation explained.  With sparse PCA things are a bit more erratic so selection is perhaps more difficult.
The keyword for this is *mixed frequency time series*. There is a state space time series literature on this following the work of [Harvey and Pierse (1984)](http://www.jstor.org/stable/2288346), and many recent contributions on mixed frequency vector autoregression models, a recent comparison of the two approaches is in [Bai, Ghysels and Wright (2013)](http://dx.doi.org/10.1080/07474938.2012.690675).
Thanks for your comment and suggestion. Then, I should take the conditions in that appendix into account and set my probability.
@miso - gung's advice here is really expert and even though the answer is somewhat dense, I'd encourage you to think through every point he made and to ask for any clarification you might need.
luciano: It might be worth trying Russ' example yourself, but also look at `table(x,y)` to see the actual table it's working with.
I too doubt that UMVUE is relevant here. What you can compare though is asymptotic efficiency, and it will be lower for the (nonlinear) least squares for the reasons you explained.
This question appears to be off-topic because it is about how to do something in R. See the help center for more advice on code-related questions.
(+1) Welcome to the site.
Besides, if the signed errors were modeled, there would be offsetting effects from the underestimations and overestimations on the regression coefficients of the IVs.
The answer at http://stats.stackexchange.com/questions/9131/obtaining-a-formula-for-prediction-limits-in-a-linear-model gives a formula.
You forgot to include a vector of 1 representing the intercept. The rest looks correct.
In the original model the ratings have a normal distribution around $U_i^T V_j$. An alternative to pymc is [Stan](http://mc-stan.org), for which I have a [somewhat-optimized BPMF model](http://git.sccs.swarthmore.edu/web/dougal/active-matrix-factorization/tree/stan-bpmf/bpmf.stan) (and [one with $W_0$ hardcoded to the identity](http://git.sccs.swarthmore.edu/web/dougal/active-matrix-factorization/tree/stan-bpmf/bpmf_w0identity.stan), and a [more straightforward but slower one](http://git.sccs.swarthmore.edu/web/dougal/active-matrix-factorization/tree/stan-bpmf/bpmf_straightforward.stan)).
This is a much more informative answer. +1
(i) To *calculate* one? No. You can do the calculation just fine - your computer won't explode or anything. The question is whether it does what you want it to (i.e. what does the result mean compared to what you are trying to achieve). (ii) So why are you standardizing? (iii) When you say 'normalize', what do you mean? (hover over your `normalization` tag to see what the tag means by the term; if that's what you mean, that would make no difference whatever to the end result)
*(In passing, I should note that your question refers to the "importance" and "predictive power" of variables in ways that suggest you see these terms as related to the betas found in a multiple regression analysis. I believe that this is a mistaken assumption--albeit a very common misunderstanding. Unfortunately, it would take quite a bit to explain why, and I'm guessing that it is peripheral to your real question, so I won't address these issues here. However, if you are interested in these issues, ask another question and I'll be happy to explain it.)*
Example: For Physics experiments done in a lab the different observations would vary due to lots of environment factors coming into play. Thus I think the disturbance in the actual value would be normal (lots of random variables influencing the actual value). Can we build upon this assumption?
As a note to your second comment. The anchors are the same as they were in the previous year of the survey. Essentially, the granularity of the scale was reduced.
Does your visual processing system have variables that are constantly dynamic (i.e. some present, some not during any given time interval)? 

I did some reading on wavelet and its applicability to time series data, but this data doesn't really seem to exhibit those types of wave patterns. Maybe I don't understand it. I'll read more tonight.


Yes, you are right about how problematic regression is for noisy data sets. The solution can be descriptive, so not so concerned about the entire model's predictive accuracy.
what tests would you do for the comparison of just two?
The update is now on CRAN for linux, windows, and mac.
@Andy Amos: I'd bet money on it. Like @whuber points out it's almost impossible that $BB'$ would have negative eigenvalues if you're computing it directly, & the zeros (from being rank deficient) should be taken care of by adding $D$ since its positive diagonal - unless some of those elements are really small. Try generating some data from a model where $\\sigma_i^2$ are pretty large and $\\sum_q B_{iq}^2 \\approx \\sigma_i^2$. The more data the better so that  the estimates should be accurate and stable. That will at least tell you if there's a problem in your implementation.
There are exponential smoothing models that don't fall within the ARIMA framework, e.g.: http://users.ox.ac.uk/~mast0315/DampedPegels.pdf
Thanks for this, and for suggesting I look up the Hauck-Donner effect.  The effect doesn't get much treatment in textbooks but seems pretty important!
+1 for pointing out the consequences. It can do wonders for getting people to switch to better methods.
Yeah, I know what you told, the question I'm asking is ''Why?''. What would happen if I used the Mean Sq? What would this resemble?
It is a very recent field so yes that may be it. Besides the Rao book I would probably check out the two journal articles cited on wikipedia that appear to be reviews (one by Rao and one by Pfefferman). They likely have bibliographies of interest. Also just see who has [cited the Rao book](http://scholar.google.com/scholar?cites=1050616381915847332&as_sdt=5,33&sciodt=0,33&hl=en), not a tiny list, but reasonable to muck through for other reading of interest. Given its contemporary nature you may need to look into journal articles as well as books.
@Harpreet You are not estimating the shape of the PDF since as @Dirk indicated it has closed form, you just specify its parameters (e.g. $\\mu$ and $\\sigma^2$ for a gaussian). It will not necessarily "fit" the data. Now, there exist several kind of non-parametric density estimates, where you only use the data at hand (plus some kernel specifications or window span, etc.); see e.g., online help for the `density` R function.
Perhaps there is something in the middle here... What about a multinomial classification? You are correct that binary classification is simpler, but as the responses indicate it may be a bad idea to try and map your setting into a binary classification. But, 3 or 4 classes might give you the granularity you need and still be relatively easy to use (lots of software packages offer neural networks supporting this; you need to use a softmax activation function in the output layer).
If you're not the one making a distinction, what reason to suppose there is one? I'd just call the latter a special case of the former ($q>1$). And note that often enough general linear models come with several more assumptions than a conditional expectation of zero.
This is off topic here and I'm pretty sure it has been duplicated on SO [approximately 100 times](http://stackoverflow.com/search?q=[r]+%2B+reshape+%2Blong+%2Bwide).
I'm not familiar with Rasch analysis. But my email is: my first name @ my lastname .ca and yes, B/F/H speak to a willingness towards complexity/change, D/E are tied by philosophy, C/I speak to how much one might care for others.
Thanks for comment. I put up the first three pages here, hoping not to violate any rights too much, but I fear the paragraph is not understandable without the rest (http://img815.imageshack.us/i/paper0.jpg/, http://img339.imageshack.us/i/paper1d.jpg/, http://img683.imageshack.us/i/paper2a.jpg/). The interesting part is on the third page and I don't really see any further information :( Unfortunately I don't have access to the original article by Godfrey and Wickens mentioned there.
Are you asking "How else could I answer those research questions, given the design?". Also, if you refine or clarify your question, please do that via an edit to the question?
@NickCox I fixed the typo, thanks
Isn't the state [customer in service, customer queued for second station] for station 1 feasible? this could occur if a new customer appears while the customer is waiting for the second station to open up.
@vitalik: So you have observations $N_1, N_2,\\ldots,N_N$ which are independent and Poisson distributed? Does your model predict that you have different $\\lambda_i$ for the different channels, i.e. $\\lambda_1,\\ldots,\\lambda_N$? Why do you want to use histograms to describe the theoretical distributions? They are used for continuous data and not for theoretical descriptions of discrete random variables...
I'm afraid I am a MATLAB user (I use the GPML package http://www.gaussianprocess.org/gpml/code/matlab/doc/), so I can't advise about R implementations, but you may find something suitable here http://www.gaussianprocess.org/#code .  If R doesn't have a decent package for GPs, someone needs to write one!
@Zen: Good point! I consider this answer as **the** answer because it made me realise $H$ sounds very much like a standard deviation, for which there exists no unbiased estimator. As for the variance of $\\hat H^2_n$, no worries: $\\mathbb{E}[(\\sqrt{f_0(X)/f(X)})^2]=1$ implies that this estimator has a finite variance.
The most obvious drawback of your approach is that it has a 50% false positive rate.
I included the figure you mention for clarity.
Wikipedia has good definitions of quasi-experiment, natural experiment, cohort study, and observational study.  Your study is an observational, longitudinal (or cohort) study.  Your study doesn't look quasi-experimental to me---it would be if students were randomly assigned to G1/G2 somehow by nature.  A picture of the "pyramid of evidence:"  http://sourcesandmethods.blogspot.com/2011/05/evaluating-analytic-methods-what-counts.html
Hm...I thought it was unambiguous given that I'm talking about Chernoff bound for absolute error...added a link in question (it seems to break in comment)...in that notation, I'm looking for epsilon as a function of m
Yes, this is possible and your interpretation is correct as well. If you are interested, there are other ways of doing this: [How to keep time invariant variables in a fixed effects model](http://stats.stackexchange.com/questions/90754/how-to-keep-time-invariant-variables-in-a-fixed-effects-model/90759#90759)
I'd suggest you check out the related links on the right hand side of the page. There are quite a few questions of very similar nature, see [Is it possible to do time-series clustering based on curve shape?](http://stats.stackexchange.com/q/3331/1036) and [Modelling longitudinal data where the effect of time varies in functional form between individuals](http://stats.stackexchange.com/q/2777/1036) for just two examples.
This might be insightful: every sequence of heads and tails corresponds to a unique number between $0$ and $1$ by encoding the heads as `1`, the tails as `0`, and putting a binary point in front of the sequence. Any infinite coin flip thereby is tantamount to throwing a dart at the line segment from $0$ to $1$. Never getting a head is the same as hitting the $0$ at the end of the segment. As in the two-dimensional dartboard case, this event is *possible* but its probability is zero.
+1 that problem twisted my brain when I first read and thought about it- and the solution is pretty simple but teaches a lot about probability.
@Zach, I've replaced, thank you for suggestion and idea of function dput() usage
What do you mean by "see if the responses in the middle ranges are significantly different than the mean"?  This needs to be clarified before this question can be answered.
I'm guessing this is some kind of self-study (you should tag it as such). What exactly is it you want? Do inference on your parameter?
To posters: It is always polite and helpful to make clear what statistical software you are using, regardless of whether you think it is obvious. (My friend Maarten knows this!)
possible duplicate of [What is the distribution of the euclidean distance between two normally distributed random variables](http://stats.stackexchange.com/questions/9220/what-is-the-distribution-of-the-euclidean-distance-between-two-normally-distribu)
Can you edit your answer. If looks like your LaTexing got messed up?
You can look at the help for guidance e.g. http://stats.stackexchange.com/help/formatting I've added the tag self-study as this appears to be a homework problem or something similar. Hint on the problem: draw the curve for your density function. Is it flat?
In the integral representation above, we _integrate with respect to the distribution of $X_1X_2$ conditional on $X_1$._The integrating variable is $X_2$. Is it because we take $X_1$ as if we knew it?
Wikipedia is right, in this case. It usually is for topics like this. I would be more leery of it on controversial topics.
Can we see all of the code? What was the glm command that you ran?
Thanks Henry!  It's what I needed.
*Elements of Statistical Computing* (R.A. Thisted, Chapman and Hall, 1988) is older, yet it contains interesting material, IMO.
Thanks. But other than the difference in underlying universe, what are the other differences? My universe is fixed so I didn't randomly sample the "much larger" universe... in this case, the mixed model is of no benefit?
Frankly, I think the claim is nonsensical. I can't second-guess why he might have said that - when someone says things you don't understand, *that person* is best placed to explain their reasoning.
@pedro That is not the case, because the errors in $x$ are not merely weights in a formula.  With errors-in-variables regression the fits will differ and the covariance estimates of the parameters will differ from ordinary regression.
@Comp_Warrior I incorrectly equated MSE with the L2 norm. They're actually slightly different. Updated my answer to reflect this.
You seem to be missing all assumptions about the distribution of $\\lambda$ among the attendees--or do you assume they all have the same value of $\\lambda$ and arrivals are independent? Also, there's no reason to suppose that both the time waiting and number of people in front of you can simultaneously be optimized, unless you are the only people to attend the concert! A clearer statement, then, of what it is you are trying to minimize is needed in order for this question to be answerable.
A minor point: you'll always get less bias, but you're not guaranteed to purchase it with higher variance (though in normal situations you probably will).  Also you'll have gotten less biased about a slightly different causal quantity of interest to the one you started with.
I read the paper on NW SE estimators. Apparently, "Newey-West" estimators are yet another derivation of sandwich based SEs. It is NOT a correction, though. But it seems like you've mostly come to the same conclusion that I recommended.
You provide good related information but I think in response to chl question, the OP made it clear that he is talking about categorical data that is not ordinal.  So your response is really not an answwer but I am not one who would give a downvote.  But I do think you should change it to a comment.
Hi - I'll check out the gam package. Thanks!
Thanks for the link to great book. I would to take a grasp of it.