tttjjj/elt_bench huggingface dataset:
A large $p$-value indicates the null hypothesis is not rejected. Based on your results you tested whether the mean in `SIFT` was not smaller than the mean of `SURF`. You get a large $p$-value indicating that your results are explained well by the null hypothesis (e.g. they are likely to occur when the null hypothesis is true). Testing the other way around would have been a better choice (since the values for `SIFT` appear to be larger than those of `SURF`). Another solution would have been to use a two-tailed t test.
Yes - for each beta coefficient, you can obtain an odds ratio and 95% confidence limit. The 1.96 will not change unless you want to calculate a different percentile for the confidence interval (90%, 99%, etc.), but as 95% is standard, there's no need to do so. And as long as you are working on the log scale, the parameters from a logistic regression model are normally distributed. Once you exponentiate them, this stops being true.
DBScan, Optics and HC can be used with custom distances, and all of them do not require number of clusers, density link is a hybrid of these approaches so shares their characteristic. Spectral clustering was your choice, I only provided you with the modifications that deal with the number of clusters.
Here is the data set that I am working on http://dl.dropbox.com/u/6967747/ratio.csv
I'm trying to use an ensemble of decision trees as a random forest to do binary classification. I'm confused on what tests I should use at the individual tree nodes to minimize entropy and maximize information gain.
@Chowza: In that case you are probably fine. That is, unless there are hidden clusters in your data that drasticaly reduce the effective sample size.
@suprvisr I can answer right there or update my response, but I think it is better to ask a new question (sth along the idea of "pros and cons of adjusting with PCA vs. stratifying") and link to this one so that people can clearly make the necessary connections.
what do you mean by "polynomial features"? features which are values of all possible polynomials of given degree?
+1 for explaining my job :)
Agree with all the statements above re: how the cases are classified.  Take a look at the ROCR package as a way to evaluate where a "good" cutoff might be, as this may help.  For a good view of what you can do with ROC, http://rocr.bioinf.mpi-sb.mpg.de/.  HTH
It is called "stratified" cross-validation. You split into k groups each of the classes, and make sure that in each training set go k-1 groups from each class, and in each test set goes 1 group from each class.
The area under the PDF equals $1$, by definition, so you must mean something else by "the curve."  What is it?
When I initially asked this question I was planning to add this answer myself, besides alternatives I didn't know. Thanks for answering, I will accept this answer as the accepted as it is a 'standard' in the ASR world!
Excellent and highly informed answer, thank you!
I would like to add that the paper cited in the paper you suggested is very nice:  it generalizes PCA to fitting a data matrix to the product of two matrices of lower rank than the original data, with a weight per data point on the fit.  [Lower Rank Approximation of Matrices by Least Squares With Any Choice of Weights, Gabriel and Zamir, 1979](http://www.ma.huji.ac.il/~zamir/documents/lower.pdf)
Thanks for your effort -- unfortunately, it did not work. (I was doing something very similar in my 3-line program: `[V,D] = eig(A); D(D <= 1e-10) = 1e-6; Apd = V*A*V';`). This approach is similar to the one by Rebonato and Jackel, and it seems to fail for pathological cases like mine.
If your hypothesis of interest is truly to compare "mood between the three groups to see if they differ" then a simple average might be all you need. But think carefully about the kinds of *specific* questions you would like to address (*before* looking at your data). Why did you collect 30 measurements per subject? What exactly are you worried that you are losing with simple means? Without more specific questions, it is unreasonable to suggest more specific analytic methods.
The question is framed ok, but the comments are getting right up to the line of argumentation and threatening to spill over to the wrong side of that line.  Be careful...this is not the place for such debate.  Create a chat room if you want to do that.
Doesn't that make sense though?  If I used a regression model where I assumed that sigma is .00001 instead of using the estimate from the data (2.3 let's say) then of course things are going to be more significant.
Indeed! I had in mind point 3 of the [measure theory statement](http://en.wikipedia.org/wiki/Leibniz_integral_rule#Measure_theory_statement) of the rule, but of course point 1 and 2 are also required. I will edit my answer accordingly.
Does this help? http://stats.stackexchange.com/questions/36015/prediction-in-cox-regression/36077#36077
@StéphaneLaurent: I do not understand the point of your first comment as I never said something to that extend. Regarding your example with the $\\Gamma$ whether or not you use $(0,\\infty)$ or $[0,\\infty)$ does not really matter in the sense that any function $g(x)$ that agrees with $f(x)$ everywhere except for a finite number of points is a member of the same equivalence class as $f(x)$ and to all intents and purposes is the same function. And as for *support* if you define as *"the smallest closed set whose complement has probability zero"* you alleviate any positivity concerns.
If we define $\\psi(x)=x^2$ for $x\\in[-T,T]$, $\\psi(x)=T$ for $|x|\\ge T$ then your problem is to find  $\\text{argmin}_w\\sum_{i=1}^n\\psi(y_i-\\sum_{j=1}^3w_jx_{ij})$. This looks similar to winsorised mean problem. I think the theory of minimizing $\\sum\\psi(\\varepsilon_i)$ is quite developed, so you can find something if you search specifically for that.
Forgive my confusion; what level of significance are you looking for if not the $p$ value itself?
+1 going to research this - thanks for suggesting
The tilde means "proportional to"
Are these lotteries for money? You'd need to control for diminishing marginal utility of money before you could elicit risk attitudes, no? I've read stuff on [Prospect theory](http://en.wikipedia.org/wiki/Prospect_theory) which discusses how to elicit these sorts of attitudes.
Can you edit your question (hitting the "edit" link in lower left) to share more details about your full model (i.e. what the predictors are, how they are coded, the sample size, etc.)?
I vaguely remember having read that someone did an analysis for some sport. There weren't any short term memory effects (apart from what is mentioned in the answers; injuries etc). Only one thing was found. If a team lost 3 times in a row, it was more likely to lose again (psychological?). To your friend you could show a genuine random walk plot and ask if he sees patterns.
What did you try so far? Have you read a little about UMVUE?
Please don't radically alter your question in a way that makes answers to what you originally asked no longer an answer - the form of the original was fine, but now my answer doesn't look like it responds to your question - people would rightly downvote my answer on that basis. You can add followup questions or clarifications without changing the question quite so much. Your new question is basic mathematics and isn't on topic here; it's *exactly* like changing from $\\sum_{i=1}^n x_i$ to $\\sum_{j=1}^n x_j$ -- changing the dummy variable doesn't alter anything.
Actually thinking about this, would the standard signed rank test work fine here, because it assumes paired values?
I'm still puzzled about the data you actually have. For each observation, you have the group, the [0,1] variable, and what else? That would make a big difference. Depending on your ultimate goal, I'd also recommend you look into hierarchical (multi-level) modeling and regression.
Thank you! But given that the factor scores used as predictors are "composite"' how one one properly interpret mlogit reg output?
To add my two cents worth on "context": if you want to estimate the proportion of people on a street who might buy a certain product being sold door-to-door, you might be happy to have a margin of error of +/- 10%; if you're wanting to estimate complication rates for a particular surgical procedure, then you might need a smaller margin of error.
To make it clearer, here you will rather need Kolomogorov-Smirnov distance than KS-test.
I guess the point is also if we're talking "Likert scale" or "Likert item" here. Scaling individual items is super-bad since they already have enough problems with validity as it is. I guess rescaling a Likert scale (i.e. an aggregate of multiple Likert items) may be okay since there are also enough people who treat Likert scales as interval data.
i agree with the see what makes sense approach, but I (at least) find that the numeric tests can indicate a range of possible solutions to examine, which can then be tested by both numerical means and their consistency with the measure and the aims of the study.
It seems like imputation would not work for covariance estimation.
The question actually seeks the distribution of the MLE of $\\text{E}(X)$ rather than the MLE of $\\text{E}(\\ln X)$ and similarly for the variance. The quantities you give are related to the answer but it's not quite so simple.
This is helpful. I've added more details in the question description. Any thoughts?
You are right that this is widely regarded as dubious, but how dubious? You would not fault yourself for trying different hammers or screwdrivers before deciding on the best for a job. How far is this similar?  On the strictest interpretation, one should specify the tests to used in advance and use only those tests. That's admirably puritan, but I'd guess it's minority practice.  In what I read, I often think about a test that has been used as just another kind of exploratory data analysis, especially when I would tend to favour a different test or no test or all (e.g. a graphical approach).
Matlab provide an example here: http://www.mathworks.co.uk/help/stats/feature-selection.html#brluyid-1
NO. I am asking for a test than can tell if there is a root(s) greater than 1 in the AR(p) process. I am not asking for integration order.
I deleted my answer, which suggested using `AIC`, because a comment made a compelling case that AIC is not generally applicable for selection of `nls` fits. I would always try to decide for a nonlinear model based on mechanistic knowledge, particularly if the data set is as small as yours.
A couple questions, not directly related to your question, but which may be helpful: 
I am not sure what you mean by "splitted the data". 

Why is calendar year categorical?
"then it would make sense to leave those cases out of the bootstrap sample, as they are a segment of the population that we do not wish to make any inferences about." I'm saying allow the bootstrap to sample these cases, remove them from the model fit upon the bootstrap sampled population. Doing this allows the effective sample size of each BS iteration to vary. This way, the $p$-value's distribution under $\\mathcal{H}_0$ depends on sample size uncertainty (i.e. not knowing how many cases in a fixed sample from an imperfect population will need to be discarded)
GradientBoosting is being applied to all the other data that was given me.
That mirror was last updated in October 2003. It's better to use http://cran.r-project.org/web/packages/
OK, here it is binned at 1% intervals. What now? http://i.imgur.com/xfyCI5o.png
Is this correct or I need to modify this.Please help
It is spelled _Somer's_.  The generalized $c$-index is simply computed by backsolving the equation I listed above.  Internal, all possible combinations of observations having different $Y$ values are examined, and the fraction of such pairs for which predictions are in the same order is the estimate of the concordance probability.  I misstated one thing: The `orm` function uses Spearman's $\\rho$ instead of $D_{xy}$.
I've made a few edits to clean the exposition up a little. I plan to come back and do a more detailed check of the calculations at the end.
It might be useful to give us more details, show plots, etc. to judge your approach. I'm still not quite certain what you've done exactly.
What has been seen cannot be unseen. I find that shadowgram to be one of those plots I'd rather forget about.
@rapio thanks. This is what I was looking for
If those ~1k students are really your entire population, then there is no inference, no question of significance, no need for normally distributed residuals, etc. You may not be interested in generalizing to *other* schools, but presumably there is a theoretical population of possible students (applicants?) who could have been offered scholarships beyond those who happened to have been.
It is also crucial to me since I use this for the extraction of static factors.
Agreed - hence 'on the list to translate', but I don't know R nearly as well, and while this bit is easy, the extended parts of the project are considerably more complicated, and have existing implementations in SAS.
I would like to respond to both of you in a single comment, but apparently I should duplicate my response.

First let me express my huge gratitudes to both of you. You helped me unlock this dilemma, and learn something while enjoying solving the problem efficiently.

Both of your comments clarified the multicollinearity problem in my case. Especially when Greg explained it as the height*weight example. I started to construct as many as possible different models, by adding one by one each variable. I don't mean stepwise regression, but different regressions where I manually added new variables.
R is too slow for what I want to do, I have millions of calculations I must do.
+1 because of the hands-on point-of-view. But, you can't hide from the fact that this approach enforces a diagonal error structure $\\Sigma_E$, ie. if $E \\sim N(0,\\Sigma_E$), then $\\Sigma_E \\sim D_E$. Yes, in the case of OLS that would come around OK but of the top of my head I can't remember any results for ridge regression. Check Izenman 2008, Chapt. 6.2 for more details.
Hi thanks for the comment. The 'id' represents the participant number in the experiment. The first six measurements from each DV come from the same id number (id==1). The next six are from id==2 etc. This might be clearer if you output the whole data, not just head(data). Hope that helps.
(+1) Unfortunately, bootstrapping the median of a small sample won't be very informative, either--and is unnecessary, because it can be replaced by a simple calculation.  (For any number $t$, ask yourself, what are the chances that more than half of a bootstrap sample will exceed $t$?  That answer is easy to come by, and now you don't need to run any simulations to estimate it.)
I have a two years of hourly data and I just wanted to see how forecasting methods work so I picked two weeks. Hence, naive model does not work for me.
Questions about error messages from software are usually off-topic on CV (see our [help center](http://stats.stackexchange.com/help/on-topic)), but are often on-topic on [Stack Overflow](http://stackoverflow.com/). This 1 may be on-topic here, depending on whether the error has to do w/ understanding rob reg or R, but it would definitely be off-topic on SO, b/c it lacks a [reproducible example](http://stackoverflow.com/q/5963269/1217536). Moreover, we can't help you here w/o 1 either, b/c it's impossible to tell what's going on. Can you add a sample of data & code that reproduces this?
(+1) Let's say my vote goes to both of you :-)
I picked 250 trees and 5 node depth and got great results but an not sure why. The data set is at https://d396qusza40orc.cloudfront.net/predmachlearn/pml-training.csv I removed seven features that were unrelated. The predictor is classe.
+1 for an *excellent* answer.
This is, at bottom, an issue of identifying and quantifying separate *components of variance.* Your weighting is appropriate when *all* variation can be attributed to the measurement error in LOS. Because you're doing a regression, there will likely be residuals: they will include a separate (independent) variance component. If, eyeballing the scatterplot, it appears the regression residuals will be larger than typical LOS SE's, then you are probably OK not weighting anything. The problem is more challenging otherwise, so first it would be good to do this check! What does it tell you?
That my null hypothesis is flat in $x_1$ and $x_2$ doesn't mean that it is flat in some transformed quantities which lead to my measurements being distributed flat in the available parameter space, right? Why do I not need to worry about this? And I am dealing with hundred millions of observations, so histograms are just easier to use for me.
@MichaelChernick, see the above edit to my response in answer to your comments.  I think it is possibe that you are thinking of regression towards the mean in a different context than I am, so I added more information and a link.
@Simone I think the distinction is whether you are evaluating the particular model, or the process used to generate the model.  In this case the former is likely to be very difficult as it sounds as if there isn't really enough data to generate a stable model (and hence the test data will be too small to give a stable estimate of generalisation).  The overlay method (or bootstrap estimates of the error bars) are more estimates of the performance of the method used to generate the model, at least the performance of the particular model should be within those error bars.
There's loess, splines, robust regression...probably other methods too.
Please, don't [cross-post](http://stackoverflow.com/q/16125368/420055) on multiple sites.
Does [the Wikipedia article](http://en.wikipedia.org/wiki/Efficiency_(statistics)) help?
The presence of an outlier that badly affects your estimate suggests that the model may not fit well. Quantile estimation is often less affected by outliers, yes, but it's not *always* "more robust" than MLE. In some situations it will also depend on the type of robustness you mean.
1: okay. 2: You're wrong, see Rubin's Statistical Analysis with Missing Data 2nd ed. CC is unbiased with MAR data, MI propagates bias in NIM. 3: Read some NEJM / JAMA articles. If you observe patients after discharge, you should do time-to-death analyses, using time-varying covariates for discharge. You can model logistic regression for 30 day discharge among survivors, but be careful of interpreting results. 3: shouldn't matter, then, use one to predict the other and vice versa, they are strong proxies 4: that's not a collinearity issue. You use outcomes as predictors in MI.
I'll let you know which it is when the papers are marked :)
Could you please clarify "it doesn't appear that $Y$ is binary". In the question I said it was continuously normally distributed so this comment is a bit confusing to me.
tahnks for the answer. I am not surety understand the procedure.
1) and 2) get the grid of crosss validation error 3) find the best pair of parameters ? i don't know the step 4 and 5 thanks
Hi Ben, I have just edited my query to include the package and the summary output of the first GLMM, thanks in advance if you can help me to understand the relevant bits!
Assuming you are thresholding at the output to get your categorical result, what threshold are you using when predicting the class? Did you try plotting P-R curve and finding a more suitable threshold?
Thank you so much for the unexpectedly fast and simple worded guidance.You must be an excellent teacher. My treatment is very simple and has already saved thousands of lives.
Can you provide an example of the outcomes you consider in your experiments? What is your sample size (in each group)?
+1, Note that there is an intrinsic ambiguity in "3% increase". You correctly state that this is a 3% increase *in the odds*, however the way OP's question is worded suggests they believe that it is a 3% change in the *probability*.
@jbowman, I will state my exact question. I think that would clarify most of the things.
@Dnaiel, note that [Bayesian HMMs](http://en.wikipedia.org/wiki/Hidden_Markov_model#A_two-level_Bayesian_HMM) are another setting in which [hyperpriors on Dirichlet-Multinomials](http://stats.stackexchange.com/questions/44144/multinomial-dirichlet-model-with-hyperprior-distribution-on-the-concentration-pa) are used.
Of possible interest: [What is the best way to identify outliers in multivariate data?](http://stats.stackexchange.com/q/213/930)
Hi, thanks for the edit. I was looking for a way to include LaTeX but a quick search didn't tell me that. My thought was that standard error might be applied to the terms of the relative risk CI calculation and propagate through to the CI, but that's just my intuition.

Here's some example SAS code I used to obtain standard error:
'proc surveyfreq data=data.data_set_name;
	tables disease*var1 disease*var2 disease*var1*var2;
	weight WEIGHT_VAR;
	cluster CLUSTER_VAR;
	strata STRATIFYING_VAR;
	strata YEAR;
run;'
This corresponds to the syntax recommended by the sample designers.
@Zen: I'd heard of bootstrapping, but never thought it would be something that happened to me. Seriously, I'm trying to make it work, but it's time consuming to resample 1000 times (each resampling having $2^{28}$ points). Even 1000 resamples of size $2^{16}$ takes me 2 hours.
can you tell us a little bit more about the design of your experiment? (Do not answer below as a comment, but instead update your question, please.)
Have a look at Zipf's law
@john Thanks for the explanation. It's very helpful.
Are these results obtained with `maixt=10000`? I get always the same result if I specify this number of iterations.
Your question title says _Multiple_ **bivariate** normal distributions instead of Multivariate normal distributions. Is there an implication that $(X,Y)$, $(X,Z)$ and $(Y,Z)$ enjoy bivariate normal distributions but that $(X,Y,Z)$ are not multivariate (or trivariate) normal distribution?
Interesting problem. Could you discretise the continuous domain and there for treat both problems as discrete? Otherwise you're right, you cannot compare log-likelihoods. In order to compare the evidence of two models you need to supply them both with the same data.
You're right, @gsk.  E.g., `pisum` (at https://github.com/JuliaLang/julia/blob/master/test/perf/perf.R) takes 7.76 seconds while a simple rewrite using idiomatic R (`replicate(500, sum((1 / (10000:1))^2))[500]`) takes 0.137 seconds, more than a fifty-fold speedup.
The use of circular graphics for tabular data with Circos has been evoked here too, http://stats.stackexchange.com/questions/3158/what-is-this-type-of-circular-link-visualization-called/3159. My opinion is that this is a good way to reduce large symmetric tables, but it is less useful for summary tables with p-values and the like.
Stepwise selection for multiple regression is a bad idea. If that doesn't make sense, or you want to understand why, it may help you to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856).
Do you want (a) each class to have equal chances of appearing (a distribution that is uniform for the classes) or (b) each class to have exactly the same number of elements in the sample (a stratified sample)?
If you're going to crosspost (which is, generally, discouraged anyway), it is always good to indicate this prominently in the post and provide a link so that the different versions can be tracked. Cheers.
Check out http://stats.stackexchange.com/tags/self-study/info for guidelines here. Here are two hints: there is no absolute reason for a confidence interval to be a 95% interval; there are many hypothesis tests other than z-tests.
I think positive (semi-) definiteness of the kernel matrix for any sample of data is pretty much it, as then the kernel matric can be interpreted as a covariance matrix computed in some feature space.
(+1) I love the implication, in the opening of the third paragraph, that linear regression is done with infinite (non-"finite") samples!