tttjjj/elt_bench huggingface dataset:
If the predictors are spatially autocorrelated then this will introduce collinearity. There are many many questions on this site that address your subsequent questions - I suggest typing collinearity (or something like that) in the search box above or looking at other questions with the associated tags.
Thanks for the response. It being the real world, I am certain that there _are_ multiple causes, as you suggest. Regulatory issues, access to funding etc etc probably also play a part. With that in mind my aim is not to present this as the only thing, but rather as an important factor in a multifactorial problem.
Interesting answer, I actually thought that the "likelihood school" was basically the "frequentists who don't design samples school", while the "design school" was the rest of the frequentists.  I actually find it hard myself to say which "school" I am, as I have a bit of knowledge from every school.  The "Probability as extended logic" school is my favourite (duh), but I don't have enough practical experience in applying it to real problems to be dogmatic about it.
BTW, having motivated you to iterate the simulation, I should offer a way to do it quickly (because on my system, which isn't shabby, it takes 24 seconds).  Here is a two order of magnitude speedup (0.5 seconds): `n <- 10^6;
system.time({
  s <- matrix(sample(1:3, 5*n, replace=TRUE, prob=c(20,15,5)), ncol=5);
  f <- function(i) ((s[,1]-i)*(s[,2]-i)*(s[,3]-i)*(s[,4]-i)*(s[,5]-i) == 0);
  x <- f(1)+f(2)+f(3) < 3
});
x.bar <- mean(x);
se <- sd(x) / sqrt(length(x));
(x.bar - 2461/4096) / se`
Babak, a wholesale give-away of reputation is viewed by the SE team (over whom nobody here has any control or influence) as an effort to game the system and it will likely reverse your actions, which is just a pain for all concerned. Although we appreciate the sentiment behind your intention, if you really plan to walk away--and obviously nobody wants that--then please consider whether you can accomplish something constructive in doing so (*e.g.*, if you are leaving due to some defect you perceive in the site or community, please point that out and suggest a solution).
Thanks @Tony_Frame. This is the method I was going to try first but wanted to see if anyone knew of a different way.
@chl: who said anything about tuning parameters? If additional things are performed, they should be repeated during cross-validation as well. Clearly modifying your algorithm until you get good cross-validated error rates is "cheating". BTW, I agree that cross-validation, especially leave-one-out, is not all that great.
Now that I have updated my questions. Would it be possible for someone to provide a more detailed answer?
Or, given a simple frequency table for `vec <- sample(1:5, 20, TRUE)` created by `table()`: `rep(names(table(vec)), table(vec))`. The result then needs to be converted with `as.numeric()` if the categories are actual numbers.
You will find a lot of information in the following pdf: http://cs229.stanford.edu/notes/cs229-notes2.pdf
See http://stats.stackexchange.com/q/1207/159
(+1) Very nice, thanks.  I think the colors might distract a bit from the purpose, but the rest was a nice idea and very nice explanation.
"PCA does not work well" please ignore this comment it not so important. the main thing here that I do not know how to tackle this classification problem.
What's alpha? Your regression doesn't contain one.
I would have suggested something along the [point processes](http://en.wikipedia.org/wiki/Point_process). But I don't know how to do it exactly, you can look at this [thesis](ftp://ftp.dina.kvl.dk/pub/Staff/Morten.Larsen/papers/thesis.ps) for some ideas...
Thank, I'll read up on that!
Apparentely, Jensen's inequality holds with equality if and only if the random variable is a point mass. Now I have to prove this! It's so hard to find a proof for this!
Your approach does make sense to me, and I guess the significance on the main effects (with the interaction term being insignificant) would imply additive.
If you present standardized effect sizes, the last issue goes away - the reader can see that it was significant, but also that the effect was small, and interpret the findings accordingly.
Excellent points!  Also, @Noah41 - since you say you obtained a "rotated component matrix," I'm betting you used principal components analysis when exploratory factor analysis proper would have been much better.  A very common mistake.  See http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi
Based on that, I believe a [multidimensional cell array](http://www.mathworks.com/help/techdoc/math/f1-86528.html#f1-87418) is what you want.  Cells in a cell array can be of any type (they don't have to be consistent among each other), and the multidimensional aspect will allow you to extend the array beyond 2D.
An example from Peter Huber, http://www.amazon.com/Data-Analysis-Learned-Probability-Statistics/dp/1118010647/ y is expenditure on cosmetics, x is income, g is gender. In each case expenditure increases with income, but women have higher intercept and lower average income. Similar to the diagram above: schematically women are black data points, men are red. I'm not clear whether the data were real.
This looks good. Thanks.
Thanks for your answer! I'm still a bit confused about the synthetic data set. Are the values from the features drawn randomly with or without replacement?
I was just summarizing the results of these regressions--I didn't force SAS to compute the results without an intercept or anything like that (is that even possible). I was just ignoring the intercept as one normally does when one is interested in the effects of predictors.
@dernier It will also tell you if there are significant differences between pairs of species: `mod <- lm(cbind(Weight, horDiam, Price, Dummy) ~ Fruit, data=myData); summary(mod, test="Pillai")`. If your test for differences will be post hoc, you will have to figure out how to implement a multiple comparisons procedure yourself, because neither `TukeyHSD` nor `multcomp` seem to work on multivariate models. It is not standard practice, in my field at least, to create groups for further analysis in this way.
Those datasets seem to be rather short - several thousands of values at most. Are there long sequences, and sequences with some extreme values to test for overflows and precision loss? Also, are there statistical properties of the "higher order" available for these sets, like skewness, kurtosis etc.?
In general, the statement in the title is not correct. It is correct for rare events because OR is approximately equal to relative risk in that situation.
Do you know if servers are operational without failures? If a magic fix makes the server not fail ever again, would the long period of non-failure show up in your data?
Note, `?pbinom` gets you the help.
Just to add, you might want to transform $N_d$, $W_{ud}$, and $L_{ud}$ in some way prior to using them, for instance by taking logarithms or constraining them to be no greater than 10, otherwise extremely prolific/older directors will tend to get relatively low scores.
Nobody knows $\\beta$, surely? How is anybody computing $X'\\beta$?
For a univariate data set, MAD is a measure of the variability (i.e. what is the "average" deviation from the "average" value, using the median function to determine averages). So as you compute it the first way, it tells you the variability among your errors, which isn't necessarily meaningful as a measure of how well your model fits. In the latter way, you are computing something very close to MAE, but using median instead of the mean. This is a valid way to determine your goodness of fit, but I don't know why you wouldn't use the more common measures of MAE or RMSE.
I still don't fully understand lme4 syntax, so I'm curious to see the answer. But I have a hunch that it's related the following difference: P is the amount of time a student spends doing homework, R is a treatment at the class level and F is the student. (We'd should also have a random effect for the class itself.) If all students are subject to all treatments R at different times, the levels of F are comparable across classes. If we measure a whole school all at once, we have different students in each class, so the levels of F in different classes don't have anything to do with each other.
Valid for what exactly? Sure, you can average them [you can multiply them, too](http://stats.stackexchange.com/a/432/7290).
could you specify the nature of data(continuous or grouped data or count data on each of variable. The two measures of correlation are quite different in terms of purpose.
Which is more powerful depends on which alternatives you consider - since they measure nonnormality differently, each has good power against slightly different things. As general tests of normality, both are excellent - with really good power against a wide range of alternatives. Neither is uniformly better. Do you need power studies?
@Scortchi you may be right. But then - looking at the table - more senile cohorts proved to be more hale guys.
@user20650 Not so to "facing in the same direction". Standardization by mean and SD preserves correlations, positive and negative, or otherwise put, correlations are based on standardization. So standardization will change no signs of relationship.
What do you mean by "determine joint and marginal distribution"? If you mean determine the theoretical distribution of some random varible, R won't be able to help you. You could obtain an empirical estimate of the distribution using R
dober dan! do you know if the official law is designed for roulette or for some other gambling devices? (slot machine, poker, etc) The third requirement sounds like autocorrelation, or it could be tested by Wald-Wolfowitz test, etc.
Thank you for the response. Speed is an issue, more because I have several thousand possible labels in a multi-label problem. The application is classifying a corpus of text data drawn from both twitter and analysts' description of certain events. I'm using tf-idf weighting and the bag of words model.
and you can introduce some non-linearity by computing for example [homogeneous kernel maps](http://www.vlfeat.org/mdoc/VL_HOMKERMAP.html).
I think you will need to expand your question to provide more detail if you expect to get a useful reply. Cheers. :)
@NickCox That is a downside compared to the more compact original chart, although you could rotate the whole graph 90 degrees if a landscape-oriented figure suited your overall layout.
@gung See the edit for brief details, including a reproducible example similar to the one with the boxplot. It's really doing nothing more clever than making use of several of the arguments to the `boxplot` function. Between `boxplot` and `boxp` you can do some rather nifty things with little effort.
I'll summarize the page I linked to.
Good point... The short paper tomswebpage.net/images/K-S_test.doc mentions the "Handbook of Parametric and Nonparametric Statistical Procedures, Fifth Edition by David J.Sheskin (Apr 27, 2011)." to offer a two-sample case construcion for D. But at the moment, I don't have access to this book.
R 3.0 (which is the current release) can access large arrays. On a 64 bit machine should be able to address that much memory if you have that much.
Normally I would not insist on the point about covering the case $y\\lt 0$, Alecos, but this is precisely the issue that was causing the OP grief, so I thought that dealing with it explicitly would be helpful in this instance.  As far as the indicator goes, your proposal doesn't work (even after replacing the "$Y$" by "$y$" in the formula so it makes sense). In order to use such notation you first have to expand the concept of a PDF to cover discrete measures, so you might as well write $f_Y$ explicitly as a measure, as in $$f_X(y)I(y\\ge 0)dy+F_X(0)d\\delta_0$$ where $\\delta_0(A)=I(0\\in A).$
It is impossible for anyone to answer this question until you explain how the 27 "actual values" were selected from the "14,000 observations".
Have you considered normalizing or scaling your data in some way so to reduce the domain?
Thanks: I fixed the link.  The previous one, which didn't work, was some kind of wild Google Docs URL.  Now I can tell the paper does the job (+1).
`mean()` in R has a trim argument http://stat.ethz.ch/R-manual/R-devel/library/base/html/mean.html
why do you say so? we do not see the $\\{x=0\\}$ part of the graph.
two *stationary* time series may be spuriously correlated as well. Just take simple textbook example, generate 100 independent realisations of Gaussian white noise process (very stationary process :) ), on average about 5 per cent of them will be *correlated* significantly with 5 per cent significance level. What is important - the theoretical background or deeper understanding of what's going on rather than a theoretical data-mining approach. Co-integration, Granger causality all concepts are useful, but sound model have to add some story behind the statistics it uses.
please be more specific. Since you do not put any conditions on the model or what are you trying to predict. Take for example two independent random variables $X$ and $Y$, regression will not increase accuracy of prediction of $Y$ using $X$, since they are not related. But that is probably not the answer you are looking for.
Can you clarify what you are asking about? I don't think this question is answerable in its current form. I am aware or robust  'sandwich' errors, eg, but those are for you betas, not for predicted y.
It would be nice to have a fully runnable example, that is, including the R code :)
The problem is that classification (as opposed to risk prediction) is an unnecessary dichotomization.
(+1) Good elaboration. Sorry I didn't see your answer before posting my comment. I think they must have been submitted almost simultaneously.
I didn't downvote this, but I can easily impute objections to it as it stands. First, there is little evidence here of research effort on matters covered by hundreds if not thousands of texts. Second, it is potentially too broad to be useful. That said, the question is, as it were, justified by the answers to date.
Sorry @RussLenth it's probably my fault I didn't explain it clearly. One replication would be 9 questions (1 questionnaire), which is 27 lines in the design matrix as each question requires 3 lines (3 alternatives). So I need 100 questionnaires with the same 9 questionnaires in each, which would be 2700 lines. (I tried those commands in any case but still had 'NA' appear after 27 lines/1 survey). I had a look through the help command but nothing mentioned that was relevant as far as I could tell. Thanks though.
I think I've figured out an answer, and also a better statement of the question, but it will be a while before I can post it...
Jack, please let me add my welcome to @gung's.  Would you mind explaining how the AD or KS tests could be used to "compare means/medians"?  I understand them to be tests of Goodness of Fit (to particular distributions), which amounts to a lot more than just comparing measures of location.
@berkay Ah so stripchart may make sense in this case! You can even superimpose individual (jittered) points onto a boxplot for each series of measurement. Either way, your y-axis = your measurement units, the x-axis = your two groups or a dummy variable coding them.
Indeed I use LaTeX. Also your approach avoids the pain of creating EPSs and importing them.
@Rosh $x$ has a lognormal distribution means that $\\log(x)$ is normally distributed.  Thus, substitute $x = \\exp(y)$ in your *original* integral.  The integrand is an exponential whose argument is a quadratic function of $y$.  Completing the square turns it into a multiple of a normal PDF, so your answer is written in terms of the normal CDF and the exponentials of the original endpoints.  There are many good approximations to the normal CDF (a multiple of the error function).
@Glen_b I've corrected the question. Do let me know if there's any more corrections needed.
I can`t caluating course. please show me.
Thanks. Do you think the beta binomial is a good model to fit the above data set? Actually the above data set can also be assumed to come from two binomial discription, the p is close to 0 and 0.5 repectively. Most of the data in this dataset follows the binomial discription which p is close to 0. Is there a better mothod to fit this dataset (estimate the p close to 0)?
To learn more about REML, see for example, http://stats.stackexchange.com/q/48671/3601 and http://stats.stackexchange.com/q/41123/3601.
If I test every individual on 20 different ways (being 5 different models and 4 different ages) then the number of higher level groups seem to be max 20 ---> doesn't multilevel modelling demands a higher number of higher level groups?
Much depends on how rigorous and formal one wants to be (and also on one's mathematical training, if any).
Thanks for answering my question! As I read from some stats articles, some researchers center or standardize the data. So I'm just confused..
On second thought: Won't `gmdistribution.fit(data,k);`, $k$ being the number of peaks you expect to have in this case, give you what you want?
@AlecosPapadopoulos      an observation $(x_1,x_2,x_3)$ of $(X_1,X_2,X_3)$
When you say throwing away values is not an option, do you mean that there's some reason you can't throw *any* away, or that you can't throw as many away as a naive approach would suggest?
There are plenty of proofs on this site.  I can point you to http://stats.stackexchange.com/a/71303 because I wrote it, but I'm sure that searching will turn up others (including others in that same thread).
How could there be an interaction? You have only 1 explanatory / predictor variable.
I'm not sure I understand that bit about the half width confidence interval. I'll also edit my post to explain more clearly what I did.
@julieth, thanks for reply. The first figure should not show any dependance. There is no any expectation of any such trend, but I was wondering if this is normal to have plots with some trend after fitting to linear model.
@Ian Interesting, I hadn't heard of the singular Normal.  Still though, in this example I don't think this distribution is necessary, because as you said (and I implied), one of the events is degenerate.  Hence, you can lop off any row/col.
This may be helpful: http://itl.nist.gov/div898/handbook/prc/section4/prc426.htm
@Peter, I think Blain has taken liking for nonlinearly "optimally transformed" variables; this procedure needs first to categorize them (e.g. scale into ordinal, before back to quasi-scale).
+1 While I agree that Tukey's test makes the same assumptions as the ANOVA, I am concerned more generally about the ANOVA itself - if the data is transformed, the hypothesis on the original, untransformed scale is no longer about means, and estimated effects won't relate to mean differences. Often that won't matter, but sometimes it might.
Thanks @peter. The 9 points are out of 210, so it's not surprising that they didn't yield much.
It's possible that your features just aren't strongly related to your classes, at least in this sample, and that the relationship that does exist is well captured by the linear methods.
I think Uri Simonsohn's argument wasn't that assumptions per se are bad, but that power analyses in general are so sensitive to assumptions as to make them useless for planning sample sizes.  However, your points are excellent, as are the references that you provided (+1).
Given that the classical bootstrap can be thought of as a computer-implemented maximum likelihood method (i.e. a non-bayesian (flat prior) technique), would it be better to rephrase your question to something like "when to use frequentist vs. bayesian technique?" Some background on bootstrap: http://stats.stackexchange.com/questions/18469/connection-between-bootstrap-both-parametric-and-non-parametric-and-maximum-li
No there are 24 lag parameters running from 1 to 24.  But you would not necessarily use all 24 in computing the Ljung-Box test because you need areasoanble number of point in the computation of each lagged correlation estimate.  The degrees of freedom for the Ljung-Box test would be the number of lagged correlations used in the formula.  It would be something less than 24.  I am not sure if fitdf refers to that or not.
@whuber How about the idea itself?
@kkp If the units of measurement are of no interest, why not simply showing the target difference A-B for each condition (superimposed on the same plot for the 4 labelled experiments)?
By the way, on the most right of Figure 2. Is there a mistake of the last two equations?
@TomekTarczynski: I have added one more point in favour of survival analysis.
Assuming that nPm means the binomial coefficient $\\binom{n}{m}$, 25P6 is the number of subsets of size 6 that can be chosen from 10+15 = 25 people, but this includes subsets of size 6 chosen from just _one_ shift, does it not?
It would be good to additionally cluster your standard errors at the individual level.
"test" $\\neq$ "report"; do you mean "test the coefficient for my variable"? Or are you more worried about measurement of the effect than testing it?
Strictly speaking, your restriction is incompatible with your definition. Perhaps $logit(x)$ is your AR(1) process? In any case, I think we need more information on what your data is and what your goals are.
As underlined by @cbeleites, the statistical sense of a sample (a set of measurements) differs from a common scientific sense, often a chunk of material (as in chemistry, geology, environmental science). Depending on field, "more samples" (scientific sense) and "more observations" (statistical sense) could even mean the same thing. Regardless of that, a bigger sample with the same design will not of itself remove or even reduce bias. Think of establishing the fraction of heavy drinkers by going into a bar; sampling more bar drinkers won't reduce the bias imparted by sampling in a bar.
@Kari Please, register your account on this site. You will then be able to link it to math.SE.
Without following individual cabbages any assumptions you'd have to make to get an answer would be very strong ones.  Do small cabbages continue as the runts of the litter or do they put on a spurt to catch up with the big ones that have already grown up?
You have received some good answers to your previous questions.  Is there a reason you haven't yet accepted any of them?
Can anyone give some feedback, please?
For the coin, see Andrew Gelman & Deborah Nolan's article in *The American Statistician*, [You Can Load a Die, But You Can’t Bias a Coin](http://www.stat.columbia.edu/~gelman/research/published/diceRev2.pdf).
Because you're dealing with a stream, I think time series analysis may actually be more appropriate than spatial methods. In spatial analysis, we typically think of each observation being correlated with all the points around it. But in your case, it seems that downstream events would be correlated with upstream events, but not vice-versa.
(c) asks for a numeric covariance. So you need to give a number.   What does $XY$ mean? What's $E(XY)$?
we have developed a very efficient and parallel clustering algorithm, if you are interested, please let me know
That in no way deals with the question, it just shifts it one adjective. How is a *normalized difference* a ratio?
@davips Generally if a statistical method is going to be used for some practical purpose, then the users will often want to have some idea of how well it works (e.g. medical screening test).  Also if you are developing a machine learning algorithm then it is useful to have an unbiased estimate of how well it performs compared with competing methods.  It is also a useful means of validating whether the method actually works (which is invalidated if the cross-validation is used both to select parameters and estimate performance).
Do you have some specific examples in mind?