tttjjj/elt_bench huggingface dataset:
@RyanThompson I agree, and have flagged it for migration.
@FeralOink that question is about a quite different distribution. Just as the uniform distribution is not the same as the normal distribution.
Hi! Thanks for the answer, The first possibility is actually the reason why. I used LOO-CV instead of stratified k-fold. The result is now the same even though I shuffled my training data.
Thanks for your reponse.Should i perform a data mining approach over my LDA result ?
I am undecisive whether this question is brilliant or complete nonsense, i.e. I am not sure whether there is a ML perspective on this discussion Can you name an example/algorithmn/field of application for each reasoning in ML ?
Just another point - I won't have time to run any more models, so I'm really just looking for a way to interpret what I have found from the models I have run. Of course, if I have run the wrong models, that's also OK - I will just report this to my teacher - I am self studying statistics in the holidays and my teacher is helping to guide me. He gave me this data to make a report on, so I just have to interpret what I found. If I did it wrong, it's OK - I guess I can also say that :)
Could you provide a bit more information about your problem: What is the dependent variable, how is it generated? What are the independent variables? Why do you think that a skewed DV would be a problem? Note that multiple linear regression makes *no assumption* about the distribution of the dependent variable.
Unfortunately, stepwise selection won't do a good job of that, except possibly under ideal circumstances.
Do I take it right that you are using the dummies as if they were continuous variables?
"Fit their data to a theoretical distribution" sounds backwards to me. *Some* people fit a theoretical distribution to their data in order to describe or understand the data, but extremely few people actually modify their data to conform to distributional assumptions (which is what your use of "fit" really means)--and their work is generally considered fraudulent.
As the question is stated, there's not enough information to answer. Do you intend the 'picking' from those ranges to be done (i) uniformly? (ii) independently?
(ctd)... I think a proper answer to this would have to get down to discussing suitable models. I am not sure I'm the best person to suggest models in this application
Welcome to the site, @Marc Schubert. Please don't sign your posts. You may notice that your username, your identicon, & a link to your userpage are automatically attached to your posts. Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about), which contains info for new users.
You should start from what you want to know/learn about these data. Also useful would be information on the number of patients. Was there a control group?
Oh, the frequency is also necessary, isn't it? This is what was strange, when I read it I thought only the frequency was necessary.
Hi Michael, thanks for your answer. I think I understand why we need control groups. The control group is incorporated in my regression equation as those who don't have treat=1. So that's not really the question here. The question is why some papers use additional control variables on top of the equation stated above. Would be great if you could answer on that or maybe someone else. Thanks guys!
Also Altman's book with Chapman & Hall/CRC "Practical Statistics in Medical Research" has a chapter on the medical literature where many statistical sins are revealed that occurred in published papers.
@Nick Standardizing to zero mean unit variance over the whole dataset actually reduces the dimension by two: it projects onto a (not *the*) sphere within a codimension-one subspace (namely, that orthogonal to the vector $\\mathbf{1}$).  So, technically, "the" and "unit" may be incorrect characterizations in this case, but the basic conclusion stands: that the dimensionality of the data is reduced.
@bart -- I wish I could offer a safe automated way to do this but I don't think it exits.  See above for edits w/ a little more detail.
On the [lme4 project](http://lme4.R-forge.R-project.org/) page, there is a [collection of articles and other research](http://lme4.R-forge.R-project.org/bib/lme4bib.html) using `nlme` or `lme4`.
Exp(B) of .986 says that the estimated _odds_ `Prob(dancing)/Prob(nodancing)` increases by .986 times (i.e. slightly decreases) in response to taking one more beer.
I'll probably go with the Wilcoxon signed-rank test, although the first option is a nice idea. Thanks for clearing things up for me.
@Srikant: Sorry, I added some links to the question.
I've merged your two accounts (as pointed out by @gung). You may want to [register](http://stats.stackexchange.com/faq#login) this one once and for all.
Cowpertwaite and Metcalfe in Chapter 3 of Introductory Time Series Using R (Springer) seem to recommend that a forecaster may want to adjust upwards a smoothing constant that has been optimised for a long and stable time series (which will give a very low estimate of the smoothing constant under most optimization routines) if there is reason to believe that recent events should be taken more into account - citing video tape sales as an example.  But perhaps this just shows the limitiations of any automatic forecasting during or just after a game changing alteration in the environment.
"Small" compared to what, @Sven?  See http://stats.stackexchange.com/questions/37522 for an account of the issues.  (You and I know that mean residuals for a linear model with intercept should be exactly zero, but nevertheless an order of magnitude difference between two computations could flag something serious going on and is worth looking into: I applaud the questioner for paying attention to this.)
There's an error in your expression for $P(Y=1)$. You should have $$ P(Y=1) = \\frac{1}{1 + \\exp \\{-X \\boldsymbol{\\beta} \\} },$$ not  $$ P(Y=1) = \\frac{1}{1 + \\exp \\{ -(X \\boldsymbol{\\beta}+\\varepsilon) \\} }$$ The randomness in a logistic regression model comes from the fact that these are bernoulli trials, not from there being errors in the success probabilities (which is how you're written it).
You should add to your list of assumptions that the variances are equal. On another note, if your alpha is set at .05, & p=.04, then you *would* reject the null, & the question of the test being "underpowered" is a non-issue. Lastly, you may find this question of interest: [Is there a minimum sample size required for the t-test to be valid?](http://stats.stackexchange.com/questions/37993/).
All R functions do not necessarily wait for a distance matrix. See e.g., the on-line help for `agnes` in the [cluster](http://cran.r-project.org/web/packages/cluster/index.html) package.
@onestop, the article in the link mentions stata, so probably stata is used. Does not matter though, since any statistical package would encounter problems in this particular case.
@RichieCotton That is very interesting, I didn't know this was done.  Thank you again!
And by close i mean kl divergence is minimised
I think this answer will be helpful http://stats.stackexchange.com/questions/68243/ar1-coefficient-is-correlation/68246#68246
(If a variable doesn't vary, it's not a variable!) You cannot validly compare two fits without knowing something about the independent variables. Imagine drawing scatterplots of the two datasets: the independent values tell you their horizontal positions. The scatterplot changes radically when you shift one group of data horizontally relative to the other and so, therefore, will the combined fit.
It's not as simple a dichotomy as that; after all, an AR can be written as an infinite MA and an (invertible) MA can be written as an infinite AR, so if either is ever appropriate, arguably so is the other.
First, if you try different models until you hit something that is better than $X$% (in your case $X=85$), you can not in general expect this accuracy on your test set, due to [over fitting](http://en.wikipedia.org/wiki/Overfitting). Second, if you only need a 85% accuracy, and over 85% of your data is "stayer", then it will be sufficient to classify everyone as "stayer". Third, if your features (demographics, I guess?) are not informative enough, this may be the best that you can do. You can probably do better, but I'm not well versed enough in decision trees or your data to comment on that.
It might help if you explain your data a bit. Do you have performance of firms before and after repurchase announcements? What does "adjust" mean?
Thanks. Welch's test and the GLM were also things we recommended to him (though he said they were too complicated), so it's good to see that we were on the right track. The ANOVA suggestion seems stranger and stranger.
@Zhubarb. To my view, intractable integration enforce us to work with the joint densities in place of the analytically marginalized it and thus to increase the dimension of the problem, digging a bit further into the curse of dim.
Dear Charlotte, this is very helpful, thank you!
Thanks Brian, makes more sense now.
Usually one-line answers aren't regarded as appropriate, and in this case, you should probably expand on it to also respond to your second question (presumably in the affirmative).
In addition to my answer, I have a suggestion for learning. While practice problems are absolutely essential, you should also study some of the basic theory behind regression and statistics in general from a cohesive source such as a textbook. Being able to carry out a regression analysis in R won't mean much if you can't identify the appropriate times to apply each method and interpret the output.
This is one multivariate logistic regression with 3 regressors, A, B, and C or 3 univariate logistic regression models? If it's a multivariate model, are you interested in testing a global null hypothesis or three one-way tests? If there are multiple tests, are you doing any size correction for multiplicity?
What problem(s) do you run into when you simply regress `lILC6` against the original five variables?  Since the third PC still accounts for 20% of all variance, with 100-78 = 22% left to explain, none of the remaining two PCs (not shown) can have more than 20%, whence each has at least 2% (and likely more): this implies the eigenvectors don't vary a whole lot--the ratio of the highest to the lowest is at at most $\\sqrt{30\\%/2\\%} \\lt 4$, indicating there should be no multicollinearity problem at all!
OK, the manual says: It is the hessian of " pi, log(zeta), log(delta), and mu", but how can I get the hessian of pi,zeta,delta and mu? So without the logs?
AYD That is not the difference. Your question is about a sample space of positive integers and it concerns the random variable $X$ which assigns to any integer the last digit of its square. It asks for the probability that $X=1.$ No answer can be determined until a probability is given for the entire sample space: that is the point of @Glen_b's answer. Many errors in probability calculations originate with the unfounded assumption that when a random variable has $m$ distinct values ($m=5$ here), then all values must be equally likely. That is what Néstor assumes, without valid justification.
I don't know what you mean by "[e]ven if" here. With no sight of the data, I can't promise more than I have done. The other side of the argument is that the coefficient of variation can seem a natural summary for distributions like the gamma or lognormal. But inferential procedures for it seem poorly developed. Nevertheless, look for references in Sokal and Rohlf, _Biometry_. Freeman, New York.
Why do you claim P(no car in 5 minutes) is .5?
Note you should usually fit the binomial GLM with logit link against raw data (your `longdata`), not the proportions as in your Model 3.
Are you asking us to guess what you can find?
Ah-ha! I wasn't aware of that. I was double-checking the outputs with a z-table and they were not corresponding with the reported p-values. I assumed reviewers would do the same thing. In the cases where I am to report adjusted p-values from a multiple comparison, is it prudent to include the z-score as a test-statistic? Thanks for your help
@PatrickS.Forscher: Added at the end. Thanks!
@Graphth, I also didn't realize there was one... I didn't find it in the "all sites" list, could you link to it here? Thanks
I wonder if it would not be better to transform your problem to be more appropriate for your question. I mean instead of countries to have: home-country, foreign-country, availability-in-home-country, all as binary variables.
I suspect you're right, since I haven't gotten any answers to the contrary.  Still seems silly to me: in operational settings problems routinely arise because of inconsistencies between the way that data is represented during training of a model and its use.  PMML is an invitation to make those problems worse by sending models off into the world detached from the context in which their training data was generated.  Ah well.
Please explain what it means to "have" both the scale and shape parameters.  I find this statement puzzling here because if these parameters are somehow already determined--which is what "have" would suggest--then the mean is also determined (as [its formula](http://en.wikipedia.org/wiki/Weibull_distribution) attests) and there is nothing you can do to change it.
Thank you for the detailed answer @cbeleites!
@whuber +1 Yes, uncertainty in the estimate of the DF is a substantial caveat. For approximately Gaussian fields, estimating the joint density with ML would do substantially better. The question of "belonging to a set of objects" without the presence of a control or alternative comparison group is a question of consistency, i.e. how consistent is the observed point $y$ with the resampling prob dist-n of a fixed set of data. If $f$ were known, then the integral can be interpreted as a probability that a newly observed point having df $f$ belongs to any subset of $\\Omega$ containing $y$ that...
"*Heteroskedasticity means that x is correlated with the errors*" -- what leads you to say this?
@Amanda I agree with drknexus that providing mean alone is not very informative. [Here](http://imgur.com/I1EHWaA) is an idea of what the `summary.formula` function might give with two treatment groups and a mix of continuous and categorical variables (Source: Frank Harrell, `rreport` package, http://bit.ly/TkD7WU).
@Khader RF importance is more reliable than IG.
Thanks for a very elaborate and thorough answer! Really appreciated. 

I will try to have thisin mind. The exercise I have mandates that the exercise is to be completed using MiniTab software. Yur answer certainly gave me a higher insight into fitted models and the actual mathematic behind MiniTab, which I fathom is fairly similar.
Excellent. Thanks very much for the suggestion Anony-Mousse. I'm reading up on setting Eps and Minpts as I've not done density clustering before. It helps to have a direction to move in. Cheers.
I don't think that "fully Bayesian" has a rigorous meaning.
The rate $\\lambda$ has nothing to do with the pdf value in $1$, the correspondence is simply a first order approximation $1-exp(-x) \\approx x$...
@Penguin_Knight I'm using the Mann-Whitney test to try to make some real-world recommendations based on the regression. Without getting too specific, y measures whether or not people performed a certain desirable activity. So, if p1 was a significant predictor of y and significantly differed between people who did and did not perform y, then I might be able to recommend addressing p1 to increase the number of people who do y. However, I think the interpretation is a bit more cloudy if p1 doesn't significantly differ among people who did and did not do y. Does that make sense?
Welcome to the site, Ernie. This question is interesting and has good potential! However, I find the exposition in the first paragraph, particularly the second half, a bit too imprecise to follow. Can you clarify what "standardized sum" you are referring to when you claim it has a chi squared distribution with one degree of freedom? What parameters of the normals, if any of them, do you assume are known? I look forward to an updated post. Cheers.
I've never encountered that rule before.  Thanks for pointing it out.
BTW I have tweaked my functions in the pipeline so that it is a bit faster now, and I proceed with the analysis as described above... I also completely agree on the philosophy of looking for the robust signal in microarray experiments. I will be coming back to this post for sure.
I tried to make the text more readable by introducing latex. Please check if there are no mistakes. And please consider rewriting the question to make it clearer.
Exactly correct!
@Jonathan glad to hear it!  It's a wonderful package. Try the `modelLookup()` command for a list of what's possible.
This is an issue of practical vs. statistical significance. If the slope is truly different from 0, even by a miniscule amount e.g. .00000000000001), a large enough sample will yield a very tiny $p$-value, despite the result having no practical significance. You'd do better interpreting the point estimate rather than the $p$-value when you have such a large sample size.
Also of interest: "[How would you explain covariance to someone who understands only the mean?](http://stats.stackexchange.com/q/18058)" and "[How would you explain the difference between correlation and covariance?](http://stats.stackexchange.com/q/18082)".
Thanks. I know both papers. I am not a great fan of Witten [not Whitten] et al., as i think there are more important papers on sparse decompositions. On randomized SVD, I especially like the review paper "Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions" (http://arxiv.org/abs/0909.4061) also coauthored by Martinsson.
This question doesn't make a lot of sense - where is your statistical/scientific question?
@chl is correct. If you'd like a simpler introduction to simple linear regression than the one in Wikipedia, I wrote one on [my blog](http://www.statisticalanalysisconsulting.com/54/)
Perfect answer, this helps me a lot!
@muratoa, yes, I know the calculus derivation, but the question asks specifically for an explanation that focuses on intuition and avoids algebra.  I would assume that the question-asker knows the calculus derivation already, but is looking for something that provides more intuition.
@Henry, I guess by complicated I mean that it may be less smooth in some parts, say something like$$Y[x]=\\sin \\frac{10}{(x+10^{-3})^2}$$ I guess I am just coming up with something that needs more samples in certain parts than others. Please don't find fault with the exact function.
@whuber in the OP I read "if $X$ is a row vector, then $X′X$ will be a singular matrix", which is right because $X$ must be full _column_ rank. But if $X$ is a row vector (not the null vector) then it _is_ full rank: $rk(X)=1$ and could never be $>1$.
At a very wild guess, no periodicity at all. Seasonality usually means variations within year; other kinds of periodicity are best named informatively. My previous comment still applies: this appears to be about how some software treats your data, and saying nothing about the software cannot help us help you.
No problem Jay... try this: informally, residuals are "what is left unexplained". The X2~X1 residuals is what of X1 is not explained by X2. Now consider the Y~X2 residuals. These reflect what part of Y is not explained by X2 (and therefore is explained by X1 and by the error term). So even if you regress the original Y on the X2~X1 residuals, these will only explain that part of Y that is left unexplained by X2. So by using Y~X2 as the dependent variable, you "remove" only that part of Y that is explained by X2- leaving unaffected that part of Y that is explained by the residuals X2~X1.
The Bhattacharyya coefficient is a measure of overlap but it is not the same, is it ?
Not quite right, the 800 people come between noon and 2:00pm and we want to know whats prob 10 of them come between 1:00 and 1:02.
You might need 7000 positives before data splitting is a competitive method of model validation.  I wasn't trying to imply that you would just use the likelihood in the training sample (although that is better than using the likelihood from a very small test sample) but rather using resampling, e.g., 100 repeats of 10-fold cross-validation or 300 bootstrap repetitions.  And regarding likelihood you can convert it to one of the unitless indexes (logarithmic scoring rule or generalized $R^2$).
@B_Miner I have modified the second version of the code to include unit root with constant.
@cardinal - I just copy-pasted from the OP - but the last digits are the user id of the person who copies the link, e.g. my link would be http://math.stackexchange.com/a/73080/3733
It has the same asymptotic properties as GLS if the weighted sum of squares matrix based on the Omega converges to a positive definite matrix (then estimated gamma also converges to the same matrix,) and if it has a limiting normal distribution.
Maybe we're talking about some form of clustered or stratified sampling?
I've summarized some of the answers plus some other material at http://robjhyndman.com/researchtips/transformations/
I can't tell you anything about the book. But I noticed that Daphne Koller / Stanford is offering a free online course on [probablistic graphical models](http://www.pgm-class.org/). You should sign up!
This question has been asked and answered on [math.SE](http://math.stackexchange.com/q/713528/15941)
http://books.google.com/books?id=kXCg8B5xSUwC&lpg=PP1&dq=Statistics+of+Extremes+gumbel&pg=PP1&hl=en#v=onepage&q&f=false
I agree about the nonlinear fit, but why search for a P value less than some threshold? The answer depends on number of replicates etc. I'd suggest searching for a response that is far enough away from the maximum to be clinically or physiologically or scientifically interesting.
Are the categories mutually exclusive, that is there is no hierarchy linking them together? Also, I'm not sure to understand the second paragraph because it seems to say that an unseen document might belong to several categories while the training set only considers a one-to-one mapping. Could you clarify, please?
+1, thanks for the edit. Note that there is nothing in nature that requires interactions interactions to be harder to detect than main effects. The power of the analysis to detect an interaction depends on its magnitude & the N used for your study. It may well be, however, that the interactions that people care about in your field tend to be smaller than the main effects, & this makes them seem less tractable.
What do you understand about it now, anything? Have you tried [Wikipedia?](http://en.wikipedia.org/wiki/Mahalanobis_distance)
A more general statement of your argument is that $E\\left[X^k\\right]$ exists and is finite for $k=1,2,3$
+1.  Of note is that there are fairly simple formal procedures to investigate transformations, such as spread-vs.-level plots (described in Tukey's *EDA*).
It sounds like a good part of your data is right-censored. Look up survival analysis and right-censoring. A maximum likelihood approach might work for you here.
Thanks for your comment!

I'm still confused though. If you end up with a higher AIC in the chosen model, why was that model chosen in the first place? This algorithm is supposed to choose models that step-by-step lower the AIC and it ends up with a model that has a higher AIC. Is the step 1 AIC calculated on the basis of a higher number of parameters, and then the step 2 AIC is calculated with 1 less/more parameter? In the case of backwards elimination the parameter effect of removing a variable should decrease the AIC, all else equal.

Still very confused by this.
I took the liberty to make your question less dependent upon Stata (shorter title and highlighting of your question in a software agnostic way).
The title strikes me as slightly misleading. The question is definitely not what I expected when I first clicked on the link.