tttjjj/elt_bench huggingface dataset:
In general, if you have *any* data with a positive mean and nonzero SD, any sufficiently large number of SDs below the mean will be negative.  Given that this is a mathematical certainty, an appropriate reaction would be "so what?"  My question is, why isn't this the appropriate reaction to your question?  Why is -2 SDs special to you?
@J. Winchester I'm afraid your question still appears to focus on AIC, and responses to the related question on AIC apply as well here. However, I feel you are also wondering whether your `lmer` syntax allows you to test the hypothesis formulated in your first paragraph. If this is the case, you can simply *edit the title and your question* and focus on this issue, rather than AIC. You might also consider adding the output of `summary(lmera.1)`. Finally, the outcome you consider is a ratio of counts (% `Sp_A`): Is it usual in your field to treat this as a continuous response variable?
(Sorry, I am unable to format a comment properly so couldn't place a link to the paper)
1. The paper says "For n = 11, r = 0.961, and the probability that the data are normally distributed is approximately 0.3" AND "...typically a probability of less than 0.10, or perhaps even less than 0.05, is required before rejecting the hypothesis of normality" I understood that it means there is a measure of "probability" of normality.
2. I am checking normality of residuals from a model fit as a part of residual analysis.
Contd...
(+1) Doesn't $\\log(\\epsilon)$ need to be *subtracted,* as you said, rather than added?  Also, because this limiting argument lacks rigor, it does not apply directly to the case of delta functions--but it still gives a nice intuition concerning why entropy and differential entropy might have different properties.
...It is not always the best idea to blindly take the model which has the optimal AIC - other considerations are important including interpretability. To help argue your case, I would definitely point out that AIC only holds asymptotically and your sample size is small. If you show an appreciation for these statistical nuances then you should be able to satisfy the reviewer.
Welcome to the site, @user824624. I edited your question to add formatting and to make the English clearer. Make sure it still says what you want.
Welcome to the site, @Marie! You may test the coefficients of `year.codedY2011` and `year.codedY2012` jointly by either a likelihood ratio test or Wald test.
Nobody wants to answer? :( Why?
Whether the power goes down depends. If you have, eg, a single non-equal group mean, and you continue to add groups w/ equal means, power will go down, but if you add unequal group means, the power will go up. OTOH, your main point is clearly right: The significance level is controlled at the stated one. +1
@Youloush: ... given that there is other, still independent, test data to validate the the final tuned model as the former test data is now used for the tuning and thus not independent any more.
I agree. I performed a quick test of the A-D test, Jarque-Bera, and Spiegelhalter's test (1983), under the null, with sample size 8, repeating 10,000 times. The A-D test maintains nominal rejection rate, and gives uniform pvals, while J-B test is terrible, Spiegelhalter is middling.
are you using R?
@Dimitriy. Can you email me (rob.hyndman@monash.edu) with your IP address and a URL where you get the problem. Thanks.
TSLS is GMM with special weighting matrix.
Do you need to do this by hand or using statistical computing software? Here's a post on the mathematics page that may point you in the right direction: http://math.stackexchange.com/questions/228971/whats-the-relationship-between-singular-nontrivial-and-linear-dependent-basic.
Hi @MånsT. Thanks for your suggestion. Of course, it is a good idea. I was thinking about a standard de facto on this field, and why it could be useful.
Btw, Dendrograms only work for a very simple case of clustering methods, namely linkage clustering.
Maybe it's a RM ANOVA with a between-subjects factor? It's as if there weren't enough uncertainty in statistics already...
I added a good reference for doing CV the right and the wrong way. From the description you have just given I can't see anything you are doing wrong.
Welcome to the site, @JamesTaylor. Would you mind giving a summary of the info in the linked paper to help readers decide whether it's what they need & in case the link goes dead?
Perhaps you could define what you mean by confusion matrix here? If the OP were to calculate the class sensitivities of the model, i.e. aim to have balance scores for class 0 and class 1 sensitivities
Thank you, I have marked this as correct - but I needed to be more clear in my question... The study in question does not apply a survival model, (or either of these models), just reports the probability of an event. Example: Sighting probability of a bird, in sampled and controlled areas, with 95% Conf Int, and sample size. I imagine one cannot treat this as means to calculate Hedges g?
I do not agree AT ALL. MANOVA seems appropriate only for the configuration of the problem, but: (1) with 107 variables (and _just_ 98 observations), statistical power will be non-existent, (2) are you prone to assume multivariate normality in a 107-dimensional space?, (3) __Strong__ multicollinearity is expected among 107 variables (even if just random ones), (4) you will have no further knowledge of the variables "causing" group discrimination. Considering, focus in p-values is simply not-right.
How are you connecting poisson with chi-square? Use `qpois`
You usually have to wear a helmet to conduct extreme machine learning.
I changed it into PCA.
@PeterFlom Oh yeah I remember that paper. You posted it on Quora. I think it is time to come up with a paper with R code. I might do that.
Thanks for the quick response.  I hate to admit this, but I don't know how to run custom packages in R (I'm very new to R).  And I can't find this package in the standard R library.  Any tips about how I can run this script?  If the answer is I just need to get up to speed on R, so be it!
@Glen_b The OP asked _"Am I doomed to just doing a linear fit in log-log space and taking its slope as my exponent?"_. The quote says (in many more words), "no".
(+1) I think whoever sent the email doesn't really understand how CART analysis works.  One of the primary advantages of CART over linear regression is that you don't need to worry about transforming the independent variables.
I really doubt that "standard error" is a "common [conventional, non-statistical] word" with a special statistical meaning differing from other uses of that word (phrase, really).  Ditto for "normalization" and "standard deviation."
Stat, this is pretty clearly a homework type question. We don't just do people's coursework for them. Please read the `self-study`  [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) to see the requirements placed on both askers and answerers for such questions here. Initially, at least, our role should lean more toward giving hints and helpful suggestions.
But note that the first formula is less prone to round-off error.
There is a small finite number of tests that can be considered: they compare the values in one summary to those in another.  One of them would be (for example) a comparison of the upper hinge of one dataset to the lower hinge of another. For sufficiently large sample sizes, this would indicate a significant difference in one population compared to another. It is related to the joint probability that $X\\gt Y$ for independent random variables $X$ and $Y$. Although you don't get much control over the significance level, these tests can be reasonably powerful against a large set of alternatives.
It could depend on your covariates. Anyway you don't know until you look.
It is geometrically correct to convert angular similarity (such as Pearson correlation) to euclidean distance by the law of cosines, as your link shows. But Spearman rho is Pearson r for _ranks_, so euclidean distance will also express ranked data, not original data. BTW, rho's traditional computational [formula](http://stats.stackexchange.com/q/89121/3277) is explicitly a conversion of the distance back into the similarity.
Actually I just took example that from the help file for map.market() by Enos and Kane.  Reflecting on it I don't see why they chose to have area show price; a more sensible measure would surely be to show total capitalisation ie price x number of shares (either number of shares in the market, or just the number of shares I happen to own depending on the purpose).  Then you would have a good intuitive use of the plot to show the importance of the different stocks.
Thank you, all the references look promising !
What do you mean by "R does not use weights to modify the MLE"? If you use the options of `weights` for `glm` in R , then all the estimations will be adjusted taking into account the weights you specify.
(a) The formula I gave is completely general; (b) $F^N$ is incredibly simple to plot :-).  The *mathematics* provides intuition far better than the results of a few simulations, because we can generalize from the math using pure reasoning, an approach that is not available from inspecting simulation results.
"score gives you the probability of the observed state sequence, marginalizing out all the possible hidden states." Is this the same as forward probability (alpha probabilities)? i.e. sum of probabilities of all paths through the HMM.
I think your p=0.049 was mistakenly taken from the chi-sq test. I used SPSS to check it, and the 2-side p value of Fisher's Exact test is 0.053; the p value of chi-sq is 0.049. That may explain why the 95% CI of RR crosses 1.
Tufte's data-to-ink ratio does not appear to be relevant in this problem, but Cleveland's studies of the speed and accuracy with which people interpret statistical graphics have a strong bearing on it. These studies suggested that people do not compare angles to each other as quickly or accurately as they compare lengths or parallel positions relative to a common baseline. One big problem with angles is that the comparison may depend on how the angles are oriented; that problem doesn't appear to be an issue with a dynamic gauge. So perhaps circular gauges are close to optimal at what they do.
I contacted Dr. Alexandre, author of the first article I referred.  His C version program output same result as my pascal.
Also note the doing bootstrapping with a small sample will lead to some biased estimates, as noted in Efron's original paper.
Thank you very much, you unlocked my brain... In fact, since I'm dealing with regression trees, using the target class variable appeared less obvious than for a pure classification task. But it now totally makes sense.
Presumabbly it is when you use a linear model when the underlying process is not linear
@PAC I just checked the textbook (p.g. 700 of Cameron and Trivedi: Microeconometrics). The standard random effect model does requires that the distribution of $\\alpha_i$ is independent of $X_{it}$. Specifically, $\\alpha_i$ has a constant mean and a constant variance. I agree that under the mean independence assumption, $E(\\alpha_i \\mid X_i)=0$, the random effect estimator is still consistent. But it won't be efficient without the homoskedasticity assumption.
You can always make the deviation 0 for any sepcific point at the sacrifice of the distance for others. If is an average of squared deviations or absolute deviations that is minimized and not all points simultaneously.  I would suggest changing "as close to each of the points as possible" to "so that the line fits the data in a way that is overall closest in a certian respect."
Note that the R `rms` package's `ols` and `Predict` functions (the latter with `conf.type="individual"` will provide confidence intervals for individual predicted values.
@Matt If you have two levels on a within subject variable you can use the difference as dv (no need to go for repeated measures models). The same logic applies if you have two two-level within subject variables. The interaction is the difference of the differences (avoiding repeated measure models). However, if one of your within-variables has more than two levels, this approach does not work anymore, but you have to go multilevel. I recommend you read the special issue of the Journal of Memory and Language referred to in chl's answer: J. Mem. Language, 2008 59(4): Emerging Data Analysis
can you explain why you think it's one larger than it should be? Do you have a simple example you could show? (Are you ignoring the existence of a variance parameter, perhaps?)
I think this might be a near duplicate of [this](http://stats.stackexchange.com/questions/72212/updating-variance-of-a-dataset)
@tashuhka: In this case, the sample mean is a 100% precise estimate of the true average. There is no sampling error here or other sources of error, thus no clear reason to add standard errors. Standard deviation of course would add useful information though.
I am not sure that your call to `wilcox.test` does a signed rank test comparing the accuracy under the two conditions, because you never tell it the pairing variable. At the very least this is an unsafe way of running the test, because it relies on the ordering of the rows in the input data.
The `timecourse` package isn't really for determining clusters, rather it's for calculating which genes are differentially expressed.
When you say a "graphical structure that has been learned from observations of real data" do you mean a probabilistic graphical model?
I didn't understand you wanted to test a submodel. It would have been more clear with a piece of code :)
Why do the _three_ variables have to be independent? In particular, where is the independence of $X$ and $Y$ used in what you have written?
@Sue, is this homework?
Haha, indeed, almost same answer at almost the same time! I think I am convinced what the best method is:)
"Repeated measures ANOVA" is virtually obsolete, and I am doubtful about GraphPad Prism having R's capabilities with regard to current best statistical practice for this situation.
What is the aim of this compsite score that two different ranks of the data you have do not solve?
Did I understand you correctly: You want to calculate the empirical kurtosis of the random variables generated by rbeta, right? So there will be finally one calculated number, which is the kurtosis? Why should there result an array of values? I am not getting this?
Five questions, even when closely related like these, make for a post that is too broad. Do you think you could edit it to focus on just one of them?
I use a linear kernel - if that matters.
Check [this](http://stats.stackexchange.com/questions/82984/how-to-test-and-avoid-multicollinearity-in-mixed-linear-model) out.
In thinking about this, I'm fairly certain they are not equivalent.  If I somehow had a `Shoe` with 10 `Decks`, and one of those `Deck` had exactly one Card, but the rest had say 50 Cards, approach 1 would yield a 1/10 probability of selecting the single card in the small `Deck` vs approach 2's probability of 1/501.
Thanks Glen_b, I am delighted with the learning of types of variables in regression models andthe answer given below by Maaten buis made me clear the concept.
0.02 is barely significant (especially if you consider the fact that you have five tests in total) and 0.11 is not very high. A generous interpretation would be that with a little more power the overall F-test would also be significant (and perhaps the first coefficient as well). A more conservative interpretation is that you shouldn't have much confidence in any of these results (including the coefficient with a .02 p value). Either way, you shouldn't read too much in the difference between .02 and .11.
Thank you Charlie, only a doubt, what do you mean with "and regressing them on an intercept term and your predictors" ? thanks
Great and clear explanation, thanks! I think it comes down to the understanding of what exactly is going on when performing an LR test, i.e. what the assumption "transformation to normality" means in mathematical terms. Your conclusion about the Wald's test overestimating is making the process a lot more transparent.
Can you define the notation?
you can use `xtabs(X6 * 10^8~ X1+X2+X3+X4+X5, data=data)` However there is an issue: since X5 = 0 always, the statistic is NA. I have to exclude X5 in the table
Hmmmm... maybe you should have a look at http://en.wikipedia.org/wiki/Percentage 1 and 100% ARE the same, as are 0.35 and 35% or 2.24 and 224%.
Shouldn't this instead be on a Stata list?
@CesareCamestre Forget the F-test, then.
You can go one step further and take logs.  Then you have *log-posterior ratio* = *log-prior ratio* + *log-likelihood ratio*
I guess that's about the answer to my question! Thanks Andy
In a business-analytic context, this is often referred to as *churn modeling* or *churn prediction*. It employs a wide variety of statistical ideas and methodologies including survival analysis, Markov-chain and stochastic-process theory, hierarchical Bayes models, etc.
John and @whuber Thank you both for your clear answers!
Everyone feels like that every now and then. You're not alone!. I learn a lot here. And the best is to get a couple of good references and keep them by your side. And, of course, study them! :)
+1 Yes, the determinants are directly related to the "hypervolume...of the ellipse defined by 1 sd from the mean."
Hah. I already added this as a comment to my answer, but it's more relevant to this answer: https://tamino.wordpress.com/2012/03/29/to-robust-or-not-to-robust-that-is-the-question/ discusses situations where OLS may be better than robust regression, even in cases with (apparent) outliers.
Because you have `normed=1` matplotlib will normalize the y-axis to represent what percentage of your data fell in that bin. If you want the counts for each bin you can simply remove the whole normed argument.
Appreciably simple and clear answer. You must be a great teacher. People like you must write a small book on Introduction to Statistics in 200 pages
@IrishStat. Yup, a lot of the early statisticians were eugenists, including Pearson and Fisher. But, they were still great statisticians.
Ok so for part d), would I condition on the first step? So my probability becomes the sum of P[T6<inf|X1=k,X0=4]P[X1=k|X0=4] from k=4 to 6?
The positions of the labels have no meaning. If you don't understand the y-axis then it's strange that you're under the impression to understand well the hierarchical clustering.
+1 Simplistic, maybe; but this looks effective and practical.  There does not seem to be any advantage to the multivariate Gaussian fitting: just use the SVD of the centered within-cluster data (which is essentially PCA on the cluster).
If you mean that you correlated the price of a stock (level) w/ another variable & the percent change in the price of a stock w/ the other variable, & got different results, that is to be expected, b/c they are different questions. Can you clarify?
@NickCox I thought that this could be due to preprocessing data...
Alecos, I cant thank you enough, thanks for breaking it down for me so well
Why not use the two-proportion z-test (http://en.wikipedia.org/wiki/Statistical_hypothesis_testing) (If I understand your problem correctly).
Many thanks - Your book is great and I have been reading it. How do I calculate epsilon (the error)? Is it just dnorm(0,sd=0.0277) ?
Have you tried Google? http://ww2.coastal.edu/kingw/statistics/R-tutorials/repeated.html
@user13506: It's not silly--it's a perfectly valid issue to raise... I'll try to edit my answer accordingly.
No problem! Sorry I couldn't reply earlier. Cheers!
From your table the variables are only weakly associated, so that any reduction of the data would lose information without obvious gain. That is apart from the fact that what you propose implies treating the scores as if they were measured; such practice divides people here and in statistical science generally. All that said, the average of two scores seems simpler than what you propose. There is lots of discussion in other threads on how to treat ordinal data.
Thanks jubo - yes, ultimately relying on in-sample data without validating on out-of-sample data is a second best solution.
Is the three variable model a sub-model of the five variable model? If so, you can take the $-2\\cdot{\\rm loglike}$ from the outputs and do it manually, if nothing else.
Having emailed Adrian Raftery (one of the authors), it appears there is no way in mclust to specify per component priors.
Welcome to our site! Please take a moment to read our [faq] and consult the markup help, then go back and edit your question to make it more readable.  In the meantime, it is quite clear your data won't be fit well: you have a large scatter of values plus a concentration of zeros.  Your approach appears doomed.  You might perhaps want to ask a more fundamental question, such as what would be an appropriate model.  If you do, please tell us where your data come from and what analyses you are really interested in.
But I would just get MSE values for each one, so how would I know if an MSE that is 3 larger for Alzheimer's for example, is a significant difference or not? As there will always be differences between datasets even from the same population.
Hint: What's the probability you're at state $i$ after exactly $i$ steps? How about that of $-i$ at step $i$? What must happen regarding other states visited in order to get there? Conclusion?
I know this is an old question, but I've just implemented this method and am frustrated with its performance after around 30 variables.  I was wondering if you could recommend more modern approaches which *are* specifically meant for computers?  Thanks!
would you happen to know how to specify heteroskedasticity on level one in MCMCglmm? Is that the R structure? What is the syntax then?
I think some attention to the formatting of this answer would be very helpful. Right now it looks to me more or less like a bunch of screen dumps from a piece of software that I'm not familiar with, with little surrounding context. I don't mean that to sound too critical; I just feel that providing some context and words in between the software output would be helpful.
The reference to multiple regression in the comment is puzzling, because this answer clearly applies only to the case of ordinary regression where one is fitting a "line" rather than a higher-dimensional surface. Moreover, you appear to have answered a different question: this one asks *only* about the case where $X'X$ is not invertible.
@onestop yes it is.
**Take your pick**: [SNAP](http://snap.stanford.edu/data/). Finding timestamps may be the challenge.
@fabians, no worries, I would not have posted mine if I saw yours. I outlined general idea, you have provided the code. Since the question asks for code, your answer is better.
Nice answer! Your latter suggestion is ordinal logit right? However I heard that there was some support for matrix response variables (clustered logit or something) which is why I asked what I did.
@mark999  Your comments are good and it looks like it gives the right answer.  You should convert it to an answer.
Another question while we are in: What is the design you are considering, I mean is there any clustering or multiple predictors, or is it a simple linear regression? This may help the reader to better understand the context of your study.
You can use `glmnet()` in the `glmnet` package to fit (generalized) linear models with lasso or elasticnet penalties. Whether this is the "best method" will depend on what exactly you are trying to do with your data (prediction, causal analysis, whatever).
I do not think that Heteroskedasticity has  anything to do with ANOVA
Uh... no. Regression does _not_ require independent variables to be normally distributed. Regression (more specifially: the various theorems on the distributional properties of parameter estimates) require that _errors_ are normally distributed: http://en.wikipedia.org/wiki/Ordinary_least_squares I will happily concede, though, that outliers in independent variables should be examined critically and obvious errors should be corrected.