tttjjj/elt_bench huggingface dataset:
Great Explanation. In this case, how can I measure the accuracy in  regression models (where we have continues data). Should we discuss Residuals, variance, pearsona-chi .. (Chech this example: http://blog.yhathq.com/posts/r-lm-summary.html)
@Jeromy:> see updated answer.
Thank you for the helpful answer. Just to make clear, why I am so concerned with "significance": this is **medical** research. The reviewer will ask for the significance and that is the only thing that will matter to him / her. The same applies to my colleagues / the boss who has to allow me to try to publish the paper. I have even reviewed a paper with p values in description of measurements in 3 patients. The reason was that the paper got rejected in another journal, one of the reasons was that the "statistics is not well done" and "p values are missing". Read 10.1371/journal.pmed.0020124.
Thanks, "Christophe Ambroise and Geoffrey J. McLachlan" research it's a good lecture. "G. C. Cawley and N. L. C. Talbot", Gene research may be too difficult for my knowledge.
Any other advice is welcome.
I think you have to reconsider and narrow down it - what you really want. The three things - PCA, LDA, clustering - are quite different. LDA needs 'supervisor' classes while clustering needs not. PCA is a feature reduction method which only marginally relates to classification.
@Andy having a large number of std away from the mean simply means that the variable is not significantly different from zero. Then it depend on the context (was is the meaning of the random variable) but in some case you might want to remove those ?
Welcome to the site, @Marz1. I don't quite follow your answer. You may want to read our [FAQ](http://stats.stackexchange.com/faq), which contains info on answering questions.
Hierarchical clustering doesn't depend on case order unless there are ties in the diatance matrix.
Your assertion that "central limit theorem states that the mean of i.i.d. variables, as $N$ goes to infinity, becomes normally distributed" is incorrect.  See my answer to [this recent question](http://stats.stackexchange.com/q/22528/6633) which raises similar issues.  Another answer to that question was posted but deleted soon thereafter, and the discussion following that answer, now also gone, discussed these issues too.
See the answers to a similar question here: http://stats.stackexchange.com/questions/16218/what-is-the-difference-between-testing-of-hypothesis-and-test-of-significance
Can you provide the code that you ran to get the kappa?
The problem is that if C is the event that the interval contains the true parameter in this particular run of the experiment, it doesn't have a long run frequency (that particular run can only happen once), and thus you can't assign a frequentist probability to it.  This is why the definition of a frequentist confidence interval is in terms of a population of replicated experiments.  You appear to be applying Bayesian reasoning to a frequentist setting, and there is a mis-match of definitions of probability.
Dear Sir, Thanks a lot for enlightenment. I have a very basic understanding of Statistics, so at times I myself find it bit difficult to understand. But at times, being part of my job, need to make my clients few concepts like VaR etc. But thanks Sir, I will try to rephrase my wordings as you have suggested that "we cannot reject the hypothesis that the data are normal" . Regards Katherine
I changed "monotonous" (= boring) to "monotonic" (= strictly increasing or strictly decreasing).  However, the second statement is not true: there are nonlinear f's that preserve the mean.  For example, when the RV is symmetric, then *any* f for which f(x - mu) = -f(mu - x) (for all x) will preserve the mean.
Hi @Tal Galili, The $t$-distribution arises as the _exact_ distribution of the test statistic when the data is normally distributed. When the data is not normally distributed that does not apply and theory only tells us that the test statistic is asymptotically normal (under the conditions provided by the CLT), which is the situation presented by this problem.
I think you go wrong at the very first formula - the Taylor decomposition of a matrix-valued function. Its derivative with respect to a matrix is a 4-dimensional object (tensor), and you can't just go around treating it like a matrix. At the very least you have to very carefully define all those notations, conversions to matrices, etc. For example, I think in that Taylor expansion, you should already have $\\text{vec}(\\mathbf{X})$ for the dimensions to make any kind of sense. But I don't know much about matrix calculus, just trying to use common sense.
This question appears to be off-topic because it is about coaxing R to do something, and not about a statistical issue.
I think I can't use the subsamples, because the total data for each set is not really large. I will try only the negative binomial and check the results. Also, I tried the Poisson, that it was my first approach, but it fitted really bad the data. So my strategy was to add more distributions although they were for continuous data. Thanks a lot for the explanation.
There is also the R [xlsx package](http://cran.r-project.org/web/packages/xlsx/index.html).
It's *discrete time series prediction*. As aplassard said, Markov process regression is well suited to it.
@FMZ, Not necessarily on the validation sample. Plus, the first model will be the best *linear* fit, but not necessarily the best among all possible models.
+1 I met with randomized algorithms this semester and fascinated with them. Let me add another nice article. Nathan Halko, Per-Gunnar Martinsson, Joel A. Tropp, "Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions", 2010, http://arxiv.org/abs/0909.4061
Ok, this is what I have thought. The multiple data points per condition are collected for two reasons. One is that the data should be more reliable this way. The other reason is, that some data points have to be discarded (subjects did not follow instruction correctly all times). The conditions are completly within subjects, so we do not have a mixed desing in this case at all. Unfortunately a repeated measure is out of question, since we have about 40 repetitions per condition in each subject. The high number of repetitions however means, we loose a lot of information when using mean.
Hi, yes I found this: http://www.ats.ucla.edu/stat/stata/output/stata_nbreg_output.htm It shows an example how to calculate McFadden's pseudo R-square using your mentioned deviance approach.
Correlation is not necessarily linear - Spearman's rho relies on the monotonic function, and yet, we refer to it as a "correlation coefficient", not "mutual information coefficient". And for a good reason: it provides an information about association between two variables. Mutual information, redundant information, mutual variance, correlation - these terms are so similar, and this question refers to _network reconstruction_, so I guess that we ended up in the wrong area with right terminology. This is quite specific question...
I like this approach, but would advise some care: $Q$ is not necessarily positive definite. The (necessary and sufficient) conditions for it to be so are described in my comment to Konstantin's answer.
I cannot expound, but I believe I've heard what sound like three different versions of the curse: 1) higher dimensions mean an exponentially-increasing amount of work, and 2) in higher dimensions you will get fewer and fewer examples in any part of your sample space, and 3) in high dimensions everything tends to be basically equi-distant making it hard to make any distinctions.
I have posted it here - http://logfc.wordpress.com/2013/06/26/pca3d/
Thanks xan for your such a complete answer.
If that does help, the observed values of the second distribution were: 47; 109; 158; 2186
Logistic regression is OK if you remember that it's supposed to be about a linear effect. It can't be Pearson because you have no continuous variables. You might be able to argue for Kendall by assigning numerical order to the variables but while correlation in concept can be analyzed none of the typical correlation coefficients work in this case.
There is no need for the sleep call in the first place and avoiding it will give as near real-time simulation as you're likely to get with Ruby. But, I still don't get why the implementation is done this way.
Perhaps try `w[x==100] <- 1000` and `w[x==-100] <- 1000` to make sure you're catching all the endpoints in the data.
I find the argument appealing but, upon reflection, unconvincing. After all, if by chance the samples are unusually far apart (which ought to happen just as easily as being unusually close), then it seems the very same logic would lead to the opposite conclusion.
+1. I usually bounce between the default R GUI and TextWrangler in OS X.
For future reference: you might be interested to check the Computational Science beta on Stackexchange for similar questions.
Anybody who doesn't use time series in R is unlikely to guess its meaning, so you must make it explicit what you mean in the question; I'd suggest replacing (0,NA,0,NA) with a sentence explaining that you have set the specific coefficients to zero.
To get the posterior mode, you use the 0-1 metric $d(\\theta,\\theta')=1$ if $\\theta\\neq\\theta'$ and $d(\\theta,\\theta)=0$. Or in other words, $d(\\theta,\\theta')=\\delta(\\theta-\\theta')$
prcomp **is not** doing orthogonal regression, at most can be used as an element of its implementation. Not to mention it is probably not worth it.
@Glen_b Yes, it does a linear interpolation by default. If the sample sizes are very different and one is small, I could see that this could be problematic. But if either a) Both samples are large (say > 100) or b) The samples are about the same size then not much seems to be lost. If one n = 1000 and one n = 100, then it's like a histogram with 100 bins.
I agree with all you have said, however t-tests are often performed "ritualistically" as Gerd Gigerenzer would put it. In practice the people performing these tests do not have the time/inclination to understand the nuances of what they are doing. For that reason I think the "unreliable" adjective may be apt. I know researchers who when you ask about the distribution (was there one high one, or what caused that big error bar?) have never looked at it.
@AyalewA. Sweet! Feel free to up vote my answer and to accept it (by clicking the check mark at the top left of it) if it works best for you.
see this related question on stopping criteria for cluster analysis: http://stats.stackexchange.com/questions/2597/what-stop-criteria-for-agglomerative-hierarchical-clustering-are-used-in-practice
No statistic is dangerous if you understand what it means.  Sean's example has nothing special to do with R square it is the general problem of being enamored with statistical significance.  When we do statistical testing in practice we are only interested in meaningful differences. Two populations never have identical distributions.  If they are close to equal we don't care.  With very large sample sizes we can detect small unimportant differences.  That is why in my medical research consulting I emphasize the difference between clinical and statistical significance.
Shouldn't it be x-variable?
what makes a test statistic 'more extreme' is the ordering imposed on the sample space by the values of the statistic - in particular, that the values most consistent with the alternative are the 'most extreme'. The only test I am aware of that orders by $f$ is Fisher's 'exact' test on 2x2 tables, but IIRC that has a one-to-one relationship with a natural ordering of a statistic
Yes, and a slight clarification is that online learning algorithms, at least as studied in Machine Learning, mostly make the assumption that your ability to store examples is very limited compared to the size of the data set. In the most limiting case, you only get to see one example at a time, and then you have to forget it after you've used it to update your classifier.
SAS has a horrific syntax that started with something similar to JCL (IBM's Job Control Language) for submitting punched-card batch jobs back in the day. It's remarkable that people are still using it, really.
thank you for your comment. The effect on other variables after this rescaling is not too big but the significance and sign of the coefficient itself changes dramatically if I multiply it by 10000 instead of 100000. Just to add one more thing (if that might help) I am using fully modified OLS for estimation of long run estimates of cointegrated equations.
Thats right, thats exactly where I'm at, however i have not learned fully how to report on the outputs, for the between subjects variable i mean. Is there nothing you think you could sod to help/point me in the right direction? basically need a mini lecture on report writing for this style of anova. If you can help i will be forever grateful!
Sorry same variables.
It'd be fun to see your code once you write it :)  Sorry, I don't remember coming across such a thing...
@Manoel Galdino: "The dependent variable isn't related to the relationship of a couple". Well, that's an empirical question, isn't it? At least, both share the same risk ("predictor (exposure) collected from 1000 homes"). Also, I agree that you could use a hierarchical model (aka multilevel model) but a SEM also works fine (especially if you have distinguishable dyads).
Throwing away some zeros or getting more ones are generally poor solutions for this.
+1, this is truly in "layman's terms", and I know you could derive it *very* rigorously if you wanted to!
I don't think this is wrong, but I do think it's dangerous advice to tell someone it's OK to exclude the main effects terms but include the interaction. You may want to read through the questions @user1188407 & I have linked.
Welcome to the site.  Could you be more specific about what you are asking.  If you're asking to be taught how to conduct regression using moderators, that's asking a lot from this format.  Perhaps you know more than you have indicated but are stuck on some specific point?
How do you use seed? Depending on your typical use case the answer ranges from yes to no.
What makes you think that the independent variable has three levels here?
@Nick: Good point! I was thinking about a situation where samples are so small that estimated variability is zero simply by chance. But maybe I am thinking wrong.
In graduate level textbooks the persistence usually is the synonym of unit root.
Yes Gustavo they would have the same means.
Well... they are pretty close to each other in the sense that both are defined on positive values; they both increase and then decrease; both have actually the same expectation; and the Kullback Leibler distance is "small" if we restrict to a portion of the x-axis... But in order to link these intuitive notions to any statistical property, I would need some rigourous definition for these features...
Thanks for your answer. I do know about meta-analysis, however I do not think it applicable here. After all, what I want to do with my data is to test for differences between experimental group 1 and 2, i.e. assign a significance level.
As far as "no challenge" in your proposed calculation goes, there are subtleties. It is essential to use an `atan2` function to compute $\\psi_\\mu$ rather than using the inverse tangent of the ratio--which would incorrectly limit the longitude to a 180 degree span. Note, too, that your formulas assume equal probabilities are associated with the points, whereas the question suggests those probabilities may vary from point to point (although it is unclear whether those probabilities are associated with points or with "visits," which might be paths on the Earth's surface).
@Ashok, interesting cases are where condition of infinite smallness is satisfied. Naturally you can consider that the interesting cases are where this condition is not satisfied.
Gaetan, I think what @gung is trying to say is that stepwise methods may cause severe damage to the initial statistical regression framework (loss function, p-values, etc.). See Frank Harrell's response and comments [here](http://stats.stackexchange.com/a/11482/930). Penalized regression, as discussed in several threads, might be a good alternative. "(...) software that automates the process" reminds me of R `fortune(224)`: no troll here, just want to highlight that you don't necessarily need to trust what software automagically gives (or offers) you as an option.
By "$H(x)$" do you perhaps mean $H(f,g)$? If not, then what is "$x$"?  BTW, it seems to me that the [definition of Hellinger distance](http://en.wikipedia.org/wiki/Hellinger_distance) makes no reference to the dimension.
You posted two almost identical questions. Please delete one.
Any reason you haven't marked John's answer as correct?
As I understand, these results would tell me if there are differences within the groups (backgrounds) and/or between them. But since I'm new in statistics I can't tell if there are differences by looking at the bunch of data I got after doing the test.
@rpierce: I am sorry, I have no idea what you mean. What is "expected variance" and what is "standard variance"? I am saying that if each $z_i \\sim \\mathcal{N}(0,1)$, then $\\Sigma_1^N z_i \\sim \\mathcal{N}(0, N)$.
Hi Glen
You have provided with the correct conceptual flaw in my approach. But as I am newbie in this environment so I tried to take the very simple approach of going through correlation matrix route. It would nice of you if you can share the algorithm of your approach.
I don't have a citation, but this indirect approach is simply not needed and adds confusion to the paper.  A bit reason not to use it is that it only works in the very special case of OLS; it doesn't translate to logistic, ordinal models, or models for longitudinal data.
Thank you for pointing out my logit mistake. I changed it to the correct one and also simplified my transformation functions. $exp(\\beta)$ is really hard to understand on a logit scale. I'm not sure this question really has an answer... Even if the method is an elegant way of avoiding values outside the boundaries it may not be that useful...
@Andy W: I added the link.
So that we can understand what you are asking, please include either a quantitative definition of "inflation" or "inflammation" [*sic*] in this question or else provide a more precise explanation of what exactly this is supposed to measure. Are you perhaps trying to quantify the degree to which an empirical univariate distribution departs from a prespecified theoretical distribution?
That's a really clever approach and this is a really interesting problem. Wikipedia has a blurb on median-unbiased estimators, for what it's worth: http://en.wikipedia.org/wiki/Median#Median-unbiased_estimators
@reox This sounds like you want to cluster your data somehow (splitting it into groups). This is more complicated and probably requires additional variables to work well. If you provide a sample plot of your data, you might get better feedback.
Thanks Frank, I'll have a look at rms. I actually ended up bootstrapping the CI's at the time as i couldn't find another way.
You welcome. Additionally, I guess you observe a significant interaction effect between Level and Lot (row Level*Lot in table Tests of Within-Subjects Effects) a statistical result that corresponds to the fact that Lot groups are changing their score in a different way in time (a remark that is strongly supported from the line plot).
60 variables is no problem. SVMs can properly deal with thousands. For SVMs the number of variables is irrelevant, as the model is specified in terms of training *instances* (because of the kernel trick) instead of training *variables*. Standard linear regression models, in contrast, suffer from the curse of dimensionality, because they must estimate a coefficient for each variable.
It can happen for data related [reasons](http://en.wikipedia.org/wiki/Linear_discriminant_analysis#Practical_use). But it can also signal a problem with the implementation. As usual, when reinventing the wheel, start by comparing your implementation to some other [ones](http://www.mathworks.com/matlabcentral/fileexchange/29673-lda-linear-discriminant-analysis)
...Or we could take 1 year sample windows to model the parameters, record the 1 day fwd est and actual observations. Then each period (day) slide the 1 year window fwd and repeat. The results of both methods will obviously differ, but which is a better model to dynamically utilize in an online fashion and/or why.
@Placida: Thank you for the information. I've spend a couple of hours reading about Random Forest and it's very interesting but sadly, implementing it is beyond of the time requeriments of the team. (I've not found a free imlementation of the algorithm in our programming lenguage)
you do not have the same predictors in the model (one less), possibly omitted RDINC?
This is a little brief as an answer - it would appear to be more like a comment as it stands. Can you expand on it? Are you referring to the `mlr` package in R? If so, please note that the question is explicitly about Matlab. If you refer to something else please clarify that.
BIC can take more than one model like this    `> BIC(m01,m02,m03)
    df      BIC
m01  6 336.3417
m02  3 355.9029
m03  2 377.4258`
Do you know which before scores go with which after scores?
[Power transformations](http://en.wikipedia.org/wiki/Power_transform) are fairly common in statistics; especially parameterized in the Box-Cox family,  but see also 'Tukey's ladder'. You might find the following paper (which discusses both references above) of some value: [here](http://www.ucs.louisiana.edu/~kxk4695/GammaR2.pdf). I believe the authors also have a book which discusses some of this stuff.
Thank you for the comment
Hi and welcome. This is standard bookwork -- and we treat such questions differently. Please see the `self-study` [tag wiki info](http://stats.stackexchange.com/tags/self-study/info), then edit to add the tag, and explain what you've tried and where your difficulties lie.
@Macro, I'm sure more has been said on the subject, but see page 4 on GAMs of this Venable's article, [Exegeses on Linear Models](http://www.stats.ox.ac.uk/pub/MASS3/Exegeses.pdf). It isn't clear exactly how non-additive main effects and interaction effects are simultaneously identified.
Hugo Larochelle also has code online. I couldn't tell if it also implements RBMs/DBNs, but this might be of interest to you anyway: http://www.dmi.usherb.ca/~larocheh/mlpython/learners_sparse.html#module-learners.sparse.classification
Almost: you are moving between exponentials and logs too quickly and have mixed them up, but you have the right idea.
Even though I gave the bounty to another respondent, I do appreciate this suggestion and intend to follow through with it. Thanks.
Also, please let me know why you think my simulation approach doesn't look right. I plotted heads & tails after 10000 simulations and they look like a normal distribution centered around 0.5*(sample size). After only 100 simulations a bit more wobbly but close. Thank you!
A pseudo $R^2$ is any measure that its authors or its users think is not $R^2$ but is presented as, or considered as, similar in role to $R^2$. The term can be positive, neutral, or pejorative and is, for the most part, used loosely or broadly with no pretentions of rigour, exact inclusivity or exact exclusivity. (I am reminded obliquely of the definition that "soil" is anything so designated by a competent authority.)
A trigonometric model can fit periodic series that have narrow peaks and wide troughs ... if you use more than one harmonic. You use sums of sin and cos functions because the amplitude and phase are unknown; the coefficients on sin and cos are related to the amplitude and phase. The benefit of working with sin and cos is that you end up with linear rather than nonlinear models. Both these issues are discussed [here](http://stats.stackexchange.com/questions/60500/how-to-find-a-good-fit-for-semi-sinusoidal-model-in-r/60504#60504)