tttjjj/elt_bench huggingface dataset:
@Glen_b: I agree with them E(X-mu) = 0 doesn't implyt that E[(X-mu)^3] = 0.
I want to test if the variability explained by the equipment parts of the model are the same for the 2 equipments.
thank you Manoel
"Statistical power focuses on whether the correlation is significantly different to zero". This is not true. To be able to speak of power, you have to assume point value for the alternative hypothesis. E.g what is the power to detect, on 0.05 alpha level, correlation of magnitude 0.2 or greater?
@Jeremy I'm precisely offering a bounty to provide a discussion of these aspects that can serve as a reference in the future. Are you willing to start?
Thanks for that, I can see how there were a few things jumbled in my head!  This definitely gives me food for thought...
+1 Field - as so often - is sort of right, but misleading. All of the usual versions of the Wilcoxon rank sum statistics are asymptotically normal and "can be converted to a Z score".
@rvl : My current models predict small revenue cases to have much bigger reve then actual (in opposite to high revenues).
@Neptune I have updated my answer to create the link. The extended version is now equivalent (aside dropping $(1-\\alpha))$, which is an unnecessary simplification IMHO).
@Conjugate prior: No. It is quite clear from the data that the best functional form varies across participants. I am running separate nonlinear regressions.
@Fr.: I've done a quick edit of my answer. See if that puts you on the right track.
Sample size is about 400 in each sample. What average correlation are you referring to? Chi-square in the model is 262.9, df = 35.
Of all the solutions, this worked best.

1. Follow-up question: is there a way to toggle tolerance with turnpoints ? Found a lot of Peaks and Valleys in the long-tail portion of the Density function.

2. Follow-up question #2: what is a good way to determine tolerance?
I haven't claimed that it is obvious, I just said that I have found this statement. So the natural question for me would be: what about other statistical tests - no matter what they are designed for.
The initial assertion is incorrect unless you specify that $X$ has zero mean and unit variance (although obviously changing the variance only rescales $Z$).  Upon adopting the same assumption for $X_1$ and $X_2$--namely, that they have zero means and their variances are equal, integration gives the PDF of $Z$ as $dz/(\\pi\\sqrt{1-z^2})$ (a rescaled Beta distribution). If they have different variances and $\\sigma^2$ is the ratio of the variance of $X_2$ to that of $X_1$, the PDF generalizes to $\\frac{\\sigma }{\\pi  \\sqrt{1-z^2} \\left(1+\\left(\\sigma^2-1\\right)\\right)z^2}$ (which can be *trimodal*).
@user1800340, you don't want the residuals vs fitted. This plot tells you there is some curvature in the x y relationship that isn't being captured by your model. To assess homoscedasticity, you want to call `plot(model, which=2)`, which will give you scale-location plot (see my answer [here](http://stats.stackexchange.com/a/52107/7290)).
@cardinal Sorry that was unclear, so let me try to rephase it. Say I have two $(t,m,s)$ nets $P$ and $Q$ that I use to generate two sequences of 100 points, $\\{p_i\\}^{100}_{1}$ and $\\{q_i\\}^{100}_{1}$. If I use a random scrambling algorithm then  $\\{p_i\\}^{100}_{1}$ and $\\{q_i\\}^{100}_{1}$ should be uncorrelated, right? Clearly this is not true if I had used a deterministic scrambling algorithm. But what if generated 200 points and only kept even entries of $\\{p_i\\}^{200}_{1}$ and odd entries from $\\{q_i\\}^{200}_{1}$. Would these be correlated? And would they still be nicely "spread out"?
@ttnphns While family-wise error rate corrections change with a redefinition of "family", false discovery rate inferences do not.
@BYS2 'Bootstrapping' is a resampling technique to help you obtain an estimate of variance in situations where conventionally calculated variance estimates are difficult/impossible to get.
@PatrickS.Forscher That's the situation exactly.  I can of course restructure the CSV files if that would be simpler for R to handle, but I can't do much major reconstruction since these are the data that I have.  (I could, for example, add a header specifying condition switches; snail / no snail, 10 / 15 degrees)
Hi Dougal, Thank you for your solution. Is there any solution in the case that X and Y are not independent?
(+1) These all are very helpful directions for the OP. I would add that Cronbach's alpha makes *no sense at all* with a multidimensional measurement instrument, but has to be computed for each scale separately, providing they have more than two items. If there're no prior hypotheses, there's no need to try CFA, IMO. It may be worth trying the item clustering approach developed by W. Revelle, also available in the `psych` package.
For a probabilistic framework of decision trees, see DTREE (url: http://www.datamining.monash.edu.au/software/dtree/index.shtml) which is based on the paper "Wallace C.S. & Patrick J.D., `Coding Decision Trees', Machine Learning, 11, 1993, pp7-22".
I don't believe there can be a theorem exactly as stated (specifically, the 'finite number of parameters' would seem to be a probelm). Where did you encounter it?
Thank you very much, this looks great!
You can test multiple contrasts simultaneously.  So for example if A had three levels and B had 2 we could use the two contrasts: C1 = (A1B1 - A2B1) - (A2B1 - A2B2) and C2 = (A2B1 - A2B2) - (A3B1 - A3B2) and use a 2 degree of freedom test to simultaneously test if C1 = C2 = 0.  It's also interesting to note that C2 could equally have been (A1B1 - A1B2) - (A3B1 - A3B2) and we would come up with the same thing.
@Rob: good comment! I'll leave my answer (even though it had a negative rating) because I think the comment would be quite useful to other more-naive folks such as myself.
@redblackbit  As an example, suppose I am interested in trying to determine whether or not my friend has ESP.  I am skeptical, so I think there is an extremely small possibility that my friend has ESP.  Suppose I flip a fair coin 10 times and he correctly guesses every time, a p-value of about .001.  Now, one of two things happened.  Either my friend has ESP, which is why he was able to correctly predict all 10 flips, or my friend doesn't have ESP and was lucky.  Confronted with this data, I still believe there is a low chance that my friend has ESP because my prior probability was so low.
No, that is not true. The "equation" that you provided is called growth in sales not forecast. A naive forecast will have a growth of 0.
You have a problem of quasi-complete separation. For ways to continue see [Allison](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.176.2077&rep=rep1&type=pdf)
"A beanplot is an alternative to the boxplot for visual
comparison of univariate data between groups. In a beanplot, the individual observations are shown as small lines in a one-dimensional scatter plot. Next to that, the estimated
density of the distributions is visible and the average is shown." Extract taken from [here](http://www.jstatsoft.org/v28/c01/paper).
(1) What is $B_0$ and $B_1$ (how to set them) ? (2) what you call $d$ is the mean distance from $x$ to the $k$ nearest points belonging to a given class $y_i$ (in which case it is $d_i$) ? or is it the mean distance to the instances belonging to the majority class in the $k$ neighbourhood of $x$ ? Please give more details. Thanks.
@Enzo, yes, the cross is literally the two terms multiplied -- interpretation will largely depend on whether `var1` and `var2` are both continuous (quite hard to interpret, in my opinion) or whether one of these is e.g. binary categorical (easier to consider.) See this answer for some interpretation examples by Peter Flom: http://stats.stackexchange.com/a/45512/16974
Thanks - you are right. I corrected the question. when you said *"the posterior is equal to the MLE **in this case**"* , are there any distributions where the posterior mode is different from the MLE when using a uniform prior?
When I started interacting with this site, Michael, I had similar feelings. With experience, they have changed. It's worthwhile knowing some $\\TeX$ and once you do, it's (almost) as fast to type it in as it is to type in anything in English. I also learned, by studying exemplary posts (such as many replies by @chl, cardinal, and other high-reputation-per-post users), that providing *references,* clear *illustrations,* and well-thought out *equations* is usually highly appreciated and well received. High quality is one thing distinguishing this site from most others.
Excellent suggestions, Hotaka. I would love to upvote your post but I don't have enough reputation points. I have noted your suggestions, the Kuder-Richardson formula should come in handy. I will read into this some more. I am glad I came here. I will eventually accept your answer, but I am hoping for a little more feedback and I hope the incentive of "correct answer" will drive more people here ;)
There are many examples in [Applied Functional Data Analysis](http://www.amazon.com/Applied-Functional-Data-Analysis-Ramsay/dp/0387954147), by the same authors.
@user333 if you can give more information about your data, it would help us more.
Try stepping back from the math a little... writing heuristically, how does $T(F)$ change for a small change in a value of $x < F^{-1}(\\alpha)$?  How about for a small change in a value $x > F^{-1}(1-\\alpha)$?
Start drawing or visit your proposed causal pathway and examine the plausibility that demographics can confound with the six predictors or cause the outcome. Don't just throw them into the regression and see what sticks. If you indeed can build a good causal reasoning, then I would consider building the original one (for the sake of comparing to others' findings), and attach a further adjusted one that include demographics.
Doesn't SPSS provide options to analyze cross-tab residuals?  See http://www.helsinki.fi/~komulain/Tilastokirjat/09.%20Ristiintaulukko.pdf for instance.
If you haven't already, it might be worth considering using some sort of polynomial lag model - an almon model for example. If you search distributed lag model, almon model, polynomial lag model, modelling sales in response to advertising, or terms like that, you should get some useful results.
I would say no. But have a look at this:http://normaldeviate.wordpress.com/2012/11/09/anti-xkcd/
Python isn't widespread in our organisation but it looks like an interesting project.  I'll see if I can pinch some ideas on how things should be done from their documentation.
It sounds like you're really asking a model selection question: given some data to model, when should one prefer a Normal distribution compared to a mixture (or more generally, how should one choose the number of mixture components)? Reframing the question like this would give you access to, oh, a few hundred related questions on this site :-).
This is the subject matter of [circular statistics](http://stats.stackexchange.com/questions/105641/best-transformation-for-sinuous-data-sets). How to include these values depends on whether the aspect is explanatory or a response variable; presumably it is explanatory. In this case consider re-expressing aspect into a value more directly related to sun exposure, for which you say it is a proxy.  For instance, the pair (slope, aspect) can be converted into a rough numerical assessment of mean total daily insolation per unit area.
I'm not sure if this is what you're looking for, but I can make some suggestions that might help. You said that the datasets are mostly very linear, so you could try to detect non-linearity using the BDS test. Note, however, that if non-linearity is present, you won't be able to say whether it is chaos or a non-linear stochastic process. In R, use the bds.test() function from the tseries package. The Hurst exponent might also be useful for your analysis. I'm not sure if these stats will give you a measure of how non-linear the data is, though.
Hi,I'll try to explain more about our data: we analyze 5 times in a row the sames gas cylinder to estimate the concentration of a given gas in the bottle.We want to study if we should take into consideration all 5 values to give an estimate of the concentration or if we should   62.970, 62.202, 61.791, 56.588, 59.155
As with any other statistic, they are random variables, and so have their own sampling variance. It isn't usual to think of them as having an error *term* (no such term is explicit in the model), but they are estimates of a population quantity - with sampling error and so while it wouldn't usually be expressed that way, could at least arguably be written with an error term.
This is very helpful, thanks. Your answer will help me get further. I understand the need for more specific questions, which I shall do in the future. Now that I have a starting point, I can get into more specific areas. Thanks again.
IMO, it does have to do with visualization (which is on-topic), so I'm open to allowing it.  That being said, I'm sure that you will have a hard time getting an answer either here or SO for this question: your best bet is the protovis mailing list.
@OtavioMacedo I would like to find a way using software and programming to solve this problem. I feel that [Linguistics.SE](http://linguistics.stackexchange.com/) is more of a theoretical approach.
You are right. It's my mistake in the formula understanding. Thanks.
Using a natural (and accurate) analogy that parameters are to distributions as addresses are to houses, one might similarly ask whether the process of looking up somebody's address is merely finding their address or actually finding where their house is. We could press this analogy in some fruitful directions. For instance, although everything that needs to be known about a house's location is in the address, someone who is unfamiliar with that area of the world might learn nothing about the house's neighborhood from its address alone. This is one reason we use "known" distributions for MLE.
Here is one possible approach. Null hypothesis: the proportion of people who want to quit is the same for high and low income people.  Alternate hypothesis: the proportions differ by income level.
Note that Z is not a function of $\\theta$.
Couldn't you also technically have far less features, if you had a highly co-linear dataset? More features would surely run the risk of over fitting the dataset, although the original question may relate to an under fitting problem.
@Michele, interpolate with `approx` function. Then apply Rob's function. See the results.
Your answer was correct, but you don't need the distribution of the result to get the standard deviation, and I've seen the $\\sqrt{2}$ ratio explained in simple terms many times without using the CLT.
Oh wow it accepts LATEX tags!  Thanks chl.
@Erik Mr. Simon's answer is correct in the sense that fixed effects models are structured as linear equation models. The random effects models are based on pure statistical assumptions. There is a difference in models. But I am giving this version in substantive sense. I am not aware of WinBUGS
This question: http://stats.stackexchange.com/q/22329/3277 is very similar
Doesnt seem to be the case here. Perhaps @Rob_Hyndman will check in.
Behacad what software was used to draw that diagram?
thank you very much, these notes is absolutly what i am searching. I just want to remark that exactly the theorem is found on its 10th page. The remark of whyber is also true. I forget the hypothesis of uniqueness for the quantile.
In EFA, you won't get a complex factor structure like that to emerge though, if you've a method artifact, because it tries to rotate to simple structure (and it ain't).
Do you have one with lots of examples that you like? In the style of Gelman and Hill, but with a bit more rigor.
You shouldn't have a look at the training error rate. 0% might just mean that your algorithm overfits.
I prefer to say math is math is math.  Math can be applied in many areas.  As long as the fundamentals are sound, the results can be good.  It can apply to systems engineering, signal processing, and econometrics with equal validity.  :)
Should the inequality sign be $<$ in your second inequality?
As many as neccessary. **Visualize your data.** If you can't see clearly cut clusters in 2d, then algorithms won't return meaningful results either.
Thank you for your fast feedback, you've addressed the incomplete part of my question by the sentence: "you want to make the claim on the original scale you would have to figure out how the change on th log scale translates to a change on the original scale". This is exactly the problem at hand, how to interpret these ln results?
It's perfectly simple to maintain anonymity while matching responses. Just as one example off the top of my head (there will be better procedures than this; it's a common issue solved many different ways), during the first test, you have pairs of cards which you give to each participant (and whose purpose you explain! They can even be drawn randomly from a bag by the participants). They attach one card to their first test and retain the matching second card to attach to the second test. You never find out who is who. If some cards are lost or forgotten, there's overlapping samples tests.
@Memming Answered in original post.
Awesome. Thank you very much for your help @whuber.
@Dilip: I agree (and nearly mentioned it, myself, but "let it slide"), though I suspect the intent of whuber's comment was to try to avoid any overtly mathematical statements in favor of intuition. That said, this is, ultimately, a mathematical question. :-)
I am interested in finding the largest differences, which are big enough to be statistically significant. The more I look for such differences, the more of them I'll find, so I still need to control for type I error inflation. The reader of the report will be willing to take advantage of the difference in performance of industries in any country if it's statistically significant.
I had not heard of DIC before, but it does look like it is related to the complexity. In fact looking at the DIC wiki page, my definition of the complexity looks very much like the quantity $p_D$, the effective number of parameters. In fact, the last line of the wiki says that $2p_D$ is the penalty due to complexity to the information criterion, so maybe I should be comparing it to the actual number of parameters of interest that I have.
This is not a professional opinion, so I'm putting it in a comment.  As a responder to questionnaires, I always wonder about the validity of results based on questions where no answer corresponds to my opinion.  If you force me to answer arbitrarily, you will get arbitrary results.  If many respondents have the same problem with a question, the *collective* response will be arbitrary, too.  (This can lead to awful decision making if you have no independent way of checking that these results are meaningless.)
I am using R. any special package I should look into? any specific topics within survival analysis that I should concentrate on?
You're conflating sample and population calculations. The $E$ there means something. If you are talking about sample calculations, consider both numerator and denominator with care.
The F-test removing a single variable is equivalent to a t-test: http://stats.stackexchange.com/questions/55236/prove-f-test-is-equal-to-t-test-squared
@Dmitrij, thanks for your comments. I've tried to remain fairly agnostic in my response regarding the utility of all-subsets regression. It seems to me that there is nearly always a better solution, but I felt that might seem like too trite of a response to the OP's question.
If you have a sample of $\\{x_i,y_i\\}$, then _if_ all $x_i$'s come from the same distribution $X$, and _if_ all $y_i$'s come from a the same distribution $Y$, _then_ sample measures (sample mean, sample covariance etc) are estimates of the theoretical statistical concepts. If you don't have this "identically distributed" feature, then sample measures describe _only_ relationships between the specific vectors of numbers.
Aurelian - google search results are tailored these days, which means I may get different results from you with the same search terms
The Pr(X > 3) you provide is different than the question askers 0.0015, you might want to revise it.
@RobinVdL I do not think there is ever a advantage to trhowing out data and thereby giving up some of the information in the data.
A color-coded scatterplot! Very interesting! I will ask a separate question.
+1 Great answer.  I also agree with the caret recommendation.
+1 Perhaps a little more relevant is the [prototype theory](http://en.wikipedia.org/wiki/Prototype_theory) of cognitive science, which provides some empirically justified explanations of how this kind of vagueness not only arises but actually is commonplace.
Yes but it also takes very little effort to use the citation on Google to find any link I might give and others. Some of the links might have the paper available to read either for free or for some small fee.
Assuming that the actual sampling distribution converges uniformly in n to the true sampling distribution, then any function of this distributoion will also converge, since you can bring the limit under the integral in the functional via uniform convergene theorem.
Hi Ranon, I added the gradient w.r.t to $W$ which is I think what you want.
In Excel, use -T.INV(0.01,1) where 0.01 means the one-tail probability and 1 is the degree of freedom.
@Macro I have revised the question. Thanks in advance.
@andrea Usually "regular" competing risks assumes that the outcome not being modeled at the time is censored - something I'd like to question in this particular case, as the counterfactual that's implied by that is absurd in my setting. Beyond that, the answer is "because I'm trying to validate a parametric mixture of two survival curves with a non-parametric approach".
@whuber (1) I'd probably put keeping the interaction in before checking VIF. VIF hurts the standard error, but the slope estimate of the main effects would still be closer to correct. (2) Thanks for the correction! I'd proceed to delete the comment.
@jbowman I'm not only looking for a robust prior, I'm looking for the whole kit, both a distribution for the data and a prior.
Thank you. Pearson correlation is what I was looking for; however, I found your mention of intraclass correlation extremely interesting, and I will surely look into it.
I think if you got the univar book, drilled a hole through it, mounted it to one end of a pole, and did the same with the multivar book on the other side, you'd have a nice zombie sledgehammer.
Could you clarify what you mean by regularised? Do you have an L2 penalty term, and if so have you searched for the optimal factor eg by cross validation?
I think it's still ambiguous, since $F_j(x)$ is not $F_y(x)$. $F_y(x)$ comes only at the end of a long process, as you see in the answer. What you can say though it's "from which $X_j$ distribution comes $y$." But you don't need to edit the question again-just to let you know.
@lmorin - I'm using R, and trying out various clustering methods using variations on `Dist=as.dist(1-cor)`.
I'd appreciate a comment why $\\frac{\\sum_j \\frac{\\sum_i (y_{ij}-\\hat y_{ij})^2}{N_j}}{k} = \\frac{\\sum_j \\sum_j (y_{ij}-\\hat y_{ij})^2}{\\sum_j N_j}$. The latter is what I'd see as overall RMSE, the former is the square of your answer. Am I misunderstanding the term 'overall'?
@allhands Maybe I'm just thick, but to me this is still unclear. Have you measured one single variable multiple times for each individual (e.g. as a time series)? How many datapoints do you have in total (all datapoints used in histograms over all individuals)? Either way, your question would be more clear if you could include a sample dataset, to show how the data is structured.
Doesn't it just mean that you divide each binary variable by the number of items in the category. If one category is of size three then the one-out-of-K coding would create three binary variables each with 1/3 the weight.