tttjjj/elt_bench huggingface dataset:
@D.W. Let the dataset consist of 100 draws from a Normal$(0,1)$ distribution ($N(0,1)$), 100 from $N(-9,1)$, and 50 from $N(1000,1)$.  The median will be around $1$ and the MAD will be around $10$, whence the 50 last draws will have standardized values around $(1000-1)/10 * 0.67)\\approx 150$, all of them apparently *strong* outliers by Peter Ellis's criterion.  It makes no sense to declare the top 20% of any dataset to be "outliers," *especially* when you expect the data to be the mixture you described.
@PeterFlom updated post to reflect your comment
Is it a "randomized clinical trial"? Can you give more information on your dataset?
@Bab This is a Bayesian question: the OP wants to make inferences about the distribution of a covariance matrix based on a prior distribution of its determinant.  Sachin, you have already answered your question: of course you can place a prior on the determinant (although the example you wrote down is neither a CDF nor a PDF except when $k=1$). Perhaps what you are really wondering is whether there are any *analytically convenient* priors that will make Bayesian updating easy. Because updating depends on the presumed distribution of the matrix itself, what can you tell us about that?
In your last paragraph, do you mean instead of using the ratio, use each variable separately (undergrad enrollment and grad enrollment)?
This asymmetry does not look *so* bad; are you sure you don't have an error somewhere else? Also can you give some more details about the data? There is not a general solution to this.
+1 Very nice idea--I will try this the next chance I get.  (GTW, *Mathematica*, for instance, makes short work of creating and saving such animations.)
A little more information about the data you have would be welcome. For instance, your response to @IrishStat's answer suggests your data might be time series, but nothing in this question suggests that this is the case--and in fact it is difficult to see how such data could usefully be construed as, or modeled as, time series at all.  It would be especially helpful to have some clearer idea of what a product's "output" is and how you expect that to be related to the other characteristics.
(Do you want to assume the $X_i$ are independent?)  The conjecture is plausible but appears to be false.  For instance, do some trials where the $X_i$ are iid with CDF $1-x^{1-s}$, $1\\le x\\le \\infty$, $s\\gt 3$.  The variance of their maximum, relative to their common variance, increases without bound as $M$ grows.
I am doing something like this now .... `Slack = [10,100,1000,10000,50000,100000,500000,1000000];
Slack = Slack';

svmtestingError = zeros(8,1);	
		
for i = 1:8
	
 svmModel = trainSVM(traindata,trainlabels,Slack(i,1));
 predictedLabels = classifySVM(svmModel,testdata);
svmtestingError(i,1) = sum(abs(predictedLabels-testlabels'))/length(testlabels);

end`
Thanks Wolfgang! This is exactly what I was looking for. Now I can try and replicate it in Excel (for my students who use Excel only).
@Srikvant.  Assume 5% is acceptable significance.  I'm seeking a more precise answer, one which exposes the idea that "A leads B" is a new statistic, the difference of pA and pB, and that it's corresponding confidence interval is not simply 2*MOE.
Then your post is so horribly misleading that you might consider rewriting it, focusing on the question which really interests you and including the relevant definitions.
I was going to ask whether you consulted the [documentation for `boxcox`](http://cran.r-project.org/web/packages/MASS/MASS.pdf), but I read it myself and realized it is utterly useless: it doesn't say anything about what is actually getting transformed.  Consulting [this recent blog by Arthur Charpentier](http://www.r-bloggers.com/on-box-cox-transform-in-regression-models/) may help set you straight.
(1) The KS test indicates the distribution of p-values is *not* uniform. The histogram makes it clear that there is a (slight) tendency towards small p-values. (2) Applying this test is invalid because the results are all strongly interdependent. (3) So, rather than asking a question about a procedure that might be poor or even misleading, why don't you edit your text to (a) provide context for the analysis, (b) state your *scientific* goals, and (c) request suggestions and advice for good statistical procedures to support those goals?
Thanks for the pointer, I am going thru it.
This question is unclear. At the moment it seems there is no any connection between input data and the number of users.
Thank you, we'll do it like this.
No, the chi-squared value is most certainly *not* proportional to the p value. And you don't seem to be plotting the statistic against the sample size, but against the replicate (with constant sample size of 100).
I'm not used to seeing designs called longitudinal when the dependent variable is measured at only one point in time.  I would probably treat this as a cross-sectional problem, but you might look into path analysis or structural equation modeling to take advantage of what seems to be a chain of potential causes and effects.
By effect size, do you mean something like an equivalent to $R^{2}$? With respect to p-values, there is a long and substantial debate around their estimation and around the implementation of them in `lme4`. Have a look at the discussion in [this question](http://stats.stackexchange.com/questions/69001/confusion-over-lmer-and-p-values) for more details.
@AdheshJosh http://www.r-project.org/
@Wayne: No, it doesn't seem to be in the internal documentation.  But see section 11.1 of [An Introduction to R](http://cran.r-project.org/doc/manuals/R-intro.pdf).
Hi Srikant, yes, the algorithm requires at least one honest party. You have no internal source of randomness at all? Then you could take the SHA-256 of the PDF version of The New York Times every day, or the SHA-256 of the volume and closing price of all the stocks in the Dow Jones Industrial Average, or the really the secure hash of anything that can be mutually observed and that you can't influence. If you want one bit, take one bit of the SHA-256. If you want a normal distribution, take the whole thing (256 bits) as a uniform deviate and use the Box-Muller transform to get a normal deviate.
Wikipedia [says](http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Kolmogorov_distribution):

$$Pr(K\\leq x)=1-2\\sum_{k=1}^\\infty (-1)^{k-1} e^{-2k^2 x^2}=\\frac{\\sqrt{2\\pi}}{x}\\sum_{k=1}^\\infty e^{-(2k-1)^2\\pi^2/(8x^2)}$$

and refers to [Marsaglia G, Tsang WW, Wang J (2003). "Evaluating Kolmogorov’s Distribution". Journal of Statistical Software 8 (18)](http://www.jstatsoft.org/v08/i18/paper). For another proof, see e.g. Theorem 2 in [these lecture notes](http://ocw.mit.edu/courses/mathematics/18-443-statistics-for-applications-fall-2006/lecture-notes/lecture14.pdf).
@whuber Thank you.  I will look at that section.  My understanding was that Program MARK more-or-less using the sine link as the default if there are no covariates, i.e., if there is only one entry per row in the design matrix (regardless of whether that matrix is an identity matrix).  I know lots of problems will arise if you use the sine link along with covariates.  Maybe I am wrong.
I think I am trying to force my thinking about it with respect to decision trees as opposed to 'general statistics'. I'm trying to compare on a variable by variable basis how well a variable can be  used to explain a target variable. If one variable has a high proportion of target variable when the target is low this is a more explanatory variable than if the target is randomly distributed around all values. I'm wondering what measure I could use to compare these variables (possibly a single variable logistic regression and compare the p-values?)
I know this answer is probably more suitable for a real tick, Zen's answer is easier for me to understand for now. Thank you very much for this demonstration.
Seems perfectly reasonable to me. Alternatively you could specify a largest difference in means that would be realistic and then compute the power for that difference.
Just curious, could you elaborate on the Field (2000) reference? Is it Andy Field: Discovering Statistics Using SPSS?
@Glen_b +1, nice!
Thank you Peter, I will try to find some readings about this.
Rpy, of course ;-)
You're aware that R won't "automatically" use all your cores, right? You need to load up a package that will implement parallel computing. My favorite for embarallel computing (as in the case of bootstrapping) is combining the `doMC` package (`doSMP` if using Windows) and `plyr`'s `.parallel=TRUE` arugment.
But is this pertinent to logistic regression? We don't need to describe the statistical distribution of either class, just the relative odds ratios (see my comment to @Henry).
Clean and simple answer. Thank you very much!
I am guessing that you are referring to this post: http://stackoverflow.com/questions/1206284/algorithm-for-hand-writing-recognition Could you please tell us why these ones did not meet your need?
it would be more helpful to figure out if you also gave us correlation between explanatory variables
Is the range you have restricted relative to the population, or relative to the possible range? (i.e. do you say that the range is restricted because it's so low, and it could go to 100%, or because in the population it's likely to be 20%).
You reminded me! - I had confused one 'rule' with another (nearly its opposite :$ ). I very much appreciate your clear explanation.
@biz14: As to copying the diagram: As SO content it is licensed under the [Creative Commons Attribution Share Alike license (CC-BY-SA)](http://creativecommons.org/licenses/by-sa/2.5/) . This means you can copy it, adapt it to your needs, but you must attribute the source here (I can send details so you can cite me by my full name). In addition, if you release/publish the result, it must be under the same (CC-BY-SA) license.
Hi M44rten: welcome to our site!
Do you have a sense of what SPSS means by "conditional effect of predictor at values of the moderator"? I have an idea but I'd rather not base my answer on a guess. Statistics terminology can get pretty dicey. Also... plot your data. Always plot your data. You are smarter than a test statistic.
For the answer in duplicate post it's far from my want , my problem is shown before , i want show and in formel proof that the expected distance between the estimated value of beta and the target have a relation with the trace of matrix .
I am not sure. Probably not.
I think you might be interested in the martingale central limit theorem.
Great point, @rolando - since the original post refers to training and validation datasets I suspect your response may actually be more usable for @Figaro.
I think it will depend a bit on the field - can you give us some more context?
The use of the phrase 'relative to' there is critical to the meaning. Because of that, it doesn't refer to the population mean but to the difference between the population and sample mean. Beware, however - I see at least one error in that article.
Definitely check the results of this script.  I am getting the same results as I would get with the Tukey test.
Is this a [tag:self-study] question by any chance?
+1 - Also see [this response](http://stats.stackexchange.com/a/68726/1036) about how to estimate the correct intercept plus other references.
Are you using ANOVA for binary data?
Just as an update, I am now pretty sure that using ML is the way to go for model selection, from what I have read in other posts. Additionally, for now I think that it is better to go with the more intuitive model which also has better residual plots, as this model better meets the necessary distributional assumptions - this is model 'SBall.gam2a'. However, I am still a little concerned by the huge gap in AIC scores.
Thanks! I was wondering if there is a formal name for this (using mean of $s^2$) or perhaps it could be derived from some existing ones.
@Willemien I want to test difference of in proportions of the two samples. One sample will be "true" 1.21% of the time, and the other will be "true" 1.33% of the time, for instance. Is this difference significant? My main problem stems from the heavy skew.
It can be represented with values 0 and 1 (binary, you see?). If it is "explanatory" is because you intend to predict/explain some other variable using this one. E.g., which is the chance of air temperatures above 30ºC when night (0) or day (1)?
@Glen_b:  cross validation might be a bit too-data intensive for my multi-level/clustered data.  Not sure I understand the second half of your comment.
Seems to me you need a multi-level model to account for the nesting of firms in countries. But it might be more complex than that, as the nesting may be partial (since many firms are international).
1) They are more important for developing accurate estimates of your parameters, not in any moral sense.  Check the link (added to the answer above, too: http://en.wikipedia.org/wiki/Heteroscedasticity.  2) The solution is not dependent on var. not being systematically related... but you could, with more work, use that relationship to get a better estimate of the var.  3) I'd just use the square of the length of the confidence interval as the estimate, or something proportional to that; R rescales the weights anyway, so it's not essential that the weights add to anything in particular.
The analogy with area in this answer is not just suggestive of what's going on, it's *exactly to the point*. There's a very direct sense in which (given a fixed total sample size $N=n_1+n_2$ - i.e. half the 'perimeter' of an $n_1\\times n_2$ rectangle), maximizing the product $n_1n_2$ (the 'area') maximizes the precision of the estimate of the difference in means (and hence, power to identify it's not zero). It's algebraically trivial, so I won't labor the point further, but you couldn't have picked a more apt analogy.
Are you familiar with some of the many measures of association, such as phi coefficient and Spearman rank-order correlation coefficient?   If not, look into them.
This is absolutely fantastic! Very thorough and succinct at the same time! Very much appreciated putting this matter to bed!
I'm not so sure. Mean & stddev assumes Gaussianity for a start, and secondly this doesn't take into account how many comparisons are being done (e.g. [Bonferroni correction](http://en.wikipedia.org/wiki/Bonferroni_correction) might be needed)
Poke around a little here and at StackOverflow in terms of what people recommend for *statistical analysis and programming*. Many of us feel that there is no real alternative to R.  But just like  beauty, this is in the eye of the beholder, so good luck.
+1. Much depends what you mean by "cope with". I take it to mean: omit missings and use the actual numbers of observations in formulas. But depending on your software, missings might cause a function to fail.
Intrinsically there is no such thing as a standard deviation that is "too high", or a data set that is "not valid".  "Too high" and "not valid" must be judgements depending on your assumptions of what is supposed to be the reality behind the data.
Indeed. Thanks for accepting the answer.
What is *c*? There's no *c* in your equation.
This is not answering what the question is asking. OP is asking how one can put a prior on $F$ when $X_1, \\ldots, X_n \\stackrel{iid}{\\sim} F$. Assuming our prior on $F$ puts probability one on distributions with a density, the likelihood is $L(F) = \\prod_{i=1}^N \\left.\\frac{dF}{dx}\\right|_{x = x_i}$. So we need to construct a prior on the space of distribution functions $F$ which are differentiable (which is infinite dimensional), and OP is asking how to do this.
@wolfies If it isn't already abundantly clear, `R` was not written by software developers!  (It was written by statisticians who based their choices on the `S` language developed at Bell Labs in the early 80s.) I would contend (although this is just a subjective guess) that there would be at least as many input mistakes if the parameter $\\sigma^2$ were used. Few distributions--if any--have universally understood parameterizations.
Welcome to CV! Do you mean $p(y|x)$, i.e. the probability of y conditioned on x? And what does `start` mean? If you're solely interested in a Python implementation, you may have better luck on SO.
To elaborate...  As I mentioned, there are a lot of people at Google who actually do care about it, as does David Mease.  (Btw, he's not at Stanford, unless there's news I missed; maybe you attended his 2007 course?)  But, you are right in that just as a lot of people don't know how internal combustion engines work, it doesn't affect their ability to drive.  Good auto engineers & researchers, however, make it work better because they do.  Same thing for the engineers & researchers at Google.  Unfortunately, that Wired article is not the most articulate presentation of Norvig's thesis.
+1 Thank you, that helped :)
@stressed That's a good way to approach the situation, because the new question does address a different aspect of your problem.  For the record, that question is [here](http://stats.stackexchange.com/q/13296/919)
'those small vaguely circular structures' are indeed the bubbles.

I was thinking of dividing the image up into partitions and counting the number of pixels in each partition. then use Pearson's chi squared test to measure how well the data fits a Poisson distribution
OK. Then the first question is whether the growth of the two teeth is independent. I am not a dentist, but it seems very unlikely that the growth of different teeth in the same mouth is independent. If it is dependent, then you need to get (ideally) a bivariate distribution of growth; if you don't have that, then some other measure of the strength of relationship would help.
Thanks for immediate reply. I need more clarification. If I have one endogenous explanatory variable and  one valid instrument, can I still use 2SLS? or do I need more than one valid instruments to ensure the legitimate use of 2SLS in my case?
@MichaelChernick, my "pessimism" above was not directed at Dixon's test when used appropriatly (that is a whole other topic), but the combination of an outlier test applied only after a non-significant normality test.  If there really could be outliers, then won't they affect the normality test first?  What is the distribution of the test statistic from Dixon's test **conditional on** having a non-significant normality test?
One thing to consider before you start is whether your model of the estimation error might be relative (they tend to underestimate by 30% say), absolute (they tend to underestimate by a certain number of hours), linear (an offset and a percentage), or some nonlinear function, and how the variability in the estimates may tend to change with actual task time.
How does either statement lead to a hypothesis of normality? It's not clear to me that you're necessarily wrong (perhaps given certain assumptions that the OP might feel happy to apply if they were explicit enough to be clearly considered), but the steps in reasoning from the question to your answer should be clear, at least.
Could you be more precise about what you mean by cross-tabulating the ORs? Your factors have more than two levels?
@jack, Friedman test ia a valid test 2+ paired samples. With 2 samples, it gives results very close to those obtained by **sign test**, so you may use this. (The latter is a little bit more liberal, in normal approximation regime.)
thanks but i don't think a decision tree algorithm may work. unless i have multiple samples of R1, R2, .... RN, there will not be enough training data for the splitting criterion/function. but a decision rule seems to be an output that i may want (i.e. if A < 8 then R1, if {B < 9,C > 3} then R2).
@paul : In optimization linearity usually refers to the constraints, not to the function itself. I (erroneously granted) referred to "linearity" in relation with the smoothness of the function and I think that is what the OP also referred at. In my answer I was mostly based on the fact he said "continuous" afterwards anyway.
This is not a common problem because it has no solution. It is on all fours with the probability of being red or any other arbitrary state with no rules given. If I say to you, Tell me the probability that my number is positive, you can't make progress with no assumptions at all.
+1 Other tests can be found by Googling "test of proportions".
It is not since the white noise is of mean zero.
+1, for noticing that this is AR(1) with zero error. Did your software detected that automatically, or did you ask to fit the AR(1) regression?
It can help to think about this geometrically. Principal components are *lines* without direction. The software outputs *vectors* to describe those lines. Literally *any* nonzero vector on a line will suffice, but typically only vectors of unit length are used--of which there are, of course, two. Furthermore, in (rare) cases there are two or more principal lines with the same eigenvalue. They generate entire *planes* (or higher dimensional linear subspaces) of "principal components." This creates even more ways to represent the answer--you can rotate the vectors in these planes.
I will send this to the author of TSA package. BTW, your "forecast" package is really helpful. Thanks.
You could enhance your graph slightly by using pseudo-transparency. Something like in [this answer](http://stats.stackexchange.com/questions/7402/how-do-i-find-the-probability-of-a-type-ii-error/7404#7404).
Also check this page http://stats.stackexchange.com/questions/81973/how-to-evaluate-select-cross-validation-method/81994#81994 about how the bias (what you refer as clustered) can be avoided.
I can understand that reading online help might seem a tiresome activity at first sight (still, it's very educative and often help to capitalize knowledge on a particular software), but could you at least accept answer(s) you find helpful for your ongoing R activities?
The z-score approach @John is referring to would be used to compare the values in a column compared to the average of that column. If this doesn't work you could weight the values using the number of games they've played in. Alternatively if you wanted to measure all three columns simultaneously to compare players you could use a multivariate statistic like Hotelling's T^2. Another idea might be to treat it as a 3x3 contingency table and perform an appropriate analysis that would tell you if there is a significant difference, but you'll have to go back and look at the cells to see which ones.
By the way, I am using R.
$Y$ is a "censored" version of $X$, not a "truncated" one, to follow established terminology -because, while it is defined in the positive half of the real line (as would be $X$ truncated from below at zero), it accumulates positive probability mass at point zero. See also this post:http://stats.stackexchange.com/questions/77878/likelihood-function-of-truncated-data/77961#77961
Thank you very much Gavin, for the clear and helpful response and also for the permute package! Looking forward when permute will be hooked up into vegan!
As an aside, you might want to consider the n=3 or n=4 case. If there are interesting transformations at these sizes, then maybe they can be generalized to arbitrary n.
This is terrible advice in general because at most locations air quality changes dramatically throughout the day. So, although you could compare the CDFs, the comparison would likely be meaningless or woefully misleading. Special circumstances could rescue you, though, depending on how these sample times were chosen and how they compare between the two datasets.  Perhaps you could tell us more about that?
@Donbeo, ice-on occurs once a year, so the response variable is at an annual resolution. The other data come in at an annual frequency as well, but in some cases could be converted to higher frequency data.
Please notice that $B-C$ in the first row is wrong: $12-8=4$, not $5$. So the variance of $B-C$ is $10.7$, not $10.3$.
@micans Thank you!
To formally test the hypothesis of a common set of parameters , you need to estimate under under the null hypothesis of no differences . Simple comparisons as you laid out simply are not formal enough.