tttjjj/elt_bench huggingface dataset:
Because all Normal distributions are symmetric, your repeated use of "symmetric[al] Normal distribution" suggests you are not using one (or both) of these terms in a sense that is usually understood. Therefore, could you please edit your question to include an explanation of what you mean by this phrase? And to what does it apply? After all, in a regression context you could be referring to the independent variables, the dependent variable, or its residuals--but which? Finally, what do you mean by "at a given point"?
A two sample t-test or Welch's t-test could compare the means at each time point, but it would not be appropriate for tests that used multiple time points.
This is a nice example of something that has been occasionally called a "delta lognormal distribution" (where the x-axis is interpreted as logarithms).  You can consider it to be a mixture of one continuous distribution (which looks almost Normal--but its precise identification is up to you) and a point distribution supported near 0.  *A mixture model should do a good job.*  In this particular case the separation between the atom near 0 and the rest of the data is so good you would be well off just removing the data at the left (less than 0.5) and estimating the density of the rest.
See http://en.wikipedia.org/wiki/Pooled_variance
Thanks for all of your help guys- yes, ANCOVA is definitely a more suitable choice in this case! And for the extra details, much appreciated.
Like doing k-means on the labeled and unlabeled set independently and finding similar clusters? Or what would you do?
+1 issues in ARIMA models noted below. But otherwise: "Simplifying a prognostic model: a simulation study based on clinical data." Ambler 2002 is the most quoted reference on this.
I was just going to check out tf-idf from reading your question.  Sounds promising.
sounds good. Thanks
That ("power against $H_a$") appears to be a particularly unhelpful phrasing. Which page and which edition does that occur on?  ... if I felt compelled to say something particularly close to that, I'd have said 'power to detect $H_a$' instead
Thanks for your help, but I should say that I don´t know much about statistic :( Do you say I should use a Chi square test?
This is a standard misunderstanding of cross-validation.  The idea of cross-validation is essentially to determine values for metaparameters: eg which inputs to use in your logistic regression (eg gender,age,smoker,interactions... ) . So basically having found which inputs give the best performance on the cross-validation data ( ie the giving the best average value on your leave-k-out data) you go back and reestimate the model with the selected inputs using all the data.
@C.Pieters The Stata postestimation command `estat recov` displays the random-effects covariance/correlation matrix, so that you don't have to square the results by hand (if you want the variances).
If it were me, I'd take advantage of the emails at the top of the paper and ask one of them what the reason for it was. It *might* simply be an issue of journal style, but in this case I don't think that's it.
You have a badly overfitted function. The advice you got about having a hold-out subsample and trying out-of-sample prediction has saved you from making a serious error. The Gamma glm advice is probably sensible, but it's no good to you until you're clear on how to go about avoiding overfitting. Consider using [cross-validation](http://robjhyndman.com/hyndsight/crossvalidation/), for example.
Homoscedasticity means equal variances. I would expect whenever homogeneity is mentioned in a statistical context, it would also imply that something is constant on average, but quite what would depend on context. As you don't explain the doubt ("perhaps"?) and give precisely zero evidence for the claim of "a lot of confusions" I can only match your two sentences by my two sentences. This gives essentially minimal substance to respond to. I'd call that a lack of research effort.
@cardinal (continued): The probability measure is uniform on the Cantor Set, but this is such a strange beastie that I'm not even sure what the $\\sigma$-algebra looks like.  Perhaps I should have said, "No density with respect to any useful measure."
Good call. I use SO and I was unaware that question even existed. It's good to have a cross-linking for times when questions might appear on multiple exchanges.
Define "correct formulation". There is not such thing. There *may* be multiple formulations of the same problem. So be more specific about where exactly you have problems understanding the differences.
May be worth reading http://stats.stackexchange.com/q/62677/3277.
I think you mean *answered*, not *asked*. The formula given in this question is incorrect; as Byron's answer nicely demonstrates. :)
Thank you for pointing the typos and providing a solution!
FYI there was a slight miscalculation earlier in the t figure I gave: i had treated the standard deviations you provided as though they were variances. I updated my answer to give you the correct t and corresponding p values, and have also provided a small r snippet for you.
Thanks for the comment *dependent on what kind of IQR you select...*  -- I hadn't realised there were so many possibile definitions for quartiles / quantiles!  I'm using Excel's built-in `quartile( )` function, and then taking `IQR := Q3 - Q1`.  My numbers come from a time series of weekly measurements over a year.  The measurements are industrial performance measures and so are from a continuous distribution.  The different populations are different product groups.  In this situation, I don't think the different definitions would be much different in practice?
The margin of error is typically proportional to the inverse of the square root of the sample size, $1/\\sqrt{n}$. The margin of error will be increased by a factor of $\\sqrt{n_1}/\\sqrt{n_2}$ when going from a sample of size $n_1(=10000)$ to a sample size of $n_2(=2000)$.
As far as I know this height is more-less arbitrary measure of "joining penalty" and thus does not correspond to any interpretable distance...
The mostly known and the simplest is the Median-Median regression, well known from smart calculators (Sigh!). Consult also Wikipedia  http://en.wikipedia.org/wiki/Robust_regression and maybe CRAN's Robust task view http://cran.r-project.org/web/views/Robust.html
@whuber - good point; I assumed the intent was uniformity, but there's no good reason in the question to assume it.
@whuber: Dave revised his question.  Under the original formulation, I misunderstood his hypothesis to be x*2=y*4 which is incorrect.
As an aside to the excellent answer you already have, note that if one of those models is appropriate, the other cannot be (one has errors on x, the other on y; if the errors are on y and x is without error, then the assumptions of the other regression model can't hold) ... in which case its AIC is meaningless (it's based on assumptions that cannot be true), and so could not be used in a comparison.
@Alfa: I supposed "stacking" k-means would give you a hierarchy of clusters, but I don't think it will do you a lot of good. At the end of the day k-means finds "clusters" of observations that are near each other in Euclidean space.  I've heard of "evidence accumulation clustering," using a hierarchical clustering built on top of k-means with several different centers, but at that point you lose a lot of speed.
Once you go into multivariate regressions, there's no way to proceed without linear algebra. The Wiki link has all the descriptions of the variables. Here, I can say that $X'$ denots a transpose of $X$ matrix. You'd have to learn how the design matrix is constructed. It's too long to explain it here. Read Wiki which I posted, it has a lot of information. Unless, you understand linear algebra, you will not be able to answer your question in a meaningful way, I'm afraid.
It's not clear what the problem is here.  If there is a significant difference in the relationship between the covariate and your response variable in different groups, why is that a problem?
There is nothing wrong with crediting a book in your native tongue, **if that is what you used**...
I couldn't follow your proof. Could that be proven rigorously? Thank You.
In addition to the great answers already given, let me mention that when using penalization methods such as ridge regression or lasso the result is no longer invariant to standardization. It is, however, often recommended to standardize. In this case not for reasons directly related to interpretations, but because the penalization will then treat different explanatory variables on a more equal footing.
(1) Why do you think it makes sense to compare likelihood for different $n$? and (2) On what basis are you characterizing "lower likelihood" as "LESS ACCURATE" in that case?
Hi Russ, I use this command: lsmeans(lmer54, pairwise~Language, adjust="tukey"). I'm also thinking of using the effect size from Cohen'd. Which one do you think is better?
The treatment factor has 40x as many degrees of freedom as the source factor (pseudoreplication?). This is undoubtedly driving down the treatment P-value.
Then what stops you from implementing it? :-)
Non-parametric Bayesian is just a way more complicated model. You need to learn a lot of parametric Bayesian to start tapping the non-parametric Bayesian issues. Frequentist non-parametric (as in, rank statistics) requires only probability on finite discrete spaces, and its set of assumptions is so much more general -- something like a continuous cdf, that's all. It's too bad that the classic non-parametric stuff from the 1940-50-s has all but died off; this has been a valuable set of techniques. These days, psychologists understand Kruskal-Wallis better than newly minted statisticians do.
Sorry for the switch in programming languages, but I have no idea how to use Stata and a lot of people on here use R.
What do the data show?
Your intuition is right--this shouldn't happen. However it can happen if two different procedures are used. Eg, 1 thing I've seen several times is someone runs a paired t-test & then presents the results w/ a bar chart w/ 1 bar for pre & 1 for post, & the CI's overlap, causing confusion & anxiety. The analogous plot would have just *one* bar for differences w/ a CI that doesn't include 0. We need to know a lot more about your data & how you got these discrepant values, as @Procrastinator suggests.
Does the Cramer-Rao bound scale with the sample size?  (Better check...)
There is nothing special people do after PCA. I think that I don't understand your goal.
What interaction? Doesn't look like your model includes that...
When you say that the Sharpe ratio is the mean divided by the sample standard deviation... "Up to a constant factor (sqrt(n), where n is the number of observations)"... what does this last part mean?  Is it really the mean divided by the standard error of the mean?
A fundamental concept to keep in mind is that all probability distributions have parameters (even the uniform distribution).  As a result, tests measuring the distance between distributions, like the KL, need distributions to be fitted in order to have something to measure at all.
The mean of the errors should be zero in general, otherwise your model is biased. It would help a lot if you would say how precisely you calculate average mean (I've just realized that this is a bit of misnomer, since average and mean are for practical purposes synonyms) of errors, and how your errors are defined precisely.
Hi, Gung, thanks lot. I would finish my modification and share the progress soon!!:)
Ah.. good point. I guess 'ignoring end points' all no causal time series can be rewritten as a casual version in this way. At least when the relationship is linear.
@user40494 if you are happy with the answer by Dino Abraham please don't forget to accept his answer by ticking the green check mark on the left-hand side of his post.
I do not have a ready compendium, but I do know of several in which it is illegal. Your state appears to be more friendly to the insurance companies in this respect. I would think that using number of years of driving experience would be one way to "penalize" teenagers without explicitly using age.
Also recognize that many (most?) academic researchers and students have free access to mature packages like Maple and Mathematica through uni machines and/or site licenses.
This is a well-reasoned answer (+1).  However, I do not understand why the numbers 8350, 8000, 6000, 5500, 5400 play such a prominent role: this seems only to distract from the argument.  How are these numbers related to the question?  If you want to compute the chance of obtaining 100 or fewer sixes in 50100 throws, why not just state its value?  (It equals $1.25\\ldots\\times 10^{-3725}$, which is easily found in Excel using its `LOG` and `GAMMALN` functions.) BTW, your calculation is not quite appropriate: you need to find the chance of a run of 100 of *something somewhere* in the sequence.
(+1) Although a bit of attention has to be paid to the convergence of the bootstrap variance estimator as mentioned in references I posted. From (1) "The natural conjecture that the bootstrap variance estimator converges almost surely to the asymptotic variance is shown by an example to be false unless a tail condition is imposed on $F$".
Alex, thank you very much for your answer!  What I'm concerned about in all these models is the fact that my individual-level measures obtained during the group discussions are dependent on the other group measures, so the independence assumption will be violated if I use most conventional statistical models.  Perhaps this isn't a concern for the group-level analysis, but it will be for the individual-level and pairwise analyses.  This is why I was considering a linear mixed effects model, some variation of which might be most appropriate in this situation.
Not rejecting the null hypothesis is not a strong conclusion: it may just be proof that your dataset was too small to pick up the existing effect.
You can read the Wikipedia statements on [Gross fixed capital formation data and time series](http://en.wikipedia.org/wiki/Gross_fixed_capital_formation#Data_and_time_series) to see some of the issues about why capital formation is easier to measure than capital stock.
I have just seen in a research paper that kld should be 
$KL(p, q) = ½ *  ((μ₁-μ₂)² + σ₁²+σ₂²) * ( (1/σ₁²)  + (1/σ₂²) ) - 2
The issue seems to be that the conditional expectation ${\\rm E}[T \\mid X = A]$ is not $1/a$ once you know that the first arrival is from process $A$.
@Gracchus: Sorry, not a C++ guy, but you may find what you need in the Armadillo C++ linear algebra library (http://arma.sourceforge.net), which is also available in R via the RcppArmadillo package.
The data set is potentially too big to read in half of it...it is in a networking context where the device doing the processing can see tens of thousands of items per second, and probably has enough memory to store only a few hundred.

Also the data is definitely not Gaussian. In fact it does not fit well to any of the common distributions.
@Karl, yes it is linear in the beginning and then nonlinear.
Specifically, there is a property of the homogeneous Poisson process that often goes by the name of the *order-statistic property*. I suspect you should immediately see its relevance. In fact it can be viewed as the *intrinsic* property that leads to the definition of a Poisson process on more general spaces than the nonnegative half-line.
This might be helpful: http://www.unc.edu/courses/2006spring/ecol/145/001/docs/lectures/lecture26.htm
Thank you for explaning :-)
@chl of all the people here you should be the least concerned about voting enough!
Removing covariates in this way biases $\\sigma^2$ towards zero, which will invalidate $P$-values and confidence limits.
Same as:  http://math.stackexchange.com/questions/572162/what-is-the-distribution-of-an-outer-product-of-uniform-random-vectors
Then without any clear identification of preferred alternatives, the chi-square is pretty much as good as you can do (if your sample sizes are large you might look at a G-test I guess, but I don't think it matters). But the number of observations required will depend on how big a deviation from uniformity you care about - it may not need to be as large as all that. How big a sample can you take?
@NicolasZhong thanks for the compliment. `libsvm` does indeed not offer multithreading. Training a single SVM is not straightforward to parallelize.
Such a model doesn't account for the within-subject term because it assumes uncorrelated errors.
I'm not convinced this is an appropriate model to begin with. Are you familiar with the [Nyquist-Johnson theory](http://en.wikipedia.org/wiki/Johnson%E2%80%93Nyquist_noise) of thermal circuit noise? If you're purposefully doing something different, it would be interesting to see the motivation. Otherwise, it might be worth it to consider a more standard model. :)
@StéphaneLaurent  I have nothing to contribute as an answer to your question but I have been reading the dialog.  Nothing you said sounded offensive to me.  Also you said you didn't know the difference between classification and clustering and I haven't seen that simple question answered. It is the differece between what the machine learning people call supervised and unsupervised learning. In classification you know all the class labels for your data and use that information to construct a classification rule for future cases that don't have labels.  In cluster you have no labelling.
Adding percentages (or counts) to a pie chart (or at the ends of the bars in a bar chart, or at the dots in a dot plot) accomplishes 2 things: 1. Serves as an admission that the plot cannot convey the information by itself; 2. Converts the plot into a poorly laid out table (with colorful background).  And in the case of the bar chart it also 3. distorts the lengths of the bars (what effect this has in the bar and dot plots is less clear).
No problem, if you're satisfied with the answer, would you mind accepting it? Thanks.
How accurate do you think your simple threshold approach is? That is, is the method of simply looking for extreme Y deviation ( < 7 ) already 90% accurate, say?
I agree to a degree... I do report data imputed from graphs as a moderator/ in a subgroup analysis. After more consideration, I think it is best to remove such studies reporting percentages - I guess that does not really answer the question?!
Actually I think the question is on topic here, falling under 'statistical computing' but it would also be on topic at SO.
Could you clarify what it means for null hypotheses to be "reversed"?  E.g., if one null is that two means are equal, do you take the other null to be that they are unequal?  (This particular example leads to trouble...)
I see thank you!
Come on, please, **show** in your question the two options you are asking about (I mean: show the matrix algebra behind them). After that, you are in right to ask whether people think one is better.
@P Schnell Many thanks for your comment. Just one question... Can I split the model into two models in a way that each model just contains one dummy variable and all the IVs? And, then interpret them separately or I should have all the possible dummy variables in just one model?
The main difficulty is that the basket proportions are published in a random PDF somewhere in the dark recesses of the website. I seem to recall finding it once and no longer needing it by the time I found it. Or that was a dream and I never actually found it.
+1, nice answer, makes mine redundant
Wow, nice explanation! Thanks a lot for your help!
if it's really about programming in R, there are quite some plot people on http://stackoverflow.com that can help you out as well.
Answered here, right?  http://stats.stackexchange.com/questions/52631/conditional-expectation-has-the-minimum-mean-prediction-error?rq=1
If the underlying distribution of your data is really a $\\chi^2(k,\\lambda)$ this is enough. For theoretical considerations, read Wikipedia on the [method of moments](http://en.wikipedia.org/wiki/Method_of_moments_%28statistics%29)
Based on some of your other posts, you know how to markup equations. The text equations here are difficult to read and the (subscripts?) are confusing - can you mark them up with $\\LaTeX$?
That sounds great (that you have in-subject knowledge that will convey intuition), though my point was more that almost any measure of nominal nominal association in a rectangular table is not in itself going to convey much intuition; I don't think it's a particular criticism of Cramer's V. For example, V is monotonic in phi; it the criticism applies to one surely it applies to the other.
+1 For showing how to compute the expectation.  But why did you choose to replace $1/i$ by $i$ and not by the tighter values of $1$, which would give the much stronger bound $E[1/(X+1)]\\le 1$? This is still trivial, since $0\\lt 1/(X+1)\\le 1$, but it can't be improved upon without more information.
For such a large sample (hundreds or thousands) even slight modifications from normality is proved to be statistically significant thus KS probably is not the right choice for you. If you have the right to choose then you should rely on simpler statistics and plots to show normality (q-q plot and skewness/kurtosis coefficients).
"stepping through some of the data analyses done in R or Stata" - any examples you like?
Michael M, yes, $\\exp(\\hat{y})$ is probably what you want and yes, you need to find out what the intercept is.  You might have to fudge it by guessing the intercept and varying it until your model appears to reproduce any graphics and tables in the paper sufficiently accurately.
Thank you ! (sample size)*(proportion diagnosed) is less than 5 for some hospitals, but there are relatively few of these....
Have you taken a look into the [`tm`](http://cran.r-project.org/web/packages/tm/index.html) package? I haven't used it, but it's a "framework for text mining applications within R", so I guess it could help. You can also look into other CRAN packages related to Natural Language Processing: http://cran.r-project.org/web/views/NaturalLanguageProcessing.html
@Tamas Once you have a statistic/measure that gives an ordering (*histogram B is more like histogram A than histogram C is*), you can have a test of equality based off it (e.g. a permutation/randomization test).
You need to read ["Averaging correlation values"](http://stats.stackexchange.com/questions/8019/averaging-correlation-values/8023#8023).  A discussion there (since deleted) indicated some people do use the Fisher transformation for averaging while others raise doubts about its meaningfulness.  One thing is clear: any mathematical combination of your three correlation coefficients does not necessarily have any predictable relationship to the correlation between shopping and weight for all ages 21-50 years.
I've used Structure for studying population substructures. Do you have a pointer describing where it can be used to analyze genetic similarity between arbitrary groups and how it compare to, say, other methods?
Thank you, these are very interesting considerations. The DAG is really, really intriguing. I was aware that comparing p-values has little meaning; it is one of the considerations that led to my question. And I was actually studying Modern Epidemiology before I switched to "Epidemiologic methods" of Koepsell/Weiss. But now you recommend it so warmly I will give Rothman another chance.
Alternatively, you just expand out $(y-X\\beta)^T\\Sigma^{-1}(y-X\\beta)$ as $y^T\\Sigma^{-1}y -y^T\\Sigma^{-1}X\\beta-(X\\beta)^T\\Sigma^{-1}y+(X\\beta)^T\\Sigma^{-1}X\\beta)$ then simplify and complete the square in $\\beta$, which then gives you result of the form $(\\beta-\\hat\\beta)^T\\text{<something>}(\\beta-\\hat\\beta) + S$ where $S$ doesn't contain $\\beta$, and where the form of $\\hat\\beta$ and "<something>" are obvious by inspection of the expanded terms