tttjjj/elt_bench huggingface dataset:
I'm not a Stata expert, but my guess would be that you have at least one case of an industry have two observations for the same year.  Maybe check out the number of observations by year...
I love this one. It is great !
You should not be doing ordinary linear regression with a "bounded (0-1) response variable".  Neither the assumption about linearity of the mean nor the constant variance assumption are likely to be true, except in some special cases, and even then usually only approximately. Is your bounded variable compositional data, or is it a proportion based on counts, or something else?
I believe that you can use the linear regression function in SPSS to do each regression; but I'm not sure how you would automate the procedure to leave out one example at each step.
what is your sample size? I'm not familiar with this `R` package but the prior becomes less and less important as the sample size increases - [here](http://stats.stackexchange.com/questions/30387/what-is-the-relationship-between-sample-size-and-the-influence-of-prior-on-poste) is a question and two answers related to that.
This question appears to be off-topic because it is about how to re-implement a programming function.
Section 26 of the [user's manual](http://www.gpower.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Psychologie/AAP/gpower/GPowerManual.pdf) appears to address this question directly.
In principle, using a synthetic dataset where ground truth is available, then it is straight-forward, as there is then no sampling mismatch; the training set is just a random sample from the underlying distribution and you can estimate the error from the distribution itself, rather than a finite sample.  For real-word datasets, however AFAICS the best you can manage is to use resampling and determine the effects of over-fitting the model selection criterion over many random test/training splits.
Your guess is right -- it's a random number generator. (Sorry, I should have been more explicit!) I'm trying to think of a good *general* way to handle the situation.  I'm reminded of a test of a different part of the system where a function was called at 200 digits of precision and the result checked against a standard correct result.  When the function was tweaked to add precision, the last decimal place changed by 1 (making it slightly more accurate) but then the test failed because it didn't match the expected, less correct data! Down the road I'd not like to end up there again. :)
thanks! this is what I look for!
& I found more info at "Brillinger, 1981, Section 9.5" as Jolliffe cited him "There is a connection between frequency domain PCs and PCs deﬁned in the time domain"
Some hints: (1) The question has nothing to do with Arizona football (soccer), so you can ignore the first two paragraphs. (2) You *cannot* use $T_n$ to construct a confidence interval--but you can use it to construct an *approximate* confidence interval for "large" $n$. (3) You should be able to recognize a standardized random variable in the expression for $T_n$: that should immediately bring to mind the Central Limit Theorem. (4) The CLT makes a conclusion about a limiting distribution.
Can you give us a link to the paper?  There is no way anyone can helpou without seeing the paper.
Right, power analysis. I think that perhaps I do indeed care about relative frequency, however. I'll try to read around that also.

Without a clearly defined number of tests to run, I have been running 2% of the tests, selected uniformly randomly, on each of the datasets. 2% is arbitrary, but also tractable on the larger datasets. It means my sample size increases with respect to the population of tests on a dataset, which may lead to more tests than I _need_ on the larger datasets...
All improper priors are admissible so long as they result in proper posterior distributions. But including prior information, even something as simple as a lower bound, can dramatically improve the specificity of the results, that is, reduce posterior variance.
Hmm, but why then are there sometimes differences and sometimes the results are exactly equal? Furthemrore, then it seems to be impossible to use `lme` or `lmer`for calculating an ANOVA (strictly speaking) as it uses a method that is similar but not identical. So is there no way of calculating contrasts for repeated measure ANOVAs in R?
+1, this is a really nice answer! One thing I wonder about: you state that there are "3 levels of nesting (location, position, sample)" & also that "Time is nested inside Sample". There is a little tension b/t these statements. Is there a way to make this clearer for me?
The CDF $F(\\cdot)$ is a real function $F: \\mathbb R \\to (0,1)$ with the property that for each $\\alpha \\in (-\\infty,\\infty)$, $F(\\alpha) = P\\{X \\leq \\alpha\\} = \\beta$. Now, $F(X)$ means we apply the real function $F(\\cdot)$ to $X$ to get a new random variable $Y$ that takes on values in $(0,1)$. Also, $\\{Y \\leq \\beta\\}$ if and only if the event $\\{X \\leq \\alpha\\}$ occurred, and so $$F_Y(\\beta) = P\\{Y \\leq \\beta\\} = P\\{X \\leq \\alpha\\} = \\beta.$$ _Now_ see if you can answer the question in soakley's comment.
Re: "normal distribution or gaussian is a scaled and shifted exp (-x^2), so the importance of exp (-x^2) comes mostly from the fact that it is essentially the normal distribution." - The **probability density** of the normal distribution is a Guassian function. The normal distribution itself is not synonymous with $e^{-x^2}$, as this comment seems to indicate.
thank you!  does this have something to do with the "identifiability constraints"?  I never understand what is meant by that.
That's a good answer @Greg.  I'd just add that, usually, in a mixed effect model the main interest is in the fixed effects. Of course, that raises the question of what effect bad estimates of the random effects has on the fixed effects.
I would add another reason to perform an omnibus test: if the number of pair-wise comparisons is large, then it can save quite a bit of computation and interpretation time to perform an omnibus test... in a way it is a "how little work can we do?" strategy. :)
@mat What is SINR?
.. continued .. I.e., it took so many generations "before" and now so many
"after" the changes made to the algorithm. 

I will have to think about the unpaired t-test approach - my stats
knowledge is not very deep (or complete in breadth, so I have found
this site to be quite useful).

Oh, I also plan to use this approach with PSOs as well down the line.
(+1) Thanks for the link.It is also possible to consider a heterogeneous correlation matrix (see, e.g. `hetcor()` from the [polycor](http://cran.r-project.org/web/packages/polycor/index.html) package). Provided the VC matrix is SDP, it should do the work--mostly in the spirit of Factor Analysis. Nominal variables might be dummy coded.
Very helpful! Thx
@strimp Because you can then solve for the values of $a$ and $b$ that standardize $Y$ (make it have zero mean and unit variance).
@NickStauner you are right. But i'm interested in profile similarity in this sense: when the attribute is a numeric one then calculate a sort of distance between the two values (e.g age, coordinates for location, etc). When an attribute is a categoric one (like religion, gender, etc) consider two users A and B with religion_A = Lutheran and religion_B=Catholic then the two users are different with regard to religion (I'm interested only in perfect matching/mismatching of the categorical attribute).The same rationale is behind the two users who like non-identical TV shows
If cosine similarity is not a clustering technique it is ok. But I have seen it everywhere coming as a document clustering technique. So if I'm having only a training dataset of one class, and I'm interested only in determining given a document, whether it is related to that class or not, what approach should I use? thanks!
Broken links for Hamilton (1968) and Morren, Gelissen, and Vermunt (2009)...bummer! Couldn't find alternative sources with Google Scholar either (didn't try regular old Google yet though).
$R^2$ says exceptionally little about fit and (*in vacuo*) nothing about prediction quality.
Some magic numbers to interpret the silhouette values: http://stats.stackexchange.com/a/12923/12359
@MingL It is true that a model that constantly predicts a student to be a persistor would classify 85% of the students in the sample correctly, but including just some of the information available to you should result in a better in-sample classification rate.  This only suggests that looking at the in-sample classification rate independent of the data obtained is not a good assessment of goodness-of-fit or predictive ability of the model.  Using the Deviance Information Criterion or cross validation to evaluate the predictive ability of the model would be much better.
My elementary teachers would always couple "Great minds think alike" with "Fools don't differ" but I prefer your choice.
@coffeinjunky I will definitely read through that paper, it is surely relevant. Have you also seen other relevant papers you can refer me to?
Hi, thanks for your input. Is there a general theory of sensitivity/specifity I could learn about in addition to this package (e.g could you link me a paper or wiki article)? I'd like to understand the theory, and I probably won't be implementing this in R.
Yes I suppose you'd have to identify them first. Is this a problem?
I don't follow how you could have 0 degrees of freedom, can you clarify that? The chi-squared test should be the right test for this situation.
Having multiple measures that are similar is OK in a factor analysis; but you could miss things for factors that are under-represented. There's no real way around that, unfortunately.
I think that you should add a "treatment" column to `x` and then get rid of `gl()`.
BTW, [here](http://www.alessandroscocciapappagallo.com/programming/calculate-mutual-information-categorical-variable-continuous-one-r/) is code in case anyone wants to try the binning method.
+1, great example of why you can't take a wild stab at a useful sample size without key info.
I'm familiar with P(A|B)=P(A and B) / P(B). I'm just slow at seeing it here. So you're saying f(y) = P(B); f(y,lambda) = P(A and B); and P(Y|lambda) is P(A|B). Good stuff! Thanks for spelling it out for me, haha.
"*according to strict scientific guidelines*" -- where are these strict guidelines? I suspect you probably mean something more like 'according to convention'... and convention is very much dependent on what area you work in.
See: http://math.stackexchange.com/questions/379378/probability-that-multi-dimensional-random-variable-is-positive/379414#379414
Absolutely true, that is why I wanted to have a large  Ar and Ma model to test the algorithm performance. But most of the books are not freely available and there is also no preview (in Google books). Therefore, was requesting for the functional form of a large AR and Ma model where the number of zero-coefficients (lags) are more than the non-zero lags.
Fantastic example - completely makes sense now! Thank you again. Just one follow up question - say you identify the last row (like in your example) is collinear based on 2-3 other variables. What do you do? Delete the variable with 'index 5' which I'm guessing in this case is d_dpi? (Note: I am running least squares and a ridge regression separately.)
One of the best ways to improve a forecast is to average different forecasts (from different models, or from different experts or both...). You may want to look around on forecastingprinciples.com or look through past issues of the International Journal of Forecasting.
Several more duplicates can be found by linking through the [tag:normalization] tag.
@JoelW. I can do either rating with category or words with category. My problem is I do not know how to predict category on both rating and document. I did txt mining with document-term matrix.
Could you post an example?  (If you put an image up somewhere on the Web and provide a link to it, any high-rep user can embed it in your question for all to see.)
This code is mistaken because it does not enumerate all possibilities with the correct probabilities. Brute force (such as `with(expand.grid(u=0:9, v=0:9, w=0:9, x=0:9), mean(u+v==w+x))` in `R`) or appeal to properties of probabilities (as in the solution `p <- rep(1, 10)/10; sum(convolve(p, p, type="open")^2)`) reveal the correct answer is $670/10000 = 0.067$. As a reality check, a quick Normal approximation to this probability can be computed as `dnorm(0, 0, sqrt(4 * (mean((0:9 - mean(0:9))^2))))`, which works out to $0.0694\\ldots$.
I knew that spearman was just pearons on the ranks. I need to see whether there can be any other routes that could cause this as well, but influential tail observations that are more linearly correlated than the bulk of the data, but which lose their influence when replaced with their rankings would definitely cause what I am seeing.
Hi, I edited my question to make it clearer, hopefully.
What do you mean by "$X_1$" and "$X_2$"? Usually a confidence interval is a statement about a *property* of a family of distributions, but (unless you are doing a Bayesian analysis with a prior) properties are not normally distributed--only the variables described by those distributions can be. If the $X_i$ are intended to designate random variables, then you need to specify what property you are estimating: the mean?  Variance?  Something else?
@mpiktas, The Ljung-Box test is based on a statistic whose distribution is asymptotically (as h becomes large) chi-squared.  As h gets large relative to n, though, the power of the test decreases to 0.  Hence the desire to choose h large enough that the distribution is close to chi-squared but small enough to have useful power.  (I do not know what the risk of a false negative is, when h is small.)
Do you want to know how to find the distribution first (with $n$ as a parameter), but feel perfectly capable of finding the limit as $n \\to \\infty$ once someone tells you how to find the distribution, or do you know the distribution (for finite $n$) already but are having difficulty finding its limit as $n \\to \\infty$?
Interesting question. For questions related to programming, such as this one, have a look at stackoverflow.com.
You could consider the relationship with AIC and [this relationship](http://en.wikipedia.org/wiki/Akaike_information_criterion#Relevance_to_chi-squared_fitting) with the chi-square fitting (under the corresponding assumptions, of course). If you want to use probabilities, then I think Bayes factors would be more suitable, but then you need to choose a prior for the parameters.
(+1) [Mosaic display](http://www.math.yorku.ca/SCS/sugi/sugi17-paper.html) are helpful to visualize residuals, as you stated in your last sentence.
@Glen_b edited again, thanks
This seems contradictory to me--can you explain?:  "Most of it appears relatively random, or the missingness seems well described by the observed data."
@Tomek, I think there is no one good way of choosing them. Usually it depends on the data. I suggest mean for the intercept and zero for other coefficients.
Specificity provided :)
What about the intensity function? The literature I'm reviewing states that 

"For any nonhomogeneous poisson process, the intensity function and the hazard function for the time to first failure have the same functional form"

Yet this just doesn't make sense to me. Are you familiar with the intensity function?
aah, I see this was produced in R :-) using `fft(c(1,1,2,3,4,5,5,4,3,2,1))` and `fft(c(5,4,3,3,2,2,2,1,2,3,4))`.
40% accounted variance is not too bad and is encountered frequently. Try also 5-dimensional solution - will it help? will it be more interpretable in the end? `some of the categories in 10-category variables have no frequency` don't quite understand it. Do you want to say you have fewer original values than 10? Then try discretization to fewer categories
Thanks - any chance this is the Rodriguez paper you're thinking of?  http://arxiv.org/abs/bayes-an/9504001
Thanks for the answer. So, just to make sure, in the original PAM, in each iteration I choose one swap that maximizes the total medoid similarity for whole dataset (sum of similarities between all points and their clossest medoids). Is that correct?
$ = -\\sigma_{x}^{2} \\cdot \\frac{1}{\\sigma_{x}\\sigma_{y}}$
$ = - \\frac{\\sigma_{x}}{\\sigma_{y}}$

So this shows that $e_{t}$ and $e_{t-1}$ are indeed correlated right?
Are you able to state which (family of) component distributions you expect? If yes, then this means to fit a mixture model to the underlying data. The main way to do this is the [Expectation Maximization algorithm](https://en.wikipedia.org/wiki/Mixture_model#Expectation_maximization_.28EM.29) which however is hardly straightforward. This is not an easy problem, at least not in generality. It is only a deconvolution problem if you start from the (estimated) densities and if the component distributions are exactly the same; and even then, deconvolution is quite tricky in its own right.
Thanks for the help! I will give this a try and I hope he agrees with this method. Have a nice day sir
+1 Thanks for the answer. However normalizing the weights in the glm fit gives different standard errors than the fit with svyglm.  The svyglm results match up with the results from SAS.
Here is another good answer about the class imbalance problem (this time the question was about logistic regression) http://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression/6072#6072
No source is given here. It does no harm and is often immensely helpful to indicate where you found such material.  But the right-hand-side of your equation gives the slope of a simple linear regression of $X$ on $Y$ {NB that way round!!!] in terms of the correlation $\\rho$ between $X$ and $Y$ and the ratio of the standard deviation of $X$ and that of $Y$. The notation $\\rho$ by convention represents a population correlation; if that is used I'd expect the notation $\\beta$ to be used for the slope.
A link explaining why random-walk needs a burn-in also would help.
There is loads of literature on this problem. Check out http://www.forecasters.org/ijf/journal-issue/320 for a few papers published in the IJF.
To the best of my knowledge, there is no proven 'mathematical' argument for excluding 'small' studies from a meta-analysis as they will get generally get a 'small' amount of weight. The main issue is really with the workload of extracting and managing data, conducting critical analysis, etc. on studies that are deemed to provide very little input into the final assessment. This is contradictory to the principles of systematically reviewing the evidence, but a meta-analysis does not particularly need to be get it's data from a systematic review.
Why do you want to use one data set over the other? If they contain the same info why not just mix them up and train your random forest on that?
Welcome to CV! I reformatted the formula using the built-in Latex formatting syntax; feel free to roll back the edit if I made a mistake. Also, do you have more detail on the question? What do the variables mean? What exactly was the context?
I just started using R recently, so I'm just running through some basic tutorials, trying to make sense of the numbers that R is spitting out.
If you want the PCA, why not call `[COEFF,SCORE] = princomp(X)` (see [here](http://www.mathworks.com.au/help/stats/princomp.html)) directly?
"*learn statistics so well that I will never need figures to display data*" --- *all* the people that I know that really "know statistics well" use graphical displays *constantly*. "*as it is far more easy to put numbers to LaTeX-document that make plots*" -- that's about the worst reason for avoiding plots I have ever seen. Firstly, it's not at all hard to put plots in LaTeX documents, and R (and RStudio) make it much easier still (via things like [knitr](http://yihui.name/knitr/), for example, which has a whole *manual* specifically on including graphics)... (ctd)
Since in an exponential distribution the population standard deviation is equal to the mean, one of "exponential, similar means, different standard deviations" will have to give. Indeed give the standard deviations look to be quite different from the means, with such large sample sizes you can't have exponential distributions.
You don't mention the type of product X is, but you might see if the Bass model is applicable.
Like David said in the comment, it seems your asking about the between group variation for `x` within `g`, which doesn't have anything to do with a model of `y = x|g`. `x` varying within `g` is neither a necessary nor sufficient condition for intercepts or slopes to vary when predicting `y`. So a second test seems appropriate to answer that question - although your example graphic sort of says it all in this example.
Maybe some remarks about the desired use?
...and why wouldn't a non-Bayesian avail herself of the additional data, too?
HI Mike - you are quite right - it was your note on ezMixed() I read and misinterpreted this for ezANOVA().
That is not always the case, ref: http://www.cochrane-net.org/openlearning/html/modA1-6.htm. I am really looking for the cut-of where you decide to use method of "difference of mean change" versus "difference of final values".
@Mars,  I think my answer does indirectly answer at least parts of yours.  The regular KS test assumes no ties and uses a theoretical model for the test statistic under the null hypothesis.  With ties the theoretical model is not exact but you can instead use simulation (permutation test, simulate from null dist, etc.) to compute the null distribution and therefore get a p-value, etc.  I would guess that that is what the simulate argument you mention does.
Could you expand on just how TLS "uses" the smallest singular vector?  It looks like [Standard accounts](http://en.wikipedia.org/wiki/Total_least_squares) of TLS are the opposite of your characterization: the smallest singular values are *zeroed out* and effectively ignored in order to obtain the fit.
Which book are you referring to in American Statistician? Lange?
@Dean The code is correct.  What you're overlooking is the rounding that occurs in Excel's display of the [Period] values: note that the displayed difference between periods is not constant, indicating that there are some fractional seconds here.  As you can see in the [Started] column, all six starts are counted, but also one batch ended at exactly 18:02:11 (row 7), which is counted in the [Ended] column.  Thus, immediately after 18:02:11, there really were only 6-1=5 batches still running.  To avoid confusion, it's probably wise to set *all* the period cutpoints at fractional seconds.
In addition to this, how am I able to use the raw data in order to know whether there is a match between two unrelated participants without knowing the probabilities of it occurring?
Could you please explain better why $w_i=a_i b_i$ in the linked formula would not fit your needs? Have you got an *additional* weighting $\\omega_i$ for the $(X_i,Y_i)$ pairs?
@Zach I'm pretty busy at the moment, but I'll see what I can do about open-sourcing it in the future. You can also email me at the address in my profile for a copy of what I have if you don't mind sorting through an undocumented/half-finished project.
Centring/scaling predictors isn't "much of a solution" because there isn't much of a problem. If three predictors take values of around one hundred, their three-way interaction takes values of around one million; it may therefore be convenient for calculation & interpretation to use linear transformations of some or all of the predictors that result in lower numbers to deal with. Such a change makes no difference whatsoever to predictions, likelihood, &c.; it's just that some coefficients are now measured in different units.