tttjjj/elt_bench huggingface dataset:
Much depends on how you think about seasonality. My patchy reading of the literature indicates that economists often regard seasonality as an uninteresting nuisance while environmental scientists often feel much more positive about it. The method of dummy variables as used in economics hinges, frequently, on data being quarterly or monthly and effects of holidays (in every sense of the word) being sometimes spikey; witn environmental data you can sometimes do well with some Fourier (sinusoidal) terms and need not resort to dummies.
@Dilip I think this answer may actually address my [question](http://stats.stackexchange.com/questions/97768/how-can-an-improper-prior-lead-to-a-proper-posterior-distribution)?
For example, variable Z follows a 2-component mixture model (with 2 density f(z1), f(z2)), with the mixing probability $\\pi$. And the likelihood can be defined as $f(z|z1,z2)= (1-\\pi)f(z1)+\\pi f(z2)$. How can I elaborate this adjusted likelihood in the bivariate case.
Uh, the spread of those residuals look okay to me. With that arrangement of x's what did you expect the spread should look like? (I'd worry more about transformations making your mean function nonlinear in x.)
Wow, thanks for going to all this trouble! I understand your qualms about bootstrapping, and I'm almost convinced. I think my main complication is the numerical covariate I have that will certainly be necessary (either by itself or in interaction with genotype), and that seems to rule out mt.maxT and plink, although I may need to look into plink again. But I will certainly dig through the references you provided!
Finding the covariance of $(Z,Z^2)$ does not require introducing $Y=\\exp(Z),$ which seems superfluous for that purpose.  Are you trying to ask a question about lognormal moments (moments of $Y$), about covariances of polynomials in a *normal* variable (such as $Z$ and $Z^2$), or something else?
Actually this is pretty close to the kind of stuff I'm looking for.
If the x axis represenst the corresponding fitted values, then you need to rethink at least two (rather three) of your answers. First hint: The average residual is always zero, no matter how bad the regression model is.
Could please add appropriate citation for the attached image, if this is not yours (otherwise, you can use $\\LaTeX$ to typeset it), and also indicate how it applies to the specific problem at hand? (The OP clearly stated that he's not too versed in mathematical statistics.)
I used the  package Performance analytics but it does not solve my problem
one constant ex. C=0.3
The regression line will be $y=\\frac{\\mu_Y}{\\mu_X}x$.  It can have a positive slope even if correlation is negative (or the other way round).  So to justify it, you need to believe that the ratio of the means carries all the useful information, and the rest is noise
Is it you that keeps ringing me unsolicited? That aside, why not start with averages across colleges for any performance measures you like? Naturally, whether the colleges provided value added for the people they took in is difficult to evaluate at the best of times. By the way, "parameter" in a statistical context means a constant to be estimated; it is not a synonym for variable or property.
+1 Nicely illustrated. Mathematically, the data are drawn from the sum of the error distribution and the (presumed) exponential distribution. It is tempting to estimate the error distribution and *deconvolve* the data to estimate the true distribution.
If there are a few outliers, then no test of central tendency is likely to be what you want. Although it is true that tests of the median are not affected by outliers (unless there are a huge number) the real conclusion is not that "this median is higher than that" nor that "this mean is higher than that" but that "this group has outliers and that group has not".
I had thought about functional data analysis, but it's not an area that I'm very familiar with. I'm working thru Ramsay and Silverman's book now.  But I don't immediately see how to deal with a multinomial outcome variable...?
The hyperbolic is a special case of the generalized hyperbolic (by setting $\\lambda=1$) and of the generalized inverse gaussian (setting $p=0$). See the respective wikipedia pages for GHD and GIG. If you still need it, the wikipedia page for the hyperbolic suggests that the Barndorff-Nielsen reference gives the mixing distribution.
@naught101 Definitions and the laws of nature (once we know them) are generalizations that I consider true. Though the former are not very interesting as in: all "true generalizations" are true.
@NickCox I used both, the use of 0.5*log((1+r)/(1-r)) was to help me (and others reading the question) understand the fisher's z transformation
For (2.), what if the response variable is not binary but continuous, will ANOVA the way to go?
Thanks Fred. Please note that I edited my first comment with some additional issues; please take another look. I have upvoted because you're clearly trying to make your question better.
(+1) For the OP's benefit, stationarity of the original series can be checked visually by examining whether the sample auto-correlation function falls off to noise quickly (exponentially).
risk differences is indeed the way to go, as that one can be understood by non-statisticians as well. Accepted.
@ArnoldNeumaier: $(X^TY)_K$ isn't computable. We only know the correlation with each predictor. $(X^TY)$ is in the "predictor domain", not in the "Y domain" (If N is the sample size and p the number of predictors, we only have p values, one for each predictor).
@chl Let's try [weighted-sampling] -- should be clear enough, IMO.
$r$ is the correlation that you _want_ $X$ and $Y$ to have, not something computed via Excel. A standardized $X$, call it $\\hat{X}$, is related to $X$ via $$\\hat{X} = \\frac{X - \\mu}{\\sigma}$$ where $\\mu$ is the mean value of $X$ viz.,the average of the $N$ cells if $X$ is stored in an array of $N$ cells, and $\\sigma$ is the standard deviation of the $N$ values of $X$. $\\hat{X}$ has mean $0$ and standard deviation $1$. Your equation thus is $$Y = r * \\hat{X} + E,$$ and $Y$ is also a standardized random variable with mean $0$ and standard deviation $1$. $aY+b$ also has correlation $r$ with $X$.
Yes Sir, I understand this. But my concern was in regard of statistical Package AMOS. In on-line materials (tutorials etc.) there is no explanation if AMOS contains  logistic regression analysis... Many thanks for your reply
(+1) Nice question. I will try to write something up for you in a few hours. In the meantime, you might try and see what happens with multiple categorical predictors that are not orthogonal (hint!).
Thanks for answering. You are right about the way I posed the question.

I am interested in testing whether a significant cluster structure has been found as a result of cluster analysis, so, I'd like to know of papers supporting or refuting the concern "about the possibility of post-hoc testing of the results of exploratory data analysis used to find clusters".
I've just found a paper from 2003 "Clustering and classification methods" by Milligan and Hirtle saying, for example, that using ANOVA would be an invalid analysis since data have not have random assignments to the groups.
Concerning k-means, do you think it would achieve a N complexity?
(1) Pollutant concentrations in natural media--air, soils, surface water, and groundwater--tend not to be normally distributed.  A good point of departure in many circumstances is to assume their *logarithms* have normal distributions.  (2) ANOVA etc. ignore the time-series nature of the data.  You need specialized tests here.  One of the simplest (yet powerful) approaches is regression of the (log) data against time, preferably using a model of degradation suggested by theory.
I was thinking of multiply/divide by two as well. However, I'm concerned that multipling by two each time a successful step occurs will end up in much more iterations. I was hoping that there is a way to do it using the gradient, since it provides some info on how steep the slope is.
David: I was not aware that the situations you were interested in involved such small values of $n$ (and you are right, the number of parameters is $(n-1)(n-m)$). May I ask why you now want this question closed? Did you solve the question? I note that there is nothing in the handout you refer to about the *estimation* problem you asked.
Is there another way you would suggest arranging the data? One that would suit another test that wouldn't have a problem with there being only 1 F-F dyad that was never seen to interact? I have data on individual dyads not just for whether or not they interact, but also who 'won' the interaction and how many times.
In what way is the data "non-normal"?  skewed? "fat tails"? outliers? You may be able correct this by using a different distribution (e.g. t or cauchy for fat tails and outliers, and rescaled beta distribution for skewness, off the top of my head), and save some power in your test.
If you give `summary(model1)` you'll see something like `Residual deviance: -2.7768e-28  on 0  degrees of freedom`
Ah, I followed the link on cross-posting in your comment and cleared it up for me. Thanks for the help!
that sounds great! Thanks for being willing to help. We have a couple of students in our lab and they might reach out to you if we need help.
Thank you, this is exactly what I was looking for.  I also did not know about column store db's.
@jsk my problem is different. My data is not a bunch of (x,y,z) points where in you are trying to apply GMM clustering using the EM algorithm. Let me simplify to 1D so you will understand. Think of a curve generated by summing up of gaussians each with a different center, different sigmas, and different height at the center. Given such a curve and given the number of gaussians, i want to solve the inverse problem of estimating the parameters of the each of the gaussian components in the mixture. Now, imagine it in higher dimensions (>=1).
Peter Ellis: I think you give a correct answer but it sounds like the OP is looking for a recursive fomrula for updating the estimate in the case of the unweighted estimate p$_n$=∑x$_i$/n the recursion would be p$_n$$_+$$_1$=n p$_n$/(n+1) + x$_n$$_+$$_1$/(n+1).
The paper asserts that P-R curves are less easily misinterpreted over-optimistically, but it says that description is outside the scope of the text.  Would you mind augmenting your answer with more material on P-R curves?
What are some good estimators when I have just 1 sample? (I've also edited the question)
Interactions make only sense across different variables. What is the interaction between two values of the same categorical variable? You can google categorical variables regression for more resources. I already included a link in my answer.
Whoa!! Thanks whuber!! It'll take me a little while to get through this completely, but thanks for taking the time and trouble!!
That's exactly right; I don't know why you think this is a partial answer. To supplement: suppose that 1 moment condition would be enough for identification of parameters, but theory provides a set of moment conditions, all of which are equally valid. In that case, rather than choosing one moment condition at random, it is intuitively more appealing to minimize some weighted average of deviations from each of the moment conditions. This is, roughly speaking, what the GMM estimator does.
@wayne,@whuber - Mean in the question is just for example and to simplify the question. By the way how amazing guesses!
Hi Etienne.  I love the teachingDemo, I will give it a look for the 3d simulation (are you talking about the dice role, or are there new stuff I do not know about?).
In general, the problem with going R on them (although I LOVE R), is that I will probably not have enough time for almost anything.  The idea is to get a "first impression" type class.  (for my regular teachings I would use R to teach concepts etc.)
Why not to exclude that binary variable and see what'll be with just continuous ones? You'll be free to choose among various distance measures. Maybe euclidean will more apt that manhattan (gower for continuous data is the normalized manhattan).
Thanks for the answer. Can you please expand a bit on what you mean. How does breaking the data into subsets help and what to do with the subsets after you train them? Also, are you suggesting using an ensemble of classifiers?
It's ok I found the answer looking at scikit-learn python code (because the glmnet source code is in Fortran and it is not my cup of tea). I will share it later if anyone is interested. Thanks anyway !
The same question is asked [here](http://stats.stackexchange.com/questions/68106/understanding-the-variance-of-random-effects-in-lmer-models), but a different answer is given.
This question appears to be off-topic because it is about diagnosing the likely cause of a heap space error in software.
Yes, R looks the way to go! I found a very complete report of a workshop on the spatstat module of R, that does exactly what I'm looking for : http://www.csiro.edu.au/resources/Spatial-Point-Patterns-in-R.html
I found that I also have to set the stop.if.error=FALSE when I put psi=NA: `o<-segmented(out.lm,seg.Z=~x,psi=NA,control=seg.control(display=TRUE, K=4, stop.if.error=FALSE));
plot.segmented(o);
lines.segmented(o)`
If the normality assumptions hold parametric confidence intervals should be available for estimating variance components in the linear model. The asymptotic chi-square statistic that you mentioned is used and under normality assumptions it is exact.
@Elpezmuerto. Yes pp plot and qqplot try to do the same job. I think QQ plot is as good as PP plot (with subtle differences). But as the great Cleveland, W.S. says, QQ plot is one of the best ways to compare distributions.
(I don't have the raw data)
@Daniel: Yes that's right. But I thought at that level the answer is "good enough". Don't you agree?
@riotburn and do I have to correct for multiple comparisons if I'll be using multiple dependent variables?
Closely related: http://stats.stackexchange.com/questions/30526.  You also want to check out the [tag:copula] tag--just click on the link here. A quick and dirty technique is to let $X$ be uniform$[0,1]$ and $Y=X$ when $X\\le\\alpha$ and $Y=1+\\alpha-X$ otherwise. The correlation is $\\rho=2(\\alpha-1)^3+1$, whence $\\alpha=1-((1-\\rho)/2)^{1/3}$ does the trick. But copulas will give you more control... .
Different software may allow different options, but there should be a principled answered, no?
It's a bit hazy on this side of the crystal ball interface, since you are using some sort of software that has it's own particular terminology. It sounds as though there was a change in the 2*log-likelihood of 6.298 when risk was added. Given the limits of what you have offered,  it appears the evidence in favor of "risk" (whatever it might be) being associated with shorter survival time is considerably stronger than was indicated by the Wald test.
Do you know which dots belong to which population? (Ie, is this cluster analysis?) Are you wondering how to *get* the line, or how to *plot* a line that you already have?
How then does the assertion relating to normality follow?
Thank you Peter. I'll do that.
There is no easy answer to this question.  There are a multitude of distributions that will resemble the above distribution.  For example, the exponential distribution will look like that or a beta distribution with the right parameters.
The closely related question at http://stats.stackexchange.com/questions/2976/ concerns clustering by similar values of the correlations themselves. Although this is not precisely what is being asked here, perhaps it offers some partial help. However, I suspect the present question has been stated ambiguously. As it is, there is an obvious and trivial solution: remove all "-" edges and report the connected components of the graph that remains. This clearly groups the similar objects and is unique; it's definitely not NP-complete. So what *really* is your question?
@Glen_b You can consult Hogg and Craig, 7th edition, page 193.
This would be easy to program in R or SAS.
+1 for point 3 alone. Cox describes the p-value as a calibration of the likelihood ratio (or other test statistic) & it's a point of view that's often forgotten.
@gung: Yes, that's right.  But the dependency of the next problem on the "current" one is weak.  New problems are randomly drawn from a bank.  The next one seen is only dependent because it might have been seen and scheduled already.
After loading your `test_data`, do this: `for (i in 1:2) test_data[,i] <- factor(test_data[,i], levels=levels(new_training_data[,i]))`. Words in `test_data` which did not appear in the training data will now appear as `NA`. Then your `predict` should work.
Thanks for fixing that, @gung
Stop paying the phone bill.
Do you assume that the constant of proportionality for both mu and nu is the same? if not the systems has many solutions right?
You have only 19 data values, they exhibit essentially no autocorrelation, and are reasonably close to normally distributed: in short, unless you have specific, quantitative hypotheses that were developed *before* seeing the data, about the only defensible conclusion you can draw is that these data look random. Your initial analysis with `bcp` got a good result. The second analysis--which amounts to a bit of "data snooping" and overfitting--effectively confirmed that result by producing untenable results. If any further analysis produces "significant" changepoints, *don't trust it.*
I changed your title quite a bit, I hope it clarifies rather than obscures your intent.
Similar version at http://math.stackexchange.com/questions/83200/tuning-the-birthday-paradox
Excuse me, I am also very interested in this topic, so can you share your multi-labeled short text database? Thanks so much!
(Somewhat tangentially) Related: [Lehmann-Scheffe theorem](http://en.wikipedia.org/wiki/Lehmann%E2%80%93Scheff%C3%A9_theorem).
I get the impression from some of your responses under my answer that you're in communication with a third party, perhaps a supervisor or some such, relaying answers and responses to them. Is this the case?
I don't understand well your last comment. On the other hand, I think I possibly got to a satisfactory conclusion: $(1-x)^n$ is the cdf  of the pdf I am looking for (in the original question). The corresponding pdf is $n(1-x)^{n-1}$ which is normalized and has nice expectation values.
(...cont.) But, for instance, Lemma 1 of [this paper on data compression](http://dx.doi.org/10.1109/LCOMM.2011.122810.102182) appeared in IEEE Comm. Lett. says otherwise. They claim that $v_i'$ are independent because rows of $A$ are linearly independent, which I thought was a mistake on the authors' side, although I might be missing something obvious. In any case, if I'm correct here, I should be able to prove that $v_i'$ and $v_j$ are not independent directly by showing $P(v_i'=1)P(v_j'=1)\\not=P(v_i'=1\\cap v_j'=1)$, for example.
@gung, thanks for clarifying.
@Ilik Something went wrong in my code (the `with(dd,` instruction was in the wrong place resulting in a weird test!). Thanks for catching that! Yes, I don't use results from the KW test but it's always a good idea to check it first, otherwise multiple post-hoc tests would be meaningless (or at least, suggestive of data snooping). **Note** I've corrected the code and nothing turned out to be significant, but I left the original image for the sake of clarity with significant results.
@chl It is possible your remark could be misinterpreted.  I *think* you are saying that one should seek non-mathematical justification of the interpretation of a particular ES, and *not* that you are saying interpreting an ES is altogether unjustified!
$b$ and $d$ are *not* random variables.
Agreed that the basis of error bars should always be explained.
Thanks you very much for your explanation. For this sollution i only need the data after the intevention? Because before the means would probably be the same between the experimental en control group. Furthermore how can i get the is βj(i), (ατ)ik, and ϵijk. Im sorry this is probably a realy dum question, but its been a long time since i worked with SPSS and had the research classes.
"*Question 1) does the assumption refer to the values of Y or the residuals of Y?*" --

Strictly speaking, *neither*, though the second is the thing you *check*.

What is assumed normal is either the unobservable *errors*, or equivalently the *conditional distribution* of Y at each combination of predictors. The unconditional distribution of Y is not assumed to be normal.
I took the liberty to add the “self-study” tag.
@whuber I've just added an example of the old and new pdf of random variable `A`. Thanks
@whuber Thanks! Just to make sure I follow you correctly: because my residuals look ok, I can stick with the 2-way ANOVA post-hocs (the Holm-Sidak-corrected t-tests) that my software provides?
I suggest you to read [this](http://dev.ulb.ac.be/sciencespo/dossiers_membres/gassner-marjorie/fichiers/gassner-marjorie-publication21.pdf) paper
If you have a more complex design you will need to incorporate that into the object you create. Perhaps you need something like: `d<-data.frame( X <- c(rnorm(50, 14,3), rnorm(35, 7,1),  rnorm(40, 15,9)), idx=rep(c("X","Y","Z"), each=c(50,35,40)))`. The principle is to make data in the same form as would be occuring in the experiment, .... before the analysis.
I wish i could upvote more. This *is* the reason i love this website: the power of specialization. i didn't know of the trapezoid distribution. It would have taken me some time to figure this out. Or I would have had to settle for using Gaussians instead of Uniforms. Anyhow, it's awesome.
**Hint**: Your code appears to be comparing the two using *in-sample* mean-squared error. Why wouldn't OLS be winning that contest?!
@rolando2 As a medical researcher, I find the sensationalist incompetence of mainstream science reporters hilarious.
Hi @whuber, these are weights that are calculated (with post-stratification applied) when sampling population and population of target inference are disparate.
Are you sure about the 0.0017.  I got something like 0.0044 with the continuity correction.  If I reject by getting 19 or less than, then I'm talking about the probability of 20 or more to not reject which would be 19.5 or more using the continuity correction, right?  Also, yes, I noticed while taking the test that the mean is inside the rejection region.  Thanks for this detailed response.  This is the sort of thing that is going to help me understand things better.
Can you give a reference/citation/description of what are `discriminant components`? The term itself seems not to be a widely accepted one, as, say, "discriminant function" or "principal component".
Whether this is urgent to you is not relevant to us. You  are not entitled to more or quicker attention than anyone else.
As discussed [here](http://stats.stackexchange.com/questions/4964/is-interaction-possible-between-two-continuous-variables), there are a lot of difficulties and various problems that can arise in interpreting the interaction between two continuous variables.
Do you have some general interests that you'd like to list? That might help guide suggestions. Applications of statistics have become pretty pervasive in a remarkably broad array of fields.
@huber I had previously commented about the strange negative phone calls. In response to your suggestion about taking roots ( meaning square roots) please see http://stats.stackexchange.com/questions/13095/types-of-inverse-transformations which shows that this specific transformation requires that the variance of the residuals be proportional to the Expected Value of Y.If this c an be verified then your suggestion of taking roots ( after making all the values positive) might be applicable.
Thank you but I'm not asking about the PSNR i ask how we calculate the SNR