tttjjj/elt_bench huggingface dataset:
I agree with the essentials in @Michelle's answer, although strictly speaking the procedure can be accomplished in some conditions with some software.  In SPSS you can choose "pairwise deletion" and it's possible you will be able to obtain some sort of solution.  But I think it's safe to say that with only 20 complete rows listwise you would not want to trust those PCA results.  There are just too many parameters/relationships being estimated based on too little data.
The residuals don't have constant variance, even though the error term does. Points with more extreme 'x'-values have smaller variance. If you want to make them comparable, you don't divide by their *average* standard deviation, but by their individual standard deviation. This is what is going on. On the other hand, `rstudent` computes *externally* studentized residuals, not internally studentized residuals.
Can you explain what you mean by "risk" of oversampling? Oversampling is an accepted method for enabling accurate estimates to be made for rarer populations, see for example http://www.statcan.gc.ca/pub/12-001-x/2009002/article/11036-eng.pdf What do you want to do, so you can get advice specific to your situation?
One way to do it is to find a seemingly applicable study then discount it by doubling the variance of its point estimate or by making the prior a mixture of a skeptical prior and the previous results.  The mixing proportion is an "applicability quotient."
@mpiktas, OK.  I'm most interested in the general question, as downvotes are more common at some sites than others.  But if you have a good answer that is specialized to the case where downvotes are rare, that'd be great, too.
Welcome to our site! For Stata-specific help, please consult our [support page](http://meta.stats.stackexchange.com/a/794/919). For general solutions, [searching our site](http://stats.stackexchange.com/search?q=moving+window+regression) turns up some useful approaches. You might also look into LOESS (aka lowess) smoothing, which automatically performs a weighted version of rolling regression. If you are indeed seeking general solutions, formulas, or advice--and not just Stata commands--then please edit your question to emphasize that.
Presumably the predictors omitted are collinear with some of those included, but that's SPSS doing good on your behalf, not being difficult. Tastes and standards vary, but in my field firing 25 predictors at 475 observations would not be regarded as statistics worth much discussion. I'd advise against fitting anything like so complicated a model to a modest dataset unless you have a really good theory-based story to back it up. Incidentally, if you have 4000 companies, what happened to the other 3525?
I believe the normalization in proteomics is similar to microarray quantile normalization. However, I find your description a bit hard to understand. Could you pls make it a bit general: as in what are your rows and columns (in an excel or csv file) and any other details (not necessarily the numbers but what are your data?).
Almost a year later, I thought I'd comment on this. The developers seem dedicated, and the things that they plan on adding to it (variational inference!) all point to Stan inevitably replacing OpenBUGS and JAGS. That being said, I don't trust it at the moment and as of now would only use it when JAGS is not an option. It seems to require *more* tweaking to get sensible results in complicated problems, rather than less. Stan doesn't use Gibbs sampling at all, and for problems where I know Gibbs will work I can't justify using it. I think in the future these issues will be resolved, though.
Understood, thank you for your quick and clear answer!
from a biological point of view, that chance is 1. The environment is bound to change to a point that no population can survive, given that in x billion years the sun is to explode. But I guess that's not really the answer he was looking for. ;-) The question doesn't make sense either. An amoebe can only divide into 2 or 0. Moral: traders shouldn't ask questions about biology.
Good job!  Here's a suggestion for the first part: when you add $n$ identical copies of $S_m/S_n$ together, it looks like the sum will have a distribution whose expectation is easy to compute using only the i.i.d. assumption.
The Wikipedia article on structural breaks provides a good place to start, and cites a variety of approaches. http://en.wikipedia.org/wiki/Structural_break
This is not going to work, since the degrees of freedom of the likelihood ratio test grows as $O(n)$ for the saturated model.
I though you just wanted to assert that one method was better than the other. The [Welsch t-test](http://en.wikipedia.org/wiki/Welch's_t_test) give you a p-value for the null hypothesis if your populations follow the [student's t-distrib](http://en.wikipedia.org/wiki/Student's_t-distribution).
@vqv You're quite right. I was too hasty to attempt an answer before leaving for the christmas break. I hadn't come across that inequality relating the Pearson correlation of three variables before. Here's another ref for it complete with 3D visualisations: http://www.jstor.org/stable/2684832 . I still think it might have *some* relevance, so I won't delete my answer, though I can't see how to fix it either.
Welcome to CV. Your paper seems to be really good (just took a glance on it). However, the Stack Exchange community is a repository of information and usually it is expected from users to not provide only links as answers (they may crash with time). So the better here, would be if you could answer the question, maybe using some parts of your paper, that surely can answer this question too. Last thing, you might want to take a look that a really good answer already exists to this thread, so it would also be important, if you could add information that it is not on other answers. Regards.
Wondering if R is matching up the rows and doing T - T instead of T  - [T-1]
What is HDV? Could you provide a reference?
I'm afraid you're mistaken. "Discrete" is not the same thing as "takes integer values".
What about in the scenario when the classifier can pass? As in it says it does not know. Can you still use McNemar's test then?
Great, thanks, it's much more readable now.
Welcome to the site, @Marc. FWIW, this question probably belonged on [Stack Overflow](http://stackoverflow.com/). This is a gray area, but I would say that Q's focusing on a data visualization (eg, nature, quality, etc) go here, & Q's on *how to make* a data vis (eg, in R) are better on SO. Since you've already got a great answer, I see no harm in letting it stay; just FYI.
Do you want to calculate the derivatives on the log likelihood?
This answer is mostly nonsense. No number of flips of a fair coin will result in a distribution of number of heads that has probabilities $\\frac 18, \\frac 14, \\frac 12, \\frac 14, \\frac 18$; indeed that is not even a probability mass function! Nor does the **number of heads in a row** have anything to do with the question.
Regarding: `Although a PCA applied on binary data would yield results comparable to those obtained from a Multiple Correspondence Analysis`, can we not convert a nominal categorical variable (let's say with N cardinality) into a collection of (N-1) dummy binaries and then perform PCA on this data? ( I understand there are more appropriate techniques)
No, I think that adding those additional frequentist error rate complications would simply complicate the issues. It is my opinion (implied in my answer) that it is essential that users of statistics understand that they can view their data as evidence OR they can worry about frequentist error rates. (Mine is an unpopular position, but I think it is a defensible one.) To set out a system that attempts to protect type I error rates only confuses because people have an intrinsic understanding of evidence but not error rates.
Which book would explain statistics like this?
I agree with @Glen_b; you can pretty much answer the question by pulling information from the ppt and some deduction, e.g., when the  slide says, as far as I can remember, "the F test is not robust against deviations from normality", that's basically telling you that normality is an important assumption of the F test, and so on.  I doubt the professor is expecting you to provide an answer that's well beyond what he/she went over in class, so I'd just play follow the leader here and regurgitate what's in the ppt and any supplemental notes you took.
@an-old-man-in-the-sea  ...and this is why the sum of squared residuals equals a _scaled_ chi-square (i.e. a gamma distribution): because the sum of squared residuals divided by their variance follows a chi-square proper. As for the degrees of freedom, it's indeed Cochran's Theorem.
There are *many* examples on this site: link through the [tag:power] tag.  The first one I followed has a nicely illustrated example by Jeromy Anglim.
It is correct that questions about random variables are usually different than questions about data. But since nobody has attempted to answer yet, why not just edit this question so that it asks what you really want to know?
When you view the interaction as just another factor in a multiple regression (which it is), this becomes the question asked (and fully answered) at http://stats.stackexchange.com/q/17110.
Have you plotted your data to see if such a sharp cutoff actually exists? Following the work of Zipf, word frequencies often follow something closely approximating a power law, so creating a [Zipf plot](http://en.wikipedia.org/wiki/File:Wikipedia-n-zipf.png) might provide some useful information. As in the example I link to, the plot is rarely perfectly linear. That said, some very large corpora like the [British National Corpus](http://en.wikipedia.org/wiki/British_National_Corpus) have frequencies that are remarkably close to Zipf's original observations.
This is not a specific programming question; it sounds like you are confused about basic statistical methods. For statistical advice, try [stats.se] instead.
all cases are of patients that are already out of hospital. We analyze their data when their treatment ist over.
One more question, The Personality like I said is derived out of Five independent factors (derived out of factor analysis) and each factor is measured on a likert scale, same goes for the three components of commitment and they are also measured on a likert type scale. The studies I have read upto now uses partial correlations and hierarchical regression to link these contructs but I somehow think there has to be a better way. So do you think SEM is better than the procedures used thus far? Thanks a Bunch, Angie
As a hint/amplification/followup of @RomanLuštrik's comment: see Pinheiro and Bates 2000; you can see relevant bits on Google Books (pp. 236, 243-244).  This is basically the "what have you tried" comment  ( http://mattgemmell.com/2008/12/08/what-have-you-tried/ ).  If you give us a more specific starting point than "I don't understand" (and show evidence of effort), it will be easier/more rewarding to help ...
I'll take that to be 'yes'. Now the question is, for a given $n$, for what value of $p$ is that standard deviation largest? [since the effect of $/n$ is the same as you change $p$, you could just think about $\\sqrt{p(1-p)}$. Then since taking the square root wont change the order, you can leave that out as well (that is, if $p_1(1-p_1) > p_2(1-p_2)$ then their square roots will be in the same order; $\\sqrt{p_1(1-p_1)} > \\sqrt{p_2(1-p_2)}$). So for what value of $p$ is $p(1-p)$ biggest?
@HongOoi That makes complete sense now. Thank you for explaining it (I'm clearly a bit slow on this). Accepting answer!
I couldn't quite follow you at the last paragraph. Isn't mathematical fact that $ V(X) = E\\left(\\frac{(X-Y)^2}{2}\\right) = E((X-E(X))^2)$? Even though the equation is interesting, I don't get how it could be used to teach n-1 intuitively?
It's not only a repost, with 114 threads already created here under the tag "variable-selection," but it's also a huge and very contentious topic without a single correct answer. And it's not something you'll likely decide once and for all based on a few hours' study. I encourage you to read widely on your question, here and elsewhere. Good luck!
Can't speak to the statistical issues as I'm inexperienced with clustering methods, but as a personality psychologist, I discourage the decision to dichotomize extraversion/introversion. It is a continuous trait dimension, most people fall somewhere nearer to the middle than to either extreme (and as such aren't always strongly distinguishable from people on the other side of the mean), and some nontrivial subset of people aren't well-described as either introverted or extraverted (nor even in-between necessarily). If you have continuous trait data, try leaving it that way if you can.
This is a confidence interval rather than a hypothesis test.  For hypothesis testing you would use power instead of interval width/half-width. What is done is that you pick the difference worth detecting and choose the sample size that makes the probability of exceeding the critical value the specified  power ( say 0.95) when that alternative distribution is the true one.
http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models
This answer suffers from the same deficiency as your previous one: it is speculative because it does not demonstrate that the resulting marginal distribution will be unimodal.
Wow! Straight from the horse's mouth, as it were...It's definitely an interesting idea that hadn't occurred to me. Thanks for the additional info.
Personally I find it rather difficult to assess normality from a histogram (or a kernel density plot). I would never rely on them as an "ultimate" evidence. QQ plots are much more powerful for this purpose.
@whuber - thanks for the responses, I agree with everything you've said. I think some of the issues you raised can be dealt with (e.g. degree of smoothing can chosen to minimizes cross-validation error). Others are more difficult and would involve making assumptions about the raw data that we don't have enough info on. So in the absence of other solutions, I think plotting the original data as I described is the best bet. Perhaps it isn't wise to quantify the peak widths at all, as this requires certain "arbitrary" assumptions.
This could be included as a further reading, rather than a mandatory one. Thanks for the link.
Hello, while I am nox expert on both methods I can say that market research does a lot of such questions and mostly uses conjoint analysis for it.
If you are unsure, I would recommend consulting the book http://www.amazon.de/Using-Multivariate-Statistics-Barbara-Tabachnick/dp/0205465250. It contains excellent "when to use method [x] " texts which are really helpful to decide such questions.
Haha, true. Tukey gets a pass though!
Thank you for your comments! The study is actually an observational registry study. Yes unfortunately the paper does not give the raw values, only the adjusted values :(
I will try to do that, but in the meantime I'm curious about how PLINK can compute the pooled OR only  with this parameters:SNP idenitifier, OR   Odds ratio (or BETA, etc) and SE   Standard error of OR (or user-defined weight field). You can realize that PLINK did't ask for allele frequencies... so there is a way to perform this...
@MHA What you are trying to achieve is still unclear. What are those independent variables (what type/how many)? What question did you ask about those pictures (e.g. were participants supposed to rank them or express preferences)? Did everyone see the same pairs of pictures? Was each picture shown only once or several times? And finally and most importantly: What do you want to learn about those pictures or those scenarios?
Sounds like a self-study question. Please tag accordingly and show more of your thinking, rather than asking us what the answer is. In any case, we can't possibly know more than you do about **what they want here** unless exceptionally one of us is your teacher.
@Eduardo `ave` is a wrapper of `lapply`+`split`, check the code (-;
I actually just realized I was missing points. I implemented k-medoids with the points I had but when I tried to compute the silhouette, I found out I had missing points
"Uniqueness of the datapoint" and "retain the knowledge" are pretty vague.  It would help to have a clearer description or at least a good example of what you're looking for.
Could you provide some more info re: data your using and the code/software your employing?
Do you assume the $A_i$ are independent? Also, what is $\\Sigma$? 
I think there is a trick that can be used: if you can find the PDF of $G$, and it is a function of $G^{\\top}G$, then you can compute the PDF of $W=G^{\\top}G$.
@GregSnow has given you a good answer below (+1). On a smaller point, you imply that the $z$-test is the same as the $\\chi^2_1$-test. This is not true. To understand this better, it may help you to read my answer here: [The $z$-test vs the $\\chi^2$-test for comparing the odds of catching a cold in 2 groups](http://stats.stackexchange.com/questions/81975//82556#82556).
@onestop's answer is good; I upvoted it long ago. To get a clearer sense of the connection between the general linear model & the *generalized* linear model, it may help you to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909) (although it was written in a different context). Assuming your errors are normally distributed, but the error variance is not known a-priori, the $t$ & $F$ tests that software will return w/ an ANCOVA will be correct; the p-value from the Wald test will be too low, unless your N is very large.
I think the Bland Altman plot and correlation are just what you want. You may also want to look at any cases where the differences are very large, to see if you can explain why they are large.
Actually, Matlab ran for a bit, then threw a major error. Looking at scaling the features down now with relieff.
Then you can do for instance: `sel <- which(cluster.object==1)` and then `seqIplot(seqdata[sel,], sortv=sil[sel])`
Added sample data. Alternatively, you can replace `#goodMood` with `mood=10` and `#okMood` with `mood=5`, if it's easier to work with a 10-point scale instead of hashtags.
Just as your other question, what you need to do depends on what you want to achieve. Asking what is “necessary” or what you “have to do” without a clear objective is not fruitful.
1+ Oh, I see. Yes, that could be the reason, although I have to verify that the categorical distribution does behave the way it should. Thank you very much!
Perhaps I'm being dense, but can't you just collapse your model into $y=a_0 + a_1x_1 + a_2z$ and go from there?
Could you augment your question to explain what you mean by being a "necessary or sufficient condition"?  It is not clear how you intend this traditionally *logical* concept to apply.
This is a good answer (+1). I especially think the activities and simulations are very useful for teaching this subject. In an introductory course I taught a few times we used a web applet that allowed students to visualize how the statistic is calculated from the sample and how the histogram of the sampling distribution begins to take shape as you repeat this many times. I think that activity helped far more than anything I ever said to the students when it came to understanding sampling distributions :)
What do you mean by "perfect fit model" and "actual model prediction"? You may want to read [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/)
I think the issue gets muddled in students' minds by the notation where commas indicate intersection and capitalization may (or may not!) mean different things. I would suggest writing $$\\begin{align*}P(A\\cap B\\mid C) &= \\frac{P(A\\cap B \\cap C)}{P(C)} = \\frac{P(A\\cap (B \\cap C))}{P(C)}\\\\&= \\frac{P(A\\cap (B \\cap C))}{P(B \\cap C)}\\times\\frac{P(B \\cap C)}{P(C)}\\\\&= P(A\\mid (B\\cap C))P(B\\mid C).\\end{align*}$$
@whuber I concur with Dan's statement. The plot is very cool!
@Jason C I think I understand your problem better. See my edits.
@DavidZ I don't believe @gung's comment to be correct, so I unfortunately I cannot explain it to you. You can simulate some data where you know $\\sigma$ and use `censreg` on it to make sure I am correct. The various marginal effects formulas can be found in most econometrics textbooks, but I don't know how to get them in R automatically.
I added some explanations for step 2. I hope these answer your questions.
This question was also asked on [math.SE](http://math.stackexchange.com/q/504631/8348).
@Skolnick: but if you don't your results could be biased, you would like to know the impact of being in the club as compared to the countries not in the club (hence you need more out of the club countries to be confident enough). If you have a particular question if the entrance into club has had an impact, than introduce one more dummy for the countries that were not originally in euro-zone.
I cannot find such a comment, By the way I added cov(X) to my question and I mean X is a matrix of variables not necessarily symmetric. If there is such a formulation for cov(AX)? (Please see X as a matrix and not a random variable)
@rajansthapit I definitely agree with whuber.
@whuber I might have simplified a bit too much in my description. I do forall i { t = gamma(a,b); sum += t; d[i] = log(t)}; logsum = log(sum); forall i { d[i] -= logsum; }. Previously, this underflowed if a was very small.
Often times the huge numbers of features can come from sets with extremely high cardinality, like the vocabulary in your document collection, the part of speech, and so on. It's also fairly common to use features from neighboring words, so it's not necessarily the case that people are thinking of lots of unique features focused only on the target token.
@JasonR Good to see you on here. Yes WLS is indeed an interesting contender. I have gone with whubers answer because of the clever polynomial factorization, and because it maintains the error structure nicely.
@xan That's a good suggestion.  (As you can tell, I haven't changed the default color scheme used by the plotting software.)  I also like to vary the styles of line symbols whenever possible so that the curves can be distinguished even in grayscale reproductions (not all journals will publish color graphics).
Thanks! Running an exact Poisson gives me a significant p-value for the rate ratio difference (similar to that for your t-test). I wasn't sure how to code the t-test, so thanks for the code, all the code actually
Very nice answer. I was going to suggest (if possible) the author provide a sample correlation matrix to see where the trouble lies. Should not be too difficult to fudge it/make it ambiguous enough to share the data. Also if one can't reproduce the problem when starting directly with the same correlation matrix that partly identifies the problem.
@John As far as I know, Jacob Cohen coined the term in his popular power analysis textbook (1971, 1988).
well re: your second question, does your derived distribution put 90% of the probability mass below 7? If so, I would think the more important thing is to convince yourself that your mcmc algorithm is behaving correctly.
It's _densities_ that can be greater than 1, which is your main point. It is never true that probabilities can exceed 1, so the answer needs editing.
None of those. The first three will inevitably bias the estimates of any coefficients (variables not selected are forced to have exactly 0 effect). What is the purpose of the model? The Lasso or elastic net methods might be usefully applied in this case which applies shrinkage to size of the absolute (or absolute and squared in case of elastic net) values of the coefficients. When a coefficient is shrunk to zero it is removed from the model. I don't know is SAS has this, but the **glmnet** package for R will do this for you.