tttjjj/elt_bench huggingface dataset:
@Peter Good point. I prefer condition indices, too, but it seems to me that VIFs are very popular now.
Interesting posting. I will look into that too. You are probably right.
@DevX: If you have too much data, you could try taking a sample and choosing variables based on the sample?
I think the OP meant "direction" in the sense of positive vs negative correlation, not the direction of any causal relationship between X and Y.
Perhaps we should discuss about the goals of your study, in order to check if nonparametric density estimation is what you really need.
I don't think the averages should change - could it be from rounding error? The decrease in "error" is expected. I believe you are estimating $\\sqrt{p(1-p)/n}$ with $p=1/6$, first for $n=20$ and then for $n=100$.
As well as editing your question to include the information in your comment above, could you add some information on what the purpose of the analysis is and what your research questions are.  Also perhaps clarify what you mean by "paired observations" in this since - where there two observations taking 80 times of each of the 29 cases?
No, except if you consider "reciprocal of the standard deviation of a chi-square random variable with $n-1$ degrees of freedom", as a "name".
An obvious lower bound is $\\Pr(T<n) = 0$, but I guess you're aware of that...
@chl About the only conceivable use of *numerical methods* in SDEs (as opposed to theoretical approaches, which have applications in areas of pure mathematics such as solutions of PDEs) is to find solutions to SDEs applied to statistical problems.  Therefore, although this question is of a general nature, I see it as directly applicable to statistics and potentially of general interest here.
There is no such rule. You can either show both or choose the one that is more relevant for scientific reason.
Generally speaking you're correct - you normally want to include lower order terms if higher order terms are in a model. I can't speak to what MuMIn might be doing though.
Why are you using a repeated measures approach?
@Chinook WLS is not the only method to address heteroscedasticity.  You could also consider using OLS with robust standard errors, which has the advantage that it does not require identifying the form of the heteroscedasticity.   Re my last para., if you do use WLS you potentially have two pieces of information that could be useful in identifying the form of heteroscedasticity as a basis for determining weightings: a) the pattern of the OLS residuals; b) the known standard error of $y.hat$.  So the latter could be relevant, but should not alone determine the weightings.
Closely related to your previous question http://stats.stackexchange.com/questions/82812/statistics-usage-for-customer-event.
Another version of Greg's answer: if 99% of all email on the Internet 
is spam, and we block 99% of all spam, what percentage of your email 
will be spam (answer: 50%). Used this in a RL presentation once, and 
the managers were shocked, and stopped complaining for almost 2 whole 
days!
Jeromy, there must have dependent variable and independent variable. Perceived ease of use, interactivity, and trustworthiness are independent variables in my study; whereas Weblog publishing software, comment system, and blogroll and hyperlink are my dependent variables. So, if the user perceived the particular corporate's weblog publishing software is easy to use; its comment system is interactivity; and its weblog blogroll and hyperlink are trustworthy, therefore the corporate Weblog is effective. And my questions that I'm facing now is SPSS output, I donno how to interpret it. Need help~
Not only is it correct to keep replicates, it is indefensible to remove them! In terms of what you did:  with 1000 data points, very small deviations from normality will give you results significant at conventional levels. Whether they are of practical importance is a different matter. You should look at a quantile-quantile plot too (normal probability plot, many other names). See `qqnorm` in R.
[Cross-posted on Mathematics](http://math.stackexchange.com/q/908048/119493).
i.e. giving you on the scale of the response variable. Left hand side plot, is the fitted values against residuals. A good model should not show any pattern. The middle and the right hand side plot is to check the normality assumption of the residuals. From the qq-plot on the right, your residuals are not normal. So the model is not good. Hope that helps.
@Macro Sorry for the late response. I am estimating each of the probabilities independently. So it should be the latter case. I am reducing a higher order MC to a k=1 MC, so this would be the case as well, right? But, $m$ states would always refer to the number of states of the input data, or would it change for the new states I create for my reduced model? Because, the reduced higher order MC has a higher states space $S'$.
+1, thanks for your suggestion! [Also, I didn’t explain the full scope of my study above. It’s a study of new molecular prognostic markers in cancer. Albumin status is one element of a previously published prognostic score that is currently used in the clinic, and the published guidelines call for it to be dichotomized based on the results of previous studies. I am trying to set up “Table 1” (*i.e.*, study population characteristics) of my paper and want to report all elements of the currently accepted prognostic score in my patients.]
This is a little unclear but the principle is easy. No whisker will (should) be drawn if there are no values in the interval between q1 - 1.5IQR and q1, and similarly for q3 and q3 + 1.5 IQR. Absence of whiskers is not a problem so long as the data match.
See http://en.wikipedia.org/wiki/Gibbs_sampling
Could you point me to some sources with a more detailed explanation or maybe try to answer the question fully so I can accept? I'm not sure I am following you but what I am grasping of your comment makes sense to me. I need a softer answer because I'm not very familiar with this theory. Not to principal components, but to their relation to what I am asking.
Alright alternatively. Well 2-way interactions means one variable has different effects on the different levels of the other. 3 way interactions will build on that. In the different levels of the two way interaction, the 3rd variable has different effects. But what does that mean? It means the effect of `c {0} vs {1}`  is different at levels of `a:b` which are l1=`a0:b0,a0:b1,a1:b0,` and l2=`a1:b1`. So we will compare `l1 c1`vs`l1 c0` and `l2 c1`vs`l2 c0`.In your case only `l2 c1`vs`l2 c0` was significant which was `a1:b1:c1` vs `a1:b1:c0` which is 0.18.I have used `c` as the third variable.
# Also, I have tried the following the above post:
mixed(mydata.mod1, mydata, type = 3, method = c("KR"))
# and 
m1 <- mixed(RT~Related*PrimeType*(1|Item)+(1|Subject), data = mydata)
# but it keeps telling me that "could not find function "mixed"" 
# am I missing the right package?
... for the moment think that at least in my field (biospectroscopy/chemometrics) few examples exist where I'd trust the optimization results to be better than a well-build model that was not subject to data-driven optimization. But then the largest study I've had my fingers in only had 80 patients, and that means that comparisons will essentially be guesswork. 500 cases looks much more promising (if they are actually independent cases, and this is not a 10 class problem).
@Srikant @whuber: the combinatorials... you're right. I started from an uniform probability, which is off course nonsense in this case. My bad
Do you only need a point estimate, or did you want an interval?
Are you sure that this is appropriate for a standard survival analysis?  It seems there are two mutually exclusive events that can occur.  Perhaps a competing risk model would be more appropriate?
I would like to estimate the intra-rater-reliability, because all tracings were done by the same person.
This seems to be not an answer to the OP's straightforward question. In none of the two links I could find any mentioning of k-medoid.
Possibly too simple a check - have you looked at your hospitals with higher incidence rates to see if the patient counts have increased more (as a proportion) for those hospitals compared to hospitals with lower infection rates? If your hospitals with higher rates are contributing proportionately more data over time, that would explain the overall rate, even though the incidence in each hospital would remain the same. All I can think is that the individual hospital contributions to the data are not remaining constant over time.
Thanks! However, they most likely didn't have a 0 problem due to the limited regions they explored and aggregation compared to the global industry level BACI-CEPII data I am using.
+1 While I most certainly agree with you, I can imagine that saying 'my claim is supported by MAGIC' might not always be helpful :-)
@Dragonwalker - you've got it, it turns it into a factor, and keeps the same name.  The values of the original variable become the levels of the factor - so you can put text into the variable instead of numbers, e.g., `x <- as.factor(c("M","M","F","M"))`, and the output will refer to levels M and F instead of 1 and 2 (or whatever.)
that's a really great hint. Thanks
robert - thanks for the reply.  With the exception of the Lessman paper, I have read all of the resources you have listed. My question was aimed more directly at techniques for data mining the data for the purpose of uncovering valuable nuggets of information.  I may ultimately use some of those nuggets in a model to pick the horses, but I haven't gotten that far yet.  Thanks again for the input.
@Alexis - There's not much point, you're right, but the question was phrased in such a way that it seemed to be coming from a student working on an assignment or practice set in a class so I proceeded as such. In that case, what other things you can do when the variance are not equal would be followed up in later lessons.
Also, should replacement be allowed. I assume it should be.
I read about GraphLab bu do you recommend another library (preferably in python) for implementing fuzzy algorithms?
However, a final provocation: let's make a 4-Likert scale from the one above. As the extremes of the scale were chosen right. Counterargument similar to Christian's: e.g. maybe (and we'll never know, if that happened or not) a higher spread of the scores in comparison to other answers happened due to the Irritation introduced.
that was a fantastic explanation!
:mihsathe . Yes it is safe. That is approximately what is shown in my first image in my second answer.
Thanks for accepting my answer chi, though your own reference list is much better and more recent. I really should have thought of a couple of them myself as I've got them on my hard drive, and may have even read parts of them...
Other alternatives worth considering are the class of cumulative probability semiparametric models, including the proportional odds and proportional hazards (log-log link) models.
I believe it is called auto.arima() in R (part of the forecast package), and not auto_arima().
+1 This is exactly the right analysis.  But why don't you finish the job and answer the question?  The OP asks whether this correlation is "high" and what it might *mean*.
@VaishakId -- I spent some time preparing an answer. Is it useful to you? Any comments?
@sviter, Why not you just unwrap a triangle of each matrix into a vector and then use some standard program which computes a (dis)similarity matrix between the vectors?
This question cannot make sense to those without access to the book. To make this an answerable question, please either (a) provide a link to those pages in the book or (b) an image of those pages or (c) a synopsis of the relevant material from those pages.
A uniform prior may not necessarily converge because the posterior is not finite. For example, a uniform prior on a normal variance parameter does not give you any kind of posterior distribution since it is not proportional to unit area. More to the point, why would you want to do this? MLE confidence intervals and Bayesian posterior densities are not identical. Even if the MLE is not estimable in closed form, any hill-climbing algorithm will do. A matrix of derivatives at the maximum gives you information about how curved the surface is, i.e. confidence.
One insight is that the correlation coefficient between any two vectors $u$ and $v$ can be computed first by separately regressing $u$ and $v$ onto the constant vector $\\mathbf{1}$ and then evaluating the correlation of the residuals, whence it makes no difference whether $\\mathbf{1}$ was included or not in the regression of $y$ against $x$.
It sounds like you really want to use GA. However, if you want to use time and machine resources efficiently, you should only use GA if you know of similar problems GA (1) solves, (2) solves more effectively than other techniques. I've watched people have fun with GA instead of solving solvable problems using less exciting techniques. In addition, you need to consider the set of features you have, which you did not describe in the question, and how you expect the fitness landscape to look.
It seems that your "equation" models a binary outcome. Now, Cox's PH model deals with survival data and what lies it the left-hand side should be an hazard rate... I think you should clarify...
For those wondering, in English versions that would be `=IF(RAND() < A1; 1; 0)`. BTW, you could also just write the logical statement `=RAND() < A1` and have the cell automatically output `TRUE` or `FALSE`.
Sorry nobody saw this sooner. I like the reference. Welcome to the site.
BMDP2V (http://www.statistical-solutions-software.com/bmdp-statistical-software/anova-analysis/)
@Glen_b: Edited above.
There is a problem. What exactly do you do with Pr [S union T]? How do you go from E [ S int. T / S union T] to the second equation?
For clarification, the difference between this solution and mine (see below), is that I test the equality of all coefficients (intercept + slope) whereas @gung test only equality in slope.
@jbowman Well, there are plenty of distributions with three parameters that do have these for any value of the parameters. An obvious example would be any actual solution but setting p(0) = 0 and renormalising. There is exactly three free parameters. There is no data, I'm trying to solve a particularly nasty recurrence relation, and I'm at the point of trial and error with solutions.
The title is also odd. If you have the actual data, there is no point in trying to use the histograms to differentiate b/t the classes.
Not surprisingly, a lot of people have looked at cheating detection in the past, including Steven Levitt, author of Freakonomics. If you want to be able to tell whether someone cheated from the answers alone, don't give multiple choice tests, and proctor the exams yourself. You might be able to reject the hypothesis that the students' work was unrelated, but you will have a terrible time proving that they didn't simply study together. Do you have a seating chart and did you verify the students' IDs, that they were sitting according to the seating chart? Can you retest the students?
@DanielJohnson, right. Let me think about it.
1. as far as I understand it, with every bootstrap sample draw, the lower bound of your p value decreases, but the actual value should asymptotically approach something not zero.
2. AFAIK, standard bootstrap test: calculate mean difference T from full data. Pool green and orange, repeatedly draw two random samples with replacement, calculate the mean difference B, and compare it to full sample mean. For which percentage of your draws was T higher than B? That is your *p*.
3. Easier to obtain and more informative might be a bootstrapped 95% CI of the mean difference.
wow, this Bergsta's random search sounds so complicated! But I guess I asked for more rigor : ) I'll research what a validation set is (at the moment not sure how it is different to the test set) and will try to implement what you have suggested. It sounds like a solution to my problem!!! By the way any thought on how the order of the training examples effects the learning by the NN?
Thank you for your contribution! The part of the question about linear regression actually is a duplicate of http://stats.stackexchange.com/questions/6920/efficient-online-linear-regression whose answers shows how to update a multiple linear regression model.  The present thread is allowed to stand because the *logistic regression* part of the question is independently of interest.  Actually, even the logistic part has been duplicated at http://stats.stackexchange.com/questions/59174/online-logistic-regression.
Thank you for clarifying that point. But the thrust of my comment still holds: since you have a sample of the objects, you can estimate the proportion of the negative ones by means of the proportion of negative ones observed in your sample. This will be a superior approach unless you have strong reasons to assume the values of the objects indeed have a Normal distribution.
(+1) Although the mathematics is essentially the same as in my answer, the focus on expressions that have probabilistic meaning provides a welcome clarity and simplicity.  Note that only the very last equality made any assumption about the distribution apart from its continuity.
Welcome to the site, @AJAY. This question appears to be *only* about how to get something done in Stata. As such, it would be off-topic for CV (see our [FAQ](http://stats.stackexchange.com/faq)), but possibly on-topic at [Stack Overflow](http://stackoverflow.com/). If you have a substantive statistical question, please edit to clarify, if not, flag your Q, & we'll migrate it for you (*please don't cross-post, though*).
Do you know how each measure is calculated?
The blue bars are the confidence interval.  I'd have to double check, but I believe that they represent a 95% confidence.  So anything within the blue bars of the graph are statistically insignificant correlation, and anything outside of the bars is significant.  The CCF also uses complete.cases in R, so missing values aren't used.  So if you've got a bunch of missing data, then the CCF might not be accurate.
Also thanks for explaining LOD in the genetics context. In my field (chemometrics) LOD is the limit of detection, for which the warning is also valid.
@soakley: could you please clarify a little bit further? I don't know, nor believe in, any specific ratio of the balls *à priori*.
Concur, the Bayesian approach is very simple for proportions, and gets directly at the salient aspect here which is the confidence interval can't go outside [0,1] and is likely asymmetrical.
Do you have any reason not to use ANOVA? This appears so natural for it that it could be textbook. If ANOVA does find differences, then use Tukey to find which pairs of groups have significant differences.
Welcome to the site, @user22299. Would you mind expanding on this w/ a few sentences to say what the package can do, & why you think it might be the answer?
That seems like a reasonable approach. Do you think there's a difference between your proposal of using a weighted mean of the director coefficients, compared to a 2 stage model where first I calibrate a model based only on other users' likes, and then on top of that model calibrating a set of "director multipliers", which would by definition have a mean of 1?

Re: collinearity, I'm keeping k small compared to the number of rows by applying some simple heuristics. I've also considered clustering the users into cohorts based on like patterns, where # cohorts << # users
Since the Cauchy is a specific t-distribution, it would be strange to select it as best. I suspect the `logLik` outputs are what they say they are: the log likelihood itself, which we wish to be as large as possible. That would select the t over the Cauchy or logistic (even when penalizing for variation in the number of parameters). (I am mystified by the appearance of a `df` parameter for the Cauchy or logistic distributions, though.) It is quite revealing that the optimal values of `m` and `s` for the standardized data aren't near their actual values of $0$ and $1$, respectively.
I have tried to use http://cran.r-project.org/web/packages/graph/vignettes/graph.pdf but it seams that the package "Rgraphviz" is not available anymore. Anyone have another solution?
If the concern is for equities, particle filters are used in estimating stochastic volatility in Gibbs Samplers.
For various reasons (basically conservative medical bias) I don't have the option of eliminating any tests, nor do I especially want to.  Think of it as maybe analogous to a credit score, where having a big credit card debt is "uncorrelated" with having a poorly-paying, apt-to-be-laid-off job, but the two together create a much more serious situation than either individually.
Sorry, meant to say "I can regress the column matrix B against the columns in A individually", meaning one-at-a-time calls to lm. Edited to reflect this.
Even with quotes, I'd argue that "PCA does clustering" is just plain wrong.
This might not need saying, but unequal variances can be something interesting in its own right, & not just a nuisance when trying to compare means.
"small sample" here is basically so small as to be pointless, basically any sample size greater than 4 will reject the null at $\\alpha=0.05$ for correlations greater than 0.9: `x <- seq(0,4); y <- seq(0,4) + rnorm(5); cor.test(x,y)`.
If you are red-green colour-blind, then know that the green line is above the red line. The green line connects open circle symbols and the red line connects plus symbols. If you are not red-green colour-blind, please try to remember this when you present graphs.
+1 I was going to post the same answer.
I think you are missing a "not" before "a multi-variate normal" distribution, so I added that. Feel free to change it back if I've messed things up.
> the problem with this method is that you can't control the ratio of smallest to largest eigenvalue (and i think that as the size of your randomly generated dataset goes to infinity, this ratio will converge to 1).
Most of the time, a singular matrix means that there is nothing going on in the subspace corresponding to the zero eigenvalues: a rank-deficient multivariate normal distribution is a point mass in the projection onto the "kernel" part of the covariance matrix. There may be other times, however, when a singular matrix flags a serious problem, e.g., when you have a combination of parameters that makes some of them unidentifiable. If you want to test $H_0: \\alpha \\beta =0$, and it is true that $\\alpha=0$, then $\\beta$ is not identified, and a Wald test for it cannot be constructed.
I believe the understanding of hypothesis testing that motivated this question is incorrect, however I don't see any reason a downvote was merited.
Are you using Linux or Mac? I can recall glimpsing over some problems installing the Mac version of lme4. There's quite a few posts on R-sig-mixed-models about this.
I was just about to edit my answer to include this following @whuber's comment :) IIRC, the fact that the zero was "filed on the wrong side" threw me off initially...However, I see [the OP wasn't all that adamant about designing the hypotheses that way](http://stats.stackexchange.com/questions/86125/hypothesis-testing-for-a-correlation-that-is-zero-or-negative/86134?noredirect=1#comment167979_86125).
I don't understand why do you need to compare the models; since they are set on different populations they cannot possibly be compatible with each other. You may want to modify the smaller model with a dummy variable so they both can be set on the same, bigger population. And then you can use AIC.
Do you mean standardizing by "suitable scaling"? I don't think that the original estimated coefficients are proper measure of variable importance.  I'm looking  measure such as the six measures discussed here:http://www.mwsug.org/proceedings/2009/stats/MWSUG-2009-D10.pdf   I am looking for R commands though. Thanks!