tttjjj/elt_bench huggingface dataset:
I like your answer too, but Wayne's similar explanation was 2 minutes faster, so I ticked his. Thank you for helping me understand this step!!
After doing a little more research, I went with the Boruta algorithm as implemented in the R package Boruta, and that helped a great deal (and conceptually is very cool as well). I ran VIF analysis as well, and that largely verified the Boruta output. So, problem somewhat solved.
1) Does the chair picks people sequentially or "all at once"? 2) You are searching for a _conditional_ probability here, i.e. P(all men | all same sex) , P(all women | all same sex)
Eigenvector u is the direction of the axis (values of u are the direction cosines relative the original axes). Xu is the data itself, the values of the principal component, the coordinates onto the aforementioned axis).
This answer hasn't been upvoted because it is incorrect: it uses an approximate rule of thumb (as if it were exact) in a case where it is inapplicable.  The answer would be true if replaced by the conclusions of Chebyshev's Inequality (which states that at least 75% of the *observations* lie within two SDs of the mean observation; i.e., at least 75% of the responses are 2's or 3's), but this will not provide much insight.
I tried to avoid using the scary term "multilevel modeling" in my answer, but you should be aware that in some cases it provides a way to deal with this kind of situation.
(3) Again for EM, when you say "specify the family of possible PDFs" for the EM, does this mean that we give it "possibilities' to work with, for example, "This data is made of two gaussians and one poisson", or "This data is made of 3 chi-squared PDFs and 1 gaussian", etc? This is confusing because it means we specify $N$, which I take it from your post is something EM gives us...
not AUC but related: on micro/macro precision-recall curves at http://stats.stackexchange.com/questions/21551/how-to-compute-precision-recall-for-multiclass-multilabel-classification
Another recent pitch question: http://stats.stackexchange.com/questions/11885/odds-for-high-in-the-card-game-pitch
The answer to Q2 is not a number. What is your homework really asking? The expected number of distinct balls drawn by Bob in three months? And does three months mean 90 days or 91 days or 92 days?
@chl i wasn't aware of those links for SPSS, SAS and MATLAB. I completely agree that the splitting sample approach is prone to overfitting, but it is less prone to overfitting than is prforming EFA and CFA on the whole sample (which happens far too often for my taste)
p.s. Jeromy, I really like your blog. It has helped me a lot so far and will certainly do so in the future :) Thanks!
Great question...  The distributions appear slightly different, but I'm not really interested in that.  I'm more interested in whether the means of the two groups is significantly different, which is why I was thinking a simple t-test.  I'm just not sure if it's ok to use that method with percentages, and also negative percentages...
You might want to read the paper "Thirteen Ways to Look at the Correlation Coefficient" (Rodgers & Nicewander 1988). As the title implies, it discusses thirteen different intuitive views of the correlation coefficient. So hopefully at least one will click :)
The connection with kriging is good, but it should be mentioned that only certain kinds of irregular time series could be viewed as having variable support. In this context, the "support" of a value is the *duration* that the value represents. E.g., if the time series consists of eight-hour total readings of particular matter in air sampling stations, obtained daily during weekdays only, then the supports are *constant,* equal to eight hours. Thus the change-of-support (or variable support) issue is different from the irregular spacing of the supports themselves.
By correlating them? I think I am missing something. To see if two variables are correlated, run a correlation.
Pedants' corner: The ratio in question is named after J.P. Mills. Thus Mills ratio, Mills' ratio and Mills's ratio are all correct renderings. But any use of the apostrophe is dangerous as it tends to be copied in the wrong place. So I suggest "Mills ratio" and this seems to be the commonest correct rendering in the literature.
I think you should start experimenting with some known decision tree model, like the one you mention. But you should not only throw at it the features you already have, you should use your insight (and maybe some analysis of correlation between features) to specifically design your own variables, like the one you mention.  Then see how well that works. Or you could prepare a competition to post it at a site like www.kaggle.com
The answer depends (*strongly*) on whether these repeated correlations are independent.  In a common case you have a set of *m* related variables and you test all m(m-1)/2 mutual correlations, which obviously are not independent.
this is a subset of [that linked question](http://stats.stackexchange.com/questions/19265/deciphering-output-from-r2winbugs). See comment in my answer here: http://stats.stackexchange.com/a/19273/5509 .
The results are probably on the log-odds scale. (They might be on a continuous probability scale.)
Yes, there are such tests. The point is that one can contemplate creating a test based on generating data consistent with the histogram, as Nick describes here, and then work out (mathematically) the distribution of the test statistic, which will *necessarily* depend only on the information available in the original histogram. For instance, a simple powerful test could be based on the maximum achievable value of the Kolmogorov-Smirnov statistic among all datasets consistent with the histogram (its "p-box"). That statistic is easily computed from the histogram cutpoints and heights.
Maybe you have a big sample size ? In regards to your initial question, I am under the impression that, at least for the binomial case, the glmer model is equivalent to the MCMCglmm model with $\\sigma_e=0$. What happens if you set a prior on $\\sigma_e$ highly concentrated at $0$ ?
(+1) But, I think there are some typos in your last paragraph. Also, you can skip right over the line invoking conditional probability with no loss whatsoever. :)
Hmmm, can you post a csv file of your data, say to dropbox or similar?
Differencing should take out a linear trend. Differencing twice takes out a quadratic trend. If you had to detrend AND difference, presumably the trend has a quadratic component (or is more complex than linear).
Thanks Zhubard I will do that
One note: Where you have 1 - p you mean p - 1. I'll fix it.
@Stephan Laurent Yes, of course, but I wanted to calculate the _integral_ for which the OP said he was stuck. And the bulk of my calculations are not about the constant term, but about manipulating the terms that contain the integrating variable.
As the latent variables are standard normal the covariance is just correlation. I think it is just this – it seems reasonable for factors to correlate when they are identifying traits / behaviours - although you can set the correlation to zero if it suits your hypothesis. If you think their may be a second level of latent factors you can examine this.
As you've stated it, Jensen's doesn't immediately show it's biased (since it could be equal). Of course, [as Wikipedia says](http://en.m.wikipedia.org/wiki/Jensen's_inequality#Measure-theoretic_and_probabilistic_form) "Notice that the equality holds if and only if X is constant (degenerate random variable) or φ is linear."
Closely related: [How small a quantity should be added to x to avoid taking the log of zero?](http://stats.stackexchange.com/questions/30728).
Basically, the results will always be biased?
"Non-linear" helps but it's so vague and imprecise I can't envision how anybody could offer more than a generic answer unless perhaps someone comes along who has solved exactly your problem before. Assuming that's a slim chance, it would help to provide more details.
I always look forward to your questions and answers, Frank. Even when you discuss topics I feel I already understand, there's usually some interesting idea or fresh perspective.
Averaging is not a good idea. Gilbert quotes Land in recommending that cubic interpolation in these tables should be adequate.  A link on the web page I referenced ("Perform linear, quadratic, cubic, etc. interpolation (used to interpolate Land's H factor tables)") will send you to a spreadsheet showing how to do that. With a sample count of $4$ the results are heavily dependent on the assumption of lognormality, so you have much more serious issues to deal with in evaluating the sensitivity to that assumption.
(1) It is clear there is no bias, because `((1:n)-0.5)/n` is reversed under the transformation $x\\to 1-x$ and so is the inverse CDF of any symmetric distribution such as the Normal. So, what evidence do you have about a biased "intercept"? Are you computing it correctly?  In your example the intercept looks *perfectly* accurate. (2) If the points in the middle were *not* denser, then the Normal distribution would be a uniform distribution.
A couple things: How is this "in `R`"? What is the distribution of the test statistic (i.e. how do you go from this to $p$-values)?
`kidneyExt.cox$loglik` returns the LLs of a particular model (here `kidneyExt.cox`). The LRT mentioned in the last line tests against a model without any predictors. You should be able to test a model without any predictors against your actual model via `anova(nullmodel, kidneyExt.cox)`.
@Kirill... Thanks I think I get it now.
Autocorrelation reduces your effective sample size and impacts the partitioning of your data. Knowing this I would gravitate towards simpler algorithms. Cbeleites' answer was good at highlighting the benefits of using ensemble methods like a random forest to accommodate nonlinearities with sample size constraints. Again, without knowing specifics about your problem it is difficult to make a recommendation, but I think a random forest would be a good starting place given that you are dealing with a binary classification and you have a large selection of input variables.
Thank you so much!!! I really appreciate that. I have one more question if you don't mind. Which values should I use for Johansen's test and estimating the VECM? level or the differenced values? Thanks again!
What $\\chi^2$ test you have in mind? $(\\theta-\\theta_{ML})(Var(\\theta))^{-1}(\\theta-\\theta_{ML})$?
If you were given just the formulas for these functions and not told that they were cdfs or pdfs (in fact, suppose that you had no knowledge whatsoever of probability and/or statistics), could you find the partial derivatives using standard calculus techniques such as the chain rule?
Can you do any investigations with the patients in the mixed category. Are there any clinical hallmarks of this group, e.g. on one particular measure of the test?
@January Good catch, likely true that Simpson's paradox will not kick in here (the effect will not reverse) but there still will be a bias (remember with binary data the variance depends on the group proportion). Pc is proportion in control group, Pt is proportion in treatment group. Also if you are not familiar with paramteric likelihood - my answer will seem (unavoidably) strange and opaque. Maybe I'll ask a question some time on how to make that less so - the O'Rourke and Altman paper seems to be a failed attempt to do just that.
Painful though it is, your speculation that a flicker might actually have different RGB components (though sometimes a similar distance from black) would be worth pursuing. Another option is to simply use the median instead of the mean, depending on your goal.
That's a good explanation, but what about the equation they provided? In your 1 feature example, where I want the estimator to be 1 degree away, (i.e. d =1) then their equation `n~1/d` would mean n needs to be approximately 1? That doesn't make much sense?
Note that use of resampling with replacement as an approximation to future data assumes that the observed data is representative of that of the data generating population. However, I believe this assumption is held rather broadly across statistical tools.
David, Post your data to dropbox.com so we can take a look.  I'm not sure why you a reaching for logs.  Transformations (like drugs) can have nasty side effects that are masking the real issue (like stress :) ). Instead of a log, you might need to consider other things like level shifts, changes in parameters, etc.
Thanks for your reply. I think that the idea was that we know one groups mean will increase over time, and the other group wont increase or dont increase as much. The idea is to show that the slope of one group differ from the other.
Anyways, I will look into repeated measurements more thoroughly.

Regards
depending on how much data you're dealing with, loading your data into [monetdb](http://www.asdfree.com/2013/03/column-store-r-or-how-i-learned-to-stop.html) might circumvent this issue.  if you make a `monet.frame` object [as shown here](http://www.asdfree.com/search/label/basic%20stand%20alone%20medicare%20claims%20public%20use%20files%20%28bsapufs%29), you can run the `quantile` function on it and get an answer from 67 million records in a few seconds.
+1 for a good and informative answer. I find it useful to see a good explanation for a common misunderstanding (which I have incidentally been experiencing myself: http://stats.stackexchange.com/questions/7022/parameter-estimation-for-normal-distribution-in-java). What I miss though, is an alternative solution to this common misunderstanding. I mean, if normality tests are the wrong way to go, how does one go about checking if a normal approximation is acceptable/justified?
@glen_b Agreed if you are referring to the "spread" of the error process as the Gaussian Assumptions refer to the characteristics of the error process not the original series.
Subsampling is not always absurd, in fact it makes perfect sense for many applications. Just because you have billions of instances, does not mean you need to use all of them to capture the information you want. The law of large numbers applies. Using less data will increase uncertainty, but not necessarily beyond acceptable levels.
In that case you can make the assumption that the user are independents and you can build one classifier per user trained only on their own history.
actually, `mcmcsamp()` was implemented for a while (when `lme4` was still in the 0.9999... series I think), but has since been replaced with `confint()`, that provides confidence intervals for an object of class `Mer`. You can use `method="profile"` for intervals based on the profile likelihood (the default), `method="Wald"` for intervals based on the normal distribution (for fixed effects only), and `method="boot"` for bootstrapped CIs. If you want p-values instead of CIs, run your model in `nlme` instead, or compute z tests by computing estimate/standard error.
+1 Interesting examples.  Are any of them accessible on the Web?
If you compare the a certain model with the *null model* using a likelihood ratio test (LRT), the answer is **yes**, it is an omnibus test. See also [this reference](http://www.stat.purdue.edu/~dasgupta/testing.pdf) (page 319).
thank you very much, it help me a lot, haji
Great thing this cross-tabulation visualization!
It would help to know more about your specific situation and the specific question you want to answer.  If you do an ABBA study (perhaps repeatedly) you might be able to calculate the difference between the first A and B, the difference between the second A and B, and so on.  You could then do a t test to see if these differ from zero.
And it never discusses the side effects that I outline in the question! I upvoted your answer, but haven't checked it as the definitive one as yet. And I'd given up on this Q!
@c00kiemonster I see; well, in the log price regression, the intercept term, as I mention above, just takes care of the _units_ of the contract. For example, if you looked at price per 100 shares instead of price per share, the multiplicative factor becomes additive once you take logs. The 'slop' gets shoved into the intercept term.
Differencing is most useful for analyzing series with unit roots, e.g., AR processes with coefficients equal to (or close to) unity. Consider the simplest such case in which $y_t$ follows a random walk, i.e., in which $y_t = y_{t-1} + e_t$. This process is non-stationary. However, the differenced series is stationary.  Ref: http://en.wikipedia.org/wiki/Unit_root
+1 ... but a nit-pick on some details. Compare the Pearson correlation and Kendall's tau on the following data:  `x <- c(17.91, 24.13, 8.58, 21.02, 5.47, 30.35, 14.80, 27.24, 33.46, 11.69)`  ... and ... `y <- c(6.06, 8.53, 3.59, 11.00, 15.94, 20.88, 25.82, 18.41, 13.47, 23.35)`.
Also check the categories for the IVs.
Thanks for the answer! Point of clarification, do you mean that I just compute the difference in scores for both Y and X?
*Significance* is a term with a very [particular meaning](http://en.wikipedia.org/wiki/Statistical_significance) in statistics. "Data" are not of themselves "significant". Please edit to ask what you want to know in plain English terms, avoiding any statistical jargon at all. (Collectively, we're relatively good at interpreting plain English 'what I want to find out' requests into statistical ideas.)
Thanks for the suggestion.  The moderators are not dichotomous.  They are continuous.  I'm just plotting the association between the predictor and the outcome at 2 different levels of the continuous moderators (+/- 2 sd of the mean).
Or 6 different colors, with means line plotted, and error bars or confidence intervals.
I am surprised that a correlation calculation returns zero, because the correlation in that case is actually undefined. Regardless, your question seems to be saying (1) "I want to use correlation to assess similarity," (2) "But I disagree that correlation is a correct way to assess similarity." What, then, do you expect in the form of an answer?
I am confused: first you say you want to calculate an average, then you deny that the average is meaningful (by pointing out problems with it), and finally you say you can't even get enough information to compute an average.  What exactly do you want to do and what kinds of data do you have available, precisely?
@Pete - see answer
In the Bates', yes, 0-mean was emphasized in the model definition; but in Gelman and Hill 2007, 0-mean is not in the model definition. Are you saying that "lmer" and "nlme" force 0-mean, since they were designed by Bates? Thank you!
Mark, if you are looking for continued help on CV, consider accepting some of the answers that people have provided you in the past by clicking the check mark next to one for any given question. People will be more likely to help if you aren't perceived as ungrateful.
I think it is neither possible nor interesting. Does it make sense to look at residuals in an ordinal logit/probit regression? They are hard to conceptualize.
If this question is statistical, it is about the interpretation of p-values, which is addressed at http://stats.stackexchange.com/questions/31/what-is-the-meaning-of-p-values-and-t-values-in-statistical-tests.
Although these descriptions are qualitatively suggestive, sadly they are incomplete: they neither distinguish covariance from correlation (the first description appears to confuse the two, in fact), nor do they bring out the fundamental assumption of *linear* co-variation.  Also, neither addresses the important aspect that covariance depends (linearly) on the scale of each variable.
I don't want to download a document. Can you post the figure to imgur.com & post a link here?
In this case is it much easier to compute P(you don't roll a 1 or 2)?  If so shouldn't you show why?
This is an unusual use of the term "closed form." Could you provide a reference?
I'm offering a bounty, because of the the answer's painstaking, unhurried style, besides the quality.
From your question, you seem to be trying to apply a method and use the result, without any further consideration. This would be "automated". Instead, use a variety of methods and then consider the results (the consideration makes it "not automated".
+1 A worked example of the ML solution appears at http://stats.stackexchange.com/questions/49443.
Because you are not a statistician--and few are!--please don't try to ask your question in a technical manner that you think will appeal to statisticians; you will only confuse both them and yourself.  Ask in your own terms: what are your data and what do you want to know about them?
+1, and we can extend this idea further: We want to make it such that people can see a graph and know (as much as possible) what is going on without having to read the caption, *and also* when they need additional information to have that available in the caption such that people can read the caption and know what they need without having to read through the paper searching for that one crucial detail.
Max can i use this for a neural network model as well ?
oops.  Reading this more carefully I see that it isn't a `glmer` issue at all.  I will say that at least the development version of `lme4` gives an error `Response is constant - cannot fit the model` , which at least gives a clue ...
The assumption is that the model is a linear mixed model. And for this case its not relevant to find a correct model for the data, rather the data is given as an example for some calculation. Now there is the possibility that you get a non positive semidefinite matrix as estimation for the covaraince. So what to do from there, if I want to figure out the covariance from the normal distributed population where the data comes from. That the sample is normal distributed is the assuption.
You are asking a very "deep" question actually! Determining the DF in a mixed model is always... *interesting*. Can you tell the the number of levels your factor variables `URBAN` and `GROUP` have? I am pretty sure that is related to your question... Here's a famous take on your general question: https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html
Some R code illustrating what precisely you are doing would help a lot. I understand that your data might be private, so I suggest simulating similar ARIMA model and then illustrating your question with simulated data.
Seems to me there might be some better way of formulating or parameterising the problem, but it's hard to tell without knowing a bit more.. could you give us some idea what the $x_i$s are and how you know the underlying model is of that form?
That's awesome, thank you very much!
This question is a duplicate.  Take a look at this question, and follow the links in the answer:  http://stats.stackexchange.com/questions/86719/using-mle-vs-ols/86724#86724
Can you try to give more specific description of what you want to cluster? or is it just a state of the art in clustering that you need?
It is possible to fit a model with a dependence between the failure time and the censoring given by an Archimedean copula. Type "copula-graphic estimator" as a keyword for Google. If for some patients you possibly have the time to the recurrence and the time to end of the follow-up, then you might be interested in semi-competing risks.
Is this a statistical question? It looks very much like a question about specific code. Please rephrase to focus on your statistical problem.
how did you set the labels of training and test datasets ?
It's simpler to note that the probability of 4 or more agreements equals the probability of 3 or fewer and, since that exhausts all possibilities, both numbers must equal **1/2** exactly.
Could you please give a little more detail in your answer, instead of referring the OP (and future readers) to online sources (which, in time, are likely to be moved or removed)?
If you found an answer here to be useful, could you please consider accepting it?
Thanks very much for your answer. I will pass it on to my colleague.
(+1) Still the fact that the log likelihood of two models is exactly the same is a bit intriguing. It might be the case that there is a mistake in the code or the bigger model is a repameterisation of the smaller one, or something else ...