tttjjj/elt_bench huggingface dataset:
Two questions. Do you have any idea about highly correlated your other variables are with year of birth? If there is little correlation then it will be hard to impute accurately, but if they are correlated you might have some success. Second question: do you have any idea about whether or not there is a pattern to the missingness? For example, are these likely to be old records, or something otherwise systemic like that?
Without having any predictors in the model there is no sense in asking for predictions.  And the `rms` package does not support intercept-only models in general.
Probably Hauck-Donner or quasi-separation.  See [this question](http://stats.stackexchange.com/q/45803/3601) and the others it links too.
@ SheldonCooper - "In image analysis it is routine to throw thousands of highly correlated features at SVM, and SVM seems to perform decently." - I would love to know more about how you do this without running out of memory. SVM have always performed quite well in prediction accuracy but with 200 input variables and over 100k samples SVM in Clementine and SPSS quickly exceeds memory limits. This may well be a another thread but would love to know how to balance SVM performance with large dimensionality & samples?
I am confused. What is assumed to be normally distributed? Your test data is clearly nowhere close to that. The ratio of two normal distributions is tricky to estimate, but if it is the ratio of two times, then at least both are positive. However, if the times vary a lot (say some tasks are much longer than others) then ratio may not be best. Doubling 1 microseconds is not equivalent to doubling 10 minutes.
Well, one of the reasons of posting this question is encouraging people like you to share their experiences with the community. Thanks for the links and sharing your experience.
$H$ is Hessian of the log likelihood function $L$, right?
Anova is ok, it's just that you can't use estimated trends to compose a hypothesis test for differences, i.e., to lower the number of degrees of freedom.  This is double-dipping.
@posdef, without a little more information on your analysis, it's a little difficult to give advice. **But**, unless you are interested in some unusual quantities or have an unusual design, then using a parametric bootstrap in a Gaussian setting is suboptimal. The core of statistical inference is centered around assumptions related to normality (e.g., exponential families) and there is some very powerful optimality theory in these cases. Without good reason, one shouldn't try to avoid it.
In the example above, you gain the ability to calculate the exact $p$, which is not given and may be useful for effect size estimation, meta-analysis, or checking for $p$-hacking. A friend and colleague of mine @rpierce has also advised me to report test statistics to ensure readers that I'm doing things properly in general, as he's caught many published articles [doing it wrong](http://knowyourmeme.com/memes/youre-doing-it-wrong) via misreported test statistics and associated $df$.
@EpiGard: +1, I have posted an answer in which I give formula to do what you recommend for computing the median survival time.
I checked Box - Jenkins and Reinsel. We can close this here. On page 56 they define the characteristic equation (same characteristic polynomial that I intended)The complex factorization gives terms 1-Gi B.  They say for stationarity that Gi must lie in the unit circle.  But it is the inverse (in the sense of complex numbers) that is the root of the equation . So the roots all do lie outside the unit circle for stationarity.  That was my confusion.
makes sense to me. I will refine the question
Just thought to add that this will also affact the interpretation of the effect of adopted on score.
It is *very difficult* to get the diagram to look nice in any automated way (the first example was likely done by the artist manually placing the nodes). Difficult to program has nothing to do with its utility as a graphical tool. You may find more motivation [on this post of mine](http://gis.stackexchange.com/a/15222/751) on the GIS site about visualizing flows. Also I give some examples of [parsets and dot plots](http://stats.stackexchange.com/a/19543/1036) on an answer to this site.
+1 Many thanks for jumping in and making this very well done effort to clarify the situation.
Thanks! In examples like plot(sin, -pi, 2*pi), is there a default spacing for sampling the range in domain?
Both are great answers, too bad I can't accept both. Thank you.
Thank you @amoeba, my code for the inertia moments is availabe on my blog: http://stla.github.io/stlapblog/posts/Variance_inertia.html
your failure classes, are they mutually exclusive?
+1, I agree w/ this analysis. FWIW, this is the explanation I was hinting at (perhaps not well) in my discussion about power in my answer to [the other question](http://stats.stackexchange.com/questions/34797/34801#34801). I do have 1 question about your version here, why do you use 32 as the mean of your error term? Is that a typo, or is it important in some way?
+1, I've deleted my answer, since your was way better and made mine obsolete :)
Is this a repeated measures design such that each subject runs in all, or many, of the conditions?  Or, is it just an independent groups, or measures, design where each subject is in one condition?
I think fabians meant Stackoverflow: http://stackoverflow.com which is for R programming, how do I do something in R questions. This place is more generally about stats. The two can blend a lot, however.
Yes, its a given exercise so I am not able to choose which model to use, unfortunately. When you say they are easy to compute, and linear combinations of the component distributions, its what I have suggested right?
Thanks for the clarification and advice, works perfectly
I don't think this kind of assessment is generally used with simple regression models. What would it tell you that you wouldn't find out from using the entire dataset to generate your regression parameters? Normally the reason to use an evaluation dataset is to prevent overfitting, but that's not an issue when you already know that your model is going to contain just one independent variable.
Cell sizes are not all equal. There's some missing data. Some of the trials had to be dropped for some of the subjects.
I have a dataset with 3 between and 4 within subject conditions. I would like to test for each and every effect, but a single regression with all interactions miss a lot of information I'm interested in. Instead I would divide the data by condition into separate datasets and run focused logistic regressions on each datasets with contrast codes coding for the differences i'm interested in.
You could perhaps try backtracking the use of the word by checking the sources if there are any known ones?
Searching the web with the package name and the error message provides some answers.
Actually, it's done all the time, when the value of the discrete parameter is "large" and the spread of the reasonable values it could take on is also "large" (but perhaps a different "large", "large" not being well-defined.)  You more commonly see this when approximating discrete variables ("fraction of people who will vote for candidate X", which is drawn from a finite set) with continuous variables.  It seems to me that with $N \\geq 72$ you are likely well within the range for which a continuous approximation is fine, unless $N\\theta$ is close to 0 or $N$.
Seems to also correlate quite well with proximity to Sweden..
You can try to look for more explanation on http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
Thanks rolando2, I'll see what I can find in the archeology literature. Converting SE to std dev is a good suggestion.
It looks like there is basically no relationship between the two things.
@whuber: What is the optimal decision boundary based on the above minimization problem?
Do you have similar numbers of males & females? What is the complete list of variables? (It seems like you may have more than just gender and reward type.) Is reward type a categorical variable?
If you give the exact value of the fitted line at the endpoints of you graph you supplied that would make this much easier. On another note, from eyeballing the regression line and the observed values, the independent (and normally distributed) errors assumption is almost certainly violated here if you're trying to do inference about $\\beta$.
To add to Nick's excellent answer, the Wilcoxon-Mann-Whitney test should be considered, and the mean should be abandoned as a summary statistic for your problem.  Though widely believed, the $t$-test is not nearly as robust against normality as many think.  The proportional odds model is a generalization of the Wilcoxon test, which can nicely handle a spike in the data.
@FrankHarrell That's cool. But I am missing how that speaks to the OP's question?
+1 for the same reasons I gave @Macro +1.
Is that a correlation matrix? The OP clearly asked to visualize a crosstabulation. I believe, though, theat map like yours is possible with it, too. The colours will tell of the cell residuals.
It is difficult to see how the relation you quote for chromotographic peaks could be true: it would imply that somehow diffusion and advection together create *less* dispersion than either one separately.  Incidentally, the [harmonic mean](http://en.wikipedia.org/wiki/Harmonic_mean) of any positive numbers $x$ and $y$ is $2/(1/x+1/y)$.
Do you know if there is a method for online training?
In that case, you could always construct an artificial dataset, e.g., in R, and dump it out into a file with the appropriate structure.  The advantage that has over a "natural" dataset is that you know the true model, so validation of the code is easy.
+1 That's a good suggestion. My current SVM implementation is not parallel.
Any poster who assumes "BMI" absolutely does not need explanation is aiming at biostatisticians, hence the tag added.
Intuitively, if $L$ truly is "luck" it shouldn't be able to predict anything!  As such, your hope is that $L(n-1)$ captures deviations (of team record relative to run production) that were *not* due to luck but perhaps to other factors, and that such factors may persist to time $n$.  Therefore your best hope is to see whether $L(n-1)$ is useful for predicting $L(n)$ (rather than $P(n)$).
And then that beta posterior becomes the prior for the next meeting and you go through the same steps to get another posterior, which becomes the prior for the city after that, ...
sample(x=c(1, 2, 3), size=1, prob=rep(1/3, 3)) generates one of "1", "2", or "3" with equal probability.
The problem (feature too) is generic. Any such test just focuses on whether a difference is large enough not to be dismissed as a chance fluctuation. Such tests in your context face two rather fuzzy but big questions: First, you can't really take the points in an image to be mutually independent, which is what Mann-Whitney and Student both assume. Second, in what sense is your image a sample from a larger population? Either way, confidence intervals for parameters of interest are more interesting and useful for large samples, even if they also face the two questions here.
Hello whuber, thanks a lot for the detailed explanation. I have one doubt, though. When $\\Sigma$ is general, the terms in the sum expansion of the inner product are not iid anymore; hence the mgf of the sum is no more the product of the mgfs. Then, how do we generalize the above analysis to a more general Sigma?
Does this thread: [Why square the difference instead of taking the absolute value in standard deviation](http://stats.stackexchange.com/questions/118/) answer your question? If there is anything you still want to know after reading that, please edit to clarify, otherwise this may need to be closed as a duplicate. On a slightly different topic, the reason for paying attention to the variance instead of just the SD, is because the variance is the parameter in the normal distribution, not the SD. E.g, variances add, but SDs don't.
The answer can be found here: [How to tell if data is "clustered" enough for clustering algorithms to produce meaningful results](http://stats.stackexchange.com/a/35760/7290). Although that is only a particular answer to a more general question, this Q may be a duplicate.
Could you clarify who '*they*' are and include examples of what they say? (Just how bad do they say it is? Let's be clear what we're discussing here.) I've seen some criticism of LSD, some of it justified, but I don't know that I've seen what you've seen, nor that what I've seen would require quite the characterization you have there.
Thanks a lot for the explanation. I have got it now.
I don't think it necessarily extends to the smaller example since you are not using cards. In it, for the cases starting with 0, there is a (1/3)(1/3)(1/3) chance for 001, (1/3)(1/3) chance for those starting with 01, and (1/3)(1/3)(1/3) chance for 021. That adds up to 5/27. Same logic applies for cases starting with 1 so that is also 5/27. Then for cases starting with 2 it's (1/3)(2/3)(1/3), or 2/27. Total is then 12/27, or 4/9 as you got from counting.
Perfect, thanks a lot!
@user110502 Note that asking for code (arguably) takes this off-topic. Your original question floats somewhere between statistics and statistical computing, but the help for CV does spell out that questions focused entirely on code belong elsewhere.
Joint confidence "intervals" are actually confidence _regions_ (see, e.g., [here](http://academic.macewan.ca/burok/Stat378/notes/moremultiple.pdf) and [here](http://www.public.iastate.edu/~maitra/stat501/lectures/InferenceForMeans-Confidence.pdf)), this is why sampling is so convenient.
You said: "I asked this question on stackoverflow.com, but I couldn't get what I want"; I imagine that your failure to include the 'r' tag on that post really didn't help. When I am there I filter by the r tag, so I didn't even see your post.
BTW KL divergence is indeed non-symmetric and does not fulfill the triangle inequality, but it does not require that one of the distributions be an approximation of the other. If you don't need symmetry etc. it should be ok to use.
Your comments indicate you are assuming that $\\mathbf{a}$ and $\\mathbf{v}$ are independent. That cannot possibly be, because there are physical limitations to speeds: that means many accelerations will not be experienced at the most extreme speeds. However, it's not easy to provide more detailed advice because you haven't articulated what you're trying to accomplish; instead, you have described an approach to solving an unstated problem. Why don't you change this question and ask instead about the problem you need to solve rather than how to implement a solution that looks invalid?
Are stochastic differential equations related to any type of Markov models?
@Neil G, there is a closed form expression for the cross-validation error of many models as well (e.g. Gaussian Processes).  As I said, it isn't the same indicator of generalisation performance, as it depends on the stochastic model (as Wahba puts it).  You gan have two Bayesian models that give the same predictions everywhere, but which have very different marginal likelihoods, but they would have assympotically similar cross-valdiation estimates.
I think this answer applies to your question as well: http://stats.stackexchange.com/questions/99777/calculating-the-variance-of-the-histogram-of-a-grayscale-image/99929#99929
Welcome to the site, @Nimonika. I took the liberty of adding $\\LaTeX$ to your equation to make it easier to read. Please make sure it still says what you want. If not you could edit it or rollback.
Could you please clarify "product agnostic" because the way I'm reading your post, it seems you both want and don't want to take the product into account.
Year	IMR
1951	82
1952	78
1953	71
1954	72
1955	71
1956	67
1957	68
1958	64
1959	58
1960	57
1961	52
1962	53
1963	56
1964	55
1965	53
1966	54
1967	48
1968	50
1969	53
1970	47
1971	45
1972	46
1973	46
1974	51
1975	45
1976	44
1977	42
1978	37
1979	38
1980	34
1981	29.5
Have you seen [Tom Waterhouse's code](https://stat.ethz.ch/pipermail/r-devel/2009-July/054106.html)?
So ... ummm ... is there some relationship between the title to your question ... and the question itself?
I would try to go for `library(boot)` aiming to bootstrap the results from `lme` fit. This link could probably help https://stat.ethz.ch/pipermail/r-help/2004-November/060444.html
please notice that first incremental point should be multipplied by $\\%10$
What statistical test are you using? Because your dependent variable is binary (yes/no), you should use binary logistic regression.

But I believe the problem may be something else. I'm guessing that one (or more) of your independent variables may have the same response across all subjects (e.g., everyone chose 1). Then the variance(s) of the variable(s) would be zero. If that really is the case, you may want to remove the variables since they were not able to solicit different responses.
$\\theta$ is just a mathematical symbol and means different things in different contexts. Sometimes $\\theta$ is used to refer to a parameter to be estimated but there is no real answer to the question "What is $\\theta$?". That's like asking "What is the letter A?". Your link even hints at this when it says *"Notice that there isn't any meaning in naming a parameter this way. We might as well call it anything else."*.
If the real question's "Should I exclude those values ?" or "How do I model those values?" it's a good one for CV - though still needs some context about what you're wanting to do to make it specifically answerable.
(+1) no matter what, this is a great question since it  can teach us a lot of how humans (are in general not be able to) deal with probabilities.
thanks sir Nick Stauner for editing my post.
Do you know how to do Bayesian VAR?
Ad 1) In linear regression, I normally understand ANOVA as a measure of goodness of fit of the model, i.e. to decide whether the model (regression line) explains substantial part of total variability. The question, whether it is equivalent to all slopes being zero, is really very interesting. Ad 2) it looks I'm getting almost the same p-values for t-test and regression ANOVA in this case. Really interesting theorem!
Hi @Vic. There is an excellent `R` package called `ROCR` for these kind of calculations. Find it [here](http://cran.r-project.org/web/packages/ROCR/). See also [the corresponding site](http://rocr.bioinf.mpi-sb.mpg.de/) with a comprehensive documentation. There is also a [paper about the package](http://bioinformatics.oxfordjournals.org/content/21/20/3940.full.pdf+html).
(+1 to both Peter and Glen)  @JRW - If you do fix the sign, I'd like to be a fly on the wall when you try to explain to your audience the coefficient you "obtained," and its confidence interval. Moreover, they might legitimately ask, Did you fix the sign and/or range of others?  If not, why not?
Yeah, this is exactly what I am doing: until now I manually split the signal into periods, so that for each of them I can define a confidence interval within which the signal is supposed to be stationary, and therefore I can use standard methods such as standard deviation, ...
The real problem is that I can not decide the expected pattern for all the signals I have to analyze, and that's why I'm looking for something more intelligent.
A few preliminaries:

1) There will be *some* relationship among the variables, you want to explore the nature of it.

2) What data have you got? How many measurements at each depth?  3) What sort of relationship are you looking for? Do you think there is one dependent variable or are all the variables equal?  4) What role does depth play here?
The `aov` function calls the `lm` function to do the main computations, so things like contrast matrices will have the same effect in both.
The solution is, that the hole thing, i.e., the tensor product, is defined row-wise. This means that $\\boldsymbol{B}_{x,i}\\otimes\\boldsymbol{B}_{z,i}$ will always have row-dimension $1$ and only the column-dimension will expand.
I think your intuition is correct, but defining a "small value of variance" could be tricky. As a first pass, I would try an ensemble approach e.g. [bagging](http://en.wikipedia.org/wiki/Bootstrap_aggregating).
Michael, $\\hat{Y}_{i}$ doesn't usually refer to the sample mean - it's usually referring to the predicted value for observation $i$ from some fitted model.
Thank you @user31265, this is what I assumed, but a colleague of mine wanted to know for sure and this thoroughly answers the question! Thanks again!
"Statisticians are people who try to come up with meaningful conclusions from messy data." - unknown
All I can say right now is that you're probably going to need some sort of pilot study. Could you describe your independent and dependent variables and what "confirming" is?
Hi, I only need to compare the models as specified above, which means that I have got non-nested models. I need to use a standard p-value approach, only one I could think about here was the Chi-square. Which means that I can only compare model 1 with all the others, but not model 2 with model 3 for example. However, the statistics favor model 1 with p-values above .72. But than the question remains, is this the best way to perform model selection using a standard p-value approach?
@GaëlLaurans, DS1 indicated that he derived eight variables *from* the drop-down emotion list. This is why I suggested linear regression. I agree that if the drop-down emotion list is to be used as independent variables linear regression wouldn't be that useful.
How is `result` coded? Also, in your code, the dependent variable is `result` but in the **R** output, the dependent variable is called `realisation`.
You comment got me thinking a bit more on the definitions so I dug through the randomForest code used in R to answer it properly. I have been slightly off to be honest. The average is made over all trees and not all nodes. I will update the answer as soon as I get time for it. Here are the answers to your question for now: (1) yes. This is how it is defined on the tree level. The sum of decreases is then averaged over all trees. (2) Yes, that was what I meant to say, but it doesn't really hold.
@Glen-b Can please give me some idea about B-Spline?
Felix, I believe the question is actually "how to establish that $c'=0$", as opposed to merely being non-significant (hence the mention of "equivalence testing" in the comments).
Might I suggest you read over some of the better posts on [tag:pca] on this site?  I believe your questions are addressed in many of them, at least in passing.  For instance, your second question concerns the distinction between performing PCA with a covariance matrix or the correlation matrix.
I have upvoted this because it's a valiant, pithy effort to respond to an extremely broad question. I have some doubts about whether "data mining" involves any statistical modeling, though, and would appreciate seeing an example or a clarification of what you mean by that phrase.
Ah, found something. Here is a question: What is the difference between your question and this one ? [Machine learning cookbook / reference card / cheatsheet?](http://stats.stackexchange.com/questions/12386/machine-learning-cookbook-reference-card-cheatsheet)
This paper looks really interesting. Unfortunately it only uses `lme` from `nlme` instead of `lmer` from `lme4`. (+1)
Such discussions are better suited for [chat](http://chat.stackexchange.com/).
Hi steffen, "whether the user has seen this ad earlier" has an effect on ctr: ctr is higher when the ad being shown to the user for the first few times (because of novelty), and lower subsequently. I should have rather put the metric as "number of times user has seen this ad earlier".
This is very hard to answer, as the datasets, problem domains, loss functions, and more affect the outcome.  A pragmatic, yet equally difficult to answer, variation might be to ask to what extent there may be differences in how skilled and unskilled practitioners fit such models.
See, for instance, here (http://www.jstor.org/stable/2291263) or here (http://stats.org.uk/statistical-inference/Lenhard2006.pdf).