tttjjj/elt_bench huggingface dataset:
in the Python community, patsy is addressing the need for "formula", which you describe, at times improving on what R offers:
http://patsy.readthedocs.org/en/v0.1.0/index.html What's nice in Python is that all these aspects are handled orthogonally. Pandas will take care of timeseries and dataframe/series representation. patsy for the formulas. numpy for array representation and vectorization. statsmodels wraps statistics algos. scipy for optimization and a bunch of other stuff. The result is cleaner interfaces. R, in comparison, is more mature, but is a hairball. ../..
This question sounds rather similar (http://stats.stackexchange.com/questions/9398/supervised-learning-with-rare-events-when-rarity-is-due-to-the-large-number-of) and Dikran gave a good answer to it.
An update. Take a look at the punitroots package. I think it may do what you want. http://road.unimol.it/bitstream/2192/163/3/ESDP11063.pdf
(1) I have no idea (2) you don't need to remove correlated predictors in linear regression (the coefficients for the correlated variables will be poorly defined but not always an issue) (3) I assumed the correlated variables would be retained in a random forest with relative importance spread between correlated variables, while in a linear model variable selection will often get rid of some of the correlated variables.
@chl: Yes, it's the same data and basically the same hypothesis.
Lot's of things are known about Fisher Information, it's integrals of fisher information that I'm not sure about. I'm not familiar with what you say about Fisher Information turning into some known divergence on integration
One-sided tolerance intervals for a normal sampling might help the understanding of this notion. An upper $99\\%$-tolerance bound is nothing but an upper confidence bound of the $99\\%$-quantile of the assumed distribution of the model. Therefore in case of a normal distribution this is an upper confidence bound of the parameter $\\mu + k\\sigma$ where $k=z_{99\\%}$ is $99\\%$ of the standard gaussian distribution.
@Navi Welcome to the site. I am appreciative of your rapid acceptance of my answer - 3 minutes may be some sort of record!.  What most participants do however is allow time both for feedback on answers, via votes and comments, and for any further answers, and then judge which answer is best. In this case, moreover, my answer only addressed the first part of your question.  So please feel free to accept another answer, and unaccept mine, if you have second thoughts.
I have added graphs to main question and an example of statement I want to make from this. Just to make it a bit clearer.
It seems that what you are asking for is actually a PhD thesis of someone! Check out [this](http://www.collectionscanada.gc.ca/obj/s4/f2/dsk2/ftp02/NQ31169.pdf).
+1, excelent answer!
@M. Tibbits is correct, as the various replies and comments show: several quite different interpretations of your question are possible.  Please clarify, because an ambiguous question has little value and would need to be closed.
Thanks, proc similarity sounds like what I'm looking for. Unfortunately I don't find a SAS trial version online so I have to keep searching for something else.
In general, a test of autocorrelation would be $H_o: \\rho = 0$, $H_a: \\rho > 0$. (Many economic time series have positive autocorrelation, so often just the one-sided test is done when working with economic data) Why are you trying to test if the autocorrelation is either more or less than zero? What is the data that you are working with? Are you actually trying to do a two sided test of $H_o: \\rho = 0$ vs. $H_a: \\rho \\neq 0$??
It's "dangerous" because it is misleading. Accuracy of 85% sounds pretty awesome, however it would be the same as predicting always the majority class. You need a combination of measures like precision/recall, or specificity/sensitivity to avoid this trap.
A diagonal matrix $D$ times a general matrix $P$ is just the sum of the diagonal elements of $D$ times the row vectors of $P$. In turn, $PD$ would is the sum of the diagonal elements of $D$ times the column vectors of $P$. Just  write a simple example of low dimension, with matrices 3x3 for instance, and you will see at once how it goes.
How do is it possible that Bernoulli deviates have a mean of 12?
The vagueness of this question may make it difficult for people to answer. Could you please edit it to explain *what* the model is attempting to do and what kind of "actionable data" are being generated?
Food for thought, however the particular example is unfair as the frequentist approach is allowed to consider the relative costs of false-positive and false-negative costs, but the Bayesian approach isn't.  The correct thing to do according to Bayesian decision theory is to give an interval of [0,1] as there is no penalty associated with false-negatives.  Thus in a like-for-like comparison of frameworks, none of the Bayesians would ever be beheaded either.

The issue about bounding false-positives though gives me a direction in which to look for an answer to Jaynes' challenge.
Please do a reasonable-sized plot. If it won't even fit on screen, it's far too big to be useful. The two sample KS test is for testing that two samples could have the same distribution, which isn't what you are doing here. What were the nature and extent of the ties (i.e. what values were tied and how many of them were there?)
You say that the variable names don't matter much as long as they are well documented... I don't want to make a mountain out of a mole-hill, but I do think they matter to some degree. Variable names which are hard to remember or hard to type have a real cost in researcher-time.  Especially if the same variable names are being used by a thousand researchers.  Thanks for your other pointers though :)
@rolando2 The words you quote come from an a set of notes I wrote in 2006 http://web.psych.unimelb.edu.au/jkanglim/Anglim2006-SPSSIntroductionWorkshop.pdf. Thinking back, I took the idea from a lecture on factor analysis that summarised the findings of Velicer, W.F., and Fava J.L. (1987) An evaluation of the effects of variable sampling on component, image, and factor analysis. Multivariate Behavioral Research, 22, 193-209.
A good place to start would be by researching "Anscombe's quartet," which is shown in the answer at http://stats.stackexchange.com/a/16131.
The development version of RStudio (0.96) now also support knitr as a weaving engine, so you can easily switch between Sweave and knitr.
I will try this. At the face of it, it looks like it may work as needed.
@Anna Ok, I have added something more feasible in such case.
I'm not familiar with the phrase model diversity. However, it sounds like what you are after is related to the concept of model _sharpness_ in forecasting. A model has _sharpness_ (a.k.a. _resolution, discrimination_) if its predictions are sensitive to differences in the input data. One could construct a measure of distance between two models based on the degree to which they exhibit sharpness on different subregions in the space of input data. One could then derive a measure of diversity as a function of average or aggregate distance between models. Is this sort of what you are looking for?
@probabilityislogic Note: I noticed a mistake in my question and I have changed the text to say "The population in which I compute my death rate are all HH".  Now for your question. For biological reaosns, there are two possible scenarios. The first is hh, Hh and HH all have the same death rate. The second is HH and Hh have the same death rate and hh is larger.
Then can you please put the information that we need to answer the question ... in the question? I'd start with an explanation of whatever that comment was trying to explain.
C.W.Helstrom's book Statistical Theory of Signal Detection (1967?) has some discussion of the equal correlation case and $X_{(n)}$ though not of the order statistics as I recall. Just remember that for _any_ (not necessarily multivariate normal) equicorrelated equivariance random variables, $$\\operatorname{var}\\left(\\sum_{i=1}^n X_i\\right)=n\\sigma^2+n(n-1)\\rho\\sigma^2 \\geq 0$$ shows that $\\rho \\geq -\\frac{1}{n-1}$.
Are you still designing this study? If so, I recommend more than two time points.
I thought of using spectral coherence (http://en.wikipedia.org/wiki/Coherence_%28signal_processing%29) but do not know how to apply it to this case
Thanks for the help. I just saw that I can do a beta regression also in SAS: http://support.sas.com/resources/papers/proceedings12/325-2012.pdf which is my commonly use software. So if I understand corretly, a beta regression is part of the GLM family
I would try something like functional pca or use a smooth version of the trajectory shape and build some clasification model upon it. Here I would suggest to try different methods to see how they perform.
"Linear" and "nonlinear" make no sense when discussing the relationship between a categorical and a continuous variable. As @ttnphns said, you can always draw a straight line, but so what?
"problems where there is a linear discriminant that can help to separate the positive inputs from the negative inputs". what does this mean?
"*To satisfy the CLT you need at least n=30.*" What is the basis for this statement? I can show cases where the distribution is non-normal yet $n$ much less that 30 gives good answers, and I can show cases where $n$ much greater than 30 is not sufficient. You can almost never have evidence that a population *is* normally distributed; indeed I doubt that's ever actually true.
Sorry, I think this link would work:  [link](http://uploading.com/files/am96bm87/dist.RData/)
While it looks like your understanding is correct, given the chisq.test function already includes the ability to simulate to get the p-value (which works perfectly well down to  very small expected values and can be made as accurate as you wish, simply by simulating more), why would you do this?
(+1) @zen. I believe $N$ is the size of the $i.i.d.$ sample of $(\\Phi_i)_{1\\leq i\\leq N}$ and the $a.s.$ convergence is indeed for $N \\rightarrow \\infty$ with **fixed** $n$. This is just to mention that you do not have a convergence theorem for (*) as $n \\rightarrow \\infty$ in the general case...Therefore the reader should really differenciate what pertains to $n$ (the number of measurements) and what pertains to $N$ (the sample size of simulated data from the posterior distribution).
Any invertible matrix for which $AA^T = S$ with $C=A^{-1}$ will work, so for example, [Cholesky decomposition](http://en.wikipedia.org/wiki/Cholesky_decomposition) is one way - it's a simple algorithm for finding a lower triangular $A$ (or equivalently, in many implementations, an upper triangular $A^T$). That is, it finds $S = LL^T$, whence $L^{-1}S(L^{-1})^T = I$. But a number of other choices are available.
A nice question but well covered at http://stats.stackexchange.com/questions/36/correlation-does-not-mean-causation
You should include them. At the very least, not including them makes your research vulnerable to criticism as to whether you did not control for C and D, and if you did not, why you didn't. Interpretation of regression models and coefficient changes depending on what's in the model, and to include stuff that you may not need in the end is a more conservative approach than to obnoxiously state that you know the true relation, and only include A and B. If you knew the true relation, you would not do any data analysis in first place, and shout at every corner that you have this divine knowledge.
I added an example.
Exactly what I was looking for. Thank you! Jay
http://stats.stackexchange.com/questions/50210/caret-and-randomforest-number-of-trees?rq=1
Thanks I found a solution:http://stats.stackexchange.com/questions/51418/assigning-class-labels-to-k-means-clusters
Logistic regression doesn't apply because your outcome variable isn't categorical. Do you have continuous data on `Intensity`, or only the grouped continuous data in bins of .04? Does the `Intensity` distribution really max out at .48?
Can you please be a litte bit more explicit about the identifiability problem?
Preferable for what purpose is the usual flip answer. In your case, there is no way I would prefer the mean +/- se plot. The cluster of outliers at about 7 on the displayed scale in the box plot is a very important detail and you can hardly make good decisions without working out how to think about them. Incidentally, I guess that the mean +/- se plot refers to quite different data, but you should clarify that. Regardless, a mean +/- se plot would hide the outliers, not a good thing for honest science or statistics.
I think I may be missing something but it sounds like you just have a usual repeated measures type of situation. Why can't you just do the usual tricks for repeated measures data (random effects or GEE)? As long as you don't have $t$ in the model, the time-varying covariate's effect should be estimable. Another interpretation of this question is that you think the covariate effects themselves are time varying - in that case you could fit each time point separately (or include a categorical interaction with time) and compare the fit of that model to a submodel that does not stratify by time.
1) What do you  mean when you say the word "model"? (because it doesn't sound like what it means in statistics) $\\,$  2) a CI is an interval for some population quantity in a statistical model. Which quantity in which statistical model, and how was it calculated?
@whuber - you are right, there is definitely a problem with that second step. The second term should be $$\\frac{P(C|A)P(A)}{P(B \\cup C)}$$ or $$\\frac{P(A|C)P(C)}{P(B \\cup C)}$$ I think...
Add them, see if they're significant, if they're not, take them out.
(+1) The second term, being the transpose of the first, must also be zero :-).
In any case, to test for the impact of habitat, you would fit 2 models - one with Habitat and 1 without - then compare with a likelihood ratio test - and the degrees of freedom would be 1. Don't do a t-test on the coefficient of Habitat; you need to compare models.
Are the same animals subjected to all 5 treatments or not? How did $\\chi^2$ not answer your question?
**What is the purpose of this simulation?** Different purposes will have (hugely) different answers.  For instance, they could plausibly include (1) testing inputs for influence in the outputs, (2) reliably predicting the output for future inputs, (3) identifying global extrema of the output, (4) developing scientific theories about the system being simulated, and (5) comparing the simulation to the results of another system or simulation.
thanks for the suggestion, I will check it out. I was hopen for an online-text, though :-)
I actually found Tabachnick rather unclear, even for subjects i knew quite a bit about. The introductory stuff on univariate stats and data cleaning was very good though
This is the practice with non-response adjusted weights, anyway. They are random because they are based on a random (with respect to the response model, not with respect to the sampling design) event of responding to the survey. Since weights enter non-linearly, there is absolutely no hope for unbiased estimation. The general spirit of consistency results may provide for consistency of $\\hat y$ when weights are consistently estimated, but I have not seen any technical results.
Some combinations (I would say "points on the 30-hypercube") do occur very frequently, it's very nonrandom. If I had to guess I'd say the most common alone accounts for > 20,000 objects.
@cardinal It pleases me so much to hear you say that. :-) Yes, hopefully this self-study investment will pay off someday. ;-)
Sounds exactly like my intuitive understanding of this term.
@ttnphns the OP almost certainly means the regression coefficients, as they are commonly referred to as factor loadings within psychology
Some intuition: In going from $Y_k$ to $Y_k^2$, do you lose any information? If so, about what? Now, consider the example of $X_n$ being a simple random walk. Is it a martingale with respect to $\\sigma(X_0,\\ldots,X_{n-1})$? What about $\\sigma(X_0^2,\\ldots,X_{n-1}^2)$?
This is an interesting question, though I think it would help to clarify a few things, particularly which are the "moving parts" and which are not. It seems you want a bound that holds *uniformly* in $p$ for each *fixed* $r$. But, what is the role of $d$ here? It shouldn't matter much, but is it's introduction necessary? One approach might be to look at things in terms of waiting times of a Poisson process and couple them to associated geometric waiting times (via taking the ceiling of each) for your binomial random variable. But that might not yield the *uniform* bound you are seeking.
Are variables produced by multiplying random variables actually mixtures? This seems at odds with the way mixtures are usually defined. For example, the product of two Gaussian random variables is distributed as a linear combination of two Chi-square random variables, rather than a mixture of Gaussians.
Kay, thank you for your detailed answer. I understand the answer is slightly dated but would you mind sharing the accompanying code if you have kept it? I am trying to learn Hierarchical Bayesian Modelling but struggling to find intuitive examples in R.
those are all cool, but I was looking for something for the whole class of charts. We went with histogram....
thank you it helps
+1 Wonderful answer! Thanks a lot. Many to learn. I have now some ideas to work on their implementation. Hope I can develop the ideas for 3D too practically and efficiently for the problem I am working on it, which is a large cubic grid (5000x5000x5000).
@cpcloud : this is not a problem of symmetry, it is whether the matrix is positive definite. Try, for example `sigma = [1.0, -1.4;-1.4, 0.2]`. `chol` bonks, but `cholcov` returns some matrix (I am guessing. I do not have the statistics toolbox).
I see, thanks, I was not sure how to convert sentences into numbers...
+1, that looks like a huge effect size that you've got there :)
Michael: I usually vote up your answers, because they are neat and useful, but I have to admit I do not like it when you answer homework questions. This is because you just give the right answer and don't let the OP think about the answers. If you don't mind, I would suggest you to give hints better than the answers: in that way, you give the OP the opportunity to THINK.
Try `help(replicate)`.
Negative weights correspond to short-selling the asset.  You borrow these and immediately sell them, adding to the cash used to buy the assets with positive weights. Any realistic theory will incorporate the (sometimes heavy) ongoing costs of paying interest on the borrowed assets, but these are often neglected for illustrative purposes.
This is a good question, because there is a subtlety: $k$ is the number of *identifiable* parameters to be estimated. For instance, although in the regression model $Y\\sim\\mathcal{N}(\\beta_0+\\beta_1X_1+\\beta_2X_2+\\beta_3(X_1+X_2), \\sigma^2)$ *five* parameters are written, nevertheless $k=4$.  (This model is equivalent to $Y\\sim\\mathcal{N}(\\beta_0+\\alpha_1X_1+\\alpha_2X_2, \\sigma^2)$ with $\\alpha_1=\\beta_1+\\beta_3$ and $\\alpha_2=\\beta_2+\\beta_3$, which explicitly needs only four parameters.)
Additionally, if your data sets are not normally distributed, then a bootstrap method may be more robust than a correction based simply on the correlation between A and B and the number of points. See [Wilcox 2009](http://www.tandfonline.com/doi/abs/10.1080/03610910903289151).
Also, draw a picture (on a number line) of the interval the variables is defined on, the interval being conditioned on, and then the interval being asked about.
@David You might have missed [this editorial](http://www.nytimes.com/2014/04/07/opinion/eight-no-nine-problems-with-big-data.html?_r=0) in today's New York Times.
I noticed that difference, but I don't think it will have much influence on the results.
+1 This answer is on the right track. It seems a little severe to suggest the evaluation of non-normality is subjective, though. The question of "amount" of non-normality bears on the consequences of departures from normality: in some applications there is little consequence--consider robust tests--and in others the consequences can be large. By considering these consequences explicitly, and the effects of non-normality on them, one can render an assessment that is more "objective"--or at least more useful and defensible.
I think your comment is basically the same as this one, which was written with less mathematical notation: http://stats.stackexchange.com/questions/32206/why-compare-sum-of-squares-with-anova-and-not-mean/32231#32231
using good features is always key to good classifier performance, but whether or not a specific feature helps in "generalizing" the classifier is not clear without context. You need to perform tests on the classifier to ensure that it works well in the intended domain and that it has not been overfitted to the training set. One way overfitting might occur is if you are choosing among a very large set of possible features. some features may seem to work very well in the training set, but this could be purely random noise. Therefore you need to test the generalizability of your classifier.
Thanks for all the precisions, I am sorry I cannot be more helpful but I am still somewhat puzzled and share @PeterEllis's view on all this. In any case, if you actually need/want to do some clustering, there are many approaches out there that can easily deal with dozens of variables and thousands of observations, there is really no need to roll out your own or be concerned about the number of combinations.
You might like to review replies to [What is the most surprising characterization of the Gaussian (normal) distribution?](http://stats.stackexchange.com/q/4364/919).  For instance, the accepted answer suggests that if you standardize the marginals to $X$ and $Y$, you need only test that (a) $X$ and $Y$ have the same distribution and (b) $X-Y$ is independent of $X+Y$.  Another reply by @Robin Girard suggests yet another test.
If you think the effects are not additive (and you have enough data), you might try creating a binary `internet only`, `newspaper only`, and `internet and newspaper` variables. You can then see if the effects of each channel reinforce each other.
Thanks for connecting the dots sir.
Because removing insignificant terms only seems to help the predictions.  It actually makes predictions worse in new data, and results in multiplicity problems that hurts type I error and confidence interval coverage.
I neglected to add that it also makes downloading your facebook data easier that eating pie.
Why did you pick these specific tags? For instance, I don't see why you picked `factor-analysis`? You could also add the `R` tag if you want this in R (and `econometrics` perhaps?).
Thank you for such careful thoughts about my code.  As someone who works with empirical data, I am proud to say I have no expertise in generating fake data, and it shows here in the shortcomings you have identified.  Although, my novice level of understanding may also be revealing itself.
@Andy W that's the one!
I think this reply could be greatly enhanced by giving details on the why and how of the `empinf` and `norm.inter` functions.
I added the relevant tag `random variable` and removed the tags `variance` and `standard-deviation` which don't directly relate to the question. A random variable might have a multivariate normal distribution, or it might (and typically will) have some other distribution. 'Deviate', as Dimitry suggests, tends mostly to be reserved for an outcome of a random variable.
For additional solutions, please see http://stats.stackexchange.com/questions/109819 and http://stats.stackexchange.com/questions/33078.
I've been using the definition that a classifier is linear if its decision boundary is linear in $x$ space. This is not the same as the predicted probabilities being linear in $x$ (which would be impossible apart from trivial cases, since probabilities must lie between 0 and 1).