tttjjj/elt_bench huggingface dataset:
@julieth: One thing I am still confused about though. Taking your example with 5 observations: if instead of classifying the held-out sample we choose to classify a random sample, then the variance will be strictly binomial. Even though trainings sets still have a majority of observations in common. So I am once again not sure that non-independence of the training sets is the correct (or full) explanation of over-dispersion of CV results under shuffling. Any thoughts about that?
You can also add the gamma distribution, parameterised by its mean and dispersion, i.e. $p(x_i)=\\frac{1}{x_i\\Gamma(\\alpha)}\\left(x_i\\frac{\\alpha}{\\mu}\\right)^{-\\alpha} \\exp\\left(-\\frac{\\alpha x_i}{\\mu}\\right)$
The green isn't really a test, it is the region that is consistent with the Null hypothesis in the one-sided test of 0 correlation where the alternative is that the correlation is negative. Actually, I wonder if your blues are too big? If you want a 95% two-sided test, the quantiles you want at the 0.25th and 0.975th such that the sum of the blue areas is 0.05 (5%). If a one-sided test, then all the evidence against the null is on one tail, in the two-sided evidence is in both tails.
@Zhubarb I'll make some shortly
This is a frequently asked question. You basically need to understand what a log model vs a non-log model is, so you should focus on the thread above. Following that, here are some more threads to read: [What if a log transformation wipes out significance in regression?](http://stats.stackexchange.com/q/79276/7290); & [Log transformed variable not significant, while variable itself is](http://stats.stackexchange.com/q/74855/7290).
What do you think the phone bill would be if the customer population was zero?
+1, I hadn't seen this thread before. I made (I believe) the same point that is discussed here after the horizontal rule in a different way in this post: [Whether to delete cases that are flagged as outliers by statistical software](http://stats.stackexchange.com/questions/15426/#38708).
@Richard There's generally a trade-off between FPs and FNs in MCP, the Bonferroni's method being prone to increase FNs, while step-down methods (like the one suggested to you by @mpiktas) will perform better in most cases. As an alternative, you might want to look at permutation-based methods which offer a good compromise for the simultaneous control of FP and FN rates. @mpiktas (+1) In R, relevant methods are found in the base function `p.adjust()`, and the [multcomp](http://j.mp/eVflBe) and [multtest](http://j.mp/hirQrK) packages have great vignettes describing all of this.
@fredrikhs: Thanks! With regard to 2, what is an example of non-portmanteau test then?
@whuber Good question. I am still working through the best trade-offs in balancing. Ideally it would be to just reduce or increase the number of states in a region, though I don't think that will be enough. An example use-case would be using population for each state value. For many values of `n`, CA would end up being the only state in its region. Regions don't have to be perfectly balanced, just looking for closest. Next I would sacrifice region shape ('s' shaped regions vs nice blocks). The geo clustering is for easy travel within a region. CA+OR+WA = perfect. OR+NV+AZ = ok. CA+GA+RI = bad.
@Chernick normal and independent and identically distributed
@Siddharth: 1. It is not possible. 2. If each point corresponds to a year, you can still determine a frequency.  For example, if you think there is a decade-long seasonality you can set frequency=10.  Maybe you should ask a separate question along the lines of "How can I automatically detect the seasonality of a time series if it is not already known"
+1, in line with this, one additional point is that numbers generated by computer algorithms, such as R & other statistical software use, are actually *pseudorandom*. They will ultimately circle back to the same values / can be made repeatable by setting the seed, whereas I believe this is not true of hardware generators.
exactly what I was after
Protovis looks to have been replaced with [d3](http://mbostock.github.com/d3/ex/chord.html).
Tests are not biased by huge sample size.
sounds reasonable, any more details?
There are infinitely many solutions in this case.  Check out `A %*% c(1, .8837, .31)` and `A %*% c(3.225, 2.85, 1)` for two examples.
I've started some discussion [in the chat room](http://chat.stackexchange.com/rooms/18/ten-fold) about making appropriate tags for such questions. I provide a useful set of related questions in there you might be interested in (there are a handful of related questions and references you might be interested in). In particular I would recommend viewing the answers to [Best practice when analysing pre-post treatment-control designs](http://stats.stackexchange.com/q/3466/1036).
If my purpose is prediction, does it not matter that the better-predicting model violates the assumptions of the statistical model? Is this model (which violates the assumptions of the model) valid anyway?
to add to mpiktas LL can be more than 1 but Rsquared cant be greater than 1.
I've seen an interesting comment in Prof. Wasserman's lecture notes at http://www.stat.cmu.edu/~larry/=stat705/Lecture13.pdf. The notation next to equation (21) on p. 6 suggests that the error you're concerned with falls off as 1/sqrt(n). Unfortunately, I don't know anything about the constant coefficient.
(+1) For this answer and the other similar one. By happenstance the examples you give in both were quite close to ones I had contrived, but did not have a chance to post. It's always a nice present to see myself thinking along the same lines as you have in the answers you post. :)
Are you two discussing the same law?  The question asks about *frequencies of events* while this reply seems to focus on the sampling distribution of a *mean*.  Although there is a connection, it hasn't yet appeared explicitly here as far as I can tell.
The details are highly relevant, Noxville, because the little information you have supplied suggests your data should be neither normally distributed nor independent. It looks like you may be forcing on them a model that gives bad information. Many in this community feel it would be remiss to give you a good solution to the question you asked when there is so much evidence that it will do more harm than good; we would rather understand what your *real* problem is so we can give you answers that actually help you out.
More on this topic and related ones is at http://stats.stackexchange.com/questions/21119/advice-on-scientifically-sound-scale-construction .
What do you mean by HAC/HC1? HAC usually stands for heteroscedasticity and autocorrelation robust standard errors. But you have cross-sectional data so there cannot be autocorrelation given that this is a time-series feature. You should first test for heteroscedasticity (e.g. Breusch Pagan test) and then you can decide whether to use robust standard errors (also known as Huber-White standard errors)
Could you be more specific? As asked, your question is not meaningful, as sufficient statistics relate to a statistical model and a dataset, while propensity scores are conditional probabilities.
@AlexTwain Yes, should have read "DBNs are directed". Even though you might intialize a DBN by first learning a bunch of RBMs, at the end you typically untie the weights and end up with a deep sigmoid belief network (directed). In a lot of the original DBN work people left the top layer undirected and then fined tuned with something like wake-sleep, in which case you have a hybrid.
you may have better luck with biogeme, another open source and widely used program for GEV models: http://biogeme.epfl.ch/
Beta regression assumes that the response comes from a Beta distribution (like gaussian regression assumes a normal).  So it is only for data between 0 and 1 but works well for things like proportions when you don't have the denominator to do logistic regression.
thanks for the tip on a way to do what I did more efficiently. can anyone comment on the meaning of the weighted r-squared that I described/proposed?
@Person: sorry I do not get it: if $X\\sim E(b)$, $P([X]=k)=e^{-bk}-e^{-b(k+1)}$ which is different from $e^{-bk}$...
Unless I'm missing something, nothing you added would imply the use of a cutpoint.  Note that a predicted probability provides its own error rate.
Actually I have often seen that kind of normalizing done with both animal trials and cell culture experiments. I agree that it shouldn't be the standard way.
I have bookmarked this question very long ago. I would be very interested to see a clear and self-contained answer that might serve as a reference in the future. Do you think you can make it?
This question appears to be off-topic because it is about finding an implementation of an algorithm in a specific language.
This looks like standard bookwork. If this is for some subject could you add the self-study tag please?
+1 Thanks for sharing. These lectures look really useful; I see that the author also has a set of other videos on machine learning and information theory: http://www.youtube.com/user/mathematicalmonk/videos?view=pl
@mbq since reinforcement learning is a branch of machine learning, I don't think so ;)
excuse me, what tests are better then mood ?
cool! thanks (also the internet is small world, thanks for your R guides)...
(+1) http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient explains the notion.
@Max - I think I got your point.. So 3 fold cross validation, divide data in 3 sets (say X,Y,Z). Take X as test and remaining (Y+Z) as training set. For next iteration take Y as test and (X+Z) as training set and so on...
Diagnosing EHEC variants and their possible sources
seems difficult, iffy.
As you know, German officials first said "it's cucumbers from Spain",
then weeks later "it's definitely sprouts" (seeds, water ... ?)
(I realize that many people want an instant answer, and are served one.)

With clear symptoms, a map may lead to a cause,
as in the famous map of the
[cholera outbreak in London in 1854](http://en.wikipedia.org/wiki/1854_Broad_Street_cholera_outbreak).
But 
[EHEC outbreak map](http://www.targetmap.com/viewer.aspx?reportId=8459) ?
Thanks a ton Juampa. :)
it does count, thanks.
@LucasReis: I added some rationale.
I tried that but found that the `summary` output are different for 2nd and 3rd line, especially for the p-value of the coefficient, I wonder this will happen if the 2 statements refer to the same dataset. I posted a question about this at http://stackoverflow.com/questions/10268689/weighted-regression-in-r
@Johannes, you might find [this](http://www.econ.uiuc.edu/~roger/research/rq/QReco.pdf) helpful, and the literature it cites. Also, Glen, non-Normal residuals are no reason to rule out using OLS; see [here](http://www.jstor.org/stable/10.2307/2336319), for example.
The question is indeed vague, because you do not define what "similar" means, and that is a very important point. For example are [0.99,1.99] and [1,2] "similar"? Are [1,2,3,4,0] and [1,2,3,4,100000] "similar"? Are non-integer points even possible?
I don't see where to go with this problem because I don't understand how you make the comments quantitative in a way that you can compare or correlate them.  i don't understand your problem.
Could you report the output of `str(data)`? Also, what is `Time`? It does not appear in your model.
I'd like to quote this zinger from the book *Numerical Recipes*: "...if the difference between $n$ and $n-1$ ever matters to you, then you are probably up to no good anyway - e.g., trying to substantiate a questionable hypothesis with marginal data."
Just to be clear, you've got 4 tables, each one if of 2 by 2, with number of "Fatality" for each population (from the people who got and didn't get treatment).  Is that correct, or are you having it in any way different then what I described?
If I divide by sd and not subtract the mean... I will get same volatilities, but different ranges right?
http://en.wikipedia.org/wiki/Randomized_response
We don't have to specify any measure to say which subsets are events. All we need is some set $\\mathscr{X}$ and a sigma-field $\\mathscr{A}$ of subsets of $\\mathscr{X}$. These are considerations about the **measurable space** $(\\mathscr{X},\\mathscr{A})$. We don't need to consider a **measure space** $(\\mathscr{X},\\mathscr{A},\\mu)$, for any measure $\\mu$ (finite or not), to answer this question.
@forecaster Perhaps we should communicate offline as it appears that you are keenly interested in software and time series . Please contact me at my email address.
judging from the graph, the problem is not the lag. What you have plotted is roughly goodness of fit. So it seems that Weibull distribution is not apropriate for your data. I see that there is a bunch up near zero, do you have zero values in your data? In that case you will need to model zero values separately. So first suggestion would be to try Weibull for non zero values. Also why Weibull, is there particular reason, some reference from similar work perhaps?
In that case it is certainly feasible to look at all 16 models that are disturbed by deleting one sample (or even at all 120 model that left out 2 samples). I think with small $n$ I'd probably go for such a systematic cv-like approach.
Perhaps you're thinking of PCA in the wrong way: it's just a transformation, so there's no question of its being correct or incorrect, or relying on assumptions about the data model - unlike, say, regression or factor analysis.
Isn't this a duplicate of [Best practice when analysing pre-post treatment-control designs](http://stats.stackexchange.com/q/3466/930)?
I would suggest using a symbolic computation package such as Maple or Mathematica.
Wow, I just did some reading on effect sizes, and I'm surprised no ones ever mentioned these before. I think I may incorporate this into my project, thanks!
I don't really agree with that, if someone considers "subjective" to be perjorative, that is their error.  Sometimes when we mean probability, we really do mean subjective personal belief - I see no reason not to call it that if that is what is actually meant (choosing to accept only long run frequencies as the definition of probability is a purely subjective choice).
Thanks for answer, this is in line what @maarten said. I quess I indeed have to motivate it
Murray's book teaches you D3.js and JSON which are important tools especially for data scientists working in Web domain; the Roadmap you cite mentions those as well - in essence, it expands your toolbox. Tufte's work is actually older than R, Python and most modern statistical packages, it's more about the general approach and methods rather than using particular software.
In Section 1.1 the authors describe the frequentist scan statistic, they explain there how "randomization testing is used to compute the *p*-value of each detected region". In Section 2 the authors describe the Bayesian scan statistic, and *note* that, by construction, "no randomization testing is necessary", which has implications on the complexity of the algorithm. They mention a particular case in which the complexity (total computation time) is reduced from $O(RN^4)$ for $R=1000$ to $O(N^4)$.
Some of the responses to this related question, [What is a complete list of the usual assumptions for linear regression?](http://stats.stackexchange.com/q/16381/930), are not limited to simple linear regression. Does that help?
Could you summarize the main points of this book?
Thanks - good points. I didn't know about  pairwise.wilcox.test!
I'm not quite sure what names you are looking for @Luca besides "between-effects" and "within-effects". I haven't seen any examples of this in practice, as random effects modelling is a much more direct way to examine the higher level effects. (Fixed effects make the lower level effects unbiased, not the higher ones, so it is unclear what is the motivation for the between effects estimator in practice.)
What data have you got about these nine points?
The title mentions that there is a gap in one variable, but from the body it seems that the gap is in both variables for which you are trying to compute the correlation. So what data exactly is missing?
Thanks for the response. I'm actually only doing this analysis on my workforce, so it's not a sample of a larger population. As far as #2 goes, how would I compute weighted averages in this case? Is there a formula available?
@ Andy W, thanks. sparkTable looks very nice, although it appears you have to decide on one type of plot per column. Would probably not be too much of an effort to achieve mixed plot types in a column manually with what sparkTable provides.
A similar question in lines of specific datasets has been closed here : http://stats.stackexchange.com/questions/38928/high-dimensional-regression-datasets
[This question](http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models) may be of interest
Benjamin, I would encourage you to go find a grad student in statistics that would consult with you. There is a lot more to this than simply having a spreadsheet create a trend line. There are a lot of considerations around how well does the line fit, how significant are the criteria you are using, etc. You need someone with a good knowledge of regression analysis to do this right. If you're writing a thesis you're going to need solid stats behind your conclusions.
1. Quality Control and Industrial Statistics - Duncan
2. Statistical Quality Control - Grant, Leavenworth
3. Introduction to Statistical Quality Control - Montgomery
These numbers are incorrect unless the marbles are returned to the urn after each draw.
@IrishStat: I can accept my own answer, but I can't upvote my own answer.
Thank you for answering a very badly worded question - you are quite right I was neglecting the joint probability
Is there an expectation that the two values will be positively correlated?
@ocram, in addition to my answer, it also stands to say that the estimate of $B_1$ will differ between the full and reduced model if the $Cov(A,B) \\neq 0$. Hence it isn't as simple as choosing the model with the most power in this case (either one of your models presented is correct and the other is wrong, or it makes little difference). But in general, what I state about the shared variance between independent variables and the estimate of the regression coefficient standard error is correct.
I added some explanation as to why I want to do this - thanks!
A coin that turns up Heads with probability $\\frac{1}{2}$ is often called a fair coin, so I suppose you could call your coin an _unfair_ coin (or a coin that is _not fair_ if you don't want to sound pejorative) if you wish to reserve the adjective _biased_ for estimators only.
Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow.
Are $e_1$ and $e_2$ the same in each equation? I think you need to use a different symbol for the errors in the second equation. And do you want $\\rho_{12}$ to be the correlation between $d_{1,2}$ and $d_{2,t}$ or between the two error series?
Comment got too long... made it an answer
@TaylerJones This assertion ("*must* calculate") is false. You can calculate the variance without computing the mean. For example, the sample variance (with the 'n-1' denominator) can be computed from the squared pairwise differences ($(y_i - y_j)^2$), for a vector y: In R the variance can be computed as `sum((outer(y,y,"-"))^2)/(2*length(y)*(length(y)-1))`. Not that I'd advise calculating it this way, but the point is that it's clearly possible without computing the mean, even implicitly.
@Procrastinator, I did check arXiv before posting the question, but it didn't seem right that there are 'only' a handful of pre-prints a day. I'm used to seeing 100+ papers every day in the arXiv categories in my field. I though that maybe the ML community wasn't really into arXiv. Can you confirm that the majority of ML papers get posted to arXiv? If so that would be wonderfully convenient, as I already peruse other parts of arXiv daily anyway.
I meant unrestricted VAR and my data is annual. thanks
Being a Geographer doesn't absolve you when doing bone-headed things with regards to statistical methodology. You'd do well to heed Frank's rebuke as he *is* an expert in this area. You claim Burnham & Anderson in support of your methodology. I am pretty much certain that in their discussion of the Model Averaging approach that B&A are clear that the models to be averaged are to be specified *a priori* from plausible scientific hypothesis, not chosen from the set of all possible models you can fit to your data. At least that was what Anderson says in his Primer book - my B&A is at home...
You could supplement your useful books with various statistics demonstrations from the [Wolfram Demonstrations Project](http://demonstrations.wolfram.com/topic.html?topic=Statistics&limit=20). For instance, [here](http://demonstrations.wolfram.com/StudentsTDistributionAndItsNormalApproximation/) is one showing the difference between the normal and Student's t distributions
Sorry, I don't think I can be much help with that.
I disagree that mean alone is non-informative for non-normal distributions. Consider Poisson.
it might help to draw a graph of age/bmr with two lines - one for men and one for women.
Since `Aptest` is a data frame I wonder what `contrasts(Aptest)` means. The `contrasts` function is not defined for data frames. Do you mean `contrasts(Aptest$Degree)`?
Your approach seems fine. It's a simple linear model and can be fitted with ordinary least squares. Categorical *responses* (outcome/dependent variables) create all sorts of problem addressed with other methods but categorical *predictors* do not require anything else as such.
(-1) These CIs are for each for individual cases?  If so, for a binary outcome, the only sensible CI for a predicted probability is [0,1].  Even though this may be a technically proficient answer.
This question appears to be off-topic because it is about choosing an appropriate statistical method; not about a specific programming question. If you don't know how to analyze your data, try moving to [stats.se]. If you know what model is appropriate for your data but don't know how to do that in R, then edit your question to be more specific.
Let the remaining variable be $y$.  Then $x^2 \\le 1-y^2$ describes the region over which you have to integrate.  Evidently the radius equals $\\sqrt{1 - y^2}$, whence the cross-sectional area equals $\\pi (1 - y^2)/2.$  That's a pretty simple formula :-).  (Remember, the theme here is geometry, not calculus...)
Concerning the 14 decimals, please see [the last line of this answer](http://gis.stackexchange.com/a/8674). :-)
I've assumed the function your referring to is in the forecast package - if that is incorrect please specify what package the function is located in.
*I've just seen on the bookstore a couple of amazing books on MCMC by some guy named Robert*: Is this a joke? Here's a fun idea: Click on the user name of [Xi'an](http://stats.stackexchange.com/users/7224/xian) (the first commenter to the OP in this thread) and follow the link to his blog.
Thinking about it, you could add some transparency (with the alpha= argument) so you can see through the negatives to the bars they are obscuring.  I can't say I love this plot as a triumph of data visualisation, however.  Perhaps it would look better with real data; and the order of the data would make a big difference to how it looks too.  Maybe you could post your final solution.
How about a probit?
The result applies even when $X_1$ and $X_2$ are correlated with unit normal margins. So your explanation only covers a subcase of the original result. However, the basic idea here is sound.