tttjjj/elt_bench huggingface dataset:
See also this related question: [practical-thoughts-on-explanatory-vs-predictive-modeling](http://stats.stackexchange.com/questions/1194/).
I inserted a new section to answer that question, zsljulius.  It concludes that the ACF approximates the correlation coefficient between the beginning and end of the series, as lagged by $k$, but that it tends to be a little smaller in absolute value and in your example grows more negative, too, because the series starts off with less-than-average values and ends up with greater-than-average values.
Can you provide a brief summary of this paper?
To add to the excellent post by @Dimitriy V. Masterov : There is a lot of week to week variation due to various promotions (even in an everyday low price chain like Walmart). Definitely you would need some other Walmart stores nearby (same seasonality in general and promotion in particular) but not too nearby (unaffected by the hurricane).
A good way to check things like this is to simulate the data using known coefficients and very small random error terms, and then check that the output is consistent with the simulation.
@Tom: The word for that is a "nonstationary process".
$\\hat y$ is not $X\\beta$, and the population mean of the $y$'s isn't the same as the sample estimate of it. Which is to say that the second thing is not actually the same thing as the first, but if you replace it with its expectation ($E(\\hat y) = E(y) = X\\beta$), the two would be equivalent.
Presumably these data are meant to be paired in the order given.
I find it helpful to try to draw the distinction between *association* (i.e.  knowing something about *X* tells you something about *Y*), and *correlation* which is an quantified measure of such association. Of course, different measures of correlation have different assumptions and properties built in.
Ace, thanks a lot.
(+1) Although this is not the accepted answer, it (so far) is the only one that correctly captures and emphasizes the distinction between ["average"](http://en.wikipedia.org/wiki/Average) as a (generic) "measure of central tendency" and "mean," which is a specific kind of average.
@Andy, Is this technique the same to what economists know as partially identified distributions (http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-00454-9)?
I don't think it is. I would remove any kind of convergence criteria and let it run for some time with c/t learning rate. Play with the values of c. Also, it is important to carefully inspect the situation when your SGD stops. Compute and look at the batch gradient at that point of time. That's all I can think of.
@StéphaneLaurent I assume you mean the prior distribution (not prior predictive) for the binomial parameter is U-shaped? Also, if the prior is proper, then I would assume the word "distribution" still applies... regardless of its interpretation. (In what way does it not match the technical definition of a distribution?)
Have you noticed that you have **4** `y_values`, but only **3** `x_values`?
It seems more likely to separate the data perhaps. The actual goal at this moment is to predict the price the product was offered for (though the goal is arbitrary; it's just a fun thing).
Could you please elaborate on what it means to you to "test" a pair of tables?  What are you trying to find out?  What does it mean to you for two tables to be "equal"?  As part of that, it would be really helpful to explain how the data arose, because that will provide information essential to determine a probability model for the tables.
Short answer is no you shouldn't do this.  Also, how many distinct "types" can you make with those $20$ features?  That is, what is the maximum number of terminal nodes a tree could possibly have for this data set?
I have altered the question and, hopefully, clarified your questions.
The basic answer is that you don't interpret main effects or interaction effects when the interaction is 'significant', you look at / interpret *simple effects* instead. We could possibly be more specific if you could state what your study, your response variable & the full regression equation is.
Thanks to all those that have provided insights, I could indeed list all possible outcomes and count the one with at least 3 heads but I like the reasoning proposed by Stat-R.
Thanks, but I need a closed form; no numerical integration.
In addition to [tag:feature-selection] that I added, you will probably find useful reference by looking at [tag:model-selection]-related threads.
Do you have multiple data points on the same system at different points in time? If so, you need to take that into account. This would be a time-series, not a cross-sectional dataset.
But you could write a script that does the splitting on your desktop and then push(es) training data and unknown test cases to rapidminer, right? I mean, doing a single iteration/repetition of a 4-fold cross validation isn't *that* much hassle, is it? And it gives you so much more information (at least if you can get the model out of Rapidminer).
"count the number of responses categorized into each number of categories". Please make it clearer what you want. A _response_ can punch but a single category. It's a _respondent_ that punches one or more categories.
@subha: My understanding of categorical distribution has it that way.  For multinomial and binomial, of course it can be.
@JEquihua: Wikipedia is a good place to start: http://en.wikipedia.org/wiki/Ensemble_learning#Stacking  The idea is simple: instead of using your held out data to choose the best model, you use the held out data to find the best way to combine all the models.  For example, you could use a linear regression to find a weighted average of your model set that performs better than any single model.
Thank you very much, that was a great pointer in the right direction!
Many hanks. I will have a look at CCA.
The size and direction of the coefficient on the propensity score is irrelevant for learning about the average treatment effect. That effect (if you have the *true* propensity scores) is what coefficient b estimates.
Am i missing something obvious, or do you not have any instrumental variables?
Something strange is going on. Look at the df in the mixed model vs. the t-test. Condition has df = 400. In the t it is 37. That is surely odd.
Yep no problem. I'm looking at "Variable noise and dimensionality reduction for sparse Gaussian processes", Snelson and Ghahramani. You can see what I've been reading lately. link here: http://www.gatsby.ucl.ac.uk/~snelson/snelson_uai.pdf
You beat me to it! I can't post any math because of a weird error so I didn't write anything. Thanks! :)
I haven't studied the Matlab algorithms, so I don't know what it's doing, but I'm wondering whether your question might be related to the one at http://stats.stackexchange.com/questions/81754/understanding-this-acf-output, which also deals with variations in how cross- and auto-correlation functions are computed.
There's something wrong here: the normal curve overlaid on the bar chart does not agree with the bars.
The problem with this designation of "true" parameters in the case of model misspecification and working probability models is that these are not necessary conditions for consistent estimation. Even under model misspecification, the marginal parameter estimate $\\beta_M$ from the working probability model $E[Y|X] = \\alpha + \\beta_M X$ can be consistently estimated even when $E[Y|X,W] = \\alpha + \\beta_C X + \\gamma W$. As Tukey said, "An approximate answer to the exact question is infinitely more valuable that an exact answer to the approximate question." We may want to estimate $\\beta_M$.
This is roughly what I was looking for.  In fact, having read your answer makes me think of another explanation: rotations are numerically stable, regardless of determinant of the covariance/correlation matrix.  And since PCA can be framed as finding the best rotation of the co-ordinate axis, it will also be numerically stable.
@kurast I'm still thinking about this; you may want to check again in a few days in case I have something more.
My impression is that the nonparametric part is not really relevant for the question, whether it's polynomials, splines, kernel functions, all of them will get "bent out of the way" if you don't put almost all the weight on the center points in the estimation. The main question, I think, is to find the settings so that a robust estimator puts most of the weight on the center points automatically.
Reference here to plotting _means_ looks like a typo. In any case, box plots don't usually show means, unless means are added. The details of box plot implementations vary a lot between software, but plotting medians as default is standard.
Just to clarify and summarise it for other people who might read this:

"cosine similarity" = "normalized correlation coefficient". The difference is that Machine Learning and Information Retrieval people would use the former term, while statisticians might use the latter.
(+1) but for the blog post, which is really good. It would be nice if you were to expand your answer here somewhat though, as this will increase the likelihood of the information remaining available.
It's not a two-sample t-test, since you're comparing a 2008 population with a 2013 sample. You're presumably meant to be doing a one-sample t-test.
http://www.healthknowledge.org.uk/public-health-textbook looks like a useful resource -- surprised i'd never come across it before. Thanks chl.
I don’t think you will have a general result: if the estimate is consistent but has huge variance, it can be near to a local maximum which is not the true maximum... to discard it, you’d have to show that (with high probability) your first estimator is in the basin of attraction of the global maximum likelihood. This seems (to me!) a very technical thing to do – and I think the proof has to be adapted to your particular case.

However I think the heuristic is good and that in general your idea will give good result. A simulation study could help to assess the properties of the method?
great explanation, very clear now.  I should of just plotted it immediately. Thanks for the help!
What is the dependent variable? Have you considered some regularized regression (e.g. lasso, ridge regression)? Such approaches are often useful when you think, a priori, that "a lot of these explanatory variables are not relevant to the prediction of the observations".
Picking the two closest to each other and averaging them seems reasonable. But do you know what exactly goes wrong and why? If you do, this could influence how you do this. For example, if you know the measurement sometimes gets stuck at 0, then you may want to just ignore any zeros.
@R_usr Sorry; I was lazy.  I edited the penultimate line to show the new parameters in terms of the original ones.
@NickStauner how is this a duplicate? "Which journal's are important" is a very different question than "what papers are must-read."
why would you say that the reliability is very low? by reliability i assume you are talking about the same reliability as referred in psychometrics. in that case, shouldn't the random error of the pre and post cancel each other out? i.e. it affects everyone randomly.
You should probably formulate this into a new question on here.  You would need to provide some context because there is no standard wide and narrow.  But to be really brief and vague, you have to decide how big an effect is important or not.  A narrow CI would be smaller than that and a wide CI would be bigger.
This question will likely be perceived as too broad to be suitable for this site, because it appears you are asking for a tutorial on time series analysis.  Please narrow its scope (or find a tutorial and return with any specific questions that occur).
You'll want to look into survival analysis methods taking into account right censoring. Also, why do exactly half the observations not have a completion time? Are you stopping data collection when that threshold is crossed?
This is really something for you to tell us, rather than vice versa. This is a site about statistics questions. "Average" amount of precision depends entirely on what field you are in (and, I bet, what you are sampling). You haven't told us either one. But suppose it's some chemistry question: Then you should be asking on a chemistry site, or, more likely, a specialty withing chemistry.
Ok. I suspected this. Are you familiar with the package? Because I have been playing around with it. I can build the model (not sure how to call it, lets call it a mobForest model) and can asess the out of bag precision etc. I have no clue about how to use it to make predictions on new observations. I have many of them for which I only have the values of the covariables which I would like to predict on.
Oh, I see. You are comparing the best overall accuracy and AUC. But they are still different concept again. The AUC is the P(predicted TRUE|actual TRUE) vs P(FALSE|FALSE), while the overall accuracy is the P=P(TRUE|TRUE)*P(actual TRUE) + P(FALSE|FALSE)*P(actual FALSE). So this depends on the proportion of the true value on your data set very much. In practice, it seems that the best overall accuracy is usually achieved when the cutpoint is near the P(actual TRUE).
If $\\Theta$ is random, and $Y>c$ for one sample, then $P(\\Theta>1/2)$ is not necessarily 1 if $c>1/2$. Could you state the problem in its entirety? Now it is not clear what does it mean that $Y>c$. Does it mean that $P(Y>c)=1$ or that for given sample $Y>c$.
I will check those papers out. Cheers.
@DimitriyV.Masterov So I could use i.year for that?
'Fit a distribution to data' is equivalent to 'estimate the parameters from data'. Some common methods include maximum likelihood or method of moments. In R see, for example, the `fitdistr` function in MASS, which comes with R (`?MASS::fitdistr`), which has an example of fitting a t-distribution. It's certainly possible to do this with a t-distribution and plot the fitted distribution. However, see the warning in the example I mentioned.
Do you mean to use a (negative) lag in the regressors?
It's not absolutely necessary; I was hoping whether there was such a technique so there's more rigour than eye-balling. If nothing exists, well, this is it, but perhaps there is.
user43075 I have taken a gamble and assumed that @IrishStat has guessed correctly (it would be my guess also), but could you confirm? [If that's not the case your question will need to be edited.]
A natural and powerful way to compare two models is to compute the (log) likelihood of the data with respect to each model: the model with the greatest likelihood wins. Your likelihood calculation will be easiest in a QM setting, where the models directly provide transition probabilities. For more information research literature about "likelihood ratios" and maximum likelihood estimation.
Also note that there is Edward's continuity correction:  Edwards, A. 1948. Note on the "correction for continuity" in testing the significance of the difference between correlated proportions. *Psychometrika* 13: 185–187.
Thank you very much!  I'll check out these resources.
So we can only find patterns that we were looking for in the first place?
Can the ROC curve be used within a regression problem? I thought it was just for classification problems.
Discussions can occur in chat, but are inappropriate on our main site.  Please see our [help] for more information about how this works.  In the meantime, I am puzzled by the formatting of your post: almost all of it is formatted as a quotation.  Have you extracted this material from some published source or is it your own, newly written for this answer?  If it's the latter, then please remove the quotations!
do you care to expand upon this comment?
@whuber there isn't a new variable. I was asking how one might find the total entropy of a system if the entropy of the subsystem components is known. Aaron has answered that I can simply add the values together!
Thanks ils for the reference! But the problem still remains.
+1 for thoughtful consideration of both pros and cons.
Yes, that's correct (assuming the waiting times are iid). But the OP isn't asking about *number* of events, he's asking about a time until an event.
Looks like I've managed to get the answer using Binomial. Many thanks for the advice Glen. I will post my solution shortly
The question looks more like a specification of project, with many subquestions...
@whuber I have also added some relevant data at the bottom of the graph. Cheers.
It is on page 425 of Hogg and Craig's Introduction to Mathematical Statistics, 7th edition.
I don't understand your question.  You mention a model and simulating to estimate but don't mention data?  Bootstrap requires resampling from the original data and bootstrap is not the same as Monte Carlo on the empirical distribution.  That is what is done in practice to approximate the bootstrap.  If you have data and you don't want to rely on Gaussian assumptions bootstrapping the data using the Monte Carlo approximation is possible.  Monte Carlo using an assumed multivariate distribution seem to be for a different situation.
+1, another variation is to use a structure plot. Instead of making all of the squares fill the space, you plot squares proportional to the area. See [this answer](http://stats.stackexchange.com/a/20900/1036) for a reference. For this it would be something like `ggplot(runningcounts.df, aes(importance, often)) + geom_point(aes(size = Freq, color = Freq, stat = "identity", position = "identity"), shape = 15) + scale_size_continuous(range = c(3,15)) + scale_color_gradient(low = "white", high = "steelblue")`
With time series data I would be very leery of multiple regression. You probably want some model that accounts for data dependence.
If you know the population mean and can use it instead of the sample mean, yes, it improves your estimate of variance, but the change to your estimate of the variance is just to use $N$ instead of $N-1$.  A useful place to look to answer your question is the justification for using $N-1$ in the first place.
Thank you Peter Flom.
What do you mean "with the same properties"? At the very least, variances and covariances will be different. If you are interested in relationships between variables, that's really important.
Very nice answer, +1 - especially the reference to Good's book, which I also like. Of course, constructing random contingency tables is not entirely trivial - this link may be helpful: http://stackoverflow.com/questions/948327/efficient-way-to-generate-random-contingency-tables
Related question: [Logistic Regression in R (Odds Ratio)](http://stats.stackexchange.com/questions/8661/logistic-regression-in-r-odds-ratio).
"My statistic isn't very big but the p-value is tiny! What gives?" is an incredibly common question here (what gives is usually large sample size). The question usually indicates (i) some lack of understanding of [what a p-value is](http://en.wikipedia.org/wiki/P-value); and (ii) a problem you probably shouldn't have been calculating p-values for in the first place (if small p-values on small effects seem wrong, it's because you're probably actually interested in effect size). Significance tests are vastly overused.
@BabakP perhaps the user was referring to the change of sign in the second derivatives, which is also referred to as curvature (imprecisely)?
Some related questions: [What theories should every statistician know?](http://stats.stackexchange.com/q/22804/10525), [What is it that a statistician does?](http://stats.stackexchange.com/q/5457/10525)
Louvain method? Which is "hierarchical clustering" in R's `igraph` package. What is "large"?
Sorry, I've edited my answer. I meant that you could go for a power of 99% or 99.9%. I did not mean to suggest you should go for a z-score that shows higher confidence. As I said in the answer, selecting a sample on the basis of the z-score it produces is what caused you to have a non-random sample in the first place.
It is difficult to see how your claim that "$V \\sim N(\\mu,\\Sigma)$" can be true: you selected $r$ points according to some *arbitrary* method and they all lie on a circle: there's just no way they have a normal distribution!  What are the $x_i$, by the way?
@Daniel: if applying this, turn off the x-axis in the `plot()` call: `plot(x, y, type = "l", xaxt = "n")` and change the two `x` s so you don't overwrite the data you created to draw the plot.
@Christopher On the contrary, the `.` in the formula tells R to use all variables in the dataframe `spambase.training` (except `is_spam`) as predictors. The model $y = \\beta_{0}$ is fit with `y ~ 1`.
Using an ARIMA model, sales does a *remarkably* good job of predicting average advertising during the current and next month. This suggests a somewhat contrarian approach of investigating the procedures used by the marketing department to allocate advertising resources: they might simply be following sales trends rather than leading them. If that's the case, your analysis of the data will be completely different (or even unnecessary), yet this insight alone (if it turns out to be correct) could have a substantial positive effect on the business. Isn't that the whole point of data mining?
Peter I think that the $t$-test is not as robust to non-normality as most commonly believe.  In effect, the central limit theorem only truly applies if one knows $\\sigma^2$, which is seldom the case.  The need for estimating $\\sigma^2$ implies that people use the $t$-distribution for $P$-values and confidence limits, and the tiniest bit of non-normality can make the $t$ ratio far from a $t$ distribution.
On its own it's not much use, but it's easy to turn it into a proportion of X's > Y's or a Z-score, both of which are reasonably interpretable.
A single answer to your confusion cannot be done because different programs (implementations) _differ in details_. The fact is that whether to rely on U1 or U2, there always a due move is done to compute the unique correct Z or the exact p-value. Don't bother your brains.
Huh? What are you talking about? Who voted you down? All I did was take guesses to cover all the options. I *did extra work as a result*, and I did not vote you down.
I put together some code to do this, just feeding random values as the probability and the actual value, and it was a straight line. I assume AUC is measuring the deviation of the "curve" from a fitted model agains the "line" or randomness?
I'm not sure what you mean by "I have a relatively small sample size (greater than 300)".
+1 for a comprehensive and humorous answer (and for referring to xkcd). In particular, you also tackled my yet unverbalized question whether there's a difference between "Bonferroni-test" and "Bonferroni-correction". Nevertheless, would you mind to explain the multiple comparisons problem in terms of my problem description? I understand that one classifiers is like a treatment group with no/blue/green/... jelly beans in the comic.
**(1)** I really think you should avoid categorizing continuous variables (either dichotomizing, or splitting into thirds). **(2)** My first guess is that you end up w/ less power w/ the tertiary split, but I'm not sure.
As I do not have the access to the article at the moment, I would say that the difference is in the assumptions. If you run regression with instrumental variables you assume that some of your variables are endogenous, i.e. they correlate with the regression disturbances. If they truly are, running the fixed effects regression will not solve this, hence your estimates will be inconsistent and biased. Are you looking for such type of explanation?
@Stat And thanks :))
With the possible exception (as you pointed out) of being able to incorporate temporal autocorrelation in nlme but not lme4.  If the data set is big enough, and if the data have this sort of structure, that could be a big advantage of nlme.
I am now confused as to why does an Academic website have an unrelated problem to Poisson and certainly makes a mistake in choice of application. Can you help me understand that, Can you also elaborate why your approach is right.