tttjjj/elt_bench huggingface dataset:
@someben And yes, it feels "hacky" but i haven't found many other methods to cross-validate time series.  Another idea is to use a "blocked bootstrap" method to re-sample your time series and create a new series with similar statistical properties.  This only works if your series is stationary, and is probably trickier to implement correctly than a simple time-series cross-validation.  http://en.wikipedia.org/wiki/Bootstrapping_(statistics)#Moving_block_bootstrap
In a very simple manner, this can be done as a control chart (just in space not time) -- namely by converting it to number of counts over discrete segments and then calculating deviation form Poissson distribution.
Standard errors are quite biased low if one removes "insignificant" predictors.  Less clear is what happens with removal of "significant" predictors.  I think it is fair to say that the $\\beta$s of the remaining variables will be biased (due to non-collapsability of odds ratios), so that even correct SEs will be SEs of the wrong quantities.
No, that's not true @Stéphane.  To see why not, consider the case of extremely large $n$ and moderate confidence, say 95%.  With $p=50\\%$, the two-sided tolerance interval should therefore be extremely close to some middle 50% of the distribution, so by definition there's only 50% chance that $X_0$ will lie within it, not the desired 95%.  That's a huge difference!  Intuitively, a tolerance interval for 95% of the population should be sort of close to a prediction interval with 95% confidence, but they still don't exactly agree.
+1 That's a nice capability.  The syntax of terms like $+0.2099013\\{b\\}$, though, is potentially confusing: there is no evident relationship between this expression and the categorical variable `g1`, nor is it entirely evident that $\\{b\\}$ really stands for an *indicator* that $g_1=b$ rather than for the numerical value of $b$!  (Here, $b$ really means "b"--the letter--which may be sufficient warning, but when the categories are coded by numbers, such as $0$ and $1$, watch out...)
What do you mean exactly by "Purchase decision which will be my dependent variable but it does not exist as it is in data"? You don't know the purchase decision for anyone who answered Q5?
In your final conclusion please do not forget you had to assume $\\Pr(B)\\ne 0$. That assumption (which rules out many practical and interesting cases) was essential; the conclusion is false otherwise.
@Glen_b - duly noted. Thanks.
Are you basically asking how topic models work?
+1 Nice explanation. I have two small comments: The dispersion parameter in binomial, Poisson and negative binomial with known shape parameter is 1 by definition of the exponential family (it's not an assumption). When you say that a different dispersion can be estimated and supplied to the summary method then one must be careful because one would venture into quasi territory which has implications especially for the likelihood.
You need to weight survey 2 to the whole population, but how to do this (and whether it is even possible) depends on the sampling method.
Unfortunately, I personally do not know any good online guide. I use "An Introduction to the Bootstrap" by Efron & Tibshirani as my main manual for the bootstrap, but if you can't get it from the local library it has a rather steep price.
Perhaps it would be a better idea use the slope (before and after), to accommodate the fact that there may be a trend?
I'm not looking for ready made software.  I'm looking for methods.
Google finds 12.3 million hits for this quotation.  After running down the first eight pages of them, I haven't found a single site that actually gives a source--they all seem to be quoting each other rather than Mr. Twain.  Does anyone know where he wrote this?  Or maybe it's apocryphal.
@whuber I think my answer was given. I am saying that you compare her performance as measured by improvement over best time to others under conditions as similar as possible. If her result is extreme relative to the others than a flag (question) is raised.  If it is typical or at least not very extreme then nothing should be said.  My point about Beamon is that even in very extreme cases where the statistical evidence of miraculous performance improvement can be shown we may not want to conclude that anything improper took place.
Note that these are all **machine learning** methods. To many "data mining" refers mostly to "unsupervised" knowledge discovery methods with a strong connection to database management for efficiency. I know that the Weka-book is called "machine learning". Did you know it was supposed to be called "practical machine learning", and it was for sales reasons renamed to the buzzword "data mining"? I recommend people to make this distinction to avoid further confusion. ML is a much more precise term.
+1 A couple of tidbits: Early on, the distribution of the square root of a chi-squared r.v., the [chi distribution](http://en.wikipedia.org/wiki/Chi_distribution) ($\\chi$-) was more widely used than now; useful for standard deviations. E.g. you can occasionally see in old papers or books people writing about the $t$ distribution as the distribution of the ratio of a normal to an independent (scaled) chi. One advantage of it (but not the original motivation for it) was that the normal approximation kicks in much faster (this is pre-Wilson-Hilferty); handy when tables tended to be quite limited.
You just write that this is a convenience sample. If you like, you can add something like "but we have no reason to believe that it is substantially different from XXXX" but you should only write this if you can defend it. (XXXX would be some larger population).

If you have any information on XXXX that you can compare to the registry, you can note that and then do the comparison. This is evidence that the sample is not biased, but it is not conclusive evidence (they could differ on some variable that you don't have).
(+1) Using parity is the way to think about this. It also helps if, as you've done, you read the question carefully enough to notice it said *with replacement*. :-) I wish the review tab would have shown there was an answer because otherwise I wouldn't have written my (hasty) comment to the OP. Cheers.
One way to look at it is to see that a kernel density estimate based on a Gaussian kernel is a mixture of normal distributions; a Gaussian mixture may be skew, or heavy-tailed, or light tailed, or multimodal. As such it doesn't assume the original distribution is of any particular form, only that it might reasonably approximated by a mixture.
@ClarkW.Griswold my guess is they've probably just done the CLT, and haven't even mentioned the Delta method. The CLT is sufficient for this question.
The first sentence of this answer misses (or obscures) a crucial point, which is that the definition of mutual exclusivity entails $A \\cap B = \\varnothing$ which is a stronger statement than $\\mathbb P (A \\cap B ) = 0$. A minor tweak to the wording would resolve this. Cheers.
This question has been treated extensively on this site, see e.g. http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi and http://stats.stackexchange.com/questions/612/is-psychprincipal-function-still-pca-when-using-rotation
@MichaelChernick that's correct. I want to compare two curves based on two-dimensional points. However, I've rephrased my question to reflect the fact that basically I want to know whether the distribution of the (x,y) coordinates of the blue points is significantly different from the distribution of the (x,y) coordinates of the red points. I'd also want to know the directionality of this difference.
Can you give a specific example of how to group observations based on the variable's distribution?
Plenty. Cheng and Church for example. You'll need to try out a few, to figure out which one matches your intuitions best. Some are designed for binarized matrixes; so CC might not be the best choice for you. It's common in gene expression analysis that the numerical value is not reliable, but you can tell whether it is "high" or "low".
The OP explicitly said he doesn't want to compute the distance matrix and the dendrogram again, so I'd suggest making this point clearer in your response.
(The $\\TeX$ will work much better when delimited with dollar signs ($).) You describe "PCA regression," but I have some doubts about its applicability, in particular concerning the last paragraph.  What matters for regression are the partial correlations between the independent variables and the dependent variable.  It is possible--even likely--that some of the principal components with *small* eigenvalues may be much better correlated with the dependent variable than some of the PCs with *larger* EVs.  Thus, no criterion based solely on the sizes of the EVs can be generally appropriate.
I'm sorry, but when 65% stayed in their group it isn't very probable, that the chance for staying in your group is 20%.
Yes, see [Probability interpretations](http://en.wikipedia.org/wiki/Probability_interpretations) for instance.
How large are your two data sets? How did you determine the performance on the training data? Is it determined by looking at the results of the classifier trained on the entire training data or is based on cross validation and/or bootstrapping? If cross validation is it a nested one, which contains the cross validation used to tune your model parameters?
I would use the block as a random intercept and test if there is a temporal trend (linear or non-linear, judging from appropriate graphs). If you had more replicates in time or space, you should also test for autocorrelation, but with only four years and four blocks you can skip that.
Thank you @JackRyan for the advice! I will revamp my work in the next month and I will let you know.
@Jase Note the definition, "Here, $\\mathcal{I}_t$ is the information set at time $t$, which is the $\\sigma$-algebra generated by the **lagged values** of the outcome process $(Y_t)$." That is, $\\mathcal{I}_t = \\sigma(Y_{t-1}, Y_{t-2}\\ldots,)$. Some authors write this as $\\mathcal{I}_{t-1}$ but that is counter to the notion of an information set **at time $t$**.
Could this be (overly) simplified as something like the difference between generative and discriminative models?
I have given the answer since the tone of the question seems to indicate that the asker already has some idea of the answer. EDIT: I removed the explicit calculation as this is self study.
It's not clear exactly what you are trying to do here. Do you mean that you want to regress a nominal dependent variable on some factor scores? Or are the factor scores the DVs? See my blog post [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/). Also, factor analysis programs output factor scores directly, you don't need to compute them.
(-1) This answer incorrectly addresses the question by misinterpreting "at most" as "equal." It is also incorrect in asserting that when two disciplines are represented they each must occur at least twice. The answer is $1-1635/4096$, obtained by summing the coefficients of all multiples of $mhg$ in the polynomial expansion of $((20m+15h+5g)/40)^5)$ to compute the chance that all three disciplines are sampled, and subtracting that chance from $1$. To find these multiples, $mhg$ can be multiplied by powers of $m,h,$ and $g$ summing to $2$, which are easily enumerated.
This is sometimes a sign of severe multicollinearity among the $\\beta$'s that are used as regressors in the cross-sectinal regressions. Maybe you should run some relevant tests.
"New observations" are new symbols. I'm working with a base which is basically product descriptions. I do not have all the possible words in my training base, so is common to appear new words (symbols) when decoding new products description.
Am I correct that your second model is not taking into account that the scores are changing over time? It is just comparing the overall group means?
Thank you for the suggestions.  I am currently working with many of those methods.  However, I need a working example of OVA.  The main reason is my mediocre coding skills especially when more than one loop is involved.
To prevent this question from being closed as potentially argumentative or too vague, please edit it to indicate *what* applications you contemplate and what performance criteria you have.  "Statistics" is far too broad to admit a coherent or useful set of replies.
Could this merely be an analogy to improving TV reception by trial and error by moving an antenna, that is the rabbit ears, around? You might imagine doing something like that with bandwidth selection.
Maybe providing a small sample of your data could help in getting more focused answers.
@frin Exactly why 632 works so well is not so easy to understand. I try to explain it by saying that it gives the appropriate coefficients to a linear combination of two estimated to keep the variance and bias both small. It keeps the bias small because it weights the apparent error rate and e0 in a way that nearly cancels their opposing biases. Understanding 632 requires soMe knowledge of the bootstrap e0 estimate and the apparent error rate (the former is a little diffucult to undersatnd and the latter rather easy).
You don't need to subtract `mean(x)` unless you also want it to have a mean of 0. Just dividing by the standard deviation will give it unit variance already.
I have added screenshots above! Is it separation?
One of the regions only has a sample size of 100.
Stationarity requires more than constant mean and variance. Weak stationarity requires that covariance function $cov(X_t,X_{t+h})$ doe not depend on $t$.
Because this is a quadratic function of $\\omega$, with linear gradient, why don't you just use the usual least-squares solution?
Diego, thank you. Actually those examples answer most of my questions.
Maybe this is a job for moderator/dummy variables?
http://stats.stackexchange.com/questions/102902/how-do-you-do-regression-analysis-on-advertising-impact/102909#102909
This is really a statistical question rather than a programming question.  If you want to do an ANOVA you need more than one observation per group.  You *can* test equality of proportions with a single observation per group (e.g. with a binomial GLM or with a Pearson X^2 contingency table analysis), but you will need to have the total number of individuals per group, not just the proportions.
Has anybody done MDPs? If so, can someone please answer my question?
+1 for pointing toward CCA (though it would not necessarily yield canonical vectors that best explain their own block) and factor-related methods.
Are you trying to identify outliers? What is it you are trying to do? Do you just want to test if the population means are the same? (BTW, [GLM](http://en.wikipedia.org/wiki/Generalized_linear_model) is a *model*, not a distribution.
+1 This reply largely avoids confusing association with causation by using the language of explanation, description, and relationship.  This lends it a desirable degree of clarity.
One practical complication with maximum likelihood for stable distributions is that a closed-form expression of the pdf is only known in a select handful of cases (normal, Cauchy and [Levy](http://en.wikipedia.org/wiki/L%C3%A9vy_distribution)).
+1, did not know about ave. When did it appear in R?
By cumulative sum do you mean rolling some with a period of k (eg the sum of the last 20 flights late times)?  Otherwise I don't see how it can decrease unless some flights come in early ie have negative "late" times - which way is it?
The confusion here isn't really w/ R, it's w/ the nature of the activity you're trying to do. What you want is to use the cumulative Weibull as a [link function](http://en.wikipedia.org/wiki/Generalized_linear_model#Link_function) to *connect* the size of your visual stimulus to the probability of it being seen. To understand more about link functions, it may help you to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909).
If "p" is the number of regressors _including_ the constant term usually to be found in a regression setup, then this formula is wrong. The last formula is correct only if we assume that the regressors are deterministic. Exactly what is contained in this book you study? Just "cookbook recipes"?
I used part of this.  The kernel stuff in R is still something that I am not yet familiar with.
Agree with PeterEllis and used one of my precious moderator flag allowances to make the request.
I'm not sure that I understand all of your points. What do you mean by "different combinations of X and Y" etc? The SE depends on X1 and X2, as well as the Y values of the training sample. I guess if the same Y occurs at different values of X1 and X2, you would get different SE's. The SE is not a function of Y alone.
What is the point of this analysis? A Cox PH model doesn't explicitly predict time-to-failure unless you're incorporating some smoothing methods or parametric modeling. What stratification/adjustment variables are in this model?
Sir,I have work on your given advice but I am unable to find any literature depicting the estimation of NB model using multivariate Newton Raphson.Will you please help me
You can model anything. The question is: what do you want to obtain from your model?
As far as I know, a covariance matrix can be computed for (a) multivariate distributions and (b) multivariate samples, but not for arbitrary $m$ by $n$ matrices such as your `A`, so it's impossible to tell what you're asking.  Could you clarify?
Good question. I agree with the quote as well. I believe there are many people in statistics and mathematics who are highly intelligent, and can get very deep into their work, but don't deeply understand what they are working on. Or they do, but are incapable of explaining it to others.I go out of my way to provide answers here in plain English, and ask questions demanding plan English answers.
As I commented on your SO post (in general you should wait for moderators to migrate your post rather than deleting and reposting yourself ...), the issue is not really type I errors (falsely rejecting the null hypothesis). The problem is that the null hypothesis is **almost never** exactly true (except perhaps in physics, sometimes), and large data sets allow you to discover that.
+1 Especially good illustration of data leakage.
+1 for mentioning NFL, but I'd argue that NFL _is_ the 'grand unified theory' of machine learning, or as close to such a thing as we're ever likely to see...
Also, I am surprised that I couldn't come up with good example papers. I'm going to keep looking and will update my answer when I find some, but it seemed like some background would be useful to the oort.
+1 Great. Please add some comments and explanation. Thank you.
Yes, the estimated standard deviation means you use a t-interval. Unfortunately, using pivotal quantities as a way to construct intervals is usually not taught at this level. (I say unfortunately because framing it that way makes it pretty obvious why it must be $t$ and not $z$.)
Would you mind explaining why such procedure should be preferred?
Hmm...All the CDs return between $1$ and $5$ percent per year, so how do you come up with such an enormous rate? After all, someone with access to a non-expiring CD yielding $5$% annually will definitely do better than this investor and their long-run rate is just $0.05$ (or, if you like, around $5000$ dollars per annum).
My data consist in facts about users economy. They are all binary so far. Some examples: owns shares, hired people for a company, owns a pension scheme, bought a house last year, received heritage, is retired, etc. If your are retired you did not hired people for a company and so on. 
Perhaps case-based recommenders are more related to this issue, but I still have not found something similar.
I think it would be a lot easier for the community to help you if you could describe your data and the model you want to fit. If you have only one observation per person and no grouping structure, how do you want to separate the between person variance from the residual variance without assuming a priori a covariance structure for the individuals (for instance, a relationship matrix)?
If your observed values are bounded and you might get fits (or predictions, if you're doing any) anywhere near a bound you should not fit a model whose fitted values are unbounded. Choose a model which gives plausible predictions.
All "extent" question are referenced to the independent variables.   There will be a few question for the dependent variables; however, I won't apply Likert scales for the DV-related questions.   Based on the answers from the IV questions, I'd like to proof/disproof the hypotheses.   Did that help?
You can find more detail here:   http://math.stackexchange.com/questions/442472/sum-of-squares-of-dependent-gaussian-random-variables/442916#442916               An R implementation is is CRAN package CompQuadForm
Garch to the seasonally adjusted data is fine (and important for financial time series). For the HP Filter, I would recommend you simulate some daily data, add to it some seaonality that might somewhat approximate the seasaonlity of your data, then apply your HP filter (or really any potential technique) and see if it can get you back to the original simulated data. Based on my experience with HP filters, the series you're left with might be smoother than the simulated series, which is not good. It will likely remove more than just seasonality.
Thanks! A few hours, or maybe even a whole weekend, worth of time investment is no problem; I'm just hoping for something that won't take *weeks* to learn. :)
This is exactly [what Elvis wrote](http://stats.stackexchange.com/a/45409), isn't it?
Yes, I am now doing something similar. I am just afraid that is not an accurate estimation. I was thinking whether this can be transformed into a Poisson Question. But I dont know whether I can do so, and how I can do so.
Thanks for the two answers! They both fairly equally helped me out. I would have "chosen both" if I could. Mike's pointed me toward some of the theoretical understanding, which was very helpful conceptually, even though Aniko's was a little more directly answering my original question. Thanks to both of you!
Thanks for your input. Will test this approach and will update. Also, do you think, i can use Kolmogorov-Smirnov Test in my case. Please see the plot, http://img254.imageshack.us/img254/9645/25971376.gif
Could you be more precise about what you mean by "look like" and why you would use PC1 rather than the unleaded data themselves?
Can't be - R^2 gets bigger when the fit is better. chi-square gets bigger when the fit is worse...
@Macro, if you don't have equality of variance then you should use the [Satterthwaite-Welch](http://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equation) correction for the degrees of freedom of the [t-test](http://en.wikipedia.org/wiki/Welch%27s_t_test).
@BerndWeiss, my apologies--there was supposed to have been a link!  I've edited my question to include the full citation and a link now.
Thanks for the answer, since some of my cells are below 5, i was about to do a fisher test, but then i came to know about Barnard test and exact package, in its tutorial, i found this "Unconditional tests (such as Barnard’s and Boschloo’s exact tests) are more powerful
alternatives than conditional tests (such as Fisher’s exact test)." that was the reason which made me think that maybe using Barnard test might be a better idea. but as you are saying a Fisher test would suffice.
I don't see where it describe how to find the full conditional distribution
@yue86231 Then it sounds like a measure of correlation would be more appropriate.
Thanks for your help! I'm not a mathematician/statistician so this confused me initially, but your explanation is very easy to understand.