tttjjj/elt_bench huggingface dataset:
Almost a duplicate of the later question: http://stats.stackexchange.com/questions/38976/deseasonalize-count-data/45995#45995.  Basically yes, stl is likely to be part of the solution but there are various pitfalls to avoid.
Exactly what I've been doing tutorials for for the last hour. It's pretty awesome.
Thanks Prof Hyndman very helpful.
Great answer, thanks. I gave it another thought yesterday and I got the feeling as well that it could actually just be a VAR(1) process I just setup the complicated way. So your answer is great in confirming that. As you wrote, now I actually have to make sure that this is what I wanted.
Poisson regression can be used to measure counts, as well. In fact, I think that's the usual use. See e.g. [Wikipedia](http://en.wikipedia.org/wiki/Poisson_regression).
D in LDA = discriminant or Dirichlet? That's a separator in itself. Whether you think you are doing statistics or machine learning will have some influence on what you most need or will find most congenial. What software do you use or intend to use?
To use SVM without feature selection, it is *vital* to tune the kernel and regularisation parameters very carefully (I use leave-one-out cross-validation and LS-SVMs and have generally found feature selection makes things worse for almost all of the problems I have looked at).  Also try a linear SVM first, rather than an RBF SVM, many problems with many features are best classified by a linear classifier (even if the underlying problem is non-linear, because of the difficulty in tuning the kernel)
Also, is it the basic version of the question (i.e. expected number of draws to collect all "N" coupons) or what?
Thanks, that was very useful. I like the two approaches that you propose.
Well, for your data, the harmonic mean is not defined! Why do you want to use an harmonic mean? You should give details of what you want to do. Harmonic mean is mostly used for situations when zero observations are logically impossible, so what is producing your zeros? truncation? true zeros? Answer will depend!
I believe you will want to take the "counterfactual" approach. Here is a recent paper of possible interest: Imai, Keele, and Tingley (2010). "A General Approach to Causal Mediation Analysis". _Psychological Methods_, 15:4, 309-334. [Link to paper](http://www.personal.psu.edu/ljk20/mediationII.pdf).
"CTMC" = Continuous Time Markov Chain, presumably.  (Using unexplained acronyms is a good way to lose many potential replies.)
`rnorm(15)` (you don't need the "0,1" since those are the defaults) gives you a sample of size 15. You need the median of that sample, right? So calculate the median and put *that* into `M`. You'll also save yourself a lot of effort if you learn to use `replicate`
I thought there would be good R packages for dyadic data, but CRAN search only produced this: http://cran.r-project.org/web/packages/dyad/index.html, which seems to offer a rather limited functionality.
We already answered that in your other post http://stats.stackexchange.com/questions/24772/finding-the-bayesian-classifier-for-a-bivariate-gaussian-distribution/24834#24834 . You just have to check inequality (*). What is the deal of doing that?
Thank you for answer. In fact I am interested in both, to find the relevant features and improve predictive performance. In my experiments (SVM for classification), feature selection significantly improved the prediction accuracy (however as you noted this may be result of over-fitting the data). I assume , that by ridge regression you mean somethink known also as Tikhonov regularization?
About your first question, yeah, a (paired) t-test wouldn't do because of the sample size (assuming you are not willing to assume a normal distribution).
Just to add some extra bits - you need to ensure normalising constants are included in the maximised log - likelihood. Also the likelihood used in multi level modeling should be the integrated/marginal likelihood (ie random effects integrated out).
@onestop In the past I have used SAS or SPSS. Currently, I'm trying to transition to mostly using R
The odd thing about this seems to be that the correlation of -0.4 suggests that  diamonds which are longer across the top are shorter from top to bottom.  This seems counter-intuitive - are sure it's correct?
I'll leave it here, but please ask future basic R questions on [StackOverflow](http://www.stackoverflow.com).
What do you mean to say by "running the same regression for 100 individuals separately"? - does this mean you have 100 datasets, each with multiple observations on the same set of response variables and explanatory variables?  Not sure how this works eg if you are interested in comparing individuals' height and weight, you only have one observation for each individual so how do fit a regression?  Do you perhaps have a time series for each individual - in which case you need more sophisticated techniques anyway.  That would be a form of longitudinal analysis, if you're looking for search terms.
There may be a textbook on nothing but the analysis of satisfaction data, but it strikes me as somewhat unlikely. If you are just wondering about how to analyze data from likert scales, you could read the threads on CV categorized under the [tag:likert] tag. There are a lot of them. If a particular issue hasn't been covered, you could ask a new question here about it.
can you give one of those or advise a particular one in the book?
What kind of confidence interval do you seek for two _numbers,_ say $x_1 = 3.8$ and $x_2 = 4.8$?  Both lie in the interval $[3.8, 4.8]$ with $100\\%$ confidence (or if you like in the interval $[\\bar{x}-\\sigma,\\bar{x}+\\sigma]$ where $$\\bar{x} = \\frac{x_1+x_2}{2} = 4.3$$ is the sample mean and $$\\sigma=\\sqrt{\\left.\\left.\\frac{1}{2-1}\\right[(x_1-\\bar{x})^2+(x_2-\\bar{x})^2\\right]} =\\frac{1}{\\sqrt{2}}$$ is the sample standard deviation.  What are you _really_ trying to estimate?
@MansT - your understanding of the problem is correct: probability of X number of events in _any_ 30 day window in the last 10 years.
@GavinSimpson, thanks for the Markdown tip, I didn't realize that I need 8 spaces instead of 4.
I understand an 'event' to be a measurable set, but obviously you cannot be asserting that $\\{2,3,5\\}$ is the same as $\\{3\\}$ or $\\{5\\}$, or even has the same probability.  What, then, do *you* mean by equivalent?  (The question probably comes down to the definition of "equivalent." I would argue that the question is ill-posed and cannot be answered as is because "equivalent" has so many common definitions.)
Ah - I didn't realize that. It still seems that the core problem is that a method which works very well on short phrases individually fails when confronted with longer sentences. You mentioned you tried using n-grams as a feature... perhaps you could try using them as a unit of analysis with words as a feature. And, aggregate findings across n-grams within a sentence.
(+1) For pointing this out.
This could be more usefully combined with your earlier question http://stats.stackexchange.com/questions/84081/is-confirmatory-vs-exploratory-statistics-induction-vs-deduction
@user31264 I don't quite know. I was citing an article that used this SSR index that I don't understand. I guess that each anemone way inhabited by one or more fishes. I guess the important part to understand this index is the book from Manly et al but I can't understand it. Thanks for your help!
Why are you using CHAID?  To the best of my understanding it is an old tree classification method that pre dates CART and lacks many of the good statistical properties of CART.
Your values are obtained from the cumulative standard Normal distribution.  For instance, in `R` you can use `pnorm` by typing (say) `pnorm(1.2)`.  The answer you will get is 88.5%.  The function `qnorm` is the inverse; *e.g.*, `qnorm(0.85)` yields the answer 1.036: that's the multiplier to use for 85% coverage.
@John usually you would not start with two random clusters, but with two random medoids. But if you already have two initial clusters, just start from step 3.
D'Agostino and Stevens' *Goodness-of-Fit-Techniques* mentions four statistics for uniformity on the circle (sec 8.10.3.2, p350): $U^2$, $V$, Watson's (1976) statistic, and Ajne's
You can simplify a bit by asking for the distribution of the ratio of $x_2 / x_1$ and subtracting 1.
It would appear that condition numbers are used here merely as a surrogate for the sample size, so why don't you just focus on sample sizes? I am baffled by two aspects of this question. The first concerns the nature of your asymptotics: what are you doing that assures the condition numbers won't grow without bound? The second is why a "typical" condition number in any field would even matter, since there is no general connection between it and the inferences one would like to draw.
By the way, just to make sure I understand your problem correctly, let me phrase it in my own words: you want to estimate an ordered probit, and the complication is that you have several outcomes, so that the ordered probit becomes a multivariate ordered probit. Does this sum up your problem? I am asking because it sounds like that to me, but it seems that there are papers out there dealing with multivariate ordered probits, so I conclude I must misinterpret sth...
The thought is that if it's decision-theoretically 'any good' i.e. 'admissible' in the technical sense used there, then it has formulation that is Bayesian. (which doesn't, of course, imply that it is admissible, that applying decision theoretic criteria make sense anyway, or that it's otherwise sensible).
I imagine that the distinction from 'constant' it's because ... while it can't be distinguished from a constant - you certainly can't prove it is a constant.
This question appears to be off-topic because it is about how to do something in R. It would be better to ask this on a list devoted to R or lme.
I put them here https://gist.github.com/4680507 Dimitriy
Regarding your second question: If you have exact colinearity you can just remove one of the variables. You don't need ridge regression.
I think these comments are not correct.  In the server world, open source rules, and the Apache web server is the most popular web server.
@whuber Thank you very much for your answer and for the SO post you've lunked to. I'm afraid I was hoping to learn how to tell them apart and tell when to use one over another using that time series for illustration purposes. I've never heard of decomposition based reciprocals :-/ I'll do some research on that.
I'll try. Which R-package do you recommend ?
Nice analogy of euclidean space!
I tried a little different approach but with the same spirit in mind, just used the curve() function and the code is as follows:

curve(ipc_values_EM\\$lambda[1] * pnorm(x, ipc_values_EM\\$mu[1], ipc_values_EM\\$sigma[1]) 
+
ipc_values_EM\\$lambda[2] * pnorm(x, ipc_values_EM\\$mu[2], ipc_values_EM\\$sigma[2]), from=-0.10, to=0.07, add=TRUE, col="blue")
-1. I agree with @whuber here. The fact that the three sample statistics are different" is not very useful to point out - assuming these data are continuous, the estimates will be different with probability $1$.  Also, I don't understand *"The key question is not whether the CIs overlap, but what the p-value is"*-these confidence intervals are based on inverting hypothesis tests; two CIs overlapping is equivalent to not rejecting the null comparing the two groups for equivalent means. Finally, I'm not sure what is the motivation for answering when you are "not sure what the question intended".
@whuber I already gave my +1, but I'd like to add that this is very instructive to read one's approach to such a problem. You're worth the bounty in any manner whatsoever.
@MichaelMayer Corrected. I'll go put on another pot of coffee...
@Shaniqueia Don't know to which papers you are refering to, but it's no rare to find very low $R^2$ when dealing with financial returns in high (daily) and ultra-high(intraday) frequency data. The majority of the models being published tend to refer to issues other than predictions... in many markets this too difficult (that's why $R^2$ are so low and often naive models like random walks perform well) so they adress other issues related to higher moments of financial returns and not necesarily to their expectation (= first moment ->prediction)
Feed $1 - m$ to a function like `qnorm()` in R. That tells you that you are so many standard deviations above or below the mean. You know the standard deviation as the root of the variance. The rest is arithmetic.
Well, Judea Pearl wrote a whole [book on it](http://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/0521773628) albeit one I find very hard to understand. Other definitions involve "producing an effect" but that doesn't help so much. It is a topic that has been very vexed.
Just do something like (for this made-up data) `cor.test(1:5, c(1,2,3,4,3)`. This example has a correlation coefficient of 0.832, but its p-value is 0.081. So clearly a sample correlation coefficient of 0.72 is simply not significant if it's based on just 5 samples. If the correlation is real, you'll likely need more data (more points) to demonstrate that.
While randomly assigning the cases to the folds is usually the best option, there are systematic ways of spitting which are also discussed as "cross validation" see e.g. here wiki.eigenvector.com/index.php?title=Using_Cross-Validation.
Compare: `x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE))`
These are explained for a simple but interesting problem (a ternary response) in plain, non-statistical language at http://stats.stackexchange.com/a/18609.
Infinitely differentiable means derivatives of all orders exist.  It is not equivalent to absolutely continuous.  Because derivatives of $\\phi$ correspond to moments of the distribution, you can turn this into a question about the existence of moments of arbitrarily high order.
Could you please suggest a stats tutorial for me (preferrably online)?
Would you break these out into separate options?
Paired-sample Wilcoxon signed-rank test is the best choice. Or, less sensitive - sign test (which is equivalent to NcNemar's in case of dichotomous data).
Depends where you look. At the level of the $f$, then the common mean is zero, but remember the entire model has an intercept (constant) term which combines additively with the $f$. So at the level of the model the $f$ describe how $y$ varies with $x$ within the groups about some non-zero common mean; the non-zero bit coming from the value that the intercept takes.
A potentially useful post: http://stats.stackexchange.com/q/26862/24000
@IdiotAbroad That's not the way you should read scientific papers, you should quote it for its main result/argument.
Also, thank you for the link. Helpful bullets
I was using predict(model, data=train). I have now switched to predict(model) and my auc has dropped to 87%. Is this a good thing or a bad thing?
Is it possible that $(a+b)^2 = a^2 + b^2 -ab$ ? it is true only when a or b is zero. Your first formula is true only when cov(dt, ft)=0
@Paul "Statistical properties" in what sense?  What role does the timing play: is it an auxiliary variable or is it the variable of interest?  I would find it *much* easier to think about your question if you were to reformulate it without the extensive speculation about types of tests and simply describe as clearly as possible what your experiment consists of and what you want to learn from it.
Are $k_A$ & $k_B$ supposed to be equal? It seems like that should be mentioned.
thanks. I made a silly mistake. Appreciate!
Thank you so much Peter, I really, really appreciate this!
I'm not so sure about that; in many cases, once you include an interaction term, A and B become unimportant.  A simple example is estimating the weight of a tree based on its height and diameter at its base; once you multiply the two terms, the individual terms will drop out (although w/o the interaction, they would both be significant.)  Often you need more than one gene for a trait to express itself (e.g., two blue-eyed parents for a blue-eyed child), or you need a gene and an environmental factor together for an effect to occur.  Not so sure about the social sciences, though...
Yes which? Continuous dependent variable? I think you need to elaborate on your problem, because if the predicted value is continuous, you can apply anything from simple linear regression to random forests and PLS. Describe your problem in detail, please.
+1 for the first paragraph but dealing with social science or medical data, the effect of 0 vs. 0.1 weeks of something is not the main worry. The point is that women who do not attempt or report breastfeeding at all might be systematically different in a lot of other respects (health problems, income, family situation, ability to stay out of work, access to health services, where they obtained information about parenting, etc.) There is really no reason to believe that these women are very similar to mothers who try breastfeeding and stop it quickly.
+1, nice question, Stephane. I don't know if you had seen this related question: [are-all-values-within-a-95-confidence-interval-equally-likely](http://stats.stackexchange.com/questions/40769/), but if not you might want to take a look at it.
How do you respond to the [comment made by Anne Z.](http://stats.stackexchange.com/questions/22445/options-for-comparing-logistic-regression-models#comment40708_22445)?
Thank you very much for replying. Would you please explain a little bit further what you mean "If you want to compare the algorithms you can embed each into the cross validation "? How do you embed them into the CV? I am confused. Thanks,
@ChrisTaylor, assuming you're using the word "correlation" in the usual statistical sense, causation does not need to imply correlation if the relationship is non-linear. If $X \\sim N(0,1)$, then $X$ is causally linked to $|X|$ but not correlated.
How many factors did you extracted and what kind of rotation did you use, if any? Also, what is the number of items and the sample size?
Generally it doesn't make sense to drop levels of a factor in this fashion. Even if you're looking to collapse a categorical variable into some smaller set of categories, I wouldn't do it this way. Sometimes levels of factors won't be very different from that of the baseline (reference) category. That on its own isn't a good reason to drop them from the model.
Please accept some answers to your questions (use the tick button next to the answer you consider best).
To give an example: Often one wants to know the P(model | data) ). Frequentist analysis gives you P(data | model) however (which then people often read as P (model |data). By assuming a prior probability P(model) you can get P(model | data) in Bayesian statsitics. But then you can debate what P(model) should be.
Thanks for the detail (I'm the same as the OP, my account's giving me issues)!  I know the same skillset is required for each category, so that's not a problem.  And while your suggestion does help me rank the users, do you know of a way to effectively generate a 'weight' - an actual, numerical coefficient - that I can calculate for each category?  Effectively to say "3 tasks in category A are worth 5 in category b" because I'll have a conversion rate.  That's what I'm aiming for.
Thanks for supporting my worries: indeed, I have though about what makes biological sense. The problem is, that I actually have two related datasets and I would like to draw conclusions from both at the same time. But in one subset, the density variable is best in the models untransformed while in the other log transformation is the best. Log transformation improves the relationship in the dataset that has the lower values for that variable, so it will be very difficult to reconcile these two datasets I think, unless I leave the variable untransformed in both.
For general-to-specific estimation I refer to the process of estimation that begins with the maximum number of (or a group of) regressors --school characteristics on my case-- that the sample allow me. Then, I substract some regressors that do not fulfill a criterion, and so on. By doing so I will get, tipically, a regression with just a few variables, all significant.

For instance, in PcGive the process is called 'testimation', given that the criterion is the t-Stat significance test.
The solution will be found by looking into non negative least squares (NNLS).
@Chris The link has passed away.
No. It's not irrelevant. In fact, they tell you explicitly in the question that it's relevant. To make that statement is to completely ignore information in the question and ensure that you will NOT get the job. It's 50% if you assume that none of your friends interviewed, because it's not representative of the actual interviewees. The more friends you have that interviewed, the closer you come to the accepted answer. The less friends you have that interviewed the closer you come to 50%.
@CSands: Thanks for answer, but if I understood correctly then in case when all observations from ninth decile shifts to the tenth decile your solution would join ninth and tenth decile and PSI would be 0, where in reality the distribution has changed significantly. Am I right?
@IanBoyd I don't think you really need to know the theoretical distribution to get that - would the median work?  And for shading in the adjacent 'standard deviations', you could just use quantiles.  If your objective is to communicate how long most pings take, I don't see anything wrong with reporting the median with the 25% and 75% quantiles.  Or, if you want to emulate the 68% that falls within 1 SD of the mean in the normal distribution, the 16% and 84% quantiles.
Partialling out more than one A-variable is done by applying the above formula of partial corr. recursively or by computing partial corr from linear regressional coefficients (which, in fact, are the "unnormalized" form of partial corr. coeffitients)
Are the features missing *truly* at random?  What is the mechanism of missingness?
Thank you for the response Mark Heckmann. I tried to transform the data as it is not normally distributed and then apply the F test. However even transformations of y^3 result in a left skew so I have now moved on to a Wilcox test...I am a bit confused though, as I have been asked to provide the degrees of freedom for this..
@StephanKolassa The books I listed are just examples of introductory statistics that I mentioned for the pedagogical value.
The models don't need to be nested to be compared with AIC or BIC.
-1 not really stats specific...
$\\epsilon_k$ is sometimes a scalar, and sometimes a vector, per-feature, at each iteration. Further, either can be constant for all time, or decaying, or adaptive ... *and* the gradients can be smoothed or scaled or not. Many possibilities, more folklore than test cases.
One need not apply more than basic probability laws to come up with an albeit trivial answer to the question if they use some logic. See my answer.
Do you got sales development from other cities to compare with?
@Russ don't worry about the downvote, it really doesn't matter, outside of the fact that it signals there's something I should address. Much more important to get at why we disagree (to the extent that we do at all) than to worry about fake internet points. You have an objection worth discussing, and I'd gladly pay ten times that downvote to have even this brief discussion. I encourage you to downvote me every time you disagree, if you'll say why. That's my chance to learn something.
@Stas As I pointed out, my functions can be made to look very "nice" by mollifying them. They can also be generalized to mollifications of large finite mixtures of atoms. The class of such distributions is dense in all distributions on the unit interval, so I don't think bootstrap regularity would be involved here.
Well, for your sentence "for every stochastic algorithm, there should be at least one deterministic algorithm that performs better", there is a counterpart written by David Mumford [link](http://www.dam.brown.edu/people/mumford/Papers/OverviewPapers/DawningAgeStoch.pdf).
Welcome to the site. This seems like a time series problem; I added that tag so our time series experts will see it.