tttjjj/elt_bench huggingface dataset:
Good answer. More explicitly: If a kernel function would guarantee that it would always be able to linearly separate data, it would be useless. This is almost exactly the definition of a function with infinite VC-dimension, which means that you will never be able to learn/generalize the correct function regardless of the amount of data. See here: http://www.cs.huji.ac.il/~shashua/papers/class11-PAC2.pdf
@AdheshJosh I've elaborated on my original answer. I hope it helps.
Please read this page of the [help](http://stats.stackexchange.com/help/on-topic), which toward the bottom says "*Please note, however, that cross-posting is not encouraged on SE sites. Choose one best location to post your question. Later, if it proves better suited on another site, it can be migrated.*"
How do you know that someone is a customer if they don't place an initial order?
@Stéphane Laurent: http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_mixed_sect036.htm
Thanks for the clarification: the appearance of dates is pure illusion, so the suggestion of autocorrelation or variogram is quite wrong here. But 8000+ square sounds awful big to me.
I've tweaked the title a little bit so that it is not a duplicate of the previous question. Feel free to modify if I've misconstrued.
Your question is a little vague about "the intervals where..." - there could be multiple intervals.  Are you interested in solely the inner 90%, i.e. symmetrically trimming on each side?  After all, from the minimum to 90%ile, 90% of the data is captured, similarly for 10%ile to the max value.
You might also want to consider the [quasi-Poisson](https://en.wikipedia.org/wiki/Quasi-likelihood) and [negative binomial](https://en.wikipedia.org/wiki/Negative_binomial_distribution) distributions.
@Anand:  Open a new R session as admin (or run sudo R on mac/linux).  Run `update.packages.` When that is finished, close R and re-open a normal (non-admin) session.  Run your code except for the "SVM" and "Confusion matrix" sections.  Then run my code.  If you still get an error, please post the line that returned an error, along with the exact error.
Hello thanks for the answer I have a question. McNemar's test seems to be according to Wikipedia for 2x2 matrix. But in my case I have 3x3 matrix cuz i have yes , no and "don't know...". So should I use another test
@NeilG: Thanks. I had hoped to get a completely analytical approach, but the DP code is quite instructional as well.
The scenario sounds like a magic recipe for irritating your girlfriend. On the other hand you've made me curious what she thinks the correct answer is (if she has a specific answer in mind, it should be easy enough to conduct a tabletop experiment which should show one of the two to be wrong).
@StasK The DF argument may be "handwaving," but it's *standard* handwaving!  For an excellent question about DF, see http://stats.stackexchange.com/questions/16921/how-to-understand-degrees-of-freedom.
Thank you, @JBowman, that is exactly the explanation I needed and the output is precisely my distribution.
PDF: The closest physical concept I know is the actual density function, this is, a normalised function that tells you how the mass is distributed in a set o points, a line, a surface or a (hyper)volume of a body or set of bodies. CDF: This tells you about the mass cumulated in a certain region or set of points in the discrete case. Similarly, the expectation is related to the [Centre of mass](http://en.wikipedia.org/wiki/Center_of_mass).
Possibly related: [this question](http://stats.stackexchange.com/questions/28251/classifying-integer-sequences-by-their-first-difference/).
Concerning confidence intervals, please consult http://stats.stackexchange.com/questions/6652/what-precisely-is-a-confidence-interval.  (In your statement, the event "the observed mean falls outside of the confidence interval" usually occurs with 0% probability.)
I actually read the book and still couldn't figure out the w's ... Thank you!
If you meant something other than what you wrote, then please edit your reply to reflect that.
Could really use some help here...I'm still not sure what the output tells me.
Edited post to add a template study.  You can do a lot of things with time series analysis as others have posted.  You can do a lot of things without time series analysis too.  Caret will be useful to you and you can read a lot more about the techniques and the package in Kuhn & Johnson's "Applied Predictive Modeling".
*Weigted by the 'importance' of $f$.
As log-likelihood and related Bayesian measures are the ideal optimality criteria, it is best to use log-likelihood-based measures to study predictive information.  This includes generalized $R^2$.  The Brier score (quadratic accuracy score) is also an excellent choice if $Y$ is binary, followed by the concordance probability ($c$-index = AUROC).  Improper scoring rules are optimized by choosing the wrong features and wrong model.  I have an example where adding a powerful variable makes the proportion classified correctly _decrease_.
Thank you for the answer. Yes, you are right. AIC does not apply any test, instead, it gives a simple measure of how good the model fits the sample and whether the model can be kept simple as well, by adding the -2*loglikelihood with 2*number_of_parameters. Maybe this explains why variables with non-significant p-values were kept in the selected model?
Thank you for this -- I'll work my way through the algorithm.
Hint: what is the formula for $\\hat{\\sigma}^2$ here?
Is the real data uniformly distributed or does it come from some unknown distribution?
More information is needed because there are many different solutions. How should the individual numbers be distributed? How should they be correlated?  Should there be any restrictions on their values (such as lying between $0$ and $1$)?
The Bayes factor is the ratio of the marginal likelihood assuming H1 to the marginal likelihood assuming H0.  In this case, the alternative hypothesis H1 is that the slope is positive, so your prior fo H1 must exclude negative values for the slope parameter as H1 says they are implausible a-priori.  One way of achieving this might be to use an exponential prior, which is strictly positive, for the slope and a Gaussian prior for the intercept.  You could then implement a "negative slope" null hypothesis by flipping the data upside-down and using the same model.

HTH
thanks for clarifying the idea. It appears that this kind of rescaling could be useful only in case of psychometric measurements of a domain such as intelligence, fear and honesty. The construct variables generally lack validity.
Could you elaborate on what this has to do with the original poster's question?
Thank you so much..
Thank you Blain Waan!
@ Richard Muallil Your grouping variable is `town` (level 2), right? `town.type` is a level 2 predictor. Does it make sense to model `town.type` as a random slope `tag(town.type|town)`?
Fred, a couple of questions: (1) when you say 'commitment' here, is that meaning 'the trait of sincere and steadfast fixity of purpose' or 'the official act of consigning a person to confinement'? (2) If the latter, are you really saying people will be confined or not based on the predictions from your model? If so I think I might wish to seek legal advice before posting an answer!

By the way, you might like to take a quick look at the latter paragraphs of the section 'How do I ask questions here?' in the site FAQ:http://stats.stackexchange.com/faq
I think I see. This is sometimes called a mixed-anova, because some of your factors are between individuals and some are within.  So you have a 2 (factor A) x 2 (factor B) x 3 (time) design, where A and B are between subjects, and time is within.  So the short answer is definitely yes. If you want to treat time as a trend, I think you're going to need a multilevel model, but your sample is small and that might lead to problems.  What software do you use?
Prediction doesn't refer to the nature of the dependent variable per se.  It refers to having a continuous result for the predictions, e.g., a predicted probability that someone will be in class B given they are in either class A or class B.  An excellent probability modeling method (e.g., logistic model with appropriate nonlinearities and shrinkage) is a good choice.  The goal of the analyst, most of the time, should be to give optimal predictions.  The decision maker should decide what to do with the predictions.  E.g., a probability of 0.5 may indicate "get more data".
Edited the post to say my preference in coordinate system. As to the shape of the cone... just a circular base. Think a polar cap, but rotated to a general direction.
@Ben Gorman, I put in some more details -- thanks!
Interesting.  I *always* use natural logs because many of the coefficients tend to be close to zero and then can be interpreted as proportional (relative) differences.  That's not possible in any other base of logarithm.  I see some merit in using other bases, but I think you need to clarify your response, because *prima facie* your interpretation doesn't use the value of the coefficient at all!
Thanks! That looks like what I need for the binomial case.
If your sample size is pretty large, the $p$-values should still be reasonable (by the central limit theorem) as long as your error distribution isn't long-tailed. If you just want to test the categorical variable in isolation, you can use a non-parametric ANOVA: http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance but, as I said, this is really becoming an entirely different question from the one posed and may be more appropriately posed as a new question or answered by searching the site for a related question.
I think the independent manipulation of $A$ would make the null exactly true. However, w/ *observational* research, you're right that the null would definitely be false: if you went to some random pool halls, & measured how fast (hard) people hit the cue ball on 1 table, & how fast object balls (eg 8 balls) rolled on another table, there is no question they would be correlated b/c patrons at the same pool hall would be more similar to each other than to patrons at a pool hall somewhere else in the world, *even though there would still be no direct causal connection* b/t the different balls.
@cardinal I was insufficiently caffeinated to notice the joke. Yes, the mnemonics are bad, but the choice of $H$ and $E$ (and $T=H+E$) is rather standard.
My guess would be that the number 30 comes from the old rule of thumb that you need at least 30 data points in order to do X. (Where I've seen X replaced with: "valid regression analysis", "to get a good estimate of variability", etc., or the point where you can switch from the t to the normal distribution.)
Sorry I went straight into importance sampling, and then I realized that usually sampling is simpler than that. I added your initial explanation to my answers. Many thanks
This is exactly what I meant, you gave me a good idea. Now I'll try to understand those test, since I have a superficial knowledge :(
- "OK, from your comments, it seems you have data on one brand across several regions, both for one month and for one year"
-> Well not exactly. The data for the particular brand is for one month. The data for ALL brands together is for one year.
- "comparable metrics" -> yes, percent of population.
 -I like the regression approach. so if i regress specific brand on the general variable, i could see which regions are above the regression line and which are below, right?
Since the mean of the $Z_i$ is known to be $0$, what is the UMVUE of $\\theta^2$ if you were given the $Z_i$'s instead of the $X_i$'s? What would happen if each occurrence of $Z_i^2$ in this UMVUE is replaced by $|Z_i|^2 = X_i^2$?
I wouldn't recommend reshape - even I don't understand how it works. Reshape2 is much much better and follows many more good development principles.
Note that "plugging in" MLEs doesn't take uncertainty in parameter estimates into account when making predictions. Profile predictive likelihoods would be a better illustration of non-Bayesian predictive inference.
Thanks ........
What are the possible values of $K$ and $N$ and how should we interpret "best" in a quantitative way?  "Best" for what purpose?
Well I've worked out that $\\mathbb{P}(X/a \\leq x) = 1 - e^{-\\lambda a x}$ and then tried to say that if $x \\in (n , n+1)$ then $\\mathbb{P} ([X/a] \\leq x)$ is the same as $\\mathbb{P}(X/a \\leq n+1)$ but I don't think this really helps!
I believe that There is no expert here, in this fórum, about your question.
Sorry
Ivan
Chapter 9 is a fairly friendly introduction with R code at the end of the chapter: http://www-bcf.usc.edu/~gareth/ISL/
As @IrishStat says, you can't associate probability values with specific future sample paths. It is not clear what you mean by "forecast a series" -- you can just string together the point forecasts for different horizons and you get forecasts for the series.

Perhaps you want prediction intervals with specified probabilities for future values -- but any software will do that, so I suspect you want something else.

Can you provide more information?
+1, excellent points.
I noticed you haven't accepted any of the answers to the (other) questions you answered so far. Once you have received a (partially) satisfactory answer, please choose the one you prefer by accepting it/clicking on the check mark to the left of the answer.
I was intending to make the intercept shared by those with the same value of controlf a fixed effect, as they are nested values (if that makes sense?) and then the season and clip were supposed to be random effects?
I am reading your edits. When you suggest using the asymptotic result, is that Wilk's theorem? I am not too sure if Wilk's theorem is valid here because one of the assumptions of Wilk's theorem is that the support doesn't depend on the parameter.
which you could write  as $\\frac{2}{3}n + \\frac{1}{2} -\\frac{1}{6n}$ to make it more obvious what happens when $n$ increases.
The uniform distribution is a special case of the dirichlet distribution.
I'm surprised to see such a jump in accuracy. How many elements do you have in your training set, and do you compute the accuracy over a testing set that does not include any elements of the training set?
Should probably be community wiki?
For some intuition--and a generalization--note that over its support $[0,\\infty),$ the exponential PDF strictly decreases. Let us assume only that the $A_i$ are independent variables having *any* distributions on $[0,\\infty)$ (possibly varying among the $(A_i)$) whose PDFs strictly decrease. Nevertheless (1) the MLE cannot exceed any of the $b_i$, making $\\min\\{b_i\\}$ a lower bound for the MLE, while (2) decreasing $k$ increases *every one* of the probabilities of the $b_i$, thereby increasing the likelihood. Now draw the (only possible) conclusion.
How are the elements in the table sampled? If $x_{11}+x_{12}=n$ is fixed by design, the conditional inference is natural. If you just sample $x_{11}+x_{12}+x_{21}+x_{22}$ observations and collect their corresponding row/column variables, unconditional inference on $p$ may be obtained using a multinomial model.
Well.... you have to make up your mind as to whether more or less parsimony is "better" for whatever your definition of "good" is. Personally, I think polynomial terms higher than 2 are hard to interpret and, if they are needed, I prefer splines. But AIC and BIC (and their variants) all have fierce partisans - if you search Google, you can find many papers. But the reason software offers both choices is that neither is that the decision on which is "better" isn't in yet.
You bet! Time between outbreaks of herpes virus. You have a much longer time to your initial outbreak because you must actually contract the virus. Your interarrival times between outbreaks thereafter are independent from one another, but come at a much faster rate relative to the index outbreak. Non-exponential interarrival times are the norm. In survival analysis, a commonly used analytic method is Cox Proportional Hazards models where you strip off any parametric assumptions about the interarrival times.
I don't think this would have a unique solution in some circumstances. It seems very close to asking for quantiles in more than one dimension. Are you sure you really want to constrain the clustering in such a way? (Imagine if you had 11 of the points in the exact same location) In a brief search I was able to turn up this [paper](http://www.sciencedirect.com/science/article/pii/S0031320399000576), but I could not tell if any R implementation existed. Try checking out the [Cran Task view for clustering](http://cran.cnr.berkeley.edu/web/views/Cluster.html).
This answer should have been made as a comment to Michael's answer or as an edit to your initial question.  No big deal - just an FYI.
By the way, you can add extra information by editing your question; it's often helpful to do so rather than appending it to a growing comment thread.
@StephanKolassa 267
I think you will get an answer soon, but maybe you could add some background information, i.e. what are you trying to show (or equivalently, what is/are your null hypothes{i|e}s), what is the sample size, etc. In the meantime, I think @caracal's brilliant response to this question [What is the NULL hypothesis for interaction in a two-way ANOVA?](http://stats.stackexchange.com/questions/5617/what-is-the-null-hypothesis-for-interaction-in-a-two-way-anova/5622#5622) might be helpful.
Hi Renee, you might want to ask a separate question about interpretation of bootstrapping for mediation. However, as a simple rule that is often applied, if the bootstrapped 95% confidence interval of the indirect effect does not include zero, then the indirect effect is deemed to be statistically significant. Preacher and Hayes (2004) explains this in great detail: http://www.comm.ohio-state.edu/ahayes/BRMIC2004.pdf
@Student That is incorrect and scarcely even makes sense.
To continue @Glen_b's wise hint, you do not have to compute any integrals at all. That is because (1) this question evaluates your understanding of the relationship between the parameters (mean and SD) of a distribution and the parameters of its sampling distribution and (2) the parameters of the parent distribution are easily found by examining the $-2x^2+10x$ term alone: use algebra to rewrite it in the form $(x-\\mu)^2/(2\\sigma^2) + c$ where $c$ is not a function of $x$.
I decided to implement my own function! thanks
Crystal-clear, and the example is perfect. Thank you!
Thanks! Does the relation between the observed power and the p-value for Z-test still apply for any test? Or what kinds of tests can it apply to?
If you really have a single categorical outcome variable and all the other variables are continuous, then the situation you describe as undesirable does not fit.  Ordinal logistic regression would place the categorical variable as the dependent, and there are no predictors that can have reference categories.  So ordinal logistic regression should give you exactly what you need:  a single coefficient for each predictor.
(cont.) To me, the only weakness of these statements is that they they do not explain the mechanism of the effect, - they just describe the effect itself, its consequence. For the mechanism, you suggest `The intution behind the terminology is that... in the simple regression case the effect of x  was being "suppressed" by the omitted z. And when we include z... the effect of x emerges, where we couldn't see it as clearly before`. This way of thinking is a bit mystic and not correct for me. How can z "suppress" something while z is absent? (to cont.)
I should have mentioned, I'm actually comparing one data set (climate observations) with an ensemble of comparable data sets (model runs). So I'm actually comparing 1.2m obs points, with 87m model points, hence the `approx()` function comes into play in the `qqplot()` function.
10% .. is that the total area of the circles, or the area of each?
I tried already, but since I have no idea of how to counter-check the result, so I wonder if anyone can enlighten me here, thanks anyway.
+1 and thanks for responding!
@jank Even though I agree with the limitation discussed by, I have to highlight one which did not receive enough attention, at least not in a sense that highlighted the problem of the approach. While the SE can be used to calculate the error of the mean inside a population, I am not sure you can use it at all to calculate the error among means which come in themselves from different populations. So I wonder if the error of the mean does not have to be calculated simply by error propagation in this last step.
Of note, `is.na()` returns `TRUE` for both NA and NaN, which differs from `is.nan()` e.g. `is.na(c(0/0,NA))` *vs.* `is.nan(c(0/0,NA))`.
Consider a Gamma distribution with shape parameter $\\alpha$. Take the scale as 1 (it doesn't matter). Let's say you regard $\\text{Gamma}(\\alpha_0,1)$ as *just* "sufficiently normal". Then a distribution for which you need to get 1000 observations to be sufficiently normal has a $\\text{Gamma}(\\alpha_0/1000,1)$ distribution.
2/2 exact same answers. So I was wondering would the same apply for a tobit or heckman selection?
Thanks David and @whuber 
I am not very familiar with JAGS, STAN, so it will take me some time to understand your answer. I have upvoted your answer for now.
No, it would become $2^{x-10}$. But I also pulled $2^{10-x}$ out of the $4^{10-x}$, so in total I pulled $2^{10}$ out of the top. Remember $1^{\\text{whatever}} = 1$.
Questions about your post:  1. You want to predict Y separately from the process of modelling the error component of X?  2. What is "an own regression line"?  3. How will you determine the size of the error component of X for any individual or group?  In other words, how can it be the dependent variable in a model without your knowing its values?
I should also add that the advertisers are only paid upon installation, so any served up but not clicked on that resulted in an installation are not tracked. In addition, due to the constraints of the store, it will not forward data indicating what led the user to install the application (only the advertiser gives that info).
I'm sort of curious how one adds fixed effects to GAMMA glms.  Anybody know a good method for doing so?
As a forecasting tool, Python lacks packages compared to R. You can only do basic stuff (or you have to recode a lot of things)
Can you change how the outcome is measured? (continuous, binary, etc.)
Aren't surveys done by surveyors? ;)
What program are you using? Also, please specify the field you are working in.
That said, I would always incluse SD and mean in an analysis - it gives the reader a better idea how your data work. But this can , in my field, be done in a different table.
Your approach is very non standard and that is probably why you don't see the approach or an answer to your question on the internet. If it is all from one population and you want to estimate the population variance or standard deviation you should just use the sample estimate of in that you can find in any elementary statistics book. There is no relevance to the fact that your data are noninteger. The accuracy of the estiamte depends on the sample size and the population variance or standard deviation. There is no one sample size that applies to every problem. But it looks like you might be c
@PeteR: Maybe it is the basic set (W looks like an Omega.)
Your equation describes the full model we're able to test, and I think that's what we'd use for goal #2. I'd like to know whether for goal #1, there's any potential to do a more powerful but less specific contrast: "any treatment involving A" vs. "no treatment". Is it possible to pool data from our four cells containing A like that, and does that affect the n we'd want for the no-treatment cell?
+1 This answer is a good read due to its clarity, authority, and consistently helpful focus on responding to the questions.