tttjjj/elt_bench huggingface dataset:
@Glen_b, so I should write something like $e^{y-x}^2$ ? I took a different approach and got $=\\int_{0}^\\infty \\int_{-\\infty}^{\\infty} \\frac{1}{\\sqrt(2\\pi)} e^{\\frac{-(y-(x-s)^2}{2}}e^{(s-2t-1)x}e^{s^2} dydx$. I'm not sure if I can use that $\\int_{-\\infty}^{\\infty} 1/sqrt(2\\pi)*e^{-(y-(x-s))^2}$ is 1 as integrand is Y~N((x-t),1).
Can you be more specific regarding what you need help interpreting?
(ctd) ... or simply to avoid the assumption if you don't think it's reasonable to make it. You could always go to a permutation test, for example, unless sample sizes are especially small (whereupon the problem is lack of suitable significance levels to use).
Is Dixon test just used to detect a single outlier? Or you are suggesting to calculate number of outliers as burstiness (i.e. more outliers means more bursty?)
To give an answer, you should expand your question and explain how exactly (formula, software) you got this result.
This is one of the clearest and most helpful answers I have received on the Stack Exchange network and it really straightens up my confusion about DF tests. Thank you.
interesting, I wasn't considering a clustering test for my data, I will give it a go. Many thanks.
I think you are on the right track, and including location as the random effect makes perfect sense.
maybe this can help you http://stats.stackexchange.com/questions/25520/how-to-do-post-hoc-tests-with-logistic-regression/25534#25534
Is this: http://www.ats.ucla.edu/stat/mult_pkg/faq/general/psuedo_rsquareds.htm helpful, or do you mean something else?
@whuber Yes, indeed. The $\\approx$ sign can work miracles!
Also, how do you define "balanced"?
Thanks, but my question is about how I get the w and b values. Also, about the decision value, I'm asking how e1071 decides which given label is positive and negative for the training data when you pass it factors.
@gung yes, that's the idea.
A mixture of two normals is not the sum of two normal random variables. It's a random variable whose PDF and CDF are weighted sums of the individuals PDFs and CDFs.
@Macro The thinking is somewhat indirect and not intended to be rigorous but I believe it is natural: quadratic loss implies minimizing variance which (via Chebyshev) leads to convergence in probability. Whence, a heuristic for finding a counterexample should focus on losses which are so far from quadratic that such manipulations are unsuccessful.
+1, a (slightly more) complete answer is that the whiskers are *up to* 1.5*IQR, if you don't have any more data within that interval, the whisker stops short. In addition, any additional data beyond that point are plotted individually as potential outliers.
$p_\\theta$ is a probability (or density) for each $\\theta$, that does not imply that the likelihood is a density function as a function of $\\theta$.
There's a nice blog post on ridge regression [here](http://tamino.wordpress.com/2011/02/12/ridge-regression/).
If you wanted to find the growth rate of your web site, you do not want an average.  You will want to do a linear regression on your points for each individual website, the slope of which will be the average increase in visitors/month.  From those slope coefficients you can get an idea of how fast your traffic is growing. This will also allow you to tentatively predict future growth and factor in other things that may change your readership.
@Mac /cc @richiemorrisroe In my view, apart from the homoscedasticity assumption (which is somewhat circumvented with Welch's t test), the problem is mostly with asymmetric response distributions (ceiling or floor effect), which often arise when using Likert items.
Luca, there are two situations. As you have written the equation, the product of densities implies independence of the three components - and yes, that's multivariate normal. But if you just mean "I have three variables that are marginally normal, is their joint distribution normal" then the answer is no. Can you clarify that the product was intended to convey independence?
http://www.r-project.org/doc/R-FDA.pdf might be useful.
Your conclusion that 'sums of random variables can be understood as conditioned probabilities' is ... pretty much a misunderstanding of what was going on. A conclusion based on adding a constant to a random variable carries over to adding two random variables *if one of them is held constant*. That 'holding one constant' is what introduces conditional distributions. If you *don't* hold it constant, the conclusion based on adding a constant *simply doesn't apply*. It's not complicated or mysterious - it's a result of adding constants and attempting to generalize to adding random variables.
@Dimitriy I find it very strange to use an arbitrary mix of colours in the same situation! I don't imply or infer anything by or from the exact colours, however quantified. But the point is only that a graded scale is well matched by a graded sequence of colours. There is some arbitrariness in the colouring of heat maps too, and indeed in many kinds of thematic cartography.
1) I'm not sure how it is possible that $p(X_2|X_1)$ is intractable but $p(X_2|f(X_1))$ is tractable.  2) Can you easily generate variates from $p(X_2|f(X_1))$ and $p(X_1|f(X_2))$?
If you use R, I can show an example
Not exactly what you want, but see bottom of p18 here for some more details:
http://lme4.r-forge.r-project.org/book/Ch1.pdf
2Irish stat stanks for excellect response. I have  read some were that we would combine oth stochastic and deterministic trends that is yt = y(t-1)+a+bt=ct ? would that be helpful
I like stripplots too, but the question is explicitly about what to do with heavy-tailed distributions.
Why do you think an ordinary interaction would not do the job? It would certainly distinguish the cases in your "suspicion" (end of 2nd paragraph).
Have you read Shmueli's article?  From reading the abstract, it seems that his article is aimed at precisely answering these questions.  Are there particular arguments he makes that you have questions about?
@Benjamin I might head that way, if the OP's responses point that direction; at the moment I can't tell.
Some negative coeff. estimates? Maybe some of the X's are highly correlated and a multicollinearity problem shows up. Try PCA or exclude some of the highly correlated variables - if any.
If a well-considered significance test is part of your strategy, and if you are in doubt as to whether one type of test is more applicable than the other, best practice is to report results from both so as not to give the appearance of cherry-picking.  As to the "well considered" part, I recommend http://stats.stackexchange.com/questions/10510/what-are-good-references-containing-arguments-against-null-hypothesis-significan.
I would try first a Quadratic Discriminant Analysis...
Perhaps because of the central limit theorem, Gaussian distributions do fit many, though by no means all, measurements of physical phenomena?  With sub-populations, one may get _mixture_ Gaussian distributions.
Have you looked into Fisher's method and variants e.g. [Zaykin et al. 2002](ftp://statgen.ncsu.edu/pub/zaykin/tpm/tpmfinal.pdf)?
Is the OP asking for a way to find the seasonality of a series? In my opinion, that is a quite complex question, and none of the above answers (I think) address that. Please correct me if I am worng, either about the question or the answer.
That formula is the Law of Total Probability (http://en.wikipedia.org/wiki/Law_of_total_probability), it applies in many cases besides EM algorithm.
(+1) Thx for the link. I like the PGP framework too.
Have you tried different values of "dodge"? For example something like that: `position = position_dodge(width = 0.90)`. See also [this post](http://stackoverflow.com/a/5256758/2378656).
Welcome to the site! For the future -- please rather ask separate questions for separate problems.
A similar question came up recently. See [this comment](http://stats.stackexchange.com/questions/39062/does-variance-predict-skewness#comment77061_39062) and the related question.
Thanks to all the replies. Much appreciated!
What you're doing wrong is using a calculator to solve your problem. The calculator is a tool. If you don't know how to do this question by hand with a smaller data set of say 5 points, you should start from there.
Ah. Thank-you very much. :)
Please post a complete reference, as well as a reference dataset.
As long as the model assumptions are valid (you **must** check this), a low $R^2$ does not affect the validity of interpreting regression coefficients.
Looks like you've got it backwards... the DVs go to the left of the "~" in your equation, and the predictors to the right. It should look more something like: manova(cbind(score_children,score_adults) ~ brad_hrs + angelina_hrs + cgi_budget + seat_cost, data=movie_eg)
Are you talking about something like an equivalence test, or are you looking more for the information provided by confidence intervals?
+1 Welcome to the site, @stachyra. This is a really nice answer: clean, authoritative, concise, & easy to understand. I hope we'll see more like this in the future.
This project seems great, especially if we predict the vpvote count very soon for the user. A further work can be to tell the user what is holding his/her question back and what improvement can make his question popular. Anyway, as always the choice of feature is really an important and challenging task and the performance of such predictions highly depends on this selection. **TL;DR I like your idea**
The null hypothesis is not always the same. The null hypothesis is just an alternative "boring" hypothesis that you compare to your "interesting" hypothesis, to see whether the data supports one over the other. Actually, "no difference" is actually a bad null hypothesis, since you know a-priori it's false. Better is "the difference is below some threshold of me caring".
@ musa .. in genomics there is a specific modeling strategy for "time course experiments". You can use it to get to see if the modeling strategy fits your data. Genes correspond to pages and individuals correspond to patients. I also listed 2 references in my previous comment.
As far as I know, Excel 2007 only allows you to have 1 million or so rows (A quick google search turns up a maximum of 1,048,576 rows in Excel 2007). I would say your best bet is to use a database program, like SQL or Access. There are simple database queries that will give you what you want (subtotals, sd's, etc). Also, I haven't really worked with it myself, but I am sure you can access datasets from Access using Excel. Honestly, though, I would just use R, and for what you're trying to calculate, it does not take long to learn some simple functions :)
I guess that is true, but to just clarify, in the end I threshold the score to obtain a class prediction.
@Jyotirmoy Bhattacharya, I've updated my answer with more details, hope it explains my intuition which motivated the initial answer.
I updated the answer, introduced a more exact notation: $g(x)$ for the blue plots, $\\lambda$ for the penalization paramater. Does that explain your concern better now?
@Henrik Your response seems to assume, implicitly, that the Wilcoxon test is *not* a permutation test.  However, *both* Wilcoxon tests (the signed rank and rank sum tests) *are* permutation tests.  What other permutation tests are you specifically recommending here?
I would be cautious about the bootstrap if any of the estimated $\\pi_i$, $i \\ge 1$, are zero, because that solution lies along a boundary of the parameter space and the regularity conditions used to justify bootstrapping might not hold. But that's precisely the case where this solution is needed: after all, if all the $\\pi_i$ are nonzero, then one could just use OLS to solve this problem.
@Cardinal Thanks. I will be re-reading this and looking for such things, as well as for the inevitable typos. You are too kind to point out other things which you surely noticed, such as some gaps in the exposition. The biggest is that I did not actually show that these ellipses are at 45 degree angles (equivalently, inscribed in the unit square); I simply assumed that. I am still looking for a simple demonstration. The other is that one might worry the skew transformation could produce a *different* distribution than the original stretch-squeeze-rotate-shift--but it's easy to show it doesn't.
@psych - without testing it, something like `+ geom_point() + ylim(0,XXX) ` should do the trick. You may want to set the `alpha` parameter in `geom_point()` to get some transparency so you can see the underlying box plot. I also recommend reading through http://had.co.nz/ggplot2/ as there are quite a few examples.
Ah yes, re-reading it now makes sense, especially when I check the comment to explain it above... clearly wasn't thinking last night :)
+1.  (But I am unable to demonstrate $\\frac{1}{n} + \\frac{(x_i-\\bar{x})^2}{\\sum(x_i-\\bar{x})^2}\\lt 1$ unless I assume $n \\ge 3$ and not all $x_i$ are the same; even then, I can only show the relationship is $\\le$, not $\\lt$. :-)  Why do you write that one cannot estimate $\\sigma$ from the data?  I thought this was routinely done by taking the root of the mean squared residual.  I must not understand your claim correctly.
+1, a quick question, how does $Var(\\hat\\beta)$ come?
I want to acknowledge and express my agreement with the caveats posted by @mpiktas and @dimitrij celov.  Analyses of prices can be--and in many cases *should* be--as complex as the economic systems of which they are a part.  However, due to the intended application (a hobby) and the plainly signaled limitations in the OP's capabilities for statistical modeling, we should place great value on simplicity, ease of use, and interpretability.  Obviously someone not yet conversant with least squares is not going to jump right in and start creating full-blown econometric models.
+1 - I would also suggest utilizing jitter for the dots (in addition to transparency). In this example I would replace `geom_point(size=2, alpha=0.4)` with `geom_jitter(size=2, alpha=0.4, position = position_jitter(height = .02))`.
WHen you write the equation for $E(Y|X=x)$ shouldn't the multiplier be a correlation coefficient? i.e. coveriance divided by the two means?
BIC is not a Bayesian tool, as it removes the impact of the prior. As a Bayesian, I am comfortable with Bayes factors, but not with AIC, BIC, nor DIC!
Sublime is a fantastic text editor.
I think using the arcsine transformation would be going backwards since the GLM with logit link is more accurate and there's no need to stabilize variances. I can set the weights in my model to anything I like, what I'm looking for is an explanation of what weighting would be the optimal one, and why.
$X_i$ isn't a probability. $X_i$ is a random variable that can (in the first case) be 0 or 1, and $x_i$ is the observed value (a 0 or a 1). But yes, as you say, the first one corresponds to m=1.
One thing is that 'longitudinal' will mean something different round here than lat/long. ('longitudinal' meaning a separate time series for each store). Suggest you refine the question you're looking to answer (as without more data it looks as though a simple approach like counting up sales at stores would be best you do).
Are these two separate questions? In the first question, are the sets of 24, 8 and 4 overlapping? Also, is this homework?
lower variance in the predictor leads to larger standard errors - they are exactly inversely proportional in a least squares model, but are only generally inversely related in GLMs like a logistic model. In the extreme case where you have no variance in the predictor, the effect is not estimable. Is this what you were asking about?
@whuber: Thank for the information. I've updated my question with an example of my data. Should the question be phrased differently? Please feel free to edit it if so. My statistical background is weak, and my statistical lingo is even weaker.
Actually, there is multicollinearity between the two variables. I'm not sure how I would be able to construct a latent variable crossing Valence and Self-relevance especially when I did a correlation, it was significant at r=.755. Thanks for any input!
Thank you Glen, I appreciate your help.
But what about the CI? How to determine its skewness based on the parameter distribution?
Are you certain of this because you know something beyond the data (i.e. Annie is the guild leader's girlfriend), or do you suspect this because it seems like some people win too often? As whuber points out in his simulation, in a fair raffle, one person could win every 3 weeks for an entire year, 1 person once a month, and 7 people every five weeks or so, then one "unlucky" person never winning even though they were eligible 52 weeks in a row. That can seem unfair. Your example of Annie has her winning 14 times in roughly a year.
thanks for the response.  I just want to predict the stock volume one day in advance.  I have six months data, however, the stock volume time series does't have weekend data.  I'll look into the  count process (poisson process) and threshold models.
I'll review this soon. Thanks. Blasted iPhone cuts off the margins of SO web pages.
I wouldn't be so quick to dismiss this approach as "clearly ... wrong," especially if $np$ and $n(1-p)$ will both be large. It has a theoretical foundation, is easy to compute and interpret, may provide a meaningful envelope around the observed percentage (especially if the square root is multiplied by a suitable constant), and could function well to enhance visual or exploratory examination of the data. One avenue to pursue would be [quality-control procedures for count data.](http://www.jstor.org/discover/10.2307/1268761?uid=3739864&uid=2129&uid=2&uid=70&uid=4&uid=3739256&sid=21104359283303)
That's a good point, but the result might not be so bad.  What one hopes is that the estimate of M*M is close enough to the true value that the perturbation of eigenvalues is reasonably small.  Thus, by projecting to the eigenspace corresponding to the largest eigenvalues, you achieve only a slight perturbation of the correct solution, still achieving the sought-after dimension reduction.  Perhaps the biggest problem may be algorithmic: since you can no longer assume semidefiniteness, you might need to use a more general-purpose algorithm to find the eigensystem.
Ah ok I think I understand. So basically we know that this function (like all distribution functions) is right continuous, but we are able to ascertain that $Pr(X=0)=0$ from the given information.
Even if the standard deviation of the observations decreases, the standard error of the estimator might increase due to the smaller sampe size. E.g. if you drop the smallest value of a right skewed distribution and consider a c.i. for the mean..
Good answer but I disagree with point 5: I can think of many different (free) software for Bayesian analysis: WinBUGS, OpenBUGS, JAGS, Stan, PyMC ... and I'm sure there are more. What I would say is that all of these software have a steep learning curve and require a decent amount of programming and statistical knowledge.
@Maxim.K Because variance can't be less than zero. Even if you were to perform a frequentist test, such as a chi-square test, you wouldn't be comparing variance relative to zero. You compare variance relative to a particular value.
By [definition](http://en.wikipedia.org/wiki/Independence_%28probability_theory%29#Two_events) of independent events. You can see that this is not satisfied. Intuitively, if $A$ occurs, then $B$ cannot occur. This reflects the dependency.
@csgillespie: sorry; it's Matlab code, as Elpezmuerto requested in his comment to the other answer.
Doing so implicitly sets a, the intercept, to be 0. For y=a+bx, that forces the line to go through the origin and will also change b. It's the same in higher dimensions. It's almost always a bad idea to omit the intercept.
What's a "hit"?  Are you making an analogy to throwing darts at a dartboard?  If so, you need additional information (or equivalent strong assumptions) to get an answer: you need to know the statistical distribution of "hits" around a target's center.
There is a lot of good info here, but it is very strongly worded & mixed w/ some things that don't make much sense to me. Eg, re #3, my understanding of the relationship b/t the z-test & the t-test is that they're essentially the same, but z is used when the SD is known a-priori & t is used when the SD is estimated from the data. I don't see how if the CLT guarantees the normality of the sampling dists this licenses the z-test while leaving the t-test invalid. I also don't think $\\neq$ SD's invalidates t if the CLT covers you, so long as an adjustment (eg, Welch–Satterthwaite) is used.
Interesting. Thank you for the detailed answer.
Exactly. Apologies if I'm being naiive!
Your uneasiness with the language of confidence intervals, but otherwise rigorous presentation, suggests you may be in a good position to give us a little more information to help you.  Ordinarily you can specify one of two properties of a CI: either its coverage or its length.  The other will be determined by the data.  It's unusual to stipulate the length, though: are you sure this is what you want?  Also, it is unclear whether you need CIs for the Q's separately or simultaneously.  Perhaps you could indicate what you intend to use these CI's for?
@rolando2 A numerical scale [0-10] will do.
The critical values are strictly monotonous in $\\alpha^*$ so "good" initial values don't change the result, they would spare you only some iterations. Maybe 0.5 is faster than what I suggested, 1.
If I'm understanding your response correctly then, at this stage, if  I do not have any significant interaction effects. I would calculate the ANOVA and report it (even when there are sig. main effects). The same applies if everything was not sig. (main effects + interactions)?
What does it mean for one set of clustering results to be "more reasonable" than another?
Please check your statements.  In the first line what is $V$? By definition a Student T has zero mean and its SD is determined by $\\nu$, so where do $\\mu$ and $\\lambda$ come from?  What, in the first two lines, is the relationship between $X$ and $Z$?  Usually Student's T is used when mean and SD are being estimated from data, but your "standardization" presupposes you *know* $\\mu$ and "$\\text{std}(X)$".  How?  And why is there even a question here, when by definition the "log likelihood ... for $Z$" is the logarithm of the density, which you have explicitly written down?
@Erogol: good question i will try to answer. Do you have a link to that dataset?
"One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is."  I'm sure you could derive analogues for Fisher information in terms of other transformations of the likelihood, but then you wouldn't get as neat of an expression for the Cramer-Rao lower bound.
"I wonder if *any* estimator would work at all" -I agree with this sentiment. This is why I said in an earlier comment that it is plausible the question has no answer. After all, to obtain consistency for the first moment, we need to say something about the tails of the distribution, but without making a parametric assumption, it is difficult to see how to do this without using the observations from the tails. I was hoping that I was overlooking something clever, but perhaps what I am trying to do is simply impossible without a parametric assumption.
@Glen_b For example http://ihome.ust.hk/~yfong/iterate.pdf Quite some effort seems to have gone into developing efficient algorithms for it. So clearly, it must be useful for something. From the literature I just can't see what without really diving into it.
I don't know the answer to this, but I do know you shouldn't use stepwise to do variable selection. See [my paper](http://www.nesug.org/proceedings/nesug07/sa/sa07.pdf)
The simple answer is: if you square a normal, you get a $\\chi^2$ and if you take the ratio of two $\\chi^2$'s, you get a variable with an $F$ distribution. But, your quotations around "real" in your question make me suspect you're asking something else. Can you provide some more details? It may help you know the context in which this question arose.