tttjjj/elt_bench huggingface dataset:
If this were true, then all distributions could be continuously parameterized with four real numbers (the mean, variance, skewness, and kurtosis). How, then, could we contemplate even discrete distributions defined on six or more values, which require five or more real parameters?
"... the discrimination of two sets of real-numbers"?  What does that mean?
@h.l.m I have edited the post for (hopefully) some clarification. Please verify that I haven't accidentically changed the meaning.
Note that it's best to carry on conversations in comments, not by augmenting the list of answers.
Say more about your situation. Surely you don't mean they have the same mean $\\mu$? Are the $\\sigma$s known? Equal? How is it that you don't know the sample sizes? Not much you can do if you don't know how much data you have...
@NightMaster769 Sorry, I should have referred more directly to your post. I realized you were _multiplying_ in order to compound the returns correctly but I didn't state that explicitly. This is after all why you were rightly concerned about using the formulas for _adding_ random variables. Nevertheless, A just compounds the "2 standard deviation monthly bad return" over 12 months. It doesn't give you "2 standard deviation annual bad return". Regarding B, your approach seems sound but is complicated compared to log returns, and begs the question "What is the distribution of Y?".
When the p-value is small, chance is not a good explanation for the estimated relationship. That's it, period: no adjustment is needed. Perhaps you should be focusing on *effect size* or measures of *importance* in your application rather than p-value.  See [this related question](http://stats.stackexchange.com/questions/7815), too.
Welcome to the site, @sedlo. I think this *is* an answer, but at present it is more like a comment than an answer. Would you mind adding some information about that macro to make it more of an answer? Nb, there is information about answering questions on CV on our [help page](http://stats.stackexchange.com/help).
Not quite answering your question, but I think that the size of n is generally more important than the ratio of n to parameters. See this discussion of [minimum sample sizes and multiple regression](http://stats.stackexchange.com/questions/10079/rules-of-thumb-for-minimum-sample-size-for-multiple-regression)
I'm not familiar with this package, but it would seem that there is some problem with `trainDescr`. Can you give us the output for `trainDescr`?
The median and the mode are not moments. In terms of your question, _you get to decide_, depending  on what you are interested in. There is far more machinery available for inferences for the mean than for the median and in turn than for the mode, but (with all sorts of reservations too small to be written in the margin) bootstrapping makes confidence intervals easier even when there is not much machinery for confidence intervals. The central position of the mean (all puns should be considered intentional) is part history, part a consequence of the central limit theorem.
@whuber Given that Bernoulli data is common, and trinomial data (say on 0,1,2 or -1,0,1) - while clearly much less common - isn't exactly ultra-rare, real data either at or very close to the limiting cases I mentioned are easily possible. It's not some bizarre never-to-be-seen contrivance, we really could get nearly 0% or nearly 100% within 1 s.d. of the mean in a real data set. I do regard "near 0%" and "near 100%" as not "approximately 68%".
I generate P(L) and P(D) from two different models. I do not know which is best.
But I slightly disagree with your comments because I see notes where they use a Taylors expansion and notation of convergence in probability and distribution. Thus people are not thinking of the sample as a fixed realization of the random variables. It seems there's a justification for expanding the likelihood and deriving convergence results but I can't find an explanation of it.
Why not cluster the square roots or logs or reciprocals of the frequencies? (A reciprocal frequency is a wavelength, after all.) If you posit a quantitative model for the relationship between variance and frequency you can use that to derive an appropriate transformation of frequency that will accomplish what you want. What, then, do you know or assume about the relationship between cluster variance and frequency?
You can have deterministic stochastic trend present together, e.g. $Y_t = \\alpha + \\beta t + Y_{t-1} + \\epsilon_t$. In this case you need to difference *and* de-trend. So it makes sense to ask whether a series is stationary around a trend or not.
To be more specific - if you make sufficient assumptions, you can compute power. For example, in the Wilcoxon-Mann-Whitney, if you assume the distributions are identical apart from location-shift, *and* you make an assumption about the common distribution form, *and* you assume a specific value of the location shift, you can either compute or simulate the rejection rate. By doing it for many values of the location shift you can even get a power curve for that set of circumstances if you wish.
Cool result indeed!
I don't quite follow your question. The expected value of a [Poisson distribution](http://en.wikipedia.org/wiki/Poisson_distribution) is $\\lambda$, the mean. That doesn't have anything to do w/ R. If you have expected & observed values, you can do a chi-squared test manually in R; I show how to do that here: [What is wrong with this chi-squared calculation?](http://stats.stackexchange.com/a/85906/7290)
Welcome to the site. You can't *prove* this formula. It's a guideline. It can be wrong or right, and whether you need more or less classes than it suggests is (at least in part) a matter of opinion.
@Andy On the first bullet, have you seen this [Puhani (2008)](http://ftp.iza.org/dp3478.pdf) paper?
The link to multicolinearity is interesting, have you seen the word colinearity used in this context in papers?
Yeah, I think that's a great application of VAR - you want to know how the FF rate is correlated with those shares over time. I would certainly recommend R: function VAR() from package vars http://cran.r-project.org/web/packages/vars/vars.pdf
@JohnRos:  I added to my answer which I think makes mine complete. In general i feel that giving someone a specific reference that provides a complete answer is complete enough. I think that the poser of the question who is really interested in the answwer should go to the trouble of looking at the references and learn that way. We are all grown ups and we don't have to be spoon fed everything.
@whuber I was talking about the sampling distribution of $r$ not about the distribution of either $X_i$ or $Y_i$. If the sampling distribution of $r$ was normal/Gaussian, then in principle it could take any number between $-\\infty$ and $+\\infty$, which does not respect the known bounds for a correlation $[-1,1]$. So the normal/Gaussian sampling distribution for $r$ cannot be strictly true but only an approximation.
Fisher's transform helps because correlations are bounded by -1 and 1, and the transform transforms them into an unbounded variable that is close to normally distributed.
I'd say NA is more of a "placeholder"; NaN is for (IEEE) arithmetic purposes.
Maybe, but do you think researchers do not manipulate regressions, use repeated testing, etc. to squeeze under the established 5% level for example...
Thank you so much for such a detailed explanation. It is really helpful. As you mentioned, there is no good reason to remove those values, so I will leave them in.  
Also, thanks for mentioning issues that should be addressed. I will have to do some thinking on that part...
(1) "Lagged" in what sense? This does not appear to be any definite *series* of data. (2) A good choice of model depends on the nature of the data: one would use two quite different models depending on whether $X$ were considered to be measured with substantial error or not, for instance. As another example, one would also choose different models depending on how the truncation of $Y$ at $0$ and $4$ occurs: are these natural limits or artificial cutoffs? Please edit your question to provide more information about the data so that it can be answered appropriately.
A quick simulation might set you at ease concerning the correctness of your result.  In `R`, for example, you could compute `n <- 10^6; var(sqrt(n)*(sqrt(rnorm(n, 1, 1/sqrt(n)))-1))` in about 1/10 second.
It should be added that the information criteria should be calculated on VAR model in levels.
Yes. It occurred to me over the weekend that the metric I was looking for cannot, in any way, be predicted. It must be established through experiential data. There are too many variables. For example, the data set given to the estimators will greatly affect their ability to make predictions (i.e. are the tasks separated by minutes, days, weeks, or months?).
Probably a bad idea, but such questions cannot be answered in the abstract! You need to tell us your real situation, and your goal!
I've been playing with the generation of random points using $p(r)$ and $p(\\theta)$ for different values of $\\alpha$, $N = 1000$ and $\\zeta = 1$. These are the resulting point densities and hyperbolic distance distributions:

[link](http://imgur.com/pgSc3e3)
Autoregression does not imply seasonality.
Sure you can, but I don't consider this or one vs. all good practise.
Just to clarify - what do you mean by "known to be similar"?  (OT: Interesting username - is it novel or derived from something?)
An even cruder approximation is as follows. For $z>0$, the tangent to $\\phi(z)$, which has slope $-z\\phi(z)$, crosses the horizontal axis at $z+z^{-1}$. For $z>1$, the right triangle whose hypotenuse is the tangent and whose base extends from $z$ to $z+z^{-1}$ lies wholly in the region that we integrate to find $Q(z)$. This triangle has area $\\frac{\\phi(z)}{2z}$ giving that $$Q(z) > \\frac{1}{2}\\cdot\\frac{\\exp(-z^2/2)}{\\sqrt{2\\pi}}\\left (\\frac{1}{z}\\right )$$ which is half the upper bound. Crude, but better than the lower bound which has value $0$ at $z=1$
@Michelle Yes, there is, but the assumption is that while this data is being viewed, we will recognize that large positive spikes in quality will correlate with some sort of machine servicing.  Often times those things take thousands of dollars or days of work, so the management that will be viewing this data will be quite aware of when it happened.
Definitely *not* a duplicate of 91961, because the joint distribution of $(X_1, Y_n)$ gives little information about $V$.
Also, you seem to know more about EP than I do...would you mind looking at this question I asked :-)
http://stats.stackexchange.com/questions/86448/directed-bayesian-network-and-factor-graphs
Yes, you're right. Here I have counting data but with large values. I should use a continuous distribution.
Agreed, Huber. Thanks!
One way would be to fit a regression model using all covariates, and then predict y values for x using some fixed value for all the covariates (either the mean, or the most common value for factors). This can be done with `predict()`.
However, as you point out, in my implementation some of the weights ended up being fantastically large even for relatively straightforward distributions.  Consequently, I could not get the algorithm to provide a better estimate than vanilla Metropolis-Hastings.
I agree. Also, I'd add that if p is the probability of an event, then it's also common to denote 1-p as q. That might help simplify describing the function.
@Stefan Wager: Why must I stay in the world of penalized regression? Why can't I use glmnet as an intermediate step as I did here to simply find features that are significant? Both GLM and GLMNET are performing classification against the same variable (mortality) - shouldn't the "true" features that are significant be the same for both? (Hope that makes sense)
To give you some intuition, consider two scenarios.  (1) Both boxes have 9999 1's; one has one 0 and another has one 3.  (2) One box contains only 0's and the other contains only 3's.  In scenario (1), you will have to pick all 10,000 numbers from each box before you can identify the one with the higher average; in (2), most routine tests will fail but the answer will be obvious after a half dozen draws from each box.  Given such a range of possibilities, can you say *anything* more about what might be in your two boxes?
This is what I suspected. Thanks for confirming.
$c_0$ is the fixed for all functions in $\\mathcal{G}$. This question is not related to any statistical tests. The background will not be easily explained and I thought it is not really relevant. I just want to see whether it is feasible to find a function $G_0$ with given conditions. Thank you very much.
@CodeKingPlusPlus there is only *one* decision boundary. You are probably referring to the *margin*.
Hehe, thank you but this is actually the article I was reading that brought me into the field and I started looking for a more mathematically detailed source for it...
If they are *random* then they aren't going to be useful as predictors.
@Gene That can be used to demonstrate a special case: because the sampling distribution of the SD from a Normal population is $\\chi^2$, which asymptotically is Normal, a CLT-like result holds in this case. However, when sampling from a Bernoulli$(1/2)$ variate, the sampling distribution of the SD does *not* approach normality.
*...and the likelihood ratio test statistic trivially reduces to* ***(through a nonlinear transformation)*** *a ratio of chi-squares...* :) (Note also that *static* should be *statistic*) :)
That is technically true but "prior odds the drug was effective" is not a well-defined concept because it neglects consideration of "how effective".  It is more sensible to describe the state of prior knowledge with a continuous prior distribution that may have more mass at zero effect than in other places but that allows for all kinds of possibilities.  The analogy is that anyone considering a diagnostic test as + or - is oversimplifying a truly underlying continuous test outcome.
Thanks so much for the sanity check! We were really banging our heads over this one!
DanTheMan - the cars are all the same year
Sure. Will elaborate in a better way
thanks for the excellent response. I'm aware of the FWER problem, and intended on using the Holm-Bonferroni method for FWER outside of the automatic tests (easy to do, well-explained in Wikipedia), but here's the rub: using exactly the same data, my t-values are different (edited in the question). Any idea why that might be? Or how to interpret that? I'm guessing it's something to do with the way the software handles the tests, but it seems to me that the way I've done it should leave these 2 test results identical (wrt the t-value).
Thanks again!!
Right, the $u_i$ are exchangeable, so once you know one $e[u_i \\bar{u}]$, you know all of them and you've found the entire vector. Thank you for the great answer.
If I'm understanding this right, you are looking at mixtures of distributions on $\\{0, 1\\}^D$ (which are restricted to be independent within cluster); I wouldn't call this a "mixture of Bernoulli distributions" since a mixture of Bernoulli distributions is trivially just another Bernoulli distribution. "Mixture of multivariate Bernoulli distributions" is a more appropriate name.
By the way, if this is for some subject, or for the purpose of self study, please add the self-study tag.
thank you for your answer. The variability of residuals of m2-model decreases with fitted values. Interestingly, in overdispersed Poisson GLMM, scatter of those residuals against fitted values is OK. Please, see my last two comments above for more info.
Clarification request:  Are you wanting to take 22 observations at one observation per year and turn them into one observation per quarter using frequency-domain methods OR are you wanting to take 22 observations per year and summarize them into 4 observations per year?  (Non-statistician here) By VAR do you mean vector autoregression OR variances?  What is TY method?
@fRed Mines2006 ! great :) where are you now ?
@Henrik - But the assignment is to find the null hypothesis of interest, and later on we need to decide if we should reject the null hypothesis at a certain significance level, so that suggests that there should be a null hypothesis.
The estimate is of course subject to other variables. But we must purify it by introducing the so-called other factors in the model. However, sometimes these factors may be of categorical nature and cause more problems than give a valid solution.
@Dualinity: I have appended a bit to my answer above. Still thinking about how to answer the question more generally (and find the time to do that).
I don't quite understand the question. Do you have data on the productivity of the respondents? If so, you can use it. If not, how could you possibly do this?
$\\lambda$ here can be seen as the amount of regularisation or the weight of the prior term as one typically has in optimisation problems. So, the bayesian translation of an optimisation problem could be something like a Gaussian prior on the parameters of interest where the covariance is specified by $\\lambda\\Sigma$. It is this $\\lambda$ that I would like to estimate from the data rather than having a fixed ad-hoc value for it. Please let me know if I can clarify this further.
Have you tried _anything_ at all to find the answers? Search for "multivariate normal" on the Internet or on Wikipedia? Look in a book on probability or statistics?
I do not remember any reference of this at the moment but you can find a discussion on this [here](http://www.johndcook.com/CompendiumOfConjugatePriors.pdf) or in the reference of the creators of the concept [here](http://www.amazon.com/Applied-Statistical-Decision-Theory-Howard/dp/047138349X).
Welcome to the site, @Femi. I edited your answer to make the English smoother. I made some educated guesses regarding what you were trying to say. Please ensure it still says what you want.
I'd say any model that involves lots of sines/cosines or decaying exponentials would make for badly-behaved problems.
Cross-posted to [Computer Science](http://cs.stackexchange.com/q/28724/9550), [Theoretical Computer Science](http://cstheory.stackexchange.com/q/25366/17082), [Cross Validated](http://stats.stackexchange.com/q/109430/43031) and [Stack Overflow](http://stackoverflow.com/q/24967944/3703627)! Please do **not** do this. Cross-posting fragments answers and wastes people's time when the answer a question that already has a good answer somewhere else that they've not seen.
Bagging involves bootstrapping.
@Benjamin How's the problem coming along?  Was I able to help you understand how to do it?
I assumed ecalculate means Recalculate, please fix if not so.
The [coin](http://cran.r-project.org/web/packages/coin/index.html) package provides *exact* p-values. The question is whether it is better to use them (compare to, say, an approximate p-value), or what to do when it is not possible to compute them, IMO.
Do you have any more information about the report?  How many people were interviewed to come up with that 82% number?  Did some fo them come from the city of interest?
**Update:** the link seems to be working again.
Age and time can often be fairly correlated, so you might be running into a multicollinearity issue, which will cause the error error to go up. This happens because you have very little independent variation in log(time).
I have updated my question, and have proposed an answer. Having pondered this further, I can see why my approach is not ideal, e.g. I would be loosing information by assuming a distribution rather than using the full posterior mcmc chain; on the other hand, my model is relatively insensitive to the choice of priors so it is difficult for me to justify using the alternative approach.
The proof is easy if you use the alternate definition of variance, half the expected squared difference between two independent observations, and consider what you get when both observations are on the same side of *a* and on different sides of *a*.
That changes the question profoundly, Lea.  Please modify your question to show that's what you really intended to ask.
@Luke that should have been in your question. :)
Is there a difference if i give the argument "simplify = FALSE" to tapply()?
In reading your first question, I gather that you wish to use $x_1...x_n$ to build a prior distribution to be used in the analysis of further observations. Is that correct?
Ooopppss sorry @cartonn, you said "on paper" ! If reading html on a screen is really not a solution for you I'll delete my answer.
@Christos - Your intuition is correct - better to substitute the known means and variances in the ks test, and, if you do so, you don't need to bootstrap the ks test statistic either.
@gung Please! The CLT does not say that sampling distribution of the _mean_ converges to a normal distribution _even if_ all the samples are drawn from a normal distribution (except in the trivial sense that a constant can be thought of as a normal random variable with zero variance).
+1 I especially appreciate the points made in #2, which are frequently overlooked.
There are many problems to model return with just linear models. First, most of the time, return are not normally distributed. So you may need to take some transformations before fitting. More than that, they are auto-correlated which is not good in linear model as you proposed. Volatility would be another problem and this list goes on! Time series models are much more appropriate than linear models ...
PCA is difficult to use in that regard. Intepretation of PCA is hard because the PCs are linear combinations of all the variables. Thats one of the downsides.
So how do I know how significant is significant? If I put in pen.value=.20 I get 8 changepoints, but if I put in pen.value=.30, I only get 5 changepoints. How do I know if those 3 changepoints are signficant? How should I know which value to stick with?
very good. I'll sum the two and see how that works!
I think we could help you if you took a step back and explained in what sense this is a GLM.  In the statistics world (and it flows through into how R is coded) people think in terms of a response variable which will be a vector with n elements, and a number (let's say r) of explanatory variables each of which will be a vector or which can be thought of as an n x r matrix.
Yes you can do this, but it will most likely require you to derive the proper correction. The correction is derived in this paper http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2629611/ in the context of penalized finite mixture modeling, but an analogous argument would suffice in other penalized models.
Why don't you write down the log likelihood and differentiate that with respect to $\\mu$? Equating that to zero and solving will show you the correct solution.
@robin, that would be the hard part. Sometimes when thousands of variables influencing something, it's hard to make sense of the patterns that emerged.
Even though we live in a facebook age, where all people, even unknown, may be considered friends, the question was quite specific - 50% of ALL PEOPLE got 2nd interview and 75% of YOUR (optimistic) FRIENDS did not get the 2nd interview. Thus I think your answer is missing the most important part. Amazon wanted to see how friendly you are :)
...Cont... At this point I am searching for published papers, in any field/discipline, where a trend in variance has been modeled at level 1 in a multilevel model. So far I have not found any, but I am pretty sure that there should be some. If anyone can point me in the right direction, or provide links to published research I would be very grateful.
As in the answers above, I find rule 3 to be somewhat bewildering. And I am not certain the z-test is the reason behind it. I suspect it's based on some sort of argument that there is an unusual covariance between your sample and the rest of the populations. I.e. if we already draw the samples that are far from the rest of the distribution due to the skewness, they won't be in the rest of the population anymore.