tttjjj/elt_bench huggingface dataset:
Good question. I found a paper that suggests two multivariate normal approximations. They mention in the introduction that the exact approach would be to simulate using conditional probabilities, but they point out that this is sensitive to the ordering of your variables and is computationally expensive. Source: http://link.springer.com/article/10.1007%2Fs10928-006-9033-1
Yes, @naught, that is a reasonable way to look at it.  But "curve" is not the same as "time series."  For instance, linear regression can be (and often is) viewed as fitting curves to data, but is separate from time series analysis, which emphasizes the structure of correlations among deviations between the data and the idealized curve.
The reason the first plot looks like a bad fit is that you plot the fitted values against the wrong time index. Use lines(fitted(fit), col='red') in every case and the results will be corrected.
In particular, your correlation matrix is already not positive definite. With r12=0.9 and r13=-0.9, r23=0.1 is unrealistic. r23 should be -.62 or more negative.
Fabulous! Exactly what I was looking for.
Efficiency in stats has to do w/ using the info available in the optimal way. If some data cluster closely to their mean & some data disperse far from their mean, the locations of the former provide more info about the location of their mean than the locations of the latter do about the other mean. If you gave all data equal weight, you would not be using the info optimally. The true amount of information in each point relative to its mean scales as 1/variance(point); w/ constant var, you treat them equally, but w/ non-constant they should be weighted w/ more weight going w/ greater precision.
I've updated my answer.
is `sd.y` the appropriate estimate of $\\sigma$ to use in the prior?
Thank you FAS! Can you recommend a resource explaining assumptions/procedure for two-sample t-tests?
Don't know about tetrachoric correlation but the only reason I sometimes square $\\rho$ is so I can handle the Kruskal-Wallis situation of multiple groups.  $\\rho^{2} = R^{2}$ where $R^2$ is the ordinary multiple $R^2$ for all the dummy variables against $rank(Y)$.  If all predictors had only one degree of freedom and I keep assuming monotonicity I'd use $\\rho$.
I don't think there is a special rule for using the same symbol to represent different things. The same could be said about $X$, it is used to represent either a random variable or a Banach space; $\\sigma$ is used for representing the standard deviation and a scale parameter ... Some symbols become popular for representing certain concepts. $\\rho$ is typically used for different types of correlation. In this case, both concepts are well-defined/demarcated but coincidentally they were denoted with the same symbol.
Excellent.  Works perfectly.  Thanks.
If that post doesn't really answer your question, @Mark, it would be helpful to use it to clarify how your Q is distinct from it. I think the answer below is somewhat ambiguous, & I worry that you may take away the wrong lesson. It would be best if you could read that thread thoroughly, & then edit your Q to state what you now understand & what you still need to know. Then you can get the best information.
thank you,thank you,thank you very much.
That is either a fairly arrogant comment, or just a fairly naive comment given how much or little you know about details of the R / S languages.
Well, this is proof that I need SE Q/A.  From what I can tell, this looks correct.  I need to run it past my professor before I really grok the answer, however.  As to your latter comment, I'm guessing that it hits on the problem I had with Spearman's Rho.
The correct calculation is described in the question at http://stats.stackexchange.com/questions/89512.  Other closely related questions include http://stats.stackexchange.com/questions/24380 and http://stats.stackexchange.com/questions/62092. Your question needs to be refined a little: to what does "one standard deviation" refer?  After all, these are *bivariate* data.
Please read the comment thread to [@mpiktas' second answer](http://stats.stackexchange.com/a/37655/) concerning why your formulation of the likelihood is not quite correct.
Welcome to the site. The power may indeed be an issue. Can you post your results, so that we could be more specific?
Can you expound this interesting idea in your question? Or at least leave references?
Thanks, unfortunately it seems that the editor doesn't support the upload in PDF format. They were vectorial.
Unfortunately I don't think the paper you cite addresses this problem.  The paper deals with when you have a set of non stochastic variables A, and finding the smallest element in A through sampling.  In the context of this problem, each element in A would be a random variable, and therein lies the kicker.  You have to find an unbiased estimator of the mean of the smallest random variable in A.
It has been claimed that the t-test itself is non-parametric because it does *not* require normality of the data, but only relies on approximate normality of the sampling distributions of the two means.  (The t-test makes no assumption about a t-distribution in the underlying data, by the way.)  When you refer to an "alternative," do you mean a test for comparing *means* of two populations or would any test for comparing their *locations* suffice? There are plenty of the latter discussed in the [tag:nonparametric] tag: have you followed those links in search of answers yet?
Could you modify your question so it doesn't contain the expression 'Gaussian with two modes', please?
Frankly, I'm not sure what the best option is for your problem. It's an interesting one!
The point of the question seems to be that the x-axis summarizes a *periodic* or *cyclic* variable.
Lemme do some edits...It now should make more sense
+1 As far as I understand it, Fisher was trying to plant the Bayesian credible interval in his frequentist garden. (He famously despised the Bayesian approach and I believe even coined the term "Bayesian" as an expression of his contempt.)
I've added the latex, but your question does not make sense. If $A$ is integer, how come it is also a random variable distributed with binomial distribution? And $p_1$ with $p_2$ are the parameters, i.e. not random variables, so their distribution is trivial. Perhaps you had in mind the estimates of these parameters?
@mike If you want to get the variance of the combined sample, you need to account for the different location parameters (or their estimates) as well.
The $t$-test for a mean $\\mu$ is the likelihood ratio test! Check out [this link](https://onlinecourses.science.psu.edu/stat414/book/export/html/247) section "The T-Test For One Mean" or Example on page 71 of [this link](http://www.stats.ox.ac.uk/~dlunn/b8_02/b8pdf_8.pdf). In a nutshell, you can get the critical value of the $t$-test using LRT.
A good course of action may be to leave this question open on stats for a while, due to its specialized nature, and then (if it gets no answers or no great accepted answer) migrate it to SO for further attention.  Migration can be accomplished by community vote and/or by flagging the question for moderator attention.
Yes, sometimes, we don't just train (a Neural Network for example) on just one set of weight. One common thing to do, if we have time, is to create several different start weight configurations and train the network on it.
Because some times, it can happen that one configuration will end up in a local optima or the weight were unfortunately not really randomize.
That's a good point. I will update the answer to reflect it.
AFAIK, only Bonferroni, Sidak and possibly Games-Howell post-hoc devices are safe to use with repeated-measures. Other traditional post-hoc tests won't do because they are ment for independent error terms. The test of spericity p-value should be 0.5 (not 0.05) or higher, for them, and this is seldom.
Thanks! I'm wondering if you're aware of an existing r implementation of this?
@BenBolker: Has this example an R version so that I can use glmmADMB package?
Yes, this is correct. If $Y_i=p_i+e_i$ with $P(Y_i=1)=1-P(Y_i=0)=p_i$, then $e_i=1-p_i$ with probability $p_i$ or $e_i=-p_i$ with probability $1-p_i$. Hence $e_i$ has a distribution with mean $0$ and variance equal to $p_i(1-p_i)$.
How is the sample data non-random?
Do you have a deterministic, but noisy process that is yielding your measurements y_i?  Or do you have a process with some hidden variable that leads to different measurements y_i?  The structure of your problem will determine what you should do to model the measurements.
@Val you are confused what linear modeling is, a linear model is any model which allows you to write the expected value of the outcome as an explicit function of one or more terms of input variable, including polynomial transformations or, as I suggested, smoothing splines. Pick up "Elements of Statistical Learning" for distinctions on what constitutes and does not constitute a linear model.
What is the question?  What do you want to determine from these data?  Are the X1 and X2 predetermined or are they observations?  What do they mean?  What is a "run"?  Does it correspond one-to-one with a single "ID"?  What does the reference to "X1+X2" mean?
An alternative to the suggestion of @Glen_b is to first center X, then create a squared version of the centered X variable. Include them both and multicollinearity will be substantially reduced. As an added bonus, model interpretation is often nicer with centered IVs.
You can't do this in SPSS with repeated measures. You need to do a multilevel model to be able to do this. You can do that in SPSS mixed models, which will allow you to have two columns and specify an ID variable, which identifies individuals.
ANOVA does not require normally distributed *data* it requires normally distributed *residuals*.
Are you aware of any real-live example/industrial applications in which a Kalman filter performs better than a simple moving average or exp smoothing in forecasting a time series provided there is no clear underlying model (so exclude models arising from the laws of physics)? In most papers the performance looks very similar (and academic papers have a positive performance bias for new, original, complex models). In most of the cases there is no knowledge of a proper linear state system model and covariances, etc, required to specify a Klaman filter...
Could you explain how these 4 predictors were coded?
Good point on filtering conditions: I checked my implementation and it turns out that I do filter the least value on lift only. I believe that filtering that way is driven by performance considerations but I will make this change to see how much worse it really is.
I think that 'biostatistics' is not a very good tag for this, but I'm not sure what is. I added tags related to issues I discuss in my answer, but I don't know that they are ideal either.
@Imran as you have seen, you will still get weight changes if you fail to reset conversion and views back to 0 after each weight change, and it may even converge towards the optimal weight given enough time, but it will not converge as quickly or directly.
Add more to your question, edit it until it better reflects your problem, what you've tried, and things like your guess about the reference group.
There are two square roots of $-1$. Mathematicians use one of them and call it $i$; engineers use the other and call it $j$. Since $j = -i$, the characteristic function $\\int e^{itx} f(x) dx$ used by mathematicians is exactly the same as the Fourier transform $\\int e^{-jtx} f(x) dx$ used by engineers
+1 for FRED, it truly is one of the most useful sources of economic data (especially financial/macroeconomic)
A [reproducible example](http://stackoverflow.com/q/5963269/420055) would help.
For more information (in addition to the duplicate thread), please review the four dozen (or so) [related questions](http://stats.stackexchange.com/search?q=%2Binteraction+%2Blogistic+%2Bregression).
the extreme tails are mucking up the density. The distribution is rather like a Cauchy. (Out of curiosity, why not use `runif`? It seems more idiomatic and seems also to be faster)
@Procrastinator Due to the independence of $X$ and $Y$, $p(x,y) = p(x)p(y)$. So, $\\mathbb{E}[X^2Y]=\\int\\int x^2yp(x,y)dxdy=\\int x^2p(x)dx\\int y^2p(y)dy=\\mathbb{E}[X^2]\\mathbb{E}[Y]=0$. Is this reasoning correct ?
another useful function is `any`. For instance if you want to check that at least one element of a vector is `= 10` you could write (`any(v==10)`).
If you are a bachelor student, simply taking the mean seems logical.
Can SPSS do this as well?
I would classify the points by the multivariate Gaussian they are most likely to belong to (which I guess is the one with highest density for that point?)
Regarding post-hoc tests, the [Mann-Whitney U-test](http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test) is a natural choice following a Kruskal-Wallis test. In R, this can be conducted with [?wilcox.test](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html). You may (or may not) want to use an alpha adjustment (eg, Bonferroni correction) to address the multiple comparisons issue.
Are you sure this is exactly the code you input? The variable is called `x` in the model and `x-vector` in the newdata. Also, there is typo in "reponse".
Physicists have created very powerful methods using their own notation, so you shouldn't hold back. I presume there's an implied summation over $j$ in the definition of $Y_i$, for otherwise there are no difficulties.  But maybe it's worth pointing out that since $Y_i$ is a bilinear form in $B_{i{*}}$ and $X$, [polarization](http://stats.stackexchange.com/a/51717/919) will re-express it as a linear combination of squares.  After that the calculations can get hairy... .  It's also a little easier when the $X_k$ and $B_{ij}$ are uncorrelated.
Even if donations are not thought of as counts, their values are positive and likely to be skewed, so something like a Poisson is still likely to be appropriate (more generally, a generalized linear model with log link). Presumably you have no zeros in your data, but if you did, the same point applies.
In general I agree with your idea. However, I think whether one follows my approach or yours depends on the school/field one comes from. If you want to get the best estimate of protein to estimate the regression coefficient for your data, you take my approach (i.e., a classical behavioral science approach). If you want to get the most accurate estimate to predict other data than the one at hand, go with your approach (i.e., machine learning approach). In this case further methods such as bootstrapping could be also beneficial to even get better estimates.
What is the question exactly?

If you are looking to understand how kernel functions help, you can check this answer - http://stats.stackexchange.com/questions/43779/non-linear-svm-classification-with-rbf-kernel/43806#43806
Can you please explain this cartoon?
Thanks. Although either way I am not able to report, based on the Wilcoxon/Mann-Whitney, that there is a difference between the means of the two groups.
Looks promising (if long)!
I'll second the vote for the SAS documentation, it really is pretty good (and voluminous).
My prediction doesn't work without normalisation. However, I'm understand that I need use normalized `Y` for denormalize `predicted.Y`
It would be nice to see the same plot on a linear scale, for comparison
+1 Thank you very much. That does the trick. Sorry about not specifying the `fitdistr` package. I'm quite new to R so I will keep that in mind for the next time. I have accepted this as the answer but as a final note, can you please tell me if there is a numerical measure that tells me how well the data could be fitted using the distribution?
The $\\beta_i$ are constants; I presume you mean their estimates. This is standard calculation; e.g. [Wikipedia](http://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties), [CrossValidated](http://stats.stackexchange.com/a/44841/805). (I simply searched in obvious ways in both places to locate those links.)
I doubt anyone would get beyond step 0: please see my comments beneath the question itself.  Note, too, that the OP asks about an *amount of money* (even though it is perversely termed a "rate"), not a count of players.
On a second thought, I may just use R to take care of the clustering (to avoid reinventing the wheel.) I posted a new question [http://gis.stackexchange.com/questions/17638/how-to-cluster-spatial-data-in-r] which is more about how the spatial cluster analysis can be done in R. Do you know about useful packages in R for spatial cluster analysis? tnx.
@peuhp It can be seen as the prior in [Gelman (2006)](http://www.ime.unicamp.br/~veronica/ME705/T2.pdf) with a Dirac delta prior $\\delta_3(\\nu)$ on the degrees of freedom. The interpretation is sort of clear, I guess (and I hope). Then, you shouldn't need a reference written by a sacred cow to use it and justify it.
Sounds like you might have a massive dataset and will be in better shape.  But a probability model can accomplish that - you can trigger when the predicted probability exceeds a certain value.  The value can be set by subject matter considerations or by a lift curve.  Dichotomizing on the front end will cause a large loss of predictive discrimination.
I updated question.
If you try to do a binary response logit in this case won't you get a huge number of duplicates? Would that be  a problem?
@JohnRos: If all the true positives in a sample comes from an alternative distribution $F$, TPR can be seen as an estimate of $P_{X\\sim F}(\\text{ reject null })$. But for FDR, because its denominator is the reported positives which is not population dependent, I don't think it can be estimate of any population quantity.
For other readers of this Q: http://en.wikipedia.org/wiki/Voigt_profile
Thanks for the helpful guidance.  Since the CV = ${}^\\sigma/_\\mu$, I can obtain the $\\sigma^2$ by ($\\mu$*CV)^2.  Once I know the variance of each estimate, I can simply add them up for the variance of the pooled value (they're independent of one another, at least I'm going to assume that).  Thanks again Glen _b, I should've known that - nothing like a good humbling to bring you back on track.
Thanks for your comment, now I ran multiple imputation to determine the missing below-LOD-values.
+1. Welcome to the site, I'm a big fan of your reviews on Amazon. Excuse my presumption in editing, mathematical notation is mostly done by putting $'s on both sides of something. If you click on the yellow-circle-? in the upper right when writing, you'll see a link for "advanced help" which will give more info; also, you can right-click on some existing mathjax (such as any of the above) & select "Show Math As -> TeX commands" to see how it was done.
Not HIDDEN Markov models here though - huh?
@Unk - I would start by making some plots of the data. How long is this list of print times?
You seem to be confusing `P` and `p`: because `R` is case-sensitive, these are two different things, whence the error message.  You must pay attention to details!
Does the HES include all hospitalisations?  Or, putting it the other way, are there instances in the CPRD that were hosptialised but weren't recorded in the HES?  I'm presuming that not all hospitalisations are in the HES because if they were you would have a straightforward logistic regression problem.  Possibly this is the answer anyway, but it is made more complex if there are randomly missing cases where it is not known if hospitilisation happend or not.
@Mahoni You're welcome. In terms of statistics, I have only come across ratio and interval. And even these two later become very interchangeable (most statistics works for interval also works for ratio variable.) So, I'd think we need not worry too much if it's ratio or interval, just acknowledging it being a continuous variables should suffice.
You need an ad display variable then. If that's what hr$hours was supposed to be then...it's not. Ads are either there or not there. Furthermore, you can't really do it with cc because what you want to do is predict subscriptions from ad display and perhaps sex. That's a completely different analysis. A couple of different kinds would work but you could do it with a chi-square contingency table with the counts entered in each cell for ad presence and sex. If you just want to test ad presence a chi-square goodness of fit would work.
That was the right answer, Gaussian mixture model to make the two classes linearly separable. Thank you.
Unfortunately, I'm not very familiar with the AUC method, so I can't say how appropriate it is here.  In the maximum score estimator, there really isn't a $p$, because you are not assuming a logistic model.  You are just deciding to predict 1 when $X_i\\beta>0$ and then finding the best $\\beta$.
Check the image in the ["Correlation and linearity" section of the wikipedia correlation article](https://en.wikipedia.org/wiki/Correlation#Correlation_and_linearity).
I was going to say the same thing.... The answers make the same major points, but different minor points, so that may be of some use of others.
@Qiang, Here's the essence of the proof: Let $X \\sim \\mathcal{N}(0, I_n)$ where $I_n$ denotes the $n \\times n$ identity matrix. Then for *any* orthogonal matrix $Q$, $Q X \\sim N(0, I_n)$. Hence the distribution of $X$ is invariant under rotations. Let $Y = X / \\|X\\|_2$ and note that $Y_Q = Q X / \\|Q X\\|_2 = Q X / \\|X\\|_2$ for any orthogonal $Q$. Since $X$ is invariant to rotations, so is $Y$, and since $\\|Y\\|_2 = 1$ almost surely, then it must be uniformly distributed on the sphere.
I'd add that one needs to consider that expertise is often so automated as to become opaque to conscious examination. Thus experts may often fail to have verbalizable awareness of the information processing steps they're taking that achieve expert performance.
Is q for AR(q) the same order that we find through model identification techniques like ACF and PACF plots? Also, if the model identification showed ARMA or MA models are better fit, Do we still find the residuals of an AR(q) model?
What was wrong when you used $n$ as the number of people in each group? Can you give the formula you have, the residual DF from whatever other source you're checking it against, the number of levels $i$ & $j$ of your 2 factors, & the number of people in each group?
Usually there is some theoretical model which is derived without statistical considerations. And then it is estimated using statistical methods. For interpretation the theoretical model is much more important than statistical one. Fixed effect and random effects differ in their assumptions, FE assumes that $u_i$ is correlated with $X_{it}$, RE assumes the opposite. Since $u_i$ is unobserved you should carefully think what kind of effects are unobserved (and constant over time) and whether they correlate with $X_i$ or not.
Interesting question -- although my perspective would be that a test of significance is of less interest than considering the effect sizes (coefficients) and confidence intervals to ask whether the two dependent variables have *materially* different values (and this would of course depend on how comparable the dependent variables are in terms of scale and so forth...)
If I had only a short period (say an hour) to discuss something like this w/ an audience that has no statistical background, I simply wouldn't try to explain these things. I would just say something like, 'we believe $X_1$ is related to whether or not we get a success, but that $X_2$ isn't'. Even 'simple' concepts like what odds & odds ratios are, are pretty foreign to most people & take a while to come to grips w/ (e.g., see my answer here: [interpretation of simple predictions to odds ratios in logistic regression](http://stats.stackexchange.com/questions/34636//34638#34638)).
There is no such thing as a good method or good algorithm without the context of a problem it is used in. Thus we do use k-means because there are problems for which k-means is an optimal solution (;