tttjjj/elt_bench huggingface dataset:
ELKI allows you to use arbitrary distance functions with k-means. Note that the algorithm may then fail to converge. K-means is really *designed* for squared euclidean distance (sum of squares). With other distances, the *mean* may no longer optimize, and boom, the algorithm will eventually not converge.
Seriously, consider using k-medoids. It actually was written to allow using the k-means idea with *arbirary* distances.
Apologies, I was typing on my phone so couldn't Latex things correctly.
Would i get anywhere with simple linear techniques (e.g. lets say I assume 3sigma = minimum value, therefore if its a single distribution 6 sigma <= maximum value) or perhaps simple clustering?
Statistically significant doesn't imply large $R^2$; with large $n$ even very tiny effects are distinguishable from chance; statistical significance is not practical significance.
**None of these questions are answerable with the information given.** In some cases, by making some quasi-reasonable assumptions, *ranges* of answers are possible. For instance, assuming the outcome ranks are independent (which is not going to be the case in most actual races), the answer to (1) is anywhere from $20$ through infinitely many and the answer to (3) could be just about anything. Would you like to say more about your assumptions and the *real* context (assuming these races are a metaphor for what you're really interested in)?
If you fit a mixed effects model then how have you not "accounted for repeated measures"?
@Andreas The variance of 1 and 2 is not the same. They differ by a factor of $n^{-2}$.
I disagree. It just makes no sense to talk about precision and recall without positives. The reason you want to mention these is to assess the quality of a model. These measures can't give any useful information, since you can't distinguish a good model from one that *always* predicts negative in your case.
@Amarald - I couldn't get that link to work but have updated my answer.
Bad practice to post identical questions on several sites: http://math.stackexchange.com/questions/548802/show-that-t-min-nx-n-in-b-is-an-mathcal-f-n-stopping-time
Here is the answer from @Ben Bolker.\\n\\nhttp://stackoverflow.com/questions/6701051/reproducing-proc-nlin-output-with-nls/6701856#6701856
Note also that the posterior of $\\mu_1+\\mu_2$ does *not* depend on $V_1+V_2$...
He doesn't have 4 data points, he has 12. And the question has answers at the link. So, what is new about your question? If nothing, then I think this should be closed.
What's your purpose behind plotting? If it is to see how well the model fits wouldn't plotting residuals make more sense?
I seem to have opened pandoras box of choice here. 



Given I'm not sure how to normalize KL-divergence to make it useful to weighting features for my classifier, MI seems like the next best choice.

So to avoid paralysis by analysis here, what is the simplest option that will improve classifier performance? Even if it's not ideal. Mutual information? Variation of information? Something else?
I lost you at the first line: the left hand side contains six symbols that could be interpreted as variables while the right hand side contains one argument, $x$, which doesn't even appear at the left! This makes it impossible to determine which of the six symbols (if any) you are even considering the variable.
"*I would like to generate via box-muller*" --- Why this and not some other thing?
+1 for "I have always been wary of using -margins- with ratio outcomes". I prefer to use the `lincom` command after a (for example) `logistic` or `stcox` model.
@Stat It should be $X^t$ (X transposed) times Y. Updated the question. Thanks.
`auto.arima` can handle missing values.
(+1) Dumb question from an interested non statistician: What about this book ([Group Sequential Methods with Applications to Clinical Trials](http://www.amazon.com/Sequential-Applications-Clinical-Interdisciplinary-Statistics/dp/0849303168/ref=pd_sim_b_4)) ? How does it compare to the others ?
I added an edit
Ok, so as far I get it, correlation is simple and everyone agrees upon it, whereas causation is far more debated.
Hi Zhao Qi, welcome to CV. Your question leaves a lot of details open. Can you give examples of how your variables are coded under the various conditions your describe? Also if you can amplify how glmnet falls short of your needs, that might also help your audience best understand what you need.
White's test is pre-programmed in Stata as `estat imtest, white`. See http://www.stata.com/help.cgi?regress%20postestimation (search page for 'White')
Robert, I wasn't attentive enough for find out what's your _question_. But here are two links that are possibly relevant for your interest. With two classes, SVD indeed [can help](http://stats.stackexchange.com/a/13246/3277) compute the discriminant. However the general (any number of classes) [discriminant analysis](http://stats.stackexchange.com/a/48859/3277) math does not use SVD.
Nice, thanks! I've been experimenting with `by`, but couldn't figure out how to transform the result into a data frame.
What are the degrees of freedom for the t?  If df is > 5 the tails are not very large and the difference from a standard normal are not that great.  Why not try a Cauchy?  Why mix the t with a normal?  Try all errors Cauchy.  Did you pick a model where the slope(s) of the regression curve (surface) is (are) large?  You could also pick a few points that would have errors that would make them highly influential for the least square regression parameters or the bivariate correlation between a predictor and the response.
(1) When a function has more than one region of local maxima, it (by definition) is *multimodal.*  Thus, as soon as you can find two locations of a *strict* local maximum, the function is not unimodal. (2) It is important to check boundary values because they can be modes, too.  For instance, a Beta$(1/3, 1/3)$ distribution is bimodal but there are *no* points where $f'$ exists, equals zero, and $f''$ is negative. (3) What would you do when a distribution is not given by a PDF? *E.g.*, you have only its characteristic function, or perhaps it is defined as a function of other distributions.
C&T Count data book has a [2nd edition](http://www.econ.ucdavis.edu/faculty/cameron/racd2/).
"Performs the best" in what sense?
@Adhesh The standard deviation is still valid for counts. In general when reporting descriptive statistics, you may want to indicate something about the distributional properties of the variable (e.g., skewness, kurtosis, presence of outliers, nature of distribution, etc.)
Could you turn your question into an actual question, please? Then your answer is a response to the question you pose.
A continued fraction of what quantity, precisely and in what variable(s)?
@stats_newb:  You are welcome.  Thanks for the correction.  I have fixed it.
@Jay: didn't see your comment sooner, but the above is applicable if the DOR are independent.  @drknexus: updated the answer for how to obtain SE.
Welcome to Cross Validated! But no. A haematologist might because he'd have an idea of the variability in these measurements, & what size differences are considered important. Look up [power analysis](http://udel.edu/~mcdonald/statsampsize.html) to see how to approach this, & then details of the calculations are easy to find for common models.
What I meant by time is that if there are 10 items I have gone through each and recorded how long (in nano seconds) it takes to access each value. So it isn't a sample as such there's a time measurement for each record. Unless that changes things significantly I think I've got a much better idea of what I need to be looking into
@MartinChaia if you are using statistical computing software to perform your analysis you can check if it supports prediction intervals for regression.
Thanks Peter - yes # of obs are sparse for very young and old. Using so many knots does seem counterintuitive, I did a mental double-take when first reading in ESL that placing a knot on *every observation* minimizes the penalized residual sum of squares. I suppose the proof is in the pudding whether a restricted cubic spline or penalized smoothing spline works better at predicting my response variable in the test dataset.
Thanks for the suggestions. I edited the main question regarding variable length of the vectors. I'm looking into your links. Thanks! :)
You may be over-thinking this: note that your estimate of $p_1$ is the mean of 606 observation, which is the same as averaging 303 estimates of $p_1$ of sample size 2, which is the same as averaging 101 estimates of $p_1$ of sample size 6.  The CLT can be used directly on your estimates, but when $p$ is near 0 or 1 the rate of convergence is poor, and people often look to continuity corrections.
What do you mean "too trivial"? They are (dependent) identically distributed random variables such that the difference is a normal. It *exactly* answers the question. The construction has the benefit of drawing out exactly how and why one might expect it to be true. :)
As currently stated, this question is only answerable by re-stating Bayes theorem: for some prior, you can compute a posterior on $\\mu$.

$$p(\\mu|y)=\\frac{p(\\mu)p(y|\\mu)}{p(y)}$$ where $y=0$, and $p(y|\\mu)\\sim\\text{Poisson}(\\mu)$.

Determining the posterior for an appropriate prior is left as an exercise for the reader, and the results are interpreted in the usual way. Are you uncertain how to interpret Bayesian inferences generally?
This is a good place for me to start doing research. By the way, do you know any website or forum that I could talk this book with others? I'll definitely have lots of questions when reading it.
While I realize this is not the case for most machines, this machine will probably interact with a fairly even distribution of the population. Right now it is just set to 5'3, and I wanted to see if I could improve that, or if not, I wanted to know what percent of the population is 5'3.
@Kate If you feel that my answer is acceptable, then please consider actually accepting it.
Without further information, I would use option 2, because otherwise your coverage probability would depend on an arbitrary sample of $x_i$'s.
In your model, the response (time) appears on the right hand side. So you are trying to predict time with time...
People who do this use the scores, not the loadings.
Whether y exhibits serial correlation is immaterial in the regression context: what matters is the nature of correlation among the *residuals.*  It can happen--and often does--that the apparent autocorrelation in the dependent variable is due solely to autocorrelation in the independent variables, but after regression, there are independent residuals.  Its seems likely that's the case for these data.  Whether it is or not is easily checked: do the regression and test the residuals!  But please don't do the differencing first: that's not the model that's needed here.
Yes and no, I would like to know in the first place if the the pwr.2p2n.test() function the right solution. If that's the case I would like to know how to calculate the effect size parameter.
@Peter - I understand and I suggest you edit your question's "I assume" sentence to read:  "I assume that, in some region of x, higher values of x have a positive effect on the dependent variable, while in some other region, higher values have a negative effect."
+1 Look into [IRLS](http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares), for instance.
Thank you, thank you, THANK YOU for including reproducible code!
Also recall that any measurement of entropy are estimates. Therefore, they have an underlying distribution if the processes that give rise to it are random!
+1, this is a very interesting take on the question.
I'm assuming that the player's true strike percentage is essentially a realization of the same distribution of strike percentages as the 1000 other players. In other words, there's nothing special about that new player, he's just another random player. Hopefully that makes sense.
cheers, I will try it over weekend and see if it helps :)
@ Stephan Kolassa, Thank you a lot! :)
Yes, there is, provided you first tell us beforehand what the relative chances of each population are of being the one the observation came from. If you do not do that, this question is unanswerable.
I'm not sure you're right that there is higher variance in the middle.  The fact that the outliers are in the central region looks to me likely to be a result of the fact that that is where most of the data is.  Of course, this doesn't invalidate your question.
Do you have the individual children's weight or just their combined weight?  And can you clarify - when you mean "the variability of the twins' weights" do you mean the variation between each set of twins, or the variation between the two members of each set of twins?
that link is broken now; please use http://yihui.name/en/2007/10/jokes-in-statistics-a-talk-to-be-given-in-cueb/ thanks!
DWin, would the specific predict.coxph() and anova.coxph() functions accomplish what you mean?
Fixed my answer, meant to say "constant values", like all 1's, instead of "constant columns". If you scale a column with constant values, you get zero SD and divide by zero gives NaN. So remove these columns or set the NaNs to zero, these variables aren't predictive anyway.
This question may be of interest: [what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/); it also covers the issue of whether normality is required of the raw data or the residuals.
I think a good statistician would not ask this question in the way you first put it, in no small part because "strong" and "weak" are vague, non-quantitative evaluations: that leaves too much opportunity to be misunderstood.  The TL;DR version *is* something statisticians are asked a lot.
I do not agree that it is a matter of speculation. Well defined motion models in Hamiltonian notation can be used for probabilistic inference problems therefore I wonder about whether these particular models can be represented in a some kind of unified framework. Actually, once this connection established in my mind, it could solve a plethora of machine learning problems. Thanks.
I agree, if first differencing is applied to remove the fixed effects then it should be applied also to the dependent variable. Otherwise also for pooled OLS with differenced explanatory variables the standard autocorrelation corrections work.
@Procrastinator Thank for this! What exactly do you mean when you're talking about the "thinning period"? The paper is also very helpful.
(+1) Of possible interest also: [An introduction to sphericity](http://homepages.gold.ac.uk/aphome/spheric.html).
What are you trying to find out? What is your research question?
It's probably worth pointing out the existence of the `arima.sim()` function here.
Using a rule like "biggest 10%" doesn't make sense. What if there are no outliers? The 10% rule would eliminate some data anyway. Unacceptable.
+1 Very well put.
Your sample size is too low by perhaps a factor of 100 for a single tree to reliably represent the predictive patterns. Your sample size is also woefully inadequate for split sample validation to be reliable and stable.
Thanks. I suspect either of these points could be it. We've done the proofs of these. It just seems like a lot of work for 1 mark. TBH, had I covered these, I would have written more for this 1 mark than for the 13 marks of the first 2 parts combined (see my comment in reply to procrastinator's first comment on my question). Having said that, have come unstuck in making assumptions of what an examiner is expecting based on the marks available for a question - but in an exam, I always allot my time based on the marks available - I don't know any other way to proceed.
@Tim.  Misread which book.  I just finished surfing multiple book sites and this book does not exist in any library that I have access to.  However Dr Roberts is at the Warwick University in the United Kingdom.   I suggest that you contact him at his email (the address in on this page --http://www2.warwick.ac.uk/fac/sci/statistics/staff/academic-research/roberts/ )
@FrankHarrell I agree, but I think this may be an easy area for the R developers to target and improve. A simple solution could be different levels of security for packages - if a package has a system call or connects on its own to the Internet the package should have a higher clearance. This could allow installations with only low-level clearance in those institutions/companies where a data leakage is of major concern. I as a user could then also do an extra check when I choose to install a high clearance package. (Btw, when is your book (RMS ver 2) scheduled?)
So, January released a package in February. I'll ... March over to get it.  :-)
Indeed, no (explicit) convolution or appeal to independence is necessary, though the latter is trivial. This can be done completely geometrically, if one prefers that approach.
@chase and @cardinal: yes Travel demand forecasting and Netflix Cinematch ( and other recommendation engines) would be great examples, if there were any documentation about how they work, or at least a description of the underlying techniques. Other examples could be from non-profit domains such as education or government.
@StatTistician You're right about the N-1 one thing -- For the code, I ended up using a formula from the wiki site, which was different from what I had originally said in that comment.  As for the pacf, I get the coefficients from what R computs (`pacf(...)$acf`, so that seems like it should give the right value.  But I confess that I'm working through this with you, so I could be wrong.  Sorry I couldn't be of more help, and thanks for talking about the topic with me.
I understand your concern.  However there was a perfect match to the theoretical results for $\\tau_3, \\tau_4$ and $\\tau_5$.  I will however also carry out the calculation for the exact solution.  That is the most honest way.
I second soakley's suggestion of a permutation test.
@Mortimer and @rm999 if you don't use @name to answer then I can know that you are talking to me (the sofware detect @robin and send me a notification somehow when it comes).
Lorenz curve: see http://en.wikipedia.org/wiki/Lorenz_curve That is easy to search for in R circles.
Is that $\\mu_{t+1}$ or $\\epsilon_t$?
Closely related: [Reference for the sum and difference of highly correlated variables being almost uncorrelated](http://stats.stackexchange.com/q/13530/7290).
This question seems to be a duplicate of [this question](http://stats.stackexchange.com/q/91493/27403)
Thank you! Looks good. Of course if anyone knows any other dataset, always welcome to point me to it.
What do you mean by “rate of the students”? What does your scale measure and can you provide some details about it (there is in fact a lot of confusion about what a Likert scale is)? Did you try searching past questions on this site (many Likert-scale related questions have already been asked and answered)?
This will give you the mutual information between the random variables, not the pointwise mutual information between the pairs.
Loosely related: [What is wrong with this chi-squared calculation?](http://stats.stackexchange.com/q/85861/7290)
Meta-response: Good point.  But your data might not behave like mine! :-)
@w1res what? You can do unsupervised learning with SVMs? I feel a bit dumb. What's the objective function?
Hard to say without knowing what they are. Maybe a problem with how are you initializing the emission distributions? Looking at your initial matrix it is clear how you "want" the HMM to work. I might try something like 1.) fit the parameters of a multivariate Gaussian to the observations in the beginning of the sequences, 2.) initialize the emission distribution of state 2 with these parameters, 3.) repeat this for states 3 and 4 using the middle and end of the sequences respectively. It could also be that the emission distribution you're using is a poor match for your new features.
I notice you have a "0% accept rate". Keep in mind that you can "accept" an answer to your previous questions by clicking on your name at the top of the screen, navigating to your old questions, and clicking the check mark next to an acceptable answer. This helps future users find an answer to their question more quickly, but also will probably make people more likely to answer your own future questions.
I read your description as having 3 IVs (age, gender, kill type)...my mistake.  I think that is because I am so used to "age" being a IV.  (Even now I am having trouble imagining how "age" can be your DV.)  I will edit my answer....
@nico - I'm not sure what you mean by generalizing to multiple groups. I gave an example for runs in three groups, but the logic applies to more. (The typical runs test in most software I've seen only allows two, but the cited papers establishes the test statistic for multiple groups. That is why I elaborated with an example with 3 groups.)
It is interesting and important to note that both of your models predict equally well, in this case.  High correlations among predictors are not necessarily a problem for prediction.  Multicolinearity is only a problem when 1) analysts try to inappropriately interpret multiple regression coefficients; 2) the model is not estimable; and 3) SEs are inflated and coefficients are unstable.
You may want to review the regularity conditions for Cramér–Rao bound properties and MLEs. Also [this article](http://www.jstor.org/stable/2282794) contains the answer.
Actually, this lead me directly to the solution. I'm going to post an answer, but thank you so much for your help!
@br69 Please [register your account](http://stats.stackexchange.com/users/login) and stop using answers for comments.
Thank you! But based on my readings (correct me if I'm wrong), a t-test is a parametric test. I do not know how to check if my data has normal distribution.
That's why I asked if the treatment is randomly allocated - if it is then the groups would be expected to be balanced for all known and unknown covariates and matching or propensity score approaches aren't likely to be of use.
@ricardh, curse upon all economists for not checking whether the term they want to use is already used in statistics. Cluster in this context means group and is totally unrelated to cluster analysis, this is why rseek  gave you unrelated results. Hence I removed the clustering tag. For panel data analysis check out [package plm](http://cran.r-project.org/web/packages/plm/index.html). It has a nice vignette, so you may find what you want. As for your question it is not clear what you want. Within group standard errors?
http://math.stackexchange.com/questions/9286/proving-int-0-infty-e-x2-dx-frac-sqrt-pi2
(+1) This is an important fact that many practitioners do not think carefully about: "p-values are arbitrary, in that you can make them as small as you wish by gathering enough data, assuming the null hypothesis is false, which it almost always is."
+1 For the implementation of the direct method. I think my implementation would be slower if $N$ is increased but I did not check how slower.
@user1172468 You really don't need to explain but it is nice that you did.
Here is the electronic copy in case you want to read it (it's very instructive and includes a lot of references): http://j.mp/b2Wbpn.
I did the edit, so you can learn from it and implement when necessary, next time. :)
I realise my answer is very similar to the others. Nonetheless, I believe it is always a good point to first check what parametrisation R uses....
Great! tks. Does the algorithm iterate through the parameters? I left `alpha=NULL, beta=NULL` and I used `opt.crit=c("mse")`  However, the number produced for the ZAN model is 2490139. I tested for arbitrary parameter of 0.5 for both alpha and beta and the number is even bigger. Is the 2490139 an acceptable value for MSE? seems rather large.. Or is this because the data set I am using is multiplicative (tested using model="ZZZ") & using a model that does not factor seasonal will produce this large number? Where as `model="ZAM"` does a much better job because it factors seasonal with HW model.