tttjjj/elt_bench huggingface dataset:
For large (many nodes) and high-dimensional data it can also be worthwhile to try a graph clustering algorithm (using e.g. tanimoto similarity and methods such as Louvain clustering, RNSC, mcl). I have some doubts whether your type of data will generate meaningful clusters (it very well may of course), but those doubts relate to clustering in general, not specifically to a particular type of clustering. PCA is definitely something to try.
Thanks Tal, I'll try corrplot now. I also wish I knew how to simplify your solution (which I linked to in the question) but I'm just a newbie in R so you know more than me. I'll update the question to clarify the solution looks complicated *to me*
Nick points out above NOT to use quarterly dummies and I agree.  Weekly dummies, Seasonal differencing or an AR52 would make much more sense.
A good example - it is very much the same as with scientific hypotheses. We're still talking about models whether they are statistical ML models or models of the universe.
Could someone elaborate about why the second formula works? `K*N/(K+1)`
Some good ideas. Thank you.
That's it, spot on.
In addition to @Tristan's answer, I just want to add that ordered and unordered factors can also be referred to as [ordinal](http://en.wikipedia.org/wiki/Nominal_variable#Ordinal_scale) and [nominal](http://en.wikipedia.org/wiki/Nominal_variable#Nominal_scale) variables
I seriously doubt the ball positions actually have a bivariate normal distribution: you should check the evidence for this. In any event, because you are studying only *radial* distributions, it appears you don't really have a spatial problem: you are merely comparing two *univariate* datasets (of radial displacements) to each other. The complication is that the $40$ observations in each dataset are clearly interdependent: not only must the mass-weighted vector sum of their locations be zero, any two balls must be separated at least by the sum of their radii.
Ah, I'd never bothered looking in to power transformations before, thanks for the pointer. I added an addendum to my question, because I realised my question was more general than intended.
+1 for the visual option.
I actually tried it the other way arounnd. I didnt't look for any clusters and I fitted a Multinomial Hierarchical Bayes and I used the betas as criteria to cluster the population but I think that it wasn't a good idea, beacause there are too many parameters: beta_0, beta_1,...,beta_m  for each individual. Assuming I perform some sort of clustering, the model im fitting still allows for errors which are a mixture of normals and the number of components doesn't necessarily correspond to the number of clusters one inputs! I don't know with what criteria to choose this parameter.
Linear discriminant score is a value of a data point by a discriminant, so don't confuse it with discriminant coefficient, which is like a regressional coefficient. See my detailed answer [here](http://stats.stackexchange.com/a/48859/3277).
@Yep, it's the CDF, which is the area under the graph below (-mu/sigma). What we've done in this penultimate step is start out with the probability we're interested in, adding things to both side to put it in terms of the thing we know the distribution of. So x(1)<x(2) while ((x(1)-x(2)-mu)/sigma) < (-mu/sigma), which is something tables or a computer can tell us.
Since your posteriors have an unknown scale (the normalizing constant), you can shift their logs (all by the same amount) without changing the relative probabilities. Just add, say 700, or, say whatever it takes to make the biggest of them 0, or whatever you find convenient.
@bizso09: as I said, this kind of correlation is possible, but I believe that people opinion reflects actual probability of the win of this or that team, so additional sentiment analysis doesn't seem to add much more info about match result.
I confirm that this is left-censoring. When $X_i < C$ I observe 0. Question edited to reflect this.
@Macro, I suppose I thought about that question differently than this one. If I remember correctly, there I was addressing the OP's belief that they needed to transform their X variables to achieve normality (although since it was merged the idiosyncrasies of that question have been lost). I'm OK w/ merging these, but I'm also OK w/ not doing so; I think there are subtle differences in emphasis. Eg, my answer doesn't make much sense in the context of the merged question (perhaps I'll delete it). Here, I think Michael's answer that logit(Y) is linear in ln(X), not X, gets at the right nuance.
PCA already does these calculations: it has reported (in slightly disguised form) the Euclidean distance if it was performed with the covariance matrix, and otherwise it has reported a quantity proportional to the Mahalanobis distance if it was performed with the correlation matrix.
user 21342 - you have been given a reasonable answer to your question. If that answer is acceptable, you should really do the right thing and accept it - I doubt anyone is going to offer a more substantive one. If it is not acceptable, you should indicate why.
This site must already have a number of closely related questions. Please search `multicollinearity`, `singularity`.
Your country-scenario seems perfectly suited for a multilevel aka hierarchical aka mixed model, where participants would be located on level 1 and countries on level 2 treating country as a random factor. If the example you gave matches your actual research question, go for multilevel; if not, maybe you can provide an example closer to your question?
your comment about *The "best" model will be choisen by the ML practitioner...* applies equally well to mainstream statistics as well.  For in most model selection procedures, one simply conditions on the final model as if no search of the model space had been done (given that model averaging is fairly new).  So I don't think you can use that as a "club" to beat the ML practitioner with, so to speak.
@DarrenCook: `sdev` implies that you're looking at the _standard deviation_, i.e. the square root of the variance (or $\\sqrt{\\lambda_i}$, using the notation from my answer), which should explain the difference. Try `pca$sdev[1]^2/sum(pca$sdev^2)` instead.
(Continued) As we're seeing, the suggestions by Mike and whuber are far different than one would normally see on SO.  FWIW, I think that changing the statistical calculation will yield the best speedups, while changing the backend math libraries will also substantially speed up the calculations (perhaps a good 12X-36X speedup, given the infrastructure, by changing the libraries, and potentially much more than that by changing the statistical methods).
Hint: what is the expectation of a constant?
It would seem that by definition the answer has to be **no**: because a gradient is *by definition* a local quantity but the global maximum is (obviously) a global property, no algorithm "that's not a global optimization algorithm" will efficiently find optima for a reasonably large class of functions.
Yes, good point. What I wrote doesn't necessarily hold.
Can you clarify the clustering goal?  For instance, do you want a discrete set of clusters, agglomerative clustering, or something else?  It may be the case that a different perspective will help.
@Matthew Yes, that is what I started doing, but if I can find a library it could be a lot easier.
@Jimj I believe I figured this out. The s.d. of the differences is not $\\sigma\\sqrt{1-r}$ but rather $\\sigma\\sqrt{2(1-r)}$. This makes the two computations agree on your small example (except for a small amount of bias). The reason the denominator of the operative effect size uses $\\sigma\\sqrt{1-r}$ and not $\\sigma\\sqrt{2(1-r)}$ is a little complicated to explain in a short comment, but the basic issue is that the power formulae derived for the independent-groups case only work nicely in the paired case if you factor out the $1/\\sqrt{2}$, because that already appears elsewhere in the formulae.
"each feature" -> there is only one, correct (1D)? also have you given it a value for `C` which enables soft margins?
Exploring the [change-point](http://stats.stackexchange.com/questions/tagged/change-point) tag might give you some ideas.
This question is off topic for SO, but can very well be asked on http://www.crossvalidated.com. I flagged it for migration, so that should be taken care of rather soon.
Glivenko-Cantelli: http://www2.imperial.ac.uk/~das01/MyWeb/M3S3/Handouts/GlivenkoCantelli.pdf
You may find this question helpful: [what-is-the-best-introductory-bayesian-statistics-textbook](http://stats.stackexchange.com/questions/125/).
Here is simple example that might help you http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/
Unless I am misunderstanding your concern (always a possibility), it seems to me the lack of a reference level for Age is irrelevant.  The odds ratio that the conversion of the estimates would put out is (probability of the event / (1-probability of the event)) where "event" is determined by the left hand side variable of your regression.  All the smoothing function does is generate a smooth, but not necessarily linear, fit to the odds ratio (equivalent to generating a smooth but not necessarily linear fit to the probability.)  There's no assumption about a reference Age in there.
things would get more interesting if you added say `rexp(n)` in place of `rnorm(n)` when generating your data. The distribution of the residuals would get WAY closer to the normal than you would think.
If your repeated-measures data is "wide" form - you have 2x3=6 variables - go to GLM -> Repeated Measures.
Thanks. I didn't think that "The logic of science" talked about this stuff too, i'm definitely going to read that book.
Thanks! Very clear.
OK THANK YOU, I ALREADY HAVE TWO
@Laurans ,I tried again but I am still confused. I used the conditional probability. A=debit card purchase, B= $20 and over. P(B|A)=P(A intersect B) / P(A). P(B)=0.44. P(A intersect B) is what I am confused about again. I came here because I am unsure of what to do. I am still unsure. For P(A intersect B), do I take .18+.14=0.32
OR .18/.44 + .14/40 = 0.41+0.35=.76
OR .18/.36+.14/.36=
OR .14+.18=.32 therefore, .32/ .36 OR .32/.84
Yes, I was mistaken in asking the question. The variance I was looking for was indeed zero. I was overcomplicating things!
Also see http://www.statdistributions.com/f/ . Across many examples, when the 2 variances used to compute *F* are divided to obtain a ratio, one gets the sort of distribution shown--IF nothing but chance is operating.  The question is, how unlikely would a given *F* be under such an assumption?
In other words, you are asking why $2\\bar{x}\\sum_{i=1}^n x_i$ = $n(\\bar{x})^2$. Factoring out the common factor of $2\\bar{x}$ reduces that to either $2\\bar{x}=0$ or $\\sum_{i=1}^n x_i$ = $n\\bar{x}$.  Now, *what is the definition of $\\bar{x}$?*
Directly relevant threads that appear to answer these questions include http://stats.stackexchange.com/questions/61798 and http://stats.stackexchange.com/questions/3734.  If that does not suffice, then please edit your question so that it more precisely and clearly states what kind of answer you need.
Thank you Tomas! This method is far more interesting from the ones I'm reading so far.
Thanks, I haven't seen that one.
I just added some data (for R and a CSV). I haven't finished choosing good colors yet (hence the Christmasy red/green), though I'm aware of the color blind issues :)
I'm glad to see you figured out the answer, @SammyJ. If you get a chance, please consider posting your answer below for future reference and so the question comes off of the "unanswered" list.
Think of the difference in time as the difference in your 'x' values on a plot.  Most types of regression analysis will take into account the varied differences.  Try your sample data using the GROWTH function in Excel, which uses exponential regression.  If you change the dates, your projected values will change accordingly.
+1 Great question!
I would be interested in learning more about what "$g(r)$" represents and how it is calculated.
@PeterFlom Thank you. When I mentioned incidence I was just talking about the number of cases divided by the number of pupils. Also, The fixed effects (and random effects) estimate have changed a bit, but their confidence intervals almost cover each other, so that's why I say they are the same. I know there might be an issue of power to detect a change, but the sample size is very large so I didn't think that would be a problem. I am really just focusing here on what it means for the random intercept to be unchanged...
I loved this book myself, but I am not sure it is the place to start trying to build an intuition for statistics.  It is a rather polemical and idiosyncratic text.
There is no reason for nonlinear least-squares here, not that it won't work reasonably well. Calculate sin(2 * pi * ToY), cos(2 * pi * ToY) in advance and feed them to `lm()` just like any other predictors. In other words, `lm()` need not see any trigonometry at all. However, you may need another model to capture marked asymmetry well. I am not a regular R user but I've often used this approach elsewhere (see http://www.stata-journal.com/sjpdf.html?articlenum=st0116).
Could you please tell us a little bit about these data and the tests? Your answer will determine whether the responses you are getting will be appropriate and useful or inappropriate and misleading.
Here is another that I like that seems so insufficiently known that it has no name attached to it, but has a similar flavor and an interesting statistical lesson: There exists a sequence of independent random variables $X_1,X_2,\\ldots$ with mean zero and uniformly bounded variance such that  $\\sqrt{n} \\bar X_n$ converges in distribution to a *standard* normal $\\mathcal N(0,1)$ (just like the CLT). However, $\\mathrm{Var}(\\sqrt{n} \\bar X_n) \\to 17$ (or your favorite positive number).
Although you have accepted a reply, I feel obliged to point out for any future readers that *no* reply can possibly be correct because the question is too vague.  It seems to ask us to read your mind; that is, to know precisely what you mean by "non-linear % correctness."  There are infinitely many solutions to questions like that and each one is tantamount to a particular way of *quantitatively valuing* almost-correct integers.  If you blindly accept a suggestion like a Gaussian, you have effectively let somebody who is *completely guessing* determine this.  (continued)
Also: bear in mind that nowhere is it chiseled in stone that we must throw away every slope estimate which doesn't have a p-value vanishingly small (it's typed on paper in lots of places, though).  You may judge it worthwhile to keep the slopes -- at least temporarily -- to investigate a different question which may be important.  On the face of it, I'd rather preserve the information in the slope estimates than throw the ties completely away.  Bottom line: we are searching for the *truth*, and the path leading there isn't always paved in steel with road signs pointing everywhere.
... A technical difference is that I use `pls::plsr` for the pls (which allows to choose from different algorithms). And I have a bunch of post-processing functions e.g. for flipping and rotating the model, which is sometimes useful for interpretation.
I did not know about fixing the dispersion parameter (+1).
(+1) Nice explanation, particularly the conditions for obtaining a good performance of the Sheather-Jones method.
Could you summarize why violin plots and/or jittered points would be useful with heavy-tailed distributions?
@paul: You have a non-stationary process here.  De-trending through a linear regression vs. time is NOT going to work.
The calculation of the MVUE for the mean of a lognormal distribution is rather complicated and involves infinite series. But check out @whuber's [Excel sheet](http://www.quantdec.com/envstats/software/ln_mvue.xls) that does the calculations for you.
Welcome to the site, @Kait. Note that purely R questions belong on [Stack Overflow](http://stackoverflow.com/), whereas CV is for questions about the statistical content. We will migrate your Q over there for you in time(*please don't cross-post SE strongly discourages this*), but 1st we should help you formulate it (here are a couple of relevant discussions: [how-to-ask-question-related-to-the-use-of-r](http://meta.stats.stackexchange.com/questions/1335/), & [how-to-make-a-great-r-reproducible-example](http://stackoverflow.com/questions/5963269/)). Can you paste your code into your Q?
A confidence interval (CI) actually tells you how reliable the 'estimation' is: "is used to indicate the reliability of an estimate". The interpretation of a 95% CI is "the probability that this interval contains the true value of the parameter is 95%". Spearman's coefficient is a measure of dependence (there are others) and it actually measures the correlation, see the [definition](http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient#Definition_and_calculation). Therefore, a 95% CI can be used to assess the degree of association by analysing the values it contains.
I found this valuable in work.  Thank you.
I appreciate that the citation comes with the stardate.
I voted to leave this question open because it appears to be asking about estimability, which is on topic here.
Good idea! Remind me to upvote this in 11 hours (used up all my votes on [cartoons](http://stats.stackexchange.com/q/423/32036))...I particularly like bootstrapping the ECDF as a way of enriching that kind of plot.
The link is not the same as a transformation, because it does not affect the error structure (!). A (nonlinear) transformation of the dependent variable will change the nature of its random variation, whereas a link can be varied independently of how the random variation is modeled. That makes GLMs more flexible than (just) transformations.
Just to be super explicit: (in my view) Glen_b is not talking about the normality of the *dependent variable*, but the normality of the *errors* in the *Y* dimension about the line of means.
I believe the answer might depend on what you believe to be "meaningful." In the sense of *potentially providing information about the data,* the median and IQR are indeed meaningful for binary variables: when the IQR is zero, you know that at least three-quarters of the data equal the median. It is unclear in what sense such information would be "practically meaningless" to you. Perhaps, then, you could edit the question to provide some guidance considering how we are to understand "meaningful"?
Another note, when finding the $\\beta$ that minimizes $(Y-X\\beta )'(Y-X\\beta )+\\gamma\\sum_j |\\beta_j |$, break the problem up into the cases $\\beta_j >0$, $\\beta_j<0$, and $\\beta=0$.
Well, theoretically you're right. But, depending partially on the data structure, a simple interaction will do the job.
If you care that small changes in your input data may result in large changes in your regression, then you need to worry about collinearity. See the many other questions about collinearity on this site.
The example in the link is about time-varying covariates, not time varying coefficients. These are too different things. To get time varying parameters the way you described use interactions, i.e. instead of model $y=x\\beta_{my}$, fit model $y=x\\beta_0+x\\cdot t\\beta_1+x\\cdot t^2\\beta_2$. In R the formulas would look like `y~x` and `y~x*(t+t^2)-t` (also possible to write `y~x+x:t+x:t^2` for the last one).
Can you put all relevant information into your question, please, rather than just in comments? You'll also find it easier to format. Then to get my attention, comment to mention it.
You can simply use the truncated version of $Z$ as regressor since it is interval scaled. If you feel that such model structure does not represent reality well enough, you could think of adding a dummy variable (1 if $Z = 24$ and 0 otherwise) to represent the 10% truncated values.
To answer the question in the title, consider using `rmultinom(1, 10^5, rep(1,4)))` to count the number of times each of four teams will be the division winner in 10^5 independent simulations. The execution of this simulation is practically instantaneous and it scales beautifully (although for more than 2^31-1 iterations you would have to make multiple calls to this function and sum the results: do so by increasing its first argument).
Katarina, what exactly would be "inappropriate" in comparing 20 patients to 1200 controls?
@gung is that better?
"Clearly therefore P(D|H)+P(D|¬H)=P(D)". I don't agree with this statement. If my hypothesis is D~N(m',v), then probability of data given my hypothesis is true is P(D|H) = N(m',v). In that case what is P(D|not H)? 1-N(m',v)? How is that going to add up to N(m,v) which is real P(D).
Your argument depends on the data being normally distributed.  If we assume the population to have a "double exponential" distribution, then the absolute deviation is more efficient (in fact it is a sufficient statistic for the scale)
Done as asked for!
By "the book written by Fausett", do you mean: Fausett (1993) *Fundamentals of Neural Networks: Architectures, Algorithms And Applications* ([Amazon link](http://www.amazon.com/Fundamentals-Neural-Networks-Architectures-Applications/dp/0133341860/ref=sr_1_1?s=books&ie=UTF8&qid=1385155863&sr=1-1&keywords=fausett+neural+networks))?
There's a nice article explaining the "alias method" here: http://www.keithschwarz.com/darts-dice-coins/
Edited detail on my 2010 paper, now accessible to all under the Stata Journal's 3 year window policy.
I double-checked and am using listwise. However, I noticed that when I don't opt to store factor scores for the analysis as separate variables, I don't get the "matrix is not positive definite" message. Very confusing, as I don't see why simple storage of factor scores should affect the correlation matrix...
Even though I can only accept one answer, I wanted to thank you for pointing out how normal distributions actually arise. It forced me to think more carefully about what it means for something to be based on theory.
@Matt It makes a different, not a huge one. Kappa goes from .45 to .48, 61% to 65% correct, mean absolute error from .21 to .26. I cut it off at >=50 examples. There's still a big range, the smallest being 50 instances and the largest 372 (not a huge dataset because I'm starting with a subset of my full set, 808 instances). If I make it >50, I get 69% correct, kappa=0.52 and mean error = 0.29, which is probably not bad for what I'm doing.
You should include the sample in your question. As whuber said, the likelihood is trivially zero for both parameter values ... which simply means neither parameter value can have produced the values $(0,1,2)$ (But neither conditional distribution seems to sum to 1 either. Is something missing?) Or are you supposed to treat each value as a single sample (giving three MLEs for three different samples)??
Do you have good reasons to believe that whatever causes the big drop is not independent on past data from that series? If it is independent, no matter what model you use, you won't be able to predit the drop. Also -- provided the drop can indeed be indicated by past behaviour of the time series, you will need a sufficiently large sample, with many examples of drops for your statistical model to learn from.
Can you imagine how a tutorial on R looks to a person who has never seen a line of code in his or her life? :)
Have you looked at the caret package? Is fast and cheap more important than accuracy of your estimate?
Why not just refer to one of the many available textbooks on time series?  They should provide descriptions of most of the tests that you aare interested in.
Thank you for pointing me to the FDR method. This looks promising. I took the liberty to add an implementation to your answer.