tttjjj/elt_bench huggingface dataset:
Because your ANOVA makes no distinction between treatment and controls, it does not appear to address the scientific question on the table.  Because temperatures have a natural order, and that order can be expected on general principles to influence growth, what exactly are the temperatures involved here (including that of the control)?
If it isn't then suppose you did n permutations.  I would simply say the p-value is less than 1/n or 1/(n+1) (whichever you prefer).
@JonathanLisic: Thanks for your suggestion. Contingency tables show indeed that position of trees is highly dependent on environmental variables. Probably it would be really better to stay far away from that spatial rabbit hole.
This seems rather strange. Which book? Are you *sure* they're not talking about standardized coefficients?
@cardinal: I'm looking for oracle properties (thus, at the least model selection consistency). The conflicting effect is indeed described in the adaptive lasso paper (Zou) for the lasso. I had the Fu & Knight paper at hand but for some reason it was not in my list of "active" papers, so I'll check it out again.
In a Bayes Net of A->B, there is a parameter $\\psi(A)$ which happens to be equal to $p(A)$, which is the prior. In a Markov Network of A-B-C, there are parameters on the cliques $\\psi(A, B), \\psi(B,C)$. Neither of these are P(A). You can also add a parameters $\\psi(A), \\psi(B), \\psi(C)$, but they are not generally $P(A)$, etc.
Nice summary.  There's a good chapter in Kuhn's book on `caret` that covers this as well.
How do you know that those are "the factors that determine it"? Did you build a model to describe your data?
"Ideal" in what sense? This sounds a bit like a substantive, scientific judgment, & I don't know if anyone here will know the pros & cons of different ways of thinking about the synchrony of laughter. If you are looking for a *statistical* ideal, we may be of more help, but could you describe the issue more fully? What is "Kuramoto's model"? Are you just trying to determine how well synchronized 2 sine waves are with respect to their frequency, amplitude, periods, or some combination?
Good question. But my initial reaction is that (because you have pretty large amounts of data) you have really uncovered evidence that those who record purchases either make different numbers of purchases on average than those who do not, or else do not record all their purchases.  This would call into question almost any inferences you might like to draw about all purchasers, so perhaps a better action to take is to note that fact (and act accordingly) rather than attempt to modify your estimator.
Hi! Actually, I did not intend to compare the relative sizes of p1 and p2. Rather, p1 and p2 respectively reflect the significance of the 2 comparisons that answer my experimental hypothesis (out of 4 possible comparisons -- yet the last two ones are of no interest for my research). That is, for my project, looking at comparisons 1 and 2 (SU and YU participants' performances across experimental conditions) seems to be more informative that looking at the global interaction (since it reflects all 4 comparisons). Does it seem correct to you to do so then?
Thanks, this looks like the cold shower appropriate for the occasion.
Do you understand the 'big O' notation?
I'm afraid that I do not know how to estimate the error process in this situation. I do not have a model that I am fitting -- I'm just taking mean values for, say, deciles, as descriptive statistics. I could make assumptions about the underlying process, that the data is drawn from a lognormal or gamma or whatever - I think the Generalized Beta type II is thought to be best fitting on U.S. data, at least for the high end of the income spectrum. I would know how to estimate the parameters of the distribution from the sample, but not the parameters of the error process.
Each point on the unit ball is, by definition, a unit vector.  Multiplying all its coordinates by the intended radius accomplishes the desired rescaling: it's just a change in the units of measurement.
Thanks very much for your thoughtful answer. I am going to have a hand at implementing the suggestions the two of you have made, and may post a new, more focused question in the future. Sorry for my delayed response, as well - things have been hectic this week.
I've seen that paper and I don't like it because the measure seems dependent on parameterization, ie, it may rank your models differently depending on whether you use regular or logarithmic units for data
The accuracies for each fold will not be independent, which will violate the assumptions of most statistical tests, but probably won't be a big issue.  I often use 100 random training/test splits and then use the Wilcoxon paired signed rank test (use the same random splits for both classifiers).  I prefer that sort of test as I often use small datasets (as I am interested in overfitting) so the variability between random splits tends to be comparable to the difference in performance between classifiers.
Maybe have a look at `multcomp` package for post-hoc testing {`glht` function]
Although the notation is a little unclear, it looks remarkably like a [Law of Large Numbers](http://en.wikipedia.org/wiki/Law_of_large_numbers). These date to [Jacob Bernoulli's time](http://en.wikipedia.org/wiki/Jacob_Bernoulli), published posthumously [in 1713](http://en.wikipedia.org/wiki/Ars_Conjectandi).
Or, even more concisely (densely) as a one-liner `x[x>=quantile(x,0.99)] <- 1e10`.
I think "usually" may be a bit extreme, but it is certainly a good point that different software does different things; and sometimes the same software does different things in different (but equivalent) programs.
Dear God, you are correct! Thank you so, so much. Everyone keeps saying "use log.likelihood" but only you really see the problem.
+1 This helps drive home the point that the question needs additional information to be solvable and it shows just what kind of information is needed.
@MichaelMcGowan I have asked the OP for clarification and afterwards performed an edit to explicitly reflect the multiclass issue, which was not obvious before the edit (IMHO).
This appears to be off-topic because it is purely a request for Stata code, with no statistical element. It doesn't belong on Stack Overflow either: requests for code with no attempt at coding are off-topic there. This would be acceptable on Statalist, talkstats, etc. There is to me no obvious one-line solution to precisely the question asked, as "terrier 1" ... "terrier 3" don't collapse to "terrier" without some extra code and some rule; what kind of rule is wanted is not clear. `duplicates`?
Thank you!  It won't let me vote up, but I appreciate your help.
I really like that 1st one. I've only recently been exposed to the glorious world of ggplot2. You've not only given me a place to take off from, but you've got me pumped on graphs! Thanks! PS I'll vote it up as soon as I have the rep to do so.
@ClarkW.Griswold Please see the edit
Are they strictly interested in survival from one season to the next, or would they prefer to model survival over time?
Thanks very much Maarten!
This question appears to be off-topic because it is only about how to use R.
@AndyW: Thanks for your reference, I have never encountered beta regression but it sounds promising.
But you can have a very simple learner in R or Weka and wrap PMML on it. Or do you mean that you want an example of a small code that parses PMML and performs the algorithm?
But you don't and can't know mu or sigma. What you know are the mean and SD of the sample, mean(S) and stdev(S).
What do you plan to do with the data after outlier detection? If your plan is to remove outliers, maybe there is a [robust method](http://cran.r-project.org/web/views/Robust.html) you could use intead.
That was a fascinating read, thank you for bringing that to my attention.
@conjectures sorry there was a typo here. I have corrected it directly in the question
On reflection, I think this question raises some important issues - ones I've been asked perhaps ten times, maybe more, when doing consulting for people attempting to do research across a variety of areas. Since in that sense it's a very important question, I'm inclined to +1 it; in particular I think it's potentially quite relevant to some future visitors.
I guess you will have to implment yourself. Maybe you can use WinBugs?
To average, or even add - *at all* - you already assumed the variables were at least interval. 

If you average t2-t4, you assumed they were all interval. Why leave t1 out of this rather sweeping assumption? The phrase "rounding back to ordinal" makes no sense to me. A rounded interval-scale random variable is simply rounded interval scale.
*"I would like to know what are the effects (repercussions for my model) of having only one replicate per environment in some of the cases"* - the question is not clear - effect on WHAT?
As a reader, I would find that a good way to represent changes in RR. Usually time would be in the horizontal axis, so I would be tempted as a reader to turn the page 90 deg when I'm reading it.
What's the treatment? (I don't see any mention of one).
Do you think you could provide a reference for your view that R² should only be used for model selection?
If you're running a Poisson regression, what difference does it make if the residuals are normal?
Appreciate the edit, thanks.
Glad you like it.  It was fun. :)  The only tricky part is getting the places where the bars should start and end (which is in the `getp` subfunction); the rest is just drawing polygons.
Definitely *not* an explanation to a layman - orthogonal basis vectors? maximising projection variance? constrained optimisation problem? Lagrange multiplier?  These are highly "jargonised" terms.  Show a layman who understands what these mean and i'll show you a mathematician/statistician
I'm not sure I follow all of this but the summed and non-summed AICs are on different scales (since the likelihood scales with the sample size, which is different in the summed and non-summed groups), so you can't compare them directly. You can only compare them when the data set remains fixed (so the likelihoods are on the same scale).
Dr. Hyndman, I am not sure if there was a change in the auto.arima function or not, but you can see in my answer that indeed the function picks up the downward trend. Did you mean to place d=0 in your last code snippet? That seems to actually produce an UPWARD trend.
More information will be required.
Because your data are not i.i.d., and the great maximum likelihood theorem about the equality of the inverse Hessian and the outer product of the gradient no longer holds, and neither of them is a consistent estimate of variances. One needs to use the sandwich variance estimator, aka linearization estimator in survey statistics, aka White robust estimator in econometrics.
"not correlated (r-squared 0.006)" means that they are not *linearly* correlated. Perhaps there is some other correlation involved. Have you plotted the raw data (dependent *vs* independent)?
Is this homework?  If so, please use the homework tag.
For SVM you can use the distance to the separating hyperplane as a measure of confidence. The further, the more confident. Points within the margin are dodgy classifications.
@andrewniesen Have you planned how to add newly added users to the queue?
I think you may need to implement gradient descent yourself in another language in cases where there is no good solution in R (for instance, some variation of regression, like regularized regression with big data)
@Michael "A table is nearly always better than a dumb pie chart; the only worse design than a pie chart is several of them ... pie charts should never be used."--Tufte.  "Data that can be shown by pie charts always can be shown by a dot chart. ... in the 1920's a battle raged on the pages of *JASA* about the relative merits of pie charts and divided bar charts ... both camps lose because other graphs perform far better than either divided bar charts or pie charts."--Cleveland. As you know, Cleveland is not prescriptive: this is as strong as he gets about anything.
Thanks, Dimitriy for that.
I should have pointed out, as @Quartz did in the question comments, that there are hardware PRNGs and some other specialized daughter-boards. In theory, you could even make/order an FPGA to do whatever you want, but that would only make sense if you had a TON of data and were doing exactly the same thing over and over and over again.
@propofol I don't understand the role of "historical probabilities" here.  My understanding is that the experiment is under your control and that you are randomizing using a random number generator; therefore, the probabilities are determined, not estimated.  Perhaps you could edit your question to elaborate on exactly how randomization is being used in this experiment.
I'm having the same problem. STL method should be able to just fill in the missing values thru LOESS, but this R implementation does not seems to do so.
Do you have repeated observations of the same units across time?
(+1) for "Let's be modest: the Euler equation is not "something we have in economics": it is a piece of mathematics that we use in economics."
I mean, in the table that you showed us, which of those variables is your dependent variable? Organization?
Were the past loans given by the same banks that are currently making the evaluation?  Do you have the borrowers credit scores from the time they received their previous loan, or only afterwards?
+1 Good advice overall.
Aren't you just asking for the Gaussian cutpoints needed to match the Poisson probabilities? That is, letting $F$ be the CDF of *any* continuous distribution--which could include any Gaussian you like--you are requiring $F(\\gamma_{k+1}) - F(\\gamma_k) = p(k|\\lambda)$, which immediately gives the full solution inductively. Given the length of your exposition I figure there must be something more going on here, but what is it?
Those categories are far from definitive.
Same as [this question](http://stats.stackexchange.com/q/38870/6633) which came from math.SE. Please ask the moderators to merge the two.
It shouldn't look like a normal because count data is not likely to have a normal shape.  Poisson and negative binomial are common shapes and can be highly skewed.  The counts could look approximately normal if they are sums of several iid count variables.
The latter. It's not just semantics. And, maybe. :)
@mkolar sorry about that, now corrected
I don't get why you dismiss an ordination (e.g., NMDS) for proportions (compositional data). You will get something similar with some typical transformations ("standardizations") of data matrices. I'm thinking of ecological abundance matrices (sites by species) where you make all sites "equally important" by dividing rows by row totals, i.e., sum of cells in each row = 1 (see default `wisconsin` transformation in vegan package in R as an example)
It seems like you are trying to find the standard deviation (variation) of the sampling distribution; it is more commonly known as the standard error.
How can you know that a sample statistic is close to a corresponding population statistic?  If you cannot (which seems to be the case), how can you "conclude" anything?
@OutputLogic which is great!  I'm not sure how much the second part of that question helps answer your question ... it is hard to explain how deterministic and non-deterministic processes interact.  Can you talk a bit more about what is confusing for you?  (I'm actually working on a paper that tries to explain this to psychology and CS people right now!)
I added some more information. I can't think of a reasonable way of  grouping the data. Grouping questions subjectively is not helpful since it does not help determine likely response. I can't cluster people by similar responses because the data is too sparse. I haven't used PLS - but will I not have the same problem with the input variables?
To generalize the first comment by @Elvis ever-so-slightly: Note that, for general df $\\nu$, if $T \\sim t_\\nu$, then $T^2 \\sim F_{1,\\nu}$ and so $T^{-2} \\sim F_{\\nu,1}$. Elvis' first comment is then recovered immediately via the symmetry of the $t$-distribution.
In case you are interested in SVM-based recursive feature elimination, see [this thread](http://stats.stackexchange.com/q/10676/930), for example.
As for spheric > euclidean: this does not hold if you work in the latitude/longitude domain. 1 degree of latitude does not equal 1 degree of longitude, they are *not* independant.
Yes, I know what it approaches as a limit; if we're going to be technical about it, it's actually the standardized number of heads that approaches a limiting distribution ... but in any case, that's not what you said before and the difference can matter. [By the way, if it had been me at the interview, my first response would likely have been 'define better in this context'. Much better to ask what might be a slightly dumb question, and be sure what question you're supposed to be answering.]
Consider including the factor $\\theta^{\\ r}$ within $h(x_1,x_2)$ (and don't forget the $r$). Also, you can *greatly* simplify the logarithms using the rule $\\log(a/b)=\\log(a)-\\log(b)$.
Thanks for your answer @jaradniemi, would you think a Cauchy prior would always avoid the particular situation given in the question?
@Luca *All* affine transformations of $\\mathbb{R}^n$ can be derived from the linear transformations of $\\mathbb{R}^{n+1}=\\{x_1,x_2,\\ldots,x_{n+1}\\}$ that preserve the hyperplane $x_{n+1}=1$. Few of those linear transformations will be "rotations," though: by definition, the matrix of a rotation is orthogonal.
no, it's only limited to two dimensional problems. There are many, much better alternative though and those generalize trivially to higher dimensions (search for robust regression here).
+1 (to you and Michael). Of interest, you don't even need [two observations](http://stats.stackexchange.com/a/1836/1036) to make inferences if willing to make a set of assumptions!
I don't think this should be closed; it is partly about  stata but also raises statistical issues.
A $p$-value refers to a statistical test, which in turn refers to (at least) a null hypothesis. Once we know which null hypothesis you want to test, we can start looking at which test you could perform, and then how to do that in your favourite program. So, what is your null hypothesis?
Maybe one approach is to ask the opposite question: When does the elastic net beat the lasso by a lot? In general, if there is a group of highly correlated features, then the lasso will just pick one of the features and ignore the others. The elastic net, conversely, will tend to use all these features a little bit; this can be desirable from a generalization point of view. On the other hand, if your design matrix X is close to orthogonal, then there may be less reason to use an elastic net over the lasso.
so the $\\Omega_i$'s are the weights? Something seems to be missing...i can make its bdp go to 0 by giving weight 0 to all except one outlier
Hey ACD I've changed the answer to reflect these two questions. The gist? You can use the same number of lags for the instrument (dated t-2 and back) *even if* you have a large number of lags in your regression equation - because the important thing is the number of lags of the *error* term, which is always going to be first-differenced. I also suggested two alternative estimators that use a larger set of moment conditions.
Certainly it's possible see `legend`
Glen, are cdf and simply density function both the same ?
Clarification is preferable, Seb.  I did not mean to suggest your answer is wrong or inferior, but only that to be generally useful it ought to explain the sense in which you have interpreted the original question.
At first blush, bootstrapping should be an easy way to do get such a CI. Do you need sample code to figure out how to do this in R?
No. You have to express the event in terms of $X$ only: note that $P(U\\leq u, U=X) = P(X \\leq u, X>\\frac12)$, and you easily have the value of this probability by considering separately the cases $u\\leq \\frac12$ and $u>\\frac12$.
@Clair.Gibbon Right!! I think I misread the name. I was referring to Glen_b. Thanks Glen!
I'm not entirely sure what your question really is... set $\\psi(\\theta, \\phi) = \\theta$  and $\\chi(\\theta, \\phi) = -\\phi$, as indicated in the lines beginning "The gamma distribution", and there you are...
There is also a post on math.stackexchange on this topic: http://math.stackexchange.com/questions/384003/l1-norm-and-l2-norm
@Brandon: if the package author cares enough to guide you, then they have given the answer in a form that will be picked up by citation("some_package")
A graph and everything... Very helpful! I'm new to this stuff; Just plunged into a job and expected to pick it up very fast. Thanks a ton for your help. Very clearly explained. :)
If you think that PCA is the way to reduce your data, it is hard to see why you do not consider Principal component 1 as the best single composite variable you can get.
(cont.) Third, regardless of the above, some software may use the *asymptotic* distribution and not the actual one, though at sample sizes of 10K this shouldn't matter too much (if ties had not been artificially induced on the data). Finally, we find the following rather strange statement near the end of that document: *Unfortunately, the properties of the KS-test in S-PLUS limit the work. The p-values for the present study were all compiled by hand over the multiple replications. A program is needed to calculate the p-values and make a judgment on them compared to the alpha level chosen.*
Oops, Sorry I wasnt clear. By return I dont mean PCA factor return.  I have those. I mean the return on NYSE index (returns data was used as variables in PCA originally) It's a factor model on equity index, where the factors are not specific like "p/e ratio" but rather statistical/unknown/pc1 and pc2.
And the data is where ?
what is the mean or median?
Could anyone explain why this was downvoted, it seems like a reasonable question to me.
Beware that the analyst is hiding steps (e.g., feature selection) from the LOO-CV.  That would make all these indexes biased.  Also I'm not convinced the LOO-CV pays the full penalty for feature selection because any two samples have an overlap of $n-2$ observations.
@AndyW `.` means all regressors, i want no regressors.
What do you mean by 'rank them'?
@csgillepsie: there is catch here. The answers about comparing distributions assume the draws from the distributions are iid. From the tone of the questions (.."mapped to line chart") this may not be the case in Tristan's application. That would invalidate all the solutions (k-s, chi-2,...) proposed.
As the logrank test is a special case of the Cox proportional hazards model, it is unclear why you are using it.  And thinking of age > 65 as a point of discontinuity in  a (highly likely to be smooth) age effect does not make sense.  There is huge heterogeneity of age within the age > 65 group, and likewise for age <= 65.
(+1) I was pretty sure that you will answer your favorite topic :) @IrishStat, may be you have also your preferred reviews (just for those who has no time to dig in search engines).
This is a worthwhile perspective, but note that there are frequentist approaches that do not rely so much on asymptotics, such as Monte Carlo simulations, bootstrapping, and permutation tests.
Ah I see exactly what you mean! Thank goodness for insight and clear explanation! So imagine that the 101st Dr has just entered the room and I would like to predict his/her next surface contact. Would this be: P(Table)= (the total # of contacts with table)/(total # of contacts made) ?
+1. Binning into classes is an ancient method. While histograms can be useful, modern statistical software makes it easy as well as advisable to fit distributions to the raw data. Binning just throws away detail that is crucial in determining which distributions are plausible. In particular, about 2/3 of your values fall in your lowest bin. (Have to add a mention of my own personal choice of program, `Stata`.)