tttjjj/elt_bench huggingface dataset:
@Eric This isn't my area of expertise so caveat emptor.  But I would go about this by first asking what level of precision is necessary.  I would look at pairs of evaluations and calculate the mean and median divergence between raters (in seconds).  If it's small then that tells you in intuitive units (seconds or fractions of a second) how far apart your raters are.  Then it's a subjective call relative to your research question.
The t-test is failing to reject the null hypothesis that your model is unbiased.  This is not the same as saying your model is appropriate, after all, the sample mean of the $y_i$ is unbiased (no need to run a t-test to test that hypothesis!) and is a different model, almost certainly, than your regression tree ended up with. The negative $R^2$ means that your model's predictive variance is less than that of the sample mean of... what?  The OOS data or the 400 in-sample data points?
It's the `subject` variable, and has the values 101X, 104X, etc.
What question would such a split answer?
Be sure to repeat the entire process 100 times to achieve satisfactory precision.
The last answer of $654/1536$ is too low. For a brute force (but illuminating) calculation, try `dice <- expand.grid(1:4, 1:6, 1:8, 1:8);
dim(subset(dice, subset=(Var1==1 | Var2==1 | Var3==1 | Var4==1)))[1]` and inspect the contents of `dice` afterwards: it shows all $1536$ possible outcomes.
(+1) Welcome to the site, Sven. First, you might note that $X-Y$ and $X+Y$ are actually iid $\\mathcal N(0,2\\sigma^2)$ random variables and $\\exp z$ is a monotonic function, so your problem reduces to finding tail bounds on $\\beta \\sigma^2 Z_1^2 / \\sqrt{2} + \\beta \\sigma Z_2$ where $Z_1$ and $Z_2$ are iid standard normal. (Here $\\beta = \\alpha / \\sqrt{2}$ and $Z_1^2$ is, of course, a $\\chi^2$ random variable with one degree of freedom, independent of $Z_2$.)
Thank you so much. I was actually trying to use the aov function and your response was of big help.
I think temperature is a bad example, because for temperature in K it is sensible to say that 20 K is twice as much 10 K. The thing with °C, °F, °R etc is that their 0 is arbitrary in the sense that there is no hard physical reason to put the 0 exactly there. Whereas 0 K is "special" from a physical point of view, and there are reasons why the 0 should actually be at 0 K.
@ntamjoachille: You could accept the answer if you think it was helpful for you.
@Denis:> 'Clumpy'=?concentrated=?with narrower tails than the gaussian?
*Mathematica*'s help suggests BetaRegularized is evaluated by quadrature, not exactly. It can be important to be aware of and understand the difference, especially from a computing standpoint, so I think this is more than just a "matter of definition." Regardless, in the present context it would seem that "evaluate an incomplete Beta integral" was not what the OP meant by "missing some formula," because it's unlikely to be any more helpful than telling him to "evaluate the CDF of the F distribution" (which is commoner in software than incomplete Beta integrals, although they are equivalent).
@Jehol updated answer. hope useful.
I'm not sure if this helps you or not, but you're taking a [convex combination](http://en.wikipedia.org/wiki/Convex_combination) of the $x_i$.
How would I use Lagrange beyond 1 input variable? Is there a general formula for Lagrange for m input variables?
Let $f$ and $g$ be functions such that $f(x)+g(y)=ax+by+c$ for all $x$ and $y$ and some constants $a, b,$ and $c$: that is what it means for the sum to be linear. Because the rhs is differentiable in $x$ and $y$, we can differentiate the lhs, giving $f'(x)=a$ and $g'(y)=b$. Integrating yields $f(x)=ax+d$ and $g(y)=by+e$ for constants $d$ and $e$.  (Comparing to the rhs shows $d+e=c$.)  Therefore *whether $f$ and $g$ are assumed to be polynomials or any kind of function at all,* they must be linear if their sum is linear.
Do you mean you want standardized parameter estimates? library(QuantPsyc)
lm.beta(mod)
If you are interested in predicting variations within a day, you need to model them. If you are not, then aggregate to daily values. What's missing from this question is quite what you want to do, based on what you consider interesting or useful for later applications. There can't be a recommendation for what to do independent of that.
Great! Now let's discuss the substance of your answer. I do not see how "continuous" = "interval/ratio." In fact, that point of view seems counterproductive, because it inhibits people from looking for nonlinear re-expressions of their "continuous" data. Next, "discrete" is a huge category. Distinguished within it are *counted* data, which have an inherent meaning and statistical properties, but otherwise there seems to be no universal equation of the form "discrete" = "interval/ratio" or anything like it. The whole nominal/ordered/interval/ratio classification is really not much help.
Thanks for making some sense of this.  BTW, I see that you have been here only two weeks and have been quite productive in contributing thoughtful answers.  Welcome to our site!
The approach of http://stats.stackexchange.com/questions/34017/confidence-intervals-around-a-centroid-with-modified-gower-similarity but instead using `cmdscale {vegan}` for PCoA and manually calculating the centroids gives the expected plots.
@DLDahly your point in (1) is not something I often find in biostatistical panel analyses. One of the assumptions behind AR-N correlation structures is that, given enough time between them, two measurements on the same individual will be as uncorrelated as two measurements between different individuals. However, the underlying major between-cluster confounders are often not time varying covariates (such as genetic markers), and assuming otherwise is very difficult (if not impossible) to assess. A lorrelogram is a very good place to start, though.
I don't know what models or methods psychometricians use or what they call them, but this is a common problem in survival analysis and also in economic data where things like income are often "topcoded" (there's a category like "makes over $180,000/year").  You want censored regression.  Look, for example, at my answer to this question: http://stats.stackexchange.com/questions/83047/modelling-when-the-dependent-variable-has-a-cut-off/83059#83059
It seems that you treat your tabular data as one long time series. This is a very grave mistake. You cannot simply ignore the structure of data.
You may want to clarify the context of your question.  What kind of standardization, what kind of studentization?  What are these values being used for?
Sorry did not specify this but my target variable is not binary it is continuous (modeling cost), most of my independent variables are binary
A related question is discussed at http://stats.stackexchange.com/questions/65640.  It explains the use of the RMS mentioned by @glen_b.
Thanks a lot for the insights. It looks like ordinary ridge regression is the way to go for my application. I've posted a follow-on question about choosing a good value for $\\delta$ (http://stats.stackexchange.com/questions/9467/regularized-fit-from-summarized-data-choosing-the-parameter)
Please articulate the need to 'classify'.  It is the only the desire to classify that makes this imbalance an issue.  If you stick to probability estimation there is no problem.
Answer to a deleted comment: If you have 4 mutually independent questions, each with a 50/50 split in the population, that is the _best_ you can do, and it will be at least as good as any 4-deep question tree (with binary responses.) This is actually mathematically trivial since any decision tree can be encoded as a (complicated) equivalent 4-item questionnaire.  Example:  Question 1: Do you support A? T/F.  Question 2:  (Do you support B AND did you answer T in question 1) OR (Do you support C AND did you answer F in question 1) etc.
+1 Thank you for your answer
@user41569 As we have the same answers I have added the derivations rather than just giving hints + the answers.
@onestop That is correct.  Indeed, the Wilcoxon test *is* a permutation test.  All we're discussing is how to compute (or estimate) the distribution of the test statistic.  The distribution can be computed exactly using combinatorial techniques; it can be estimated through simulation; and it can be approximated, usually with a Normal approximation when the amount of data is large enough (m+n of 10 or greater is usually good enough) and the data can be thought of as realizations of a continuous RV.
(+1) for mutual information. Additional remark: a) I suggest Information Gain as special case of mutual information. b) Automatic feature selection will not only remove the redundant but also all features which have a negative impact on class discrimination.
Although we could migrate this to SO where the `R` community likely could give you a good answer to the *programming* issue quickly, I would like to suggest instead that you edit your question to ask whether this approach is appropriate and how one might correctly perform a regression with these kinds of responses.
@James: Alternatively, this might argue for more education in the field, not less...
Do you only have two leafs in total - one control and one treatment?  If so, it is not going to be possible to separate out the effect of the treatment from individual leaf randomness.
Again, what do you mean by "important"? I don't think it's likely a meaningful concept for clusters.
Is there any problem rewriting your model in terms of $\\alpha_0=\\beta_0^2$ and $\\alpha_1 = \\beta_0 \\beta_1$, which *is* linear?
I agree, this is more appropriate for Cross Validate. But a word of advice; you need to write a question that provides a reproducible example. Nobody knows what "emp" looks like making it is difficult to provide relevant advice.
Before we can talk about unresolved problems in statistics, we need to define statistics. Dimitriy Masterov gave an answer involving econometrics, and Aksakal, involving data science. In mathematics, Hilbert's problems were compiled at a time when there were may be what, 100? 200? top mathematicians in the whole world, and most of them would agree that yes, each of the 23 problems is both an important one to solve, and a cute one to have on one's resume. These days, there are way more statisticians, and they are too busy to coordinate.
It would help to say more about those "feature functions."  If, for instance, they form a mixture distribution, then the sampling process is reduced to choosing one of those functions randomly and then sampling from it.  This could be hugely more efficient than attempting to sample from the density using, say, rejection sampling.
Given the problems with retrospective power (see my answer) I haven't given much thought to the aspects of the question pertaining interactions but I do have a question: Do your factors represent experimental manipulations or measured/observed variables?
Yes, spot on. Exactly what I meant. You gave me a direction to keep going. Thanks.
i would have thought a good approach would be to find the largest term in the hypergeometric sum and truncate around this term.  this way you can factor out the largest term and work with terms less than 1 in the sum.  you could also use laplace method on the integral representation (suitably parameterised to avoid boundary issues).
One goes from the first expression to the second by letting $\\frac{n_r^i}{n_r}=p(x_i)$, noting that $\\log_q(z) = \\log(z)/log(q)$, interpreting the $\\log$ as $\\log_q$, and inserting the minus sign that should be there.
To quote xkcd 552, "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'."
Can you explain what your data set looks like and what your model is a little better? Do you have 20 observations with seven variables, or 140 observations with some sort of correlation structure?
A useful fact given in these references is that non-overlapping 84% confidence intervals *do* approximate a 95% level test.
In those cases small departures will be detected.  As long as the analyst recognizes that in practice the population distribution will not be exactly normal and rejecting the null hpyothesis is just telling him that his distribution is slightly non-normal, he won't go astray.  The investigator should then judge for himself whether the assumption of normality is a concern or not given the slight departure that the test detects.  Shapiro-Wilk is actually one of the more powerful tests against the hypothesis of normality.
THX a lot for your answer! How do you conclude that $\\alpha$ is $\\beta_0$?
This post seems to have stopped before it reached an answer to the question.  Did the last part somehow get lost?
Why are you doing PCA rather than FA? And, if you are trying to confirm the number of factors, why not do confirmatory factor analysis?

If you actually do PCA, then you get principal components.

Yes, any of the things you mention could happen.
Are you interested in all methods of data mining, or just one method in particular?
@Michelle I reread your first question and I think I answered a different question. You were talking about the second method described in paper, right? I think the authors of the papers just mean a y-intercept, not an intercept with a value of zero.
@steffen Yes, how would you do this in R? One thought I have is to normalize all the factors and multiply each by its appropriate weight. But I'm not sure whether dist() already does the normalization. If so, then I would need to override it, I suppose.
You are not looking for a probability distribution (for instance, which is the probability that you sample all N cells taking y cells after growth?), but asking for the smallest y which gives you at least one of each N cells. In this case (N-1)x+1 cells would be the smallest value you need to take. If this is too expensive, you could define a threshold and look for the minimum number of cells such that the probability you are missing one of the N cells is smaller than your threshold...
Here is the link to implementation in R (http://www-etud.iro.umontreal.ca/~botev/kde.R). I understand your question now. Mine is why you want to do that ? I would perform a MC as described to get an estimate of the error of the kde, but not to calculate a kde for NB, which uses maximum likelihood estimates of all variables per definition ! I think that performing the algorithm as described by you and calculating the kde directly from the training data (without any subsampling) will not affect the result of NB.
@ttnphns- I have to reduce dependent variables in SPSS as  e.g my DV is business development and for it I asked 8 likert scale questions such as increase in sales, profit, size, labour.....If I will not reduce it, it means there is eight models of increase in sales, profit, size....and no model of business development.
@Macro I will check it out thanks!
@MaartenvanStrien I like the explanation in the regression book by Draper and Smith. I think they give a very understandable explanation which involves graphics and not just statistical jargon.
Exactly right. Aggregating prematurely is likely to obscure important relationships. If meat prices are going up in concert, is it because energy or transportation prices went up in prior intervals? I see no mention of methods for examining auto-correlation or cross-correlation which would clearly be needed for a proper treatment.
It has nothing to do with identifiability.
Referring to your link, look under the "Classical CLT" heading or the Lindeberg-Levy CLT. This is the situation you describe. No additional moment finiteness is needed beyond the second moment. :) The Lyapunov variant drops the *identical distribution* assumption. Since we give something up, we have to add back some structure to retain the result. One way to do this is to require a slightly higher moment (though this is just a special case of full Lindeberg-Feller). :)
the difference between procedures is that one assume same variance for both groups.  The separate analysis assumes different variances.
(+1) but I'm not sure how applicable these are for me. The distributions $F$ and $G$ are both **highly** multimodal, and moreover are supported on a very nonlinear set. I can make no distributional assumptions at all, and I am looking for more than a difference of means so I'm not sure distance-based non-parametric tests are applicable either.
(+1) Nice, Dilip!
Are those 20% all lumped around taking 50%? Or are they spread out from 1% to 99%?
likelihood ratio chi square is a better choice than pearsons chi square when cell counts are small.  you take $2O_i\\log\\left(\\frac{O_i}{E_i}\\right)$ instead of $\\frac{(O_i-E_i)^2}{E_i}$.  much less sensitive for small $E_i$
A t distribution might not be a good choice, because it is symmetric whereas asset returns tend to have strong skew.  At a minimum, consider modeling the *logarithms* of the returns rather than the returns themselves.
Oh, snap. Yeah, that makes perfect sense. How would I standardize the dependent variable to make the coefficients more comparable?
$w^*$ certainly only has one value in reality. I'm looking for what I can say about its possible values based on the measurement(s) taken though, which I guess is a Bayesian perspective. The finite nature of the beach and the sampling without replacement aren't really big deals. They are true of my real problem, but an approach that assumed them away would still be useful.
It depends. If there is only one reason a feature could be missing, then you could just use your software's missing-value code for those instances. Then if an analysis requires oldest son's age, you use only the non-missing values (and the analysis only applies to that sub population), but if you're analyzing whether or not they have a son, you count the number of missing and non-missing values (possibly for each value of some other factors). But if you are concerned about why they are missing, you need another variable in the dataset with a code for why it is missing.
thanks you so much once again. will be sharing my answer based on your method.
It seems the question about ratio-level comparisons still hasn't been answered.  Even if we know that AUC is, say, .6 using just x1 and .9 using just x2, we can hardly say that x2's importance is therefore 50% greater.  Nor, I think, that it's (1 - 10%/40%) = 75% greater.  Nor can we do something analogous using just sensitivity or just specificity.  I also have doubts about the Wald statistic's applicability here.  Most helpful might be explanations of standardized coefficients (see Scott Menard's online book).
This is a variant on [Correlation between discrete and continuous variables](http://stats.stackexchange.com/questions/62741/correlation-between-discrete-and-continuous-variables). It is much better (a) to pay attention to comments already given (if appropriate, explain why they are wrong, irrelevant or otherwise not what you want) (b) not to spawn multiple versions of the same question.
Actually the model you are describing is exactly the same as the one described in the wikipedia entry since $\\sigma^2 = \\beta^{-1}$ (recall that if $X$ is gamma distributed, then $1/X$ is inverse-gamma distributed). You are then using the prior $p({\\bf w},\\beta)=p({\\bf w}\\vert\\beta)p(\\beta)$, which is a hierarchical prior. The posterior is $p({\\bf w},\\beta \\vert {\\bf t}) = N(...)p({\\bf w},\\beta)$. Then, yes, it is analogous to the normal-gamma case.
Thanks to all for helping me!!!
@Anony-Mousse: I have treated the ordinal data (1: strongly agree, ..., 5: strongly disagree) as intervall data and then performed the pca. Afterwards I have done a cluster analysis (k-means) on the raw data as well as on four factors. Both results were quite similar.
Can you not algebraically simplify your function? To: $Y=\\frac{IC_{50}M + BX}{IC_{50} + X}$
Hi Russ, it does not work in lme4 either: > err <- data1.frame$peak_Mid - predict(lmer50)
Warning message:
In data1.frame$peak_Mid - predict(lmer50) :
  longer object length is not a multiple of shorter object length
well, to be so simple as to be facile, because the sum of 3 uniforms has quadratic segments in its p.f., and once you get two or more uniforms you have a peak at the mean. A quadratic peak is "smooth"... and the joins between quadratic pieces are at 1 and 2, so it *can't* kink at 1.5 ; there are other ways of arriving at the same conclusion
$S$, $A^c$, $A_1 \\cup \\ldots \\cup A_n$ are subsets of $S$, don't you agree ?
+1 for showing just how easy it is to do that way (much easier than what I did, though as expected, the final result doesn't yield any further simplification of the probability mass function).
@Harokitty It's briefly described in Agresti's "Analysis of Ordinal Categorical Data", [section 3.2.2, p49, equation 3.8](http://books.google.com/books?id=VVIe4BPDR7kC&lpg=PP1&hl=de&pg=PA49#v=onepage&q&f=false). Alternatively in Agresti's "Categorical Data Analysis", section 9.4, p323, equation 9.12.
Thank you for your answer. The intuition you mentioned actually makes sense. I would consider those cases you mentioned as more significant. But is there a way to express this idea more mathematically rigorously?
@cardinal , even without a formal definition, it would be possible for a small number of people to have an appreciable impact on voting patterns (which is what I assume whuber is referring to in this context). Jon Skeet's average post was only 5 something upvotes. If all of a sudden one person decides to upvote all of his answers, that could have a pretty substantial impact given the low average score to begin with.
One problem is likely to be when an MLE is on the boundary of the parameter space.
+1, that was a nice account of it. I changed `B^-1 W` to `W^-1 B`. Your picture with discriminants as axes is similar to my own [plot](http://stats.stackexchange.com/a/22889/3277);I believe that you used the same "nonorthogonal rotatation according to the normalized eigenvectors" approach.
What you're attempting to do there is fit a *mixture* of two lognormals by taking logs and fitting a mixture of two normals. This is fine. I'd say it's actually the "right" way to do it.
@Lynn Do you mean that you want to know the % of variance ($R^2$) in your DV explained by all predictors in your final model, or during model selection?
@JeromyAnglim [The study of this paper](http://home.cc.umanitoba.ca/~kesel/cis_semipart_akp_aug_2007.pdf) seems to be close to what you're looking for (with random predictors).
I would recommend the same thing at such low sample size. It might not be entirely rational, but I just disregard almost any statistical test running on less than 30 observations.
(last comment), thanks for the pointer to R.  I tried it without too much luck, but was able to get something very similar in Excel using transparencies in the data points, so areas of higher concentration are darker/lower concentration are lighter.
OK, so it seems that the graph tells you quite a lot. There is an overall pattern and a striking exception you can interpret. You win both ways.
Actually I think you've answered it for me.
Thank you for the answer. With this procedure i can't access the order of the terms in a document. The document term matrix is just a bag of words approach. What i'm doing is the extraction of all n-gram statistics up to tri-grams. I could try to process the documents 3 times for all the n-gram types. But i thougt there might be a better way to produce n-gram language models.
You mean a specific computational solution?  Grid searches are a standard methodological tool.  A quick search showed several instantiations: http://exposurescience.org/heR.doc/library/heR.Misc/html/optgrid.html http://rss.acs.unt.edu/Rdoc/library/R.basic/html/gridOptim.html ... but there were many more.
Here is the head(data)