tttjjj/elt_bench huggingface dataset:
To spell it out, the main pattern here is curvature, indicating a distribution that is positively skewed. The two highest points are broadly consistent with that.
By the way, there was an answer here as well, which I even replied to, but now it's gone. If I have accidentally deleted it I'm sorry.
Thanks "user".  This doesn't answer my question - but it is a valuable reference for me - thank you.
In some of your answers, @January, you seem to abuse acronyms _without_ explaining them. That is not a good practice because many statistical abbreviations are ambiguous (e.g., GLM = 'general linear model' or 'generalized linear model'?) and because not every one knows a given acronym which you knows.
+1 I like this idea but am surprised it works *so* well with the example data (from an exponential distribution). Do you have some intuition as to why that is? In some sense it does *great* near $0$ but it is missing the "lump" of values near $1$ in the actual data, so I wonder whether there's not some kind of trade-off between good accuracy at low values and low accuracy (or, equivalently, large bandwidths) at high values.
Have you tried making a 11-class classification?
@whuber Correct. @harvey-motulsky A negative R^2 value **is** a mathematical impossibility (and suggests a computer bug) for regular OLS regression (with an intercept). This is what the 'REGRESSION' command does and what the original poster is asking about. Also, for OLS regression, R^2 **is** the squared correlation between the predicted and the observed values. Hence, it must be non-negative. For simple OLS regression with one predictor, this is equivalent to the squared correlation between the predictor and the dependent variable -- again, this must be non-negative.
Are you asking if it makes statistical sense to do this, or how to write the code to do it? If you're asking the latter, please show that you have answered the relevant questions from the [Stack Overflow question checklist](http://meta.stackexchange.com/questions/156810/stack-overflow-question-checklist).
This is true and close to the point I was trying to draw out. The first sentence of the last paragraph is referring to constructing an unbiased estimate of a nonlinear statistical functional from, e.g., a single random sample. This is quite different from constructing an unbiased estimate of a full distribution from a random sample of the function itself. :-)
An anonymous user (who likely does not have the rep to create comments) has suggested www.metcardio.org as "a very interesting website."  It appears legitimate, but is heavily focused on medical applications.
I am not sure but I think it is possible to interpret it as a stocastic gradient descent optimization procedure. I'll think about that...
Also, Did you mean to say "Only if you think of 'effect' as inherently nonlinear instead of linear?
What I meant to ask was. I read somewhere that "with non-linearity, the effect of X on Y depends on the value of X and X somehow interacts with itself". Does this mean that X interacts with itself(X)? or does it mean that X interact with other variables(X,W  etc.) if any?
Although this is what I also usually cite (+1), I am not sure whether or not one needs to revise this recommendation by one decimal place, given the recent recommendation of Valen Johnson in [PNAS](http://www.pnas.org/content/110/48/19313.short): "Make 0.005 the default level of significance [...]. Associate highly significant test results with P values that are less than 0.001."
Is this another reason for not binning that @FrankHarrell can add to his [list](http://biostat.mc.vanderbilt.edu/wiki/Main/CatContinuous) ?
Good question, although it looks like your links to the other threads on this site are missing!
In the $p >> n$ case, LASSO selects at most $n$ of the $p$ predictors. Depending on your goal, this might be undesirable. In stepwise regression, usage of AIC or BIC minimization as selection criterion has a one-to-one correspondence to particular p-values, depending on the quantiles of the chi-squared distribution. For example, AIC minimization corresponds to a p-value of 0.157 for variables with one degree of freedom.
It's a bit unusual to determine if kappa is statistically significant, because that just tells you if it's different from zero. Usually you want kappa to be large (ish), not just larger than zero.
Most statistical methods assume normality, not of the data, but rather of an assumed random variable, e.g. the error term in a linear regression. Checking involves looking at the residuals, not the original data!
I just discovered that this result is known as Samuelson's inequality: http://en.wikipedia.org/wiki/Samuelson's_inequality.
I recommend leaving one out for interpretation, unless the constant is excluded. Consider the case where you have a single binary indicator representing two groups (one treated, one not treated). In OLS, the parameter estimate yields the interpretation that the effect of the binary on y is the point estimate when the indicator is set to unity. So, when the binary  is set to zero the prediction would only be the intercept. To have a variable representing both groups, the model would have to remove the intercept. If OP does remove the intercept, then he/she can put all of them in.
Ah, the assumption of equal variances is the *population* variances not the *sample* variances. You can infer that the population variances are equal... via eyeball test, or by some other, say, statistical test.
Right, more certain estimates should be given more weight. I suspect there are means to render this notion in the bayesian framework by playing with the risk functions. From an application perspective, you can naively incorporate such a notion by discarding (or down-weighting) items with less than a pre-defined number of reviews. You can also incorporate other elements like date of reviews, reviewers profiles etc. At a certain point you might want to switch to a [Learning to Rank](https://en.wikipedia.org/wiki/Learning_to_rank) approach to combine all those elements.
@user1738154, to *center* a variable you subtract the mean of the variable from each value. Thus the mean of the new variable is **0**, but the SD is the same as before.
+1 for C++; while it is quite easy to embed C/C++ to R I often wrap my codes and run them inside R -- then it is nicer to pass parameters, do live visualisation and obviously analyse results without thinking of output file format.
I do not see an easy way to link the covariance and the Inclusion-exclusion principle. I do not think there is a connection.
A related term to search for is intermediate outcome for bullet 1. [See here for related discussion](http://stats.stackexchange.com/a/19615/1036). Bullet 2 is a bad idea as well, for related (although more obvious) reasons.
@user23599 That student might have used three degrees of freedom instead of four. It's not possible to determine from your question whether that is correct or not.  BTW, a good way to avoid questions you fear might be stupid is to [do a quick search first](https://www.google.com/search?q=distribution+definition+statistics).
@Bogdan, please convert your comment to answer.
@Dbr In that case you should edit the question to make it accurate/correct. You are not using all the information that has been given to you. What about the information on the earnings of someone who is 66 inches?
@Pegah: I am confused. It is easy to prove that $X^TX$ is positive (semi-)definite for any $X$. Are you alluding to some numerical issues?
Could you show the code of what you did?
You can also use a formula interface for glmnet through the caret package.
Er... I'm not familiar with real stats by a long shot... but isn't "exposing all hidden variables" by definition impossible? How do you know when there's no more "hidden" variables?
+1 good question.
@Leon-Alph Don't forget to mark the answer as accepted if it answers your question.
@nikie there are principles, such as MaxEnt and transformation groups for defining priors in an objective way.  In that case the computation gives the same answer regardless of who or what is performing the calculation (a robot would come to the same conclusion).  Frequentist statistics is not without subjectivity either, for instance how to set $\\alpha$ for hypothesis tests, but the intention there was to remove the subjectivity from statistics in the sense of "personal probability/belief" rather than "degree of plausibility" (IIRC).
@Glen_b `glht` is in the `multcomp` package. @anne: Please post at least the call of the `glm`. It seems as if the variable `treatment` was not in the modell. But without more specific details, we can only speculate.
(1) This sounds a lot like [routine bookwork](http://stats.stackexchange.com/tags/self-study/info), as might be set for an assignment. (How does this question arise?) $\\quad$ (2) You might find the discussion (and the diagram) [here](http://en.wikipedia.org/wiki/Simpson%27s_paradox) useful.
@AhmetYılmaz Nothing says "Thanks" like an upvote.
This is known as the "change of support" problem: [Google it](https://www.google.com/search?q=change+of+support+geostatistics) for some good introductory references.
Hmm, I don't think that's what I am doing. Each "scenario" is different. For example, I am looking at about 300 genes, which I need to rank by the frequency with which they are mutated in samples. Gene A is let's say mutated 9/10 times, while Gene B is mutation 99/100 times. Thus, Gene B is ranked higher in terms of mutations that Gene A. Would this still apply? @MLaz
I'm not sure why you would care about whether or not the eigenvalues are equal. There is a lot of research on identifying what are the most important eigenvalues. For instance, the literature on Random Matrix Theory looks at the eigenvalues of the covariance matrix as following the Marcenko-Pastur distribution and frequency identifies the ones beyond what random matrices' eigenvalues would look like as the significant ones.
@Srikant-vadali: Good point. I suppose I should add that I did a PhD in Stoc Proc and have found it very helpful in my new field of systems biology.
@AlexWoolford I edited my response to include Stochastic Gradient Boosting. It's a similar model, but tends to perform better on regression tasks. The common implementations of SGB also include confidence interval tooling.
Note that you don't actually know $A$ and $B$, only their sum, which isn't sufficient to calculate $A^2 + B^2$ in your last line.
you sound strange. "Two way" ANOVA or ANCOVA needs 2 independent categorical vars ("factors") by definition. Your data is one-way ANCOVA. Next, because your predictant is only 2 levels (fixed, I assume) you'd better go to Binary Logistic regression procedure instead of GLM.
This seems conceptually different than the answer above.  I'm still not clear as to how transforming my variables to PCA coefficients allows me to regress on 2+ dependent variables?
I cannot speak to the prerequisites, but I do know Michael Lavine. He is a wonderful lecturer, able to convey advanced ideas clearly to undergraduate audiences. If you're comfortable with the mathematical material in the beginning (e.g., the statement and proof of the first theorem at p. 12), then you should feel encouraged. Otherwise, if you find this notation alien and unfamiliar, you will need more of a mathematical background to read it (at the level of an "introduction to modern math" course typically offered to sophomores considering a math major).
Perfect fit is usually an easily detectable corner case. Hence not all of the statistical software checks for that. See for example the help page of nls function. It specifically states that it should not be used with perfect fit data.
Welcome to the site @user2488177! I edited your post a little bit (under review) but there are still something unclear. For the following part, the rule `b1=b1+w,b2=b2+w` doesn't work, please revise it. "If TRUE, say for `b1[2:4],b2[5:7]`, then a change point is recored
   (C1), then `b1=b1+w,b2=b2+w` (i.e. `b1[5:7],b2[10:12]`). For the new
   b1 and b2, if TRUE again, then another change point is recored (C2),
   and b1 and b2 slide by w (`b1[10:12],b2[13:15])`; say TRUE again, then C3
   recored, then `b1[13:15],b2[16:18]`, and TRUE again then C4 recorded and
   b1, b2 slide by w."
You should get some samples from the dataset. For example, separate 10000 rows and test some techniques on it before using them on the whole data. Regarding the methods, you probably don't need anything "fancy" right now, just getting the correlation matrix should be enough to start. Also, it isn't a machine learning problem by some definitions (that means: don't worry about it too much).
It is an assumption.  I used the term mimimal to say that i am making as few assumptions as possible to reach a definitive answer.
Please forgive the sensitivity of a biostatistics professor but if I wanted to build a house I would study carpentry first.
(1) Presumably $f$ is a bivariate probability density function. (2) The question asks for a probability, which (by definition) is a *number.* The answer you propose depends on some undefined variable "$x$": this makes no sense. (3) The answer is fairly nasty: it will help to write it in terms of the number $x$ satisfying $x = \\exp(-x)$. If this is a probability homework problem, it seems to miss the pedagogical mark by requiring the evaluation of such an integral. Are you sure you copied it correctly?
It depends on what you mean by "difference as a percentage value" -- percentage of what, exactly? That is, what is it you want to measure? And, very much relatedly, what is it about the problem that makes it necessary that the error stay between 0 and 1 (100%)? Is the 'target' some kind of gold standard you want to measure the deviation from, or is it that you want to measure how far the target is relative to the actual? (And if you want *either* of those, how does that fit with the 0-1 restriction?)
That plot is horrible! Some smoothing, as in earlier answers, are necessary.
What is your research question? Whether you group or not will be affected by the question. What is your independent variable?
+1 @fmark I'm glad you started a new question.  I think this is the right format for your questions to be discussed to your satisfaction.  Comments just aren't enough.
Think about what the parameter estimates mean in a linear regression. Why do you suppose one is called "intercept"?
Done! Hopefully it makes some sense in the context of what I want to do now...
@Chris one thing you can do for any method, is to test the performance (in cross-validation) for subsets of variables and conclude about the importance of variables from that (for example, leave one variable out on each run. This is not necessarily comprehensive, but it is often useful in practice and is easy to do for small datasets and simple models.
Thanks for your response. Yes the variables at hand are independent variables and it slipped my mind that the two model that you showed are so similar.
How are you going to find the probability distribution of Y given X? How does what you did / found differ from 'having a model'?
Whether higher or lower is better depends on the implementation. In `SAS` I believe lower is better (it says which is better on the printout). In `R`, as well, a lower BIC is better. From the `R` documentation for AIC: "When comparing fitted objects, the smaller the AIC or BIC, the better the fit. "
Search our site for [delta method](http://stats.stackexchange.com/search?tab=relevance&q=delta%20method).
+1 Heh, good idea.
BFGS uses an approximation of the information, and it is scaling using that. Newton-Raphson scales using the information. In these cases, the gradient is multiplied by the inverse of the information (or an approximation to it).
+1: You got the correct answer.
what dimensional space are you requesting? have you tried it for smaller k?
@jbarlow: Sorry if my previous remark was unclear. If $p(n) = k n$ for some $0 < k < 1$, then there is no way your process could have more than $\\lceil k^{-1} \\rceil$ steps since a uniform random number between zero and one would definitely be smaller than $p(n)$ for any $n > 1/k$. The quantity $p(n)$ as a function of $n$ is very closely related to what is called the *hazard function* in the subfield of statistics known as *survival analysis*.
In your simple situation (one X) consulting with r is justified. In general situation (several Xs) the answer is no.
Great thanks for your answer, I will post another question.
Could you expand on the third point? - non-linearity before non-additivity.
Thank you! I see what you mean with a bimodal distribution.  Follow up question: What if the variances of the data are different, (heteroscedasticity?)  Say.. all jockeys are small right, but the heights of basketball players ranges a lot.  Maybe for them, tall <- rnorm(100,78,10).  How does a situation like this change your assumptions on either $y_i$ or $\\epsilon_i$?
Thank you for your help =)
An illustrated worked example of one possible method to identify effective nonlinear re-expressions of regression variables is provided at http://stats.stackexchange.com/a/35717.
My only thought is to ask a subject matter expert what the important predictors of the response variable are likely to be, without telling the expert about the differences you found, then adjust for these predictors.
(+1) Your first sentence reminds me of [World’s Greatest Pie Chart](http://zoomdoggle.com/2008/09/worlds-greatest-pie-chart/) :-)
Likelihood is not the same as probability. The probability of getting $k$ Heads in $10$ independent tosses of a coin that turns up Heads with probability $p$ is $\\binom{10}{k}p^k(1-p)^{10-k}$. The _likelihood function_ of $p$ is $\\binom{10}{k}p^k(1-p)^{10-k}$ where $p$ is the variable and $k$ is the number of Heads that have been observed; that is, the data.  In your case, the likelihood function of $p$ is $p^{10}, 0\\leq p \\leq 1$, which has a maximum value at $p=1$. To estimate $p$ via Bayes' theorem, you need to describe what you know about $p$. Are the other $9,999$ coins fair?
@Adam, Could you give a reference for the studies showing problems with the LRT ?
I'm sorry if I offended you. Of course it is not a matter of faith that is why I edited my question with what I believed was counterexample while still stating "please correct me if I'm wrong :)" because I'm completely open to being wrong.
Hi Andy. I've updated my question to include an example. Hope it's clearer now.
The last remark makes it sound like you contemplate the alphas differing among the distributions. Are you going to fix the domains (aka "scales") of the distributions or not?  A quick *Mathematica* calculation indicates the PDF includes, as one of its terms, the product of $x^{-\\alpha-\\beta}$ and the difference of a Beta$(-\\alpha,1-\\beta)$ distribution in $1-1/x$ and a Beta distribution in $1/x$. It is unimodal for $0\\lt\\alpha\\lt\\beta\\lt 1$. This result would not hold for larger $\\alpha$ and $\\beta$, so are there any limits on the possible values of the parameters in which you are interested?
... but here's the Wikipedia link on funnel plots: http://en.wikipedia.org/wiki/Funnel_plot
Welcome to the site. I removed your signature because the site adds it automatically
What $R$ does when there is no intercept is that it calculates $$R^2 = 1 - \\frac{\\sum_i (y_i - \\hat y_i)^2}{\\sum_i y_i^2}$$ instead (notice, no subtraction of the mean in the denominator terms). This makes the denominator larger which, for the same or similar MSE causes the $R^2$ to increase.
@ayush. I describe the calculation in the third paragraph.  In terms of actually calculating the resulting Chi Square p value, you can use an Excel formula.  If I remember correctly it is either CHIDIST or CHITEST.
The data are daily with frequency 5. That isn't high frequency.
You would need an intercept of $-\\infty$ to do that!  An intercept of 0 corresponds to a probability of $\\frac{1}{1 + \\exp(-0)} = 1/2$, not $0$, when all independent variables are zero.
(+1) it is straightforward to see how to generalize to a much wider class of correlation structures: Simply tie successive random variables together via your favorite copula, then conditionally sample the current value from the copula given the previous value. One may, in this way, use copulae with explicitly parametrized measures of association, be they linear correlation coefficients or other measures such as Kendall's tau.
@ ttnphns (significant driver, I suppose a better word would be "determiner")
In your model $p_{0i}, p_{1i}, r_{0j}, p_{1j}$ are already latent variables.
.../... A way out of the conundrum you seem to be dug into might be to introduce the mid-point measure $P+Q$. Since $P\\ll P+Q$ and $Q\\ll P+Q$, the quantity $\\eta(P,Q):=\\kappa(P|P+Q)+\\kappa(Q|P+Q)$ is always finite. Furthermore $\\eta(P,Q)=0$ iff $P=Q$ and $\\eta$ is symmetric. Hence $\\eta(P,Q)$ indeed measures a kind of "distance" between $P$ and $Q$.
Interesting topic but I wonder how well the technique you've chosen matches your research questions.  It fits as a way of identifying stronger and weaker predictors of each individual's vote.  But you talk mostly about the group-level results--the percentage who vote a certain way.  These can be compared using a much simpler method such as a proportions test, which can yield confidence intervals for percentages and/or their differences, or a Chi-Square test.
To omit a next myth about "bad CV", this is a terminology problem -- Harrell's "cross validation" means N-fold CV, and "bootstrap validation" means resampling CV. Obviously I agree that this second flavor is more stable and overall nicer, but this is also a type of cross validation.
But how to explain this then? (Let's just take df>2, so that we have some basic moments). I first thought the simulation may not stable, I mean, if I run it again, the mean(exp(0.05*rt(1e6,df=4))) will change, but the result is almost the same. I am really lost... Moreover, some other "small" factors can also make this like 0.2. Is there a link between the m and the "samll" factor? I really want to know more on this, are there some references studied this question?
Mysteriously disappearing results should be automatic rejection, imo. I'm sure this happens a lot "behind the scenes" (i.e. before the paper is submitted), but this is clear evidence of "cherry picking" that normal readers of the paper would never know.
If your goal is to predict, a higher R-square may be due to overfitting. Try to compare the RN predictions with Regression predictions using cross-validation.
possible duplicate of [What are the differences between Factor Analysis and Principal Component Analysis](http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysi)
And just to be clear - an interaction can be made significant by the omission of a non-significant main effect term.
Andreas, could you share some information about the purpose of this modeling?  Absent that, you will only get generic answers that might or might not be appropriate for the question, which makes it difficult (if not impossible) for people to provide the great answers we look for here.
+1 Because statistical language can be confusing, especially to beginners, I would like to point out that scaling *does* change the distribution: but it does not change its *shape* (which remains Normal) and that of course is what you are contending.
@Glen_b. It was a typo. I have changed it
This works perfectly, an easy method of achieving exactly what I wanted.
That's defenitly a valid point. We are *ver* reluctant to throw away data.
I'm having trouble following this. First, do you have 100 or 1000 variables labeled "Y"? Are you using "Y" for the predictor variables & "X" for the response variable? (That's OK, but it's the opposite of how it's usually done.) Is the main issue here that you have a multivariate situation (ie, multiple response variables)? Is this supposed to be a $p>>n$ problem? Re different shapes, note that the distribution of predictors is irrelevant, but if the issue is w/ response variables, can you say more about what the distributions are? (Eg, I think a random variable can be normally distributed.)