tttjjj/elt_bench huggingface dataset:
For the percent explained, it depends on the type of study you use, but all options will present this information. For example, linear regression will tell you the amount of deviance explained by each variable. A discriminant analysis will tell you the amount of variance explained by each axe (which is a combination of different variables). Confidence intervals can be made on the estimates of the variables (how correct are we?), and it usually requires more assumptions on the data. I would suggest creating a new question once you've decided what method to use to get a specific answer.
The groups represent gene classifications. So, each value is a count representing a gene which falls into that classification. The two datasets are different groups of genes, and I want to find whether the datsets are significantly different based on how the genes fall into these groups (I ideally don't want them to be)
A log likelihood ratio is operated on the same data.  One of the distributions can be the same general form as the other, but its parameters were fitted to data when some other criterion was true.  For instance, you could have one distribution whose parameters describe healthy production process variation (not necessarily Gaussian) and another that is fit to current production process values, and operate both on current production process values compare the log-likelihood ratio to a threshold value indicating possibility of excursion.  It can the actual to be compared with the ideal.
Better a rough answer to the right question than an exact answer
to the wrong question, an aphorism, due perhaps to Lord Kelvin, that I heard as an undergraduate in applied mathematics. --D.R. Cox
For three tests you can test each at the 0.05/3 = 0.0164, then the FWER is guaranteed to be less than 0.05 based on the Bonferroni bound.
Thanks for the suggestion. My model does appear to be suffering from a degree of heteroscedasticity which lme has a convenient method of handling so I think this may be a better option taking this also into account. I will in the near future be needing to model a binary response variable in which case I will need the lmer function, so any further suggestions would be useful.
@robin That's a good point, but I constructed this example specifically so that for potentially long periods of time these two processes are *perfectly* correlated.  The issue is not one of dependence versus correlation but inherently is related to a subtler phenomenon: that the relationship between the processes changes at random periods.  That, in a nutshell, is exactly what can happen in real markets (or at least we ought to worry that it can happen!).
Did you try to validate your classification? Do you have samples of data which allow you to compare results with your map? For example, databases providing a threshold value involving number of cattle produced per hectare (or acre). If cattle/ha > 1  = 0 (not degraded), if not = 1. Your way of thinking makes sense to me, but you need to validate your model somehow.
Perhaps you could email Nate Silver (nrsilver@gmail.com) and ask him yourself?  That blog contains an email link.
+1 beautiful, useful, and comprehensive answer!
Nicely done, but then wouldn't we need a measure of "out of placeness"? E.g. for 3 of the 14 time points, the orange line is the highest. Is that "out of place"? Or the orange line's rise from time 1 to time 2 is very rapid, by eye, it might be the most rapid rise of any of the 20*13/2 or so "rises" in the whole set. Is that "out of place"? 

If we define a particular type of "oddness" it will be easy to come up with a measure of it. But a measure of overall oddness seems very hard indeed.

All this is not meant as criticism at all (I +1ed your answer)
Because it's a *definite* integral.
The arrows appear to point to the locations where the second derivative of $f$ is largest: is this what you mean by "the point on the curve at which the horizontal component of the curve meets the oblique component"?
Got it--checked.
Certainly, you can train on whatever you want. But are you having a lot of success with training on a symmetric distribution of the target variable and predicting for a skewed one? If so, it seems like an exceptional situation then - would love to see a counter-example on this. In general, I would want the training data to be representative of the testing dataset, across all variables, as much as possible. Regarding the point with the stratified sampling, I was thinking about it as a way how to deal with skewness during prediction, for example if you do cross-validation for parameter tuning.
Thanks for your answer Julián, I will read further on it. An additional question though - if we were looking at ecd f(3) as you proposed, I can see that 35% of the sample obtained 3 or below. But could you say that a score of 3 is in the 'top' 65%? Intuition tells me that a score of 3 is in fact the top 80% (16 scores of 3+ out of 20 = 0.8). Is my logic flawed here? Just trying to understand the relationship between the 'top' and 'bottom' percentages. Thanks
+1. Very nice explanation. Feel free to dismiss this as ultra-picky, but I prefer df to $df$ as notation. Composite symbols are generally better as roman (think exp not $exp$, log, sin, cos, etc. Otherwise somebody wastes time asking what $d$ and $f$ are which are multiplied together.
Just curious, what defines the trend line? is it the relationship between time and concentration?
I have used $f(x|\\mu,\\sigma)/f(\\mu|\\mu,\\sigma)$
@Jacob It is almost certain that the two distributions are different, so get over it!  The crux of the matter is whether the difference is large enough to be of concern.  That suggests additional information is needed; namely, the purpose of your determination.  If you conclude both methods have "similar" service times, what then?  If due to chance (and unknown to you) your conclusion is erroneous, what implications would that have?  Your answers to these questions would indicate how to compare the two empirical distributions you have.
Ah. Yes Gotcha. Thanks a lot!
I think it is valid to use [AIC](http://en.wikipedia.org/wiki/Akaike_information_criterion) for comparing these models. The Wikipedia entry mentions "There are, however, important distinctions. In particular, the likelihood-ratio test is valid only for nested models whereas AIC (and AICc) has no such restriction". On the other hand, it is more intriguing why are you comparing such diferent models **""I would fit a normal, an exponential and a uniform distribution"**? They are different in terms of shape, support, ...
Greg, can you give, please, a numeric example in line with your code? For people not using `R` to be able to repeat your idea outside of `R`?
Argh...yes...I mixed up sensitivity and specificity.
Hi Matt, really tnaks for your help. There's less fog on my mind now!
Hi Tal, Is it possible to upload my dendrogram "x" as a ".RData" file/object?
Because according to my regression coefficients IV(a) has more influence on the mediator than IV(b) does. And now I would want to know whether this difference between the regression coefficients is significant
@LOL same answer as mine with the same order of choices and the same examples... hah what are the odds? I guess we all think alike
Thank you.  Walking through the source to cor.test with my data helped me see I was screwing up the Fisher's Z transform.
Ok, think of it like this.  What is the average of `4,6,8,0,0,0`? Well, $\\frac{4+6+8+0+0+0}{6} = 3$.  We could instead have thrown away the zeros and found the conditional expectation $\\frac{4+6+8}{3} = 6$, then we multiply by the probability of being non-zero $6 \\times 0.5 = 3$.  Does that help?
This is a nice perspective. Thanks!
I wouldn't call $\\phi$ a measure of correlation. I think you are always safe calling it a measure of association, but the former usage will often cause problems to people who think they know what correlation is and will wonder if you do too. Who's right and who's wrong about where to draw the line is a bigger question, but my recommendation is not to do that.
I don't see how this can be done in the *same model* because using different IVs makes the models different. But, as @ttnphns pointed out, you can break a multinomial regression into several models.
This is also known as a biexponential function: see https://www.google.com/search?btnG=1&pws=0&q=biexponential+model
Hmm, ok, and +1 for pointing them towards considering the original units rather than just proportions, but I think this might be an overly literal interpretation.  On the other hand it's true you have answered what they asked whereas my "10 or more" would be assuming they really meant to ask something slightly different.
Section 5.4.6. And you can extend the idea using orthogonal polynomials to code the ordinal predictor.
What do you mean there are not described as points?
People could use different pictures of the same person.
Canyou clarify answer.  You start by suggesting that heteroskedasticity is a problem.  And, you move on by suggesting that multicollinearity is more of a problem and that heteroskedasticity is Ok.
Have you consider this is a discrete choice problem? http://en.wikipedia.org/wiki/Discrete_choice
Thanks, that was an interesting paper.
I agree , it looks like either a first differnce or an AR(1) model with coefficient near 1.0. The plot of the orifiginal residuals suggests (in general) a tendency for a positive residual to follow a positive and a negative to follow a negative. Based upon these results I suggest constructing a TRansfer Function using the original data as important lags in one or more of the predictors may have been omitted. In this case the residuals can suggest autocorrelation whereas the more correct fix is to extract more structure from the X's.
thank you so much for the help, I will check it immediately ;)
@yannick: Real-world significance is a domain knowledge problem, not a statistical one. That is, I can tell you that I have some weight data and there is a statistically significant 10 gram difference in mean weights between two groups, at the 95% level, but does that have real-world significance? For a minnow, yes, for adult men, no. If you're talking real-world significance, I'd imagine comparing to SD or determining quantiles would answer your question even if that doesn't seem rigorous and leaves room for someone to disagree with you.
I suppose it was the only thing that came to mind. If I want to create a confidence interval for the parameter based solely on a sample of say size 30, given the MME is a function of the sample mean which has a t-distribution (technically I don't know the variance), I figured that was the route to take. I could be way off base though.
what do you mean a computer task that has three different conditions? Would a simple ANOVA be the appropriate analysis?
Instead of that (cumbersome) **sapply** call in your first code snippet, you can always just use **pmax**.
erf is not a very handy function, provided you don't want to rather use it for its derivative.
It is not obvious to me what fit2 is. For example, what do you get when you regress 30 values of Y on the factor X having 30 levels?
Closely related question (duplicate?): http://stats.stackexchange.com/questions/45182/recoded-items-as-separate-factors-in-factor-analysis
@rocinante I agree. I would not recommend doing that for a real world case.
Oh! I almost wrote the wrong reply then!
Many implementations of DT accept missing data. Those data are kept as a separate category which can be merged with other categories.
Hat tip to @PeterEllis, whose comment on the original answer I shamelessly included in this answer.
I have edited the text for clarification. I am assuming that the poster wants test the equality of location.
This article may be of help: http://www.tqmp.org/Content/vol08-1/p023/p023.pdf
You need to look at R2 adjusted, not just R2, since you are estimating models with a different number of parameters.  That said, I would argue that "Is Model 1 statistically better than model 2?" is too vague of a statement--- what is "better"? Less residual sum of squares? Does R2 really tell you which is "better"? What does the domain science say? Have you checked that both meet assumptions of OLS linear regression? I would use AIC to compare the candidate models, but there is no associated tests, only rule of thumbs. Perhaps look at the cox test (in the lmtest package of r)
Yes, you can and may use. CATPCA (i.e. iterative PCA with optimal scaling) can take any categorical or discrete variables. A variable is quantified into interval-level variable either (1) linearly, (2) monotonically, (3) nonmonotonically, or (4) split into a set of dummy variables. The choice is yours. For an ordinal variable you normally should use (2), and for nominal variable, (3) or (4). Note that if you use (4) for all your variables, CATPCA becomes identical to Multiple Correspondence analysis (MCA=HOMALS). Please search `CATPCA` on this site for more.
(+1) Thanks for providing links and reasons for your recommendations.  Welcome to our community!
You're basically correct - you are trading analytic error for monte carlo error.  The parametric bootstrap is also an approximate posterior sample.
what is the difference between your question and this one: [Clustering with a distance matrix](http://stats.stackexchange.com/questions/2717/clustering-with-a-distance-matrix)
Of possible interest: [A mathematical formula for K-fold cross-validation prediction error?](http://stats.stackexchange.com/q/17431/930), [Variance estimates in k-fold cross-validation](http://stats.stackexchange.com/q/31190/930).
if she has the ability to start the next haircut earlier than scheduled, she is not allowed to do so?
@user603 That's easy for finding a median, but for multiple regression it gets much harder. The OLS solution involves solving a linear system which is as easy as it gets.  (Historically, the first efforts along these lines, c. 1755, indeed minimized the sum of residuals. There's a pretty geometric solution in the univariate model, but I don't think it generalizes to multiple explanatory variables.)
I've used R and SAS before, but they're more acclimated with Excel, so I figured I should stick to that. This gives me a good starting point. Thanks for the help! The sample graphs look more like what I would expect to show to a client.
@DimitriyV.Masterov please see my edit which I hope makes more clear what I actually have done. Sorry for the earlier confusion. Would really appreciate your advice on my interpretation in light of this edit.
`lm(y~x+f:x)` ...
$\\sqrt{Var(x)} = 20, \\sqrt{Var(y)} = 500,000$
Sure. That's what regression does. you are saying that dv1=a*dv2+b*iv+ error and dv2=c*dv1+b*iv. So fit the regressions and see if the coefficients for iv are similar. To do this "right" you probably need a simple structural equation model.
@suncoolsu: I menat parameter estimate $\\hat{b_{p}}$
Not that the concordance probability (AUROC) is often not sensitive enough to compare methods.  It is best for describing pure predictive discrimination for a single model.  It is better to use the gold standard (deviance-based methods such as generalized $R^2$ or likelihood ratio $\\chi^2$) for comparing two or more predictive methods.
Subtracting the mean changes the null distribution of the KS statistic.  @David Wright raises a valid objection to the application of the KS test to histograms anyway.
A word on notation: I have made some edits to this post. While everyone is entitled to their own approach to mathematical notation, it's often useful to have conventions. It is quite uncommon to denote a *distribution* by the letter $X$ or $Y$, and it *is* quite common to denote a random variable by those letters.
Hi Jeremy. Could you please clarify this post? My concern is that (i) the triangle inequality is not what you've written and (ii) the equation does not match with either the triangle inequality or the description you've given. I suspect these mat just be typos.
If you expand the distribution of the sum in an [Edgeworth expansion](http://www.amazon.com/Bootstrap-Edgeworth-Expansion-Springer-Statistics/dp/0387945083), you find that $F_n(x)=\\Phi(x)+n^{-1}g(x)+o(n^{-1})$ uniformly in $x$ as $n\\rightarrow\\infty$ (since the uniform distribution is symmetric), so $c/n$ sounds about right. Because of the $o(n^{-1})$ term, that doesn't give you a bound though...
Because there are many ways to "make a nonlinear fit," please provide some details of what you are doing, including what program you are using.  (Doesn't the program's documentation explain its output?)
Apache Mahout could work, at least according to the bin/mahout svd command. It is very unfortunate they don't have a simple example how to do svd programmatically.
when you say "such as in SPSS" do you really mean "in SPSS" or do you mean (as it appears to ask) "in some statistics package, of which SPSS is an example"? Because if you mean what you seem to say, you'll probably get answer for how to do it in R.
Answer [here](http://stats.stackexchange.com/questions/80399/can-someone-sort-me-out-regarding-the-calculation-of-auc/80404#80404).
Can you please explain what you mean by "regions"?  What, exactly, stops you from being able to ignore the regions and do one AnOVa?
Great answer!!! thanks a lot!
Nice interpretation. Thank you for that.
you need to know what is $\\hat{p}_1$ and $\\hat{p}_2$ knowing age $\\leq$ 50
Isn't $Y\\sim U(0,b)$ and, therefore, $1/(1+Y)$ bounded by $(1/(1+b),1)$ with probability 1? Because $b>0$, the expectation is finite anyways as you said.
Here is a mirrored version of the original vignette, in case it helps: http://bit.ly/x6eI4x.
And actually, I'd forgotten a bit too.  It's `ctree` to make a single tree, `cforest` to make a random forest and `mob` to make the model based leaves.  And fyi, cforest is fun, but horribly slow to predict with.
Do you want to compare the model fits, or test for differences between the terms across models (which may or may not be possible if you have different sample sizes). Have you considered using list-wise deletion to make the sample sizes comparable (if you have missing data issues rather than different samples due to non-overlapping IVs)?
Also, please do not say "item loads onto a factor". More correct is "factor loads onto an item": loading is the regression coef. in predicting items by factors, not vice versa.
@Christopher: !! With 16 years (of daily closings?) you have enough data not only to find a correlation, but also to explore how it has been changing over time.  (This I believe is the spirit of @Stephan Kolassa's reply.)
Also, R-squared here is just the square of the Pearson correlation, to be calculated using the appropriate function or command in your favourite software.
Yes, that the CEO's education and prior work experience in various fields causes diversity of partners.
If you use $\\LaTeX$ you can run `latex(cph result)` to see the entire model statement including a table of underlying survival probabilities you can plug in.  There are like a series of intercepts in the model.  You would have to program that part; the `Function` function does the more complex linear predictor part of the model.  If you want survival estimates at a single fixed time horizon then reading that off the $\\LaTeX$ output and programming it is very easy.  Be cognizant of the `cph` function `time.inc` argument.
@broiyan:  Sounds close. I think technically the precise value obtained, if re-obtained, would be included in the probability.  Therefore, I would go to the expense of extra words to make that clear by saying "as extreme or more extreme".  Also, because you are phrasing it in a frequentist framework, I wouldn't talk about the probability of that event, but the proportion of times you would expect it to occur if the experiment were repeated over and over.
Thanks man! When i saw your answer, then my thought was: Of course! How stupid could i be?
@AurelianTutuianu There is no specific input/output variables or question for classifier. Just in general is it possible?
Your question seems to end mid sentence. Did you lose part of it?
@probability So what *is* the gambler's expected return?  I couldn't find it in your reply :-).
The full model holds all possible independent variables (predictors) candidates.
Right, you already know the bias. Would this strategy also work for a dice where one face was heavier (again, you are aware that it is)?
@Jon You need the average of the sum of squared differences for an estimate of prediction accuracy. I edited my answer to make that clear.
OK, so power is not a problem. But this is still not generally a good idea. In fact, one of the few good reasons to lump categories would be low N in a category. You don't have that problem. Even 1% of 19,000 is plenty
Is that $y_1 \\lt -x_1$ or $y_1 \\leftarrow x_1$ or possibly the R code  `y1 <- x1`?
@Adam: I added a link to zoo in my answer.  That page also has a link to zoo's R-forge project page.
I just wanted to say that this is a good example of a question that might be asked on the Public Health and Epidemiology proposal site here:
http://area51.stackexchange.com/proposals/34565/public-health-epidemiology Perhaps Kaish follow the proposal there, and post a shortened version of the question as an example?
PCA axes are orthogonal to one another, and therefore uncorrelated.  Or at least that is my understanding.  Can you or someone else please provide a source or example that shows otherwise.
@PeterEllis I don't know the exact definition of the estimation theory, but what I mean is something like the maximum likelihood (ML) estimation, maximum a posteriori (MAP) estimation, and minimum mean squared error (MMSE) estimation, etc. See [here](http://http://en.wikipedia.org/wiki/Estimation_theory). I don't know if this kind of estimation overlap with regression, or one belongs to the other?
Too broad as stated.
"based on set A" means that all of those M*P elements are sampled from A. M*P can be smaller than N.
@Peter I am not qualified to tell whether this answer is correct or not, but I think Tom makes an excellent point.  It is a little troubling that such a negative answer is also, by its very nature, difficult to support: how does one supply a reference supporting an assertion that no valid references exist? :-)  Nevertheless, it arouses my sympathies because I have encountered many algorithms in statistics--particularly in statistics applied to business--that are complex, sophisticated, undocumented, and wholly bogus, yet it can take a *lot* of work to discover that last fact.
My understanding was that if you take one variable (black color mobile device) and modify it (apply a blie skin), then it is considered as one variable with two levels (black and blue color). However, if you take four "different" mobile phones (same brand: Nokia) - Lumina 930, Dual Slim, Nokia XL Dual, Asha 503), then we are dealing with 4 different variables. This is because one model may be touch screen, one works with two simcards, the other may have keypads etc.
I have to go through all the steps myself, as my data requires the use of different steps depending on its nature. I prefer the PCA function generally too, but I would like to know the answer anyway :)
(+1) But note that your inequality is tight only for even $n$ and can be improved by accounting for the SD of the $S_j$.  (For instance, when all the $S_j$ are equal to one another, clearly the sum is $0$.)
@MrFlick Thank you.
Additional threads with related (or even the same) questions can be found by searching our site for [probability row](http://stats.stackexchange.com/search?q=probability+row).  They indicate the probability you seek is $112/2^{15}\\approx 0.003417969$. By looking at when the streak begins, I can find $2^5 + 2^4 + 2\\times 2^3 + 2^2\\times 2^2 + 2^3\\times 2^1 + 2^4\\times 2^0$ = $112$ distinct ways in which it can occur, each with probability $2^{-15}$.
I think no one got the question. The guys is asking if there is a machine learning algorithm that can learn 10000 features (assume F1) to produce 1500 outputs (assume O1). So if we assume that we have an experiment that work as follows F1 has -> O1, F2 -> O2, F3 -> O3 , .... then can this be done using machine learning, if so how.
I wish I can help further