tttjjj/elt_bench huggingface dataset:
@wildetudor, the 1 column for the 2 level could be done using either a t or an F (the F will just be the t squared).  The 3 level needs an F.  So any number of levels can be compared with the F, the t is a special case for when you only have 2 levels.
Your question needs some clarity. It appears you are talking about a hazard ratio, not a hazard rate. Is there a reason you've chosen that particular regression equation - there are many ways to approach a survival analysis. And what do you mean by "when" can we assume this - do you want a set of circumstances where it's true, or a means to assess whether its true in your case?
I recommend moving this to statistics.SE.  What you're describing appears to be change point detection.  I'm sure the good people on that site can advise on useful CP algorithms.
No, those cannot be interpreted as p-values. Do you need p-values? From your given goal (identify interesting points) I didn't think so. You might want to [read up on robust regression](http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-robust-regression.pdf) in order to understand these weights.
"always" is a big word, but looking at the percentiles (with possible adjustments) for computing confidence intervals fits more naturally within the logic of the bootstrap.
1+. Thanks! Please, see my update to the question. I have a couple of questions I hope you can clarify.
Right, the rows are sorted by **y**.  The columns are sorted by the sums of the products.  _Whoops, I see I made an error in the column labels of the image ... they should read x3, x1, x2 (just edited answer to fix this)._  X3 had the smallest product sum, x2 had the biggest.  Why does it work?  It's just a very simple approach to seriation (see for example, [this link](http://cran.r-project.org/web/packages/seriation/vignettes/seriation.pdf)).
The explanation of eigen values is correct. Explanation on ratio is not perfect. 
The eigen values are calculated of scaled version of $Z'Z$.When there is no collinearity at all, the eigenvalues, condition indices and condition number will all equal one. As collinearity increases, eigenvalues will be both greater and smaller than 1, and the condition indices and the condition number will increase.
I've realized from this that I have a lot of work to do on pre-processing the data before I start the analysis I've outlined here. Reading the response from @nadya I think it's clear I need to look at some kind of spatial aggregation, but that will be challenging as it's wrong to aggregate land and ocean data. Then I need to look at gap-filling strategies. Then (and only then) can I start to look at the mapping / visualization work.
There is a noteworthy economic literature on statistical measurement of race and sex discrimination in, e.g., labor markets, that might be useful.  A seminal work is:
 
The Statistical Theory of Racism and Sexism
Edmund S. Phelps
The American Economic Review
Vol. 62, No. 4 (Sep., 1972), pp. 659-661
Published by: American Economic Association
Article Stable URL: http://www.jstor.org/stable/1806107
English is not a romance language.
Do I get it right you are trying to fix the output instead of the underlying model?
His question might go a bit deeper assuming he means that maybe the assumptions do longer hold. In any case he'll have to specify.
(1) Perhaps [NIST](http://www.itl.nist.gov/div898/handbook/eda/section3/eda362.htm#CDF) or [IPSUR](http://cran.r-project.org/web/packages/IPSUR/vignettes/IPSUR.pdf). (2) The number of events in ten trials can't have a Poisson distribution because you can't get more than ten out of ten whereas a Poisson variable is unbounded above.
This answer is incorrect.
Also, if I use only beta values, how do I determine if one is signifcantly greater than the other?
This wikipedia page http://en.wikipedia.org/wiki/Kolmogorov_extension_theorem helps you ? Is your question a general question about existence theorem in mathematic ? or are you talking about a particular existence theorem ?
It is not theoretically sane to do k-means on binary ("present" vs "absent") data because means are meaningless for data which cannot be thought of as continuous. You should compute Gower similarity and then do hierarchical or k-medoids clustering.
Cumulated payment amounts ($C_1=Y_1$, $C_2=Y_1+Y_2$, $C_3=Y_1+Y_2+Y_3$, ...)  are not independent, but highly correlated. It would be much better to deal with that issue directly (why fit a *cumulative* distribution function. to *cumulated* data?) before trying to build a model for it.
thank its very useful
@user777: As long as one's nested inside the other.
+1 This is worth reversal badge (-;.
@seanv507 typo is fixed, but that doesn't change the question.
An interesting and thought-provoking thesis, but ultimately less than helpful in this context. Moreover, its truth would seem to rest on an idiosyncratic and limited idea of "reality-based." (To justify that allegation of *limited,* consider--among many examples--what it would take to derive distributions like the hypergeometric or Benford from the three distributions named here.)
(+1) Of note, another interesting discussion on formative vs. reflective models can be found in Chapter 3 of *Measuring the Mind*, by Denny Borsboom (Cambridge, 2005).
@ Cardinal: Thanks for the opportunity for clarifying (1) In my observed set (i.e. the DNA sequence with my target 6 mer) I find up to 7 times the 6 mer. (2) No they are not aligned. (3) the frame is not relevant to my problem. Basically I expect to find each 6 mer in about every other sequence by chance, and finding 2, 3, or more times the 6 mer in the 2100mer sequence will be very rare.
you case looks like a classical case of categorical data analysis. Can you please elaborate what do the fractions represent? An elaborate description of the data and problem will help us answer your question better.
Except your own answer misses out the most important issue, which is that the "effect size" is only an *estimate* of the true effect size.  The estimate is so uncertain that the true value might easily be as small as zero.  Estimated effect sizes are very dangerous if not reported with some indicator of the uncertainty associated.  I agree a p-value is a very poor indicator and much better is a confidence interval, but I would hate to see someone naively reporting the estimated effect size on the basis of your answer that it is more important than the p-value.
This question: http://stats.stackexchange.com/questions/1826/cross-validation-in-plain-english has two great answers (highest score), i think they could help you in your question. The second is exactly what you want. OBS.: I'd like to write this as a comment, but I can't do it, so i've answered it.
@Dbr You can do both with `splom()` (from the `lattice` package) or `pairs()`, i.e. plot all your variables (predictors + response) in a *scatterplot matrix*. You can then add local smoothers, highlight possible outlying values etc.
There is no problem w/ having a non-significant linear effect in a regression model. You may find the following thread helpful for understanding these issues: [Does it make sense to add a quadratic term but not the linear term to a model?](http://stats.stackexchange.com/questions/28730/)
Thank you for the answer. I forgot to optimize sigma_n. I only optimized sigma_f. This caused the strange behavior. But your displayed thoughts are very useful. Thanks
Do you have the quality values per class at least, or do you only have a global measure only?
Did you not perform the Cochran-Mantel-Haenszel test precisely because the evidence for an odds ratio different from one might be weak for each stratum considered individually, but strong for all considered together?
Well $\\mathbf{X} \\prime \\mathbf{X}/\\sigma^2$ is $\\chi^2 (n)$. It remains to prove the independence though.
I don't quite follow your setup. What is your response variable, & what are your predictors? Are you just correlating a lot of variables?
best one line summary!
If you fit a model of a straight line with constant gaussian error to a bounded variable, obviously this is going to happen. The correct way to deal with it is not to fit a model you know to be wrong, but one that's at least plausible. If y's can only be positive, you don't fit a model that says they can be negative.
+10 on item 4 :)
It works! thank you very much for your help.
Do you mean [reversible jump MCMC](http://en.wikipedia.org/wiki/Reversible-jump_Markov_chain_Monte_Carlo)? In my opinion it is impossible to find an R package that does reversible jump in a generic situation, simply because the jumps are very specific. For learning Gibbs on generalised linear models, you can use the [R code available on the webpage](http://www.ceremade.dauphine.fr/~xian/BCS/) of our book [Bayesian Core](http://www.amazon.com/gp/product/1441919392?ie=UTF8&tag=chrprobboo-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=1441919392).
Can you post a tiny sample of your dataset so that readers get a better idea of the data structure?
@gung I'm asking about how to select a model using PROC ARIMA
Stepwise regression destroys all aspects of statistical inference (bias, type I error, false standard errors, inflated regression coefficients, inflated $R^2$, false confidence interval coverage).  Breaking apart components of categorical variables is even worse.
Related (and the source of the quotation): http://stats.stackexchange.com/questions/24900/negative-binomial-regression-and-heteroskedasticity.
One important thing to remember in time series analysis, is that in time series analysis, not only the ordering of when the observations occur matter, but also the time interval/frequency have to be the same. That is, you cannot mix daily/weekly/monthly data.
(i) What does it mean for the *data set* to be of type 'center' or 'spread'? Isn't that actually dealing with the form of hypothesis about the data? (ii) you cannot determine that data is normal. Nothing can tell you that. I  presume you mean you're doing some hypothesis test of normality?
+1. Based on your scatterplot you wrote that you can estimate *scale* (which I called *gain*) and *offset* but I have a third parameter called *dispersion* that is not accounted for...
Heh, I kept the date out on purpose because I thought it would be amusing to notice the time span.  :)  The books by Michael Berry and Gordon Linoff are pretty good and will appeal to statisticians (for the broader exposure rather than learning statistical techniques).  If you want a sense of the fuzzy, "enterprise" side of this field, skimming through one of the books on a vendor product, like SAS's Enterprise Miner or SPSS's Clementine, may help.  I wouldn't recommend buying them unless you're going to work with the product itself.
Ah, I am very sorry then, but your estimation that research has dropped off is most likely correct.  However, as I'm sure it seems to you, it seems likely to me that the multivariate case would at least be as impacted by seasonality/trend as the univariate, if not more so.
Uhm. Unless you show the results of the two methods, along with the input matrix, that question would look much as purely `R` question.
I don't understand what you're looking for that isn't already covered in my response to your previous question, "[Interaction/Moderation effect in ANOVA - variable is not one of the factors](http://stats.stackexchange.com/questions/100926/interaction-moderation-effect-in-anova-variable-is-not-one-of-the-factors)". This appears to be a duplicate of that question.
I am just asking for a generalized solution, no particular problem specified
Potential edges are all the relationships which could exist, out of which only a few can actually be found in the empirical data.
@Frida: ttnphns' comment is on-target. See http://en.wikipedia.org/wiki/Linear_discriminant_analysis third paragraph. In some sense it's luck that PCA turns out to be useful so often.
Excellent answer Nick, thank you very much!
The denominator is used to account for the variance of raw residuals which then makes Pearson's residuals approximately of unit variance (there are other methods to achieve this). Please note that there's a component `stdres` for standardized residuals.
Data and/or summaries of the models would allow one to help you better.
I'd just use the term "Wald-test". But if that's too complicated, I'd stick with "$t$-test".
Often it is better to plot power curves rather than just computing a single value of power.  Have the different effect sizes on the x-axis and the resulting power on the y.  You can include a few different curves representing different sample sizes.  These can show that while the initial conditions don't give the power desired, a few small changes would, or they may show that the experiment is not worth pursuing with the available resources.
That's indeed a great reference and will be helpful when extending the cheat sheet with R examples.
thanks, @Patrick. i gave you an upvote yesterday for your thoughtful response. waiting a bit to see if others have ideas to contribute.
I have not kept up with the literature regarding optimum split configuration.  But some general principles apply.  For the validation sample you need $n$ large enough so that you can estimate the calibration curve with great precision, then you need to see that what's left is more than adequate for reliable model fitting (using, say a 20:1 ratio of events:candidate parameters if you don't penalize).
I considered the route you mentioned with the 2 different Y axis, but what I didn't like about it was: if a new series was introduced that wouldn't fit on one of the two Y-axis, i would probably be stuck. thank you for the suggestion, and maybe another time i would consider this more :)
Why do you want to remove variables in your model?
Unless I'm missing something, $\\text{RidgeRegression}(Y - X \\beta_0, X, \\lambda)$ gives you $\\theta^*$, but you want $\\beta^*$, so you actually want to replace $\\text{RidgeRegression}(Y, X, \\lambda)$ with $\\text{RidgeRegression}(Y - X \\beta_0, X, \\lambda) + \\beta_0$.
@Nico You're right to wonder that.  The references to percentages and, indeed, to chi squared are red herrings.  In the new situation the observations are total dollar returns on page visits, not a percentage of anything (so of course a chi-squared calculation is wholly invalid).  In the old situation the observations are counts of "conversions" and visits for each page.  Chi squared enters as one (of several) possible approximations to a *test of proportions.*  (In both situations it's necessary that conversions be serially independent, which is a dubious but often necessary assumption.)
I think you have taken a good deal of machine learning dogma for granted.  You are making a large number of unwarranted assumptions.  One of them is that people actually need a forced choice into a categorical bucket.  They may claim to want this but they really don't need this in most situations.  Choices don't have to be forced.  A great choice is "no decision, get more data".  Prediction of an underlying continuous quantity is usually what is needed.  It would be worth your while to study optimum (Bayes) decision theory.  If you can provide a concrete example I'll comment further.
A related question: http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues and a blog post is referenced by one of the answers: http://www.cerebralmastication.com/2010/09/principal-component-analysis-pca-vs-ordinary-least-squares-ols-a-visual-explination/
I think I am mostly interested if the population males differs from the population of females whether they go and ask the teacher or not. Just a formulation thing, or is this different?
@Flask I see your point. Indeed, if you have a linear regression model of the type ${\\bf y} = {\\bf X}{\\bf \\beta} + {\\bf \\varepsilon}$, where each entry $\\varepsilon_j\\stackrel{ind.}{\\sim} \\text{Normal}(0,\\tau^{-1}= \\sigma^2)$, then ${\\bf \\varepsilon}$ represents the residual errors, which is a random vector, $\\tau$ is the precision of the entries of this random vector, $\\sigma^2$ is the variance, and $\\sigma$ is the standard deviation.
Thanks. But you took here an infinite order of an MA model as well (at least of the same order $q$). So why is this then more parsimonous than an AR model?
Feels strange to see if zero is among the credible differences as surely you expect there to be some difference between the folds. One suggestion would be to calculate point estimates of $\\beta$ for each fold and look at the spread of these.
Why do you need to take the relation between parent and child into account? What is wrong with just putting everything in at one level? What are you worried about?
How many observations are you working with?  A little bit more information would be helpful, what are you predicting, are there any dependencies between the observations etc...
Thanks for your answer. All models I have estimated so far for this project are frequentist models. Thus, I have a strong preference to stick with the frequentist approach and to avoid confusing reviewers anymore. I see that Bayes would make my life in this case probably easier. I'm rather surprised that I don't find too much information on such a model (and of course implementations).
I'd second the answer of @AdamO.  In addition, you need to choose the measure of obesity that reflects the disease or outcome risk of interest.  In some cases this is BMI; in other cases it is waist circumference or weight.  But never categorize a legitimate continuous variable.
How do you handle face recognition without occlusions ?
This past question asked about correlation instead of covariance, but it is related: http://stats.stackexchange.com/questions/35941/effect-of-log-transform-on-pearson-correlation
Re your e.g.: It's a common misconception that a mixture of two Gaussians is necessarily bimodal. A mixture of two Gaussians with equal standard deviations is bimodal only if the difference in their means is at least twice their standard deviation. See: Schilling, Mark F.; Watkins, Ann E.; Watkins, William (2002). "Is Human Height Bimodal?". *The American Statistician* 56(3): 223–229. http://dx.doi.org/10.1198/00031300265
Are these figures classification accuracy on an independent test set?  How big is the test set?
"*Is the difficulty of this the reason why no one does hypothesis tests anymore?*" -- your assertion is untrue, since lots of people do hypothesis tests, as even a casual perusal of the questions posed here on CV show (indeed, they're still used far more than they ought to be, I think). The reason isn't the difficulty of explicitly computing the algebraic distributional forms, since we can carry out tests without being able to do that.
Have you had a look at 'scatterplot3d' ? http://cran.r-project.org/web/packages/scatterplot3d/scatterplot3d.pdf
$\\lambda$ is one minus the coeff on $Y_{i,t}$? But this is a dynamic panel that violates the strict exogeneity assumption. You should use Stata's `xtabond` command.
Oh yeah, I should have thought of that.
Q1: keep me updated if you got an ansver. Q2: you may run PCA first.
If anyone know of techniques which could exploit this property, that'd be great.  for example, provided the covariance matrix wasn't too non-stationary, perhaps I could estimate the covariance matrix for the LDA from many time points, and feed this to the classifier for a single time point?  the idea with using PCA was partly that since the PCA decomposition could be made over the entire dataset (all time points) that would 'implicitely' calculate the covariance matrix better than it can be estimated across trials at a single time only.  But perhaps that doesn't make sense!
@Tailltu I think, ideally, editing the question itself to lend clarity is preferable. That way your question would show up in searches by other people who may have an issue similar to yours.
Each variable takes all the sums of squares "available" to it when it is fitted, and every subsequent variable takes what is left conditional on prior variables having already taken what they could. Coefficients in the final model (and their standard errors) are effectively as if fitted last.
+ very nice and thorough answer.
That makes sense regards the geometric distribution. If i were to revert back to mixed effects model my data for each individual would be 0,0,0,1 (i.e. many zeroes before a 1) is this ok? Also I would need to add in individual as a randome effect do to 'some' repeated measures... not all individuals have repeated measures
`principal axis eigenvalues are based solely on the shared variance among the variables` only in the end, when iterations have stabilized communalities. Vicious circle: one either have to know number of factors already, to stabilize them, or know true communalities in advance (which is so seldom in practice). Cattell and "experts" are realistic when they estimate approximate number of factors from PCA eigenvalues.
Read the first chapter of Michael Greenacre's [Biplots in Practice](http://www.multivariatestatistics.org/biplots.html). Then read the rest of it when you understand that much! It is in essence a demonstration of how to do the type of interpretation from a biplot that you are asking for.
It doesn't seem clear what you're concerned about. You can get no error term (the interaction) with or without sphericity. What makes you think your second paragraph is necessarily true?
Thanks for solving the Q&R riddle for me.
It would be better to know more about the data first.  For instance is that buildup around 0 due to a whole lot of 0 values or just values that are near 0?
The first thing you'll need to do is verify whether or not this is a convex log likelihood.
Your solution seems to be focused on robust estimation of the location(mean/median). My question was rather about estimation of scale with the property of consistency with respect to retrieving the SD when the data generating distribution actually is normal.
It's sort of a variation of the coupon collector's problem, but instead of n coupons, there are not-quite-arbitrary vectors in an n-dimensional vector space.
@gung Thanks for catching that.  I fixed the example.  However, there is a typo in the figure: it looks like the value of HTT at line iii should be 4, rather than 2.
You are correct (+1).
Hi, welcome to the site. Please learn how to use LaTeX to format equations. I just submitted an edit to your question to format the integrals properly. Furthermore, this appears to be self-study/homework, please add the appropriate tag.
thanks, actually i just managed to depict it geometrically to understand what exactly is happening. Thank you so much anyway
Is there a way to report numerical findings of an autocorrelation, e.g. similar to an ANOVA or t-test?
Ordinal regression for continuous or ordinal $Y$ generalizes the concept of rank methods, through the use of semiparametric models.  Though one never really knows on what scale interaction should be analyzed, interaction tests from e.g. the proportional odds model are worth a look.  The R `rms` package's `orm` function makes it efficient to do ordinal regression on continuous $Y$ by efficiently handling thousands of intercepts.
You can not eliminate human judgment -- nor would you want to. That said, making a model explicit does help reveal assumptions and open them up for discussion.
(+1): I cannot count the times I thought I've understood something, but then I failed to explain it to someone else in easy words. Example: p-value ;)
Ok yes, you're completely right -- comparison. So it looks to me like there's a difference between Female and Male misdiagnosis. I'd like to make a fair and quantitative comparison instead of just pontificating about the differences. I wish to compare female and male misdiagnosis for the "IBS" category. Thanks.
I don't think there is a sound way to handle missing values in random forest. I know there are other tree based methods (even CARTs themselves) that can be built with surrogate splits on the predictors (as I understand it you are trying to predict time series with a RF so the lagged variables are your predictors). Another thing that can be interesting is a semi-supervised random forest which I read about sometime ago. Look it up.