tttjjj/elt_bench huggingface dataset:
Hi Gavin.  A study design: Let us say I have heights of kids and their parents - and I wish to estimate the 95 quantile of the kids height from the parents one.  But the kids I have are also divided to kids from different countries, and I want countries to be a random effect. We might also want to account for gender and age, for example.  Any further advice would be great. :)
This question opens up the broader, and potentially more useful to you, question of whether $R^2$ is an appropriate metric for a classification model.  Typically we'd avoid regression (`rpart`) for binary outcomes and resort to classification, as you've done; the metrics of choice for classification are mostly variants or transforms of the fraction of 1s your model identified as such and the fraction of 0s your model identified as 1s.  Are you sure that $R^2$ is what you want or need?
+1. Another solution, *based only on what is visible in the output*, is `sm$sigma^2 * sm$fstatistic[3]/(1+sum(sm$fstatistic[2:3]))`. That is, from the antepenultimate row you read off the $8.173$ and $58$ df and in the final row count the number of parameters ($1+1$), giving $8.173^2\\times 58/(1+1+58) = 64.57$.
I gathered that was the intent, but in its current form your reply is easy to misunderstand.  E.g., is it a sin to assess the "impact" of "one-time pulses" or is it a sin not to do that?  This is sufficiently vague that a good case could be made for either interpretation!  (Yes, it's a sin, because one-time pulses might just be outliers to which you don't want to grant undue influence and accounting for them all could overparameterize the model; no, they need to be included because their effects may linger for a long time and ignoring that could bias estimates of other parameters.)
@MaxG What I mean by adding a term or several is essentially adding a time variable - one that makes sense for your study - or several in a more sophisticated form (polynomials of time for example) - which allows the constant hazard assumed by the poisson model to "flex" a bit. I would have used a likelihood ratio test to check for over-dispersion as well, so you should be good, though it seems your model may be having some issues converging (see the warnings about iteration limits reached). Comments on the similarity between CRR and Poisson in my answer.
You got some good responses here. If any of them helped you, please consider accepting one of them. It's what keeps people answering questions :)
I think that in your case you are not really comparing two models. You have a problem in entropy estimation in the initial experiment because you should wait to see a longer sequence.
Welcome to the site, @Shyam. I have added the 'homework' tag, because your Q looks like it is. If it's not, you can edit your Q to remove the tag w/ my apologies. Note that homework gets special treatment here: we provide hints to help you work through the problem, instead of a full solution, so that you can learn from the process (see our [FAQ](http://stats.stackexchange.com/faq)).
By comparing with Amazon, if there is correlation then what would be the benefit of the equation? if there is no correlation then does that mean it is not valid or just another way of ranking? Many thanks
Also, I have posted a few good papers and books that discuss the use of AIC [here](http://stats.stackexchange.com/questions/18746/can-the-aicc-corrected-akaike-information-criterion-measure-be-used-for-post-h/33429#33429).
You should probably take a course in regression, there is much more to it than just looking at the output, at least when the output is so limited. You have assumptions to check, graphs to make, etc.
@AlecosPapadopoulos I actually haven't. I am under the impression the derivation there consists of algebraic steps typically taught in math courses lacking the explanation I'm looking for. Can you say otherwise?
@Glen_b: Okay, but can you do it with the minimum number of flips per generated variate?
Also, given this is a homework question, you should add the `self-study` tag.
What is the policy on cross-posting? The exact same question was asked on math.stackexchange.com: http://math.stackexchange.com/questions/15214/finding-the-change-point-in-data-from-a-piecewise-linear-function
No there is no MODE. See [this NABBLE discussion](http://spssx-discussion.1045642.n5.nabble.com/Computing-a-Variable-in-SPSS-Mode-of-Many-Variables-td5713850.html#a5713855) for code on how to generate the MODE across cases. It tends to be a bit hairy because the mode can easily be multiple values.
Strictly, the residuals are only estimates of the unknown and unknowable errors or disturbances, so even if normality is correct in principle, you can't get exactly normal residuals in practice. More importantly, normality of errors is the least important assumption in these methods!
Is this a real question? If yes, I recommend any standard text (ESL by Friedman-Hastie-Tibs is freely available online). If not, best question EVER!
Djamillah -- when you do that with expectations, it works because expectation is a linear operator. Variance isn't
This question was simultaneously [crossposted](http://math.stackexchange.com/q/570625/7003) to math.SE, where it has an accepted answer. For future reference, please do *not* crosspost simultaneously. [This meta post](http://meta.stackexchange.com/q/64068) contains more details regarding the philosophy behind this.
Thanks for your response, Glen_b. I'm splitting my larger sample to avoid training a classifier on data with a dependent variable that has a highly-skewed value distribution. Also, because this way I can limit the amount of data that I have in memory at any given time. By averaging values, I'm trying to determine whether the t-statistics across samples are reliably different than zero. What is the situation in which you can summarize information from many p-values? Why are effect sizes preferable? How do I take into account the dependence between my samples?
@PeterEllis - Whoops, yeah it was a typo.
Since you prefer the answer on math.SE to the ones given here, why don't you delete this question?
@whuber Thanks!
I might be misreading the question, but it seems that in spirit it is asking for more than how to code a purely nominal explanatory variable.  (The clue for that comes from the reference to ordinal regression.) Perhaps the best answer is "use some kind of dummy coding," but maybe, on the other hand, the question that's really lurking here concerns the choices one makes for modeling data that appear to be categorical but might have *some* structure: perhaps not an order, but something that would allow one to use many fewer parameters than categories. But maybe I'm being too imaginative ...
If the number of +s is about the same as the number of -s the differences could be just due to chance.  If they are very difference there could be a real difference in performance.  But I think a better way would be to construct a 2x2 contingency table of predictions and test for discordance in the contingency table or you could apply Cohen's Kappa test for agreement/disagreement.  If you want to compare performance look for significant differences in their error rates.
FWIW, the `I(foo)` form in `R` models basically means to compute the `foo` expression without interpreting it as special 'model language'. So for your case, it is equivalent to making a dummy variable `newvar <- TreeVol^-0.82`, appending it to `test` and adding a `newvar` term to your model.
This doesn't feel like a programming question, but rather one about how to interpret the results of a logistic regression. As such, I think this is off topic for SO. I would suggest consulting a statistics textbook, or even better a statistician.
Okay, apparently FPGrowth_association_rules from SPMF does this. Let's see what it will produce...
I don't get it, are you asking a new question here? Do you want to know how to do a one sample t-test in R? How are normal data w/ mean=25 & SD=4 related to a Bernoulli? If this is a totally different question, it would be best to start a new question, instead of deleting most of your old one. If the older phrasing let to a misunderstanding & answers that aren't really what you need, we prefer you edit *below* the original text.
+1 - Another reason is to identify interaction effects with treatment, but that takes more than just "controlling" for other factors like the OP mentioned.
+1.  The `R` implementation of Tukey's HSD is `TukeyHSD`. In using it, you will discover that **this question typically has no unique or definite answer**--there often is just a continuum of slight but insignificant differences among the groups, even though the extreme groups are significantly different.
I could think of some options here, but maybe it's better to open another question on what software exists to fit regularized GLMM models (or even better, specifically the kind of GLMM you want, which I think is a logistic regression).
If I remember correctly, it is often more correct to say that the CI covers the true value AT LEAST 95% of the times the experiment is repeated. That is, the exact CI often cannot be derived.
+1 for FDA, this was what first came to my mind too.
Yes, that does it -- thank you.
Maybe I'm missing the point, but isn't this *the* use case for k-means? You know you have three classes and that the future temperature for some class will be nearby to previous measurements for that class. The fact that you also have time data probably can be incorporated to make the problem even easier since temp won't swing wildly in 60 seconds... unless someone sets your sensor on fire.
I think the point of the question is how to define a straight moving average for points at either end of the time series and not how to find the best fitting timne series model of a form such as an ARIMA.  This is just a class exercise.  I am not sure that there is a unique answer to this.  Nor do I quite understand the logic of m=3 and m=5 cases to extend them to general m.
Normally SSX is defined such that $\\text{SSX} = \\text{Var}(X)\\times (n-1)$ not $\\text{Var}(X)/ (n-1)$. Please check.
I haven't worked with endogenous/exogenous variables. Having said that: Have you thought of using `lme` instead of `lmer` so that you can define an AR=1 correlation structure for your errors and see if that takes care of your endogeneity?
How strong is a "relatively strong ... correlation structure"?  With one million observations, even correlations of 0.99 shouldn't be a problem, with the only caveat having to do with the condition number of the matrix $X^TX$.
I realize the the wilcoxsign_test documentation states that it is for repeated measures designs, which is what I am doing, while the wilcox_test is documented as useful for independent data sets. However, the wilcox_test has a blocking variable documented in the formula, which indicates that it should handle handle pairwise comparisons as well.
Thanks @rapaio. Your functional description near the end finally made things click. Greatly appreciated.
Thanks. But I'm looking for a tool available in R or SAS. Then ten distributions above are just one example, I have a lot of series of distributions to cluster.
Fair enough, Xavier: now I understand the thrust of your question better.  Thank you for the clarification.  And this issue actually has more than academic (or "byzantine") applications.
@Jonathan You may try looking at a plot featuring the fraction of real object form some class as a function of a fraction of votes for this class from the forest. I was working on one problem which required confidence score and it turned out that I managed to obtain a very nice logistic curve.
that's not a task for classification. You want to use Poisson (or even the usual LS) regression here.
+1 for your try , but i am still very confused
5) Is this homework? Then please add the `self-study` tag.
No, there is no problem with negative means and variances are always positive.
Related: http://stats.stackexchange.com/questions/2306/feature-selection-for-final-model-when-performing-cross-validation-in-machine
@user1180428: I've edited my answer, which may now provide a possible alternative for you.
@rolando Yes, but if the data are subsequently analyzed as if they were from a random sample, they will usually be biased.
Yes, I have sample size and Cohen's d
I am cloudy on the details (and far from being even a novice when it comes to the NegBin) but recall Joseph Hilbe discussing this in his book Negative Binomial Regression (2nd Edition). He comments on p.9 that $V$ is a natural expression of the Poisson-gamma mixture view of the NegBin. After their 1989 book Nelder developed the *kk* macro for GenStat, in which he favours a direct relationship between $\\alpha$ and $\\mu^2$ with the variance $V = \\mu + \\alpha \\mu^2$ and that this direct parametrisation has proven very popular recently.
Your question is pretty vague and as it stands is more of a request for an introduction to plotting with R and less of a specific problem. If you provide sample data either by making it up or by using `dput()` to paste the contents of your actual data into the question - people can provide specific tips. Otherwise, [this tutorial](http://www.harding.edu/fmccown/r/) addresses lines and bar charts well.
+1 Great answer. One of the best and most accessible explanations of the Wilcoxon-Mann-Whitney test I know. Thank you.
@Daniel Are you saying that*every* participant consistently answered every question as predicted by the model?  That's what 100% means to me.
I'm not sure about your screenshots. Could you add or link the full Octave code?
The CLT is not needed to demonstrate any of this. In fact, the relationships among Normal distributions of differing means and variances follow immediately from their definitions (or are actually used as definitions).
Particularly good the way you related it back to effect size and practical (rather than statistical) significance. I'd just add that there are a couple of things to watch out for: (i) when the population about which you wish to make some inference is not the population you have completely sampled (people often say 'I've sampled the population' ... but it's not the exact one they want to talk about); and (ii) when you sample a large fraction of your finite population of interest, even small biases in sampling (etc) can loom very large, dominating the (likely quite tiny) variance of an effect.
Supporting this derivation, it is also discussed by Richardson (2011, Educational Research Review), attributing it to Cohen (1965 and 1973).
@Hans Ivers, do you have any input on this related question of mine: http://stats.stackexchange.com/questions/96151/calculating-cis-for-eta2-via-z-scores-sample-size/97079?noredirect=1#97079
I would recommend using `nlme`'s `lme` or `lme4`'s `lmer` function. They are better documented and with cleaner syntax. In the long run their flexibility will definitely prove beneficial. I think what you are doing appears sensible. Just be weary of *multiple testing* and *data dredging* issues.
Thanks. your answer probably exceeds my needs cause I'm just trying to estimate the out of sample forecasting errors, and using a RW model, the forecast k step ahead  y(t+k) it's equal to y(t).And this is what I wrote in my code but I obtained too much small errors, mach smaller than the paper I'm trying to replicate. For this reason  I think that I did a mistake. But I can't find it!
Great reference there - the Cox paper.
No, it is not correct to do data dredging to decide which parameters to remove from the model.  The value of random forests, like elastic net, is that it incorporates the right amount of shrinkage.  Starting over with a subset of the variables found in a way that was not masked to $Y$ will cause bias.
I plan to do the following then. Train three models : model 1 from my example, model 2 from my example and model 3 (which is model 1 + three dummy variables corresponding with the three interactions that are significant). And the I will see which one performes best... Does this seem reasonable to you?
Are you looking for a domain-specific answer, or a general answer of why this kind of thing is done in a linear model? Non-domain-wise, I believe it's common to have age and age-squared in survival-type studies to model the relatively linear failure rate during a subject's prime years, followed by a rapidly increasing failure rate as the subject reaches "old age". Would this apply in a genetic association study if some characteristic was associated with old age?
Because Levene's statistic is a ratio of *squares* constructed from those absolute residuals, and is referred to an F distribution, it is not immediately apparent that it should be any more robust than other tests based on ratios of squares!  You may be thinking of more robust variants, such as the [Brown-Forsythe test](http://www.itl.nist.gov/div898/handbook/eda/section3/eda35a.htm).  See a good discussion by @chl at http://stats.stackexchange.com/questions/2591/when-to-use-nonparametric-test-of-homoscedasticity-assumption.
Both look like very good ideas. In fact, the two approaches you propose even *complement* one another. I will leave the question open still a little while.
Honestly, I find the information about probability distributions on Wikipedia to be convenient & useful. Here is the [list of probability distributions](http://en.wikipedia.org/wiki/List_of_probability_distributions), which contains some, brief, information on where they are typically used.
For an AR process to be stationary the roots of the AR polynomial must be outside the unit circle. Thus if the model is an AR(1) the coefficient must be absolutely less than 1.0 . All AR processes are not stationary.
Did you look at log-linear models as well? Should be loglin in R.
@Glen_b Happy to add the self-study tag, although this was related to a simulation study I am working on where I am drawing standard deviations from a Gamma, and then wanted the mean of the variances (i.e., squared Gammas).
I have the list of values of random variables and the corresponding CDFs are represented as list of values.
(+1 yesterday) A couple of things might be taken into account in (1.) above. Since the lower bound of the frequency range of interest is pretty small, one alternative to consider is simply to apply a low-pass filter and subtract the mean. Second, instead of an IIR filter, one might consider a linear-phase finite-impulse-response filter in this instance. I say this because I suspect that to predict gestures by using the multi-axis accelerometer signal, one will want to keep the movement synchronized independently of frequency. (cont.)
Antony, if you *had* stopped the study when the desired significance was reached, then you'd be on thin ice like you say. But my understanding is that you collected the data and then analyzed it using standard, accepted techniques. Post hoc power is an attempt to re-analyze the same data and is completely unnecessary (plus it is *not* an accepted, standard technique in the eyes of most statisticians). Power is for prospective use only.
@TalGalili come to the dark side - we have cookies. Incidentally, if you're ever doing some experiments with millisecond response time recording, then Linux is about the only way you can go (thats what happened to me).
@Chris: your "item" is the latent variable that drives the correlation between two dependent variables (acidity and consistency). "fact that multiple data points come from the same subjects" means that here is a random effect of the subjects.
Randomizing the chosen side probably doesn't matter, I was just concerned about **A)** Accounting for any possible true bias in the dice and **B)** Making sure any one single anomalous session (that 1 in 20 due to true randomness I talked about) isn't taken as a confirmation of his psychic ability. Probably 6 sessions, one for each number, would be sufficient, as long as my **X** (and confidence level) is high enough.
Hi there, do you mean that you are looking at the distributions of final scores for games? I'm not following what you mean by results of large iterations of games.
thanks a lot for this great post! I haven't yet fully understood your way of doing it but it's now much closer to what I'd like to have. I think I can adjust the values in here: x <- rbind(mvrnorm(n1, c(-2,-1), sigma(0, c(1/2,1))),
           mvrnorm(n2, c(0,1), sigma(pi/3, c(1, 1/3)))) to achieve the data I want! thanks very much again for that great peace of code!
Mbq: thanks for sharing your thoughts. I have tried a k-fold cross-validation and the results are average of the k-runs. HF is a subset of NF, but several features are merged as  one single feature.
I played with that idea, but what if a new user comes along later? Then 85% similar at time t+1 might not mean the same thing as 85% similar at time t.
You might want to use the robust option to calculate heteroscedasticity robust standard errors. People with little schooling tend to have less variance in their earnings than those with lots of education, so the error variance is not going to be constant. Also the rural dummy can be problematic because the choice of living in cities is potentially endogenous.
The last sentence is wrong: confusion tables for $N$ classes are usually of size $N \\times N$. From that you can derive $N$ $2\\times2$ tables for "class a" vs. "not-class-a", but that is not so common although sometimes done implicitly (e.g. specificity calculations).
By average F1 I mean the sum of F1s of the classes divided by the number of classes. Yes, I have seen that there are methods, which give more weight to new examples, but I don't know how to decide the weights or which concepts are defined as old. Moreover, I would like to ask why the trend is to increase with respect to the amount of data. If we have a model that has high bias, then increasing the training set will not help, right?
Elaine...I'm your desnity.
I think what you want is to see if there is an interaction, & if so, what simple effects exist.
The first half of [this answer](http://stats.stackexchange.com/questions/71260/what-is-the-intuition-behind-conditional-gaussian-distributions/71303#71303) (through the statement "Choosing an angle of 45 degrees makes the ellipse symmetric around the square's diagonal (part of the line y=x)") explains this geometrically, perhaps in much more detail than you really wanted :-).
@Glen_b - I don't think that's exactly what I am using (although it could be the formula just looks different to me). A description of what I am doing is provided [here](http://www.unc.edu/courses/2007spring/enst/562/001/docs/lectures/lecture28.htm). I will look into stepping through the code to see what it does. Mainly I'm interested in finding a way to explain this since it contradicts my understanding of CIs.
+1 That's a creative idea.  It sounds like the "Gower Similarity" is a scaled [Hamming distance](http://en.wikipedia.org/wiki/Hamming_distance).
Hi thanks for the answer. I'm basically trying to fill up missing values for the age of passsengers in a dataset (positive mnumerical variable).  I thought as a start I could use a linear model for a rough estimation which would improve on just filling up with the median.... I thought about using glm or a transformation but I wanted to find out first if a "truncated " linear model could do the trick.
(+1) I can confirm that R's `kmeans()` expects raw data. Also, using an euclidean metric with k-means clustering based on standardized values (z-scores) is equivalent to relying on Pearson's correlation distance.
Can you say what you are going to use the model for?  I am skeptical that you are really only interested in prediction.
Prediction does not necessarily have to be about the future.
If you look up Frank Critchley's 1986 paper "influence functions in principal components" (can't remember the exact name of the paper).  He defines the influence function for ordinary regression here (which may or may not prove my answer wrong).
@NickCox. I get a variance of 2 though. I just did a ~ N(0,1) and b ~ N(0,1). Generated 100 samples for each. Let c = a + b and var(c) = 2. By properties of variance. var(a+b_ = var(a)+var(b)+2cov(a,b). Since a and b are indepenent cov(a,b) = 0. var(a) = 1 and var(b) =1 so it is equal to 2