tttjjj/elt_bench huggingface dataset:
@Timo: That looks generally right except that the needed sample size shouldn't depend on the population size (many populations are infinite after all). For just a rough estimate of the variance, 30 or so should be good enough. Rushdi I've been reading a little on this topic recently and it actually gets pretty complicated; google "stopping condition" and see the kind of scary stuff you get... but if you just want to know *about* how much data to take, I'd say take a sample of 30, estimate the variance, collect more data, if it's not enough, repeat.
And here's something from SAS about it.  http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_calis_sect102.htm I've never seen anything in R.
You may find the discussion at http://stats.stackexchange.com/questions/4220/probability-distribution-value-exceeding-1-is-ok sheds some light on your question.
You could use bootstrap for the error rate estimation.
I wasn't being precise. There's an example of a slightly more precise version of the argument [here](http://www.stat.tamu.edu/~hart/611/neyman.alt.pdf).
Is [this (pdf)](http://www.stat.washington.edu/~raftery/Research/PDF/socmeth1995.pdf), the Raftery paper you are referring to?
Another important thing to keep in mind; the performance of many classifiers is not linear in the amount of training data, but will typically be a sigmoid like curve. That means 500 more tagged questions could be almost just as good a benefit as 5000. My advice is to work in small steps.
My short course is aimed at that audience, among others.  Info including handouts is at the web site for the full semester version of the course: http://biostat.mc.vanderbilt.edu/CourseBios330.  One of the many things I cover is why it is unreasonable to anti-log logistic regression coefficients to get odds ratios; this is in the context of allowing effects to be nonlinear and getting e.g. inter-quartile-range odds ratios.
@Peter The 2/3 rule (part of a 68-95-99.7% rule) is good for a huge variety of datasets, many of them non-normal and even for moderately skewed ones.  (The rule is quite good for symmetric datsets.)  The non-finiteness of the SD and mean are not "nonsense."  Taleb's example is one of the few non-contrived situations where the Cauchy distribution clearly governs the data-generation process.  The infiniteness of the SD does not derive from the possibility of missing the wall but from the distribution of actual hits.
Hi iampat and welcome to the site! Would you mind elaborating a little bit on why you're suggesting Additive Logistic Regression and what it is? Even just a few sentences would be great (plus, links with context tend to get far more up votes, so this would help you too :-)
+1.  This is a nice example of the importance of being clear about one's standard of "proof."  In many applications the one you invoke here--the "act as if" standard, if I may call it that--is so weak that nobody would accept it as "proof."  I do not deny its utility, though, and advocate this kind of approach to support rational decision making.  (But maybe Bayesian methods are better... :-)
I do not understand how it is possible to reparameterize. If I have a model whith e.g. the following shape:
a X + b Y + c (d Z + e W)
What would be possible a reparameterization?
Or perhaps do you have a reference I could read?
Ad 3) - so in REML it is as I expected, i.e. it uses the `SSE/df` instead of `SSE/n`?
I think genetic algorithms might do well (although they tend to be quite slow, in my limited experience). Local minima are indeed a problem: using several starting points with gradient climbing algorithms is always advisable. But the problem is not only multiple local mínima, also that quite often solutions are in the boundary of the feasible parameter space, e.g. some variances of the noises are estimated to have value zero.
Andrew Gelman has a paper that uses surveys from Gallup and GSS simultaneously to examine [state level effects of death penalty opinion](http://andrewgelman.com/2011/10/death/). Perhaps that would give some useful references to this problem.
Having too many rows does not result in overfitting. Having too many columns does.
What you need is a basic understanding of probabilities, odds, & odds ratios. It might help you to read my answer here: [interpretation of simple predictions to odds ratios in logistic regression](http://stats.stackexchange.com/questions/34636//34638#34638).
OK, that's not a bad assumption.
Both of the answers by Srikant and whuber were equally informative in helping me answer this question. I wish I could give both an accepted checkmark, but I chose whubers response simply for the added context in helping to demonstrate how I would integrate given my prior distributions.
Of course, that assumes that you **know** the probabilities of your events. In the real world, after a certain number of trials you have to point out that your calculations give you a 99.999% chance of seeing the unlikely event at least once by now, and you still haven't seen it, so perhaps it's less likely than you thought (or maybe even impossible).
If by "inconsistent" you mean that the model with the lowest Brier score isn't the one with the lowest AIC, the answer is there's no reason to expect them to be. AIC's based on a likelihood measure of fit, & more importantly it penalizes bigger models to avoid overfitting - that's the point of it.
Thanks for that; it's perfect!
The day Open Source Language will replace MATLAB will be the best day to the engineering world.
thanks so much now it works @ Patrick Coulombe
Durations are usually modeled with survival analyses like the [Cox model](http://en.wikipedia.org/wiki/Proportional_hazards_models) b/c they are usually censored (ie they have observations that didn't stop freezing b/c the sound stopped / the study ended before they stopped freezing). Do you have any such censored observations? Another possibility, since you divided by total time is, if you don't have any 0%'s or 100%'s is to use Beta regression for continuous proportions. Both of these are fairly advanced.
I'm not R user. I used SPSS, where I performed hierarchical procedure exactly the way I described. I saved 20- through 2-cluster solutions and compared them with silhouette statistic, which showed that 2-3 clusters were most reasonable. Other settings may yield other results, of course. Clusters in your data are far not clear-cut, so I think you can't escape comparing several solutions for interpretability in order to choose one.
The linear interpolation of $E(Y_i |X_i, G_i)$ does not have to agree substantively with that of $E(X_i|Y_i, G_i)$. For example, if 

$$ Y_i = \\beta_0 + \\beta_1 X_i + \\beta_2 G_i + \\beta_3 X_i G_i + \\varepsilon_i $$ 

is the correct model then rearranging terms will show you that $E(X_i | Y_i)$ depends non-linearly on $G_i$, so there's no reason to think inference about the coefficient of $Y_i G_i$ in a linear model for $X_i$ should agree with that of $\\beta_3$ above.
(1) there was no for data analysis plan this study? even the IRB document should contain some minimal elements of a data analysis plan. (2) what do you feel your options are? (3) you haven't really described how pain is evaluated. Sounds like 6 ordinal categories, but unclear (4) have you done a simple plot of the data?
I believe this question is not about unexpected signs of estimates, but simply about the behavior of software that seems to graph the results of the regression in an unexpected way.
A few more Qs: 1) Do you get a p-value from a log likelihood test. 2) If the factor is insignificant, is it then okay to read p-values from other factors within the global model while this (multi-level) factor is still in the model?
You are asking for a [prediction interval](http://stats.stackexchange.com/search?tab=votes&q=%22prediction%20interval%22). Your second thought leads to  non-parametric prediction intervals (which I believe have not been mentioned on this site before).
@Glen_b The mistake I made was that I computed the sum up to 5 instead up to 3.
Peter, did you catch that the question is about a *finite* population of known size and that the sample (apparently) is without replacement?
@ttnphns In the post, it says how each item was scored: "Always", "sometimes", "Never". That sounds like ordinal data to me!
I didn't get what you meant by "it is not right to choose the one with the bigger P". In a CPD (such as $ P(L \\mid G) $ ), any one of the assignments is chosen at random, not the one with the highest probability.
Hi: Here is the link to the file containing the output (   http://www.mediafire.com/view/?208z1tw155e6juy   ). The R square is quite high - is it "too good" for humanities research? Appreciate your views and thank you.
How about + or - 10% if the data is 15% Democrats and 12% Republicans (I'm concerned on figures only). I'm still quite confused if we add and deduct 10% of those numbers. Will they make sense? Thanks a lot!
This is an interesting question, but I'm not sure if you'll get an answer.
@Mike Seems like you have two SE accounts, no?
+1, thanks Aniko for explaining this. As an alternative to eyeballing KM-plots to find change-point candidates, one could consider a model with time-varying effects, see e.g. the R package timereg.
Thank you for this clarification. I really appreciate your help with this one.
All of these are at least somewhat suspect. They seem to have clear bounds, which isn't really consistent w/ the standard OLS regression model (but may not pose too much of a problem in practice). Also, there may be some non-linearity--ie, mis-specified functional form. The wiggly-ness of the fitted line on the plots is driven by a smoothing parameter, you don't necessarily know how literally to take it. You can get another perspective on that same information by getting the `acf` & `pcf` of those residuals. What are these data / the models?
thank you, looking.
Your answer kind of implies that supervised learning is preferable to semi-supervised learning, where ever feasible. Is that correct? If not, when might semi-supervised learning be better?
good that's what I thought, did not know the correl formula. Any thought about the second question (should I be bothered that an infinite number of $\\beta$ maximize the correl) ?
@user27271 Yes - the software used is discussed in the above chapter. It runs with _Mathematica_.
I'm not sure what you mean by "strong, linear, growing correlation".  Can you show the plot?  It's perfectly reasonable to plot residuals against fitted values.  In general, you want there to be no relationship--a flat horizontal line running through the center.  In addition, you want the vertical dispersion of the residuals to be constant from the left side of your plot to the right.
You're right; sometimes the results are so obvious that applying a formal test would be overkill--and actually woudl look suspicious ("what's this guy trying to hide with his statistics?").  But *by themselves,* knowing only what we have been told so far, these data are **not** convincing.  For instance, maybe the treatment was *completely without effect* but there happened to be two strains of this disease in State 2 (but not in State 1) and the second strain was much more virulent.  The point is that any "proof" must rest on *scientific* argument, not a statistical one.
What is large performance variation? 1%? 10%? 25%? Over how many datapoints?
@Charlie: Using AIC for model selection is not, in general, consistent, even for very simple versions of the problem. BIC sometimes is.
K nearest neighbor, naive Bayes, logistic regression, support vector machines, in roughly ascending order of complexity.
4 experts sat down around the table, discussed each products and rank ordered each product (in excel). Then they took first 1% of products and said "these products are excelent", then they took following 10% and said "these products are very good"... etc
Could you make your title more explicit and descriptive? It seems your question pertains to trichotomizing an item/score (splitting into 3 groups).
(+1) Good update.
@carlos, You can learn a lot about a function by running `?cutree`. ;)
+1 and I would also add that simulating solutions can be really useful practice for when one needs to use simulations for actual work
Hey sorry didn't quite understand your answer. In my cross validation, i split my train set into train/test sets, and then run either a grid search of min finding algo on a train set, and optimize parameters according to the test set. in grid search, i can average the fit over the cross validated parameters, but can't in min finding algo. when trying to predict a new test set, i can't optimize on the new test set and optimizing on the whole train set would just overfit the training data. i probably misunderstood your answer though.
@Arvin  You should understand that the normal distribution has a unique _definition_ $$\\frac{1}{\\sigma\\sqrt{2\\pi}}\\exp(-(x-\\mu)^2/(2\\sigma^2))$$ (where $\\mu$ is the mean and $\\sigma$ the standard deviation) that everyone agrees on, but how it is _denoted_ (whether as $N(\\mu,\\sigma^2)$ or $N(\\mu,\\sigma)$ or $N(\\mu,1/\\sigma^2)$ (cf. comment by whuber) or Gaussian$(\\mu, \\sigma^2)$, etc) is different depending on the user.
Both of these are really excellent.
IRT won't let you uncover 'good' items to devise a scale easily. First of all, you have to identify the dimensionality of your questionnaire, and apply basic item-level analysis (item difficulty, discriminatory power, item-scale correlation) and test-level analysis (reliability or internal consistency). Second, be careful with $\\chi^2$ tests as they will be impacted by large (or low) $N$. Third, adding extra parameters to your model (discrimination, and/or guessing) should be motivated by theory, and not data driven. Could you perhaps tell us more about item content/wording and the examinees?
@whuber Just had a quick look. Yes - almost all of the timing is `FullSimplify`. Under Mma 9, the total time is 15 minutes. Under Mma 10, the same calculation took 10 hours - aha (running overnight). There does seem to be some bug or problem with `FullSimplify` under the new Mma 10.
The logic is a little subtler than laid out here.  When a result is significant at the 95% level, its p-value could be 0.05 or it could be practically 0.  In the latter case one might expect *no* significant tests in 100 independent random samples.  Another subtlety is that the p-value *observed* in one sample is not the same as the p-value that would be attained in the long run of repeated independent samples.  Also, the implicit presumption of independence of the p-values is rarely satisfied; usually the p-values reported for a dataset are correlated.
Are you sure it is about a *statistical* test and not a test from medicine, chemistry etc?
No; $\\gamma$ is the minimum; that is explicit in the page cited.
Cross posting is discouraged. You should probably close the question on Math.SE, especially if you haven't yet received an answer and you feel that this is the better place.
I've been pondering on this same question for nearly 12 months. See  http://stats.stackexchange.com/questions/21530/do-random-forests-exhibit-prediction-bias and http://www.kaggle.com/forums/t/1106/random-forests-newbie-question. I agree that the problem seems to be that outliers are not being represented well by the averages that are used as prediction at each node. Prediction adjustment or model-in-node seem like reasonable approaches, but not sure what the standard approach is.
Just highlighting a point made by Penguin_Knight that seems to get lost in the discussion: If, for the most part, all firms from a given country also have the same legal system, you can't put both countries and legal countries as predictor in an OLS regression model. From what you said until now, this seems to be your main problem at the moment.
Well, the number of Facebook searches may spike upwards based on this article. ;)
polr() - http://stat.ethz.ch/R-manual/R-patched/library/MASS/html/polr.html
It's also really only centered on two population t-tests somehow. This whole explanation seems to be wrong and misleading. In fact I would stop using that textbook and probably the software as well. ugh
Obviously my problem is to compute the standard errors for the coefficients. I am sure there is software for this, but I need to learn how to do it myself.
This question is very confusing because it asks about **four** distinctly different things: comparing "trends," comparing means, comparing another measure of location (implied by the Wilcoxon test), and comparing distributions (via the K-S test). Please tell us a little more about what these "paired histograms" represent (including how you obtained the original data and how you chose the histogram bins) and as much as you can about how you really want to compare these data: what is your ultimate objective?
Thank you for your answer, Peter. It is appreciated.

Yes, the data set is organized by patient and I have other variables which are there to record other outcomes (side effects from the drugs) as well as other possible covariates. I will have to read up on how to generate a propensity score as I am not familiar with it.
Gael, thanks for making the edits, I think they add clarity as this question is at the intersection of applied statistics, reliability engineering and decision making. Also thanks for adding the hyper-link to the e-handbook . . . I'm a newbie at this site.
"Is it that the product of two Gaussian PDFs does not describe the distribution of the product of the corresponding variables?" Yes, as StasK points out in his answer, multiplying two Gaussian PDFs (and using the same argument in both: $f_X(x)f_Y(x)$) rarely results in a PDF.  When it does give a PDF, or if we use $c\\cdot f_X(x)f_Y(x)$ where $c$ is chosen to give a PDF, the result is a _Gaussian_ PDF, while $XY$ is definitely _not_ a Gaussian random variable. Note that $f_X(x)f_Y(y)$ is a _joint_ PDF that ignores the correlation between $X$ and $Y$ and treats them as independent Gaussians.
Nice example, I would have thought some implementation of running quantiles was available in R (to avoid the problem with bins alltogether). Kind of reminds me of [bag-plots](http://rgm2.lab.nig.ac.jp/RGM2/func.php?rd_id=aplpack:bagplot). Also see Rob Hyndman's [extension](http://robjhyndman.com/papers/Rainbow5.pdf) in his Rainbow package.
This is probably what I would do. It may run into problems if the means are close to either bounds and it can be more difficult to interpret. @asdf needs to consider why these should be on the same scale (do they measure the same thing)?
"Predict the outcome with X set equal to zero and compare the difference? Standardize the coefficients and take the value of the beta coefficient as the percent contribution?" -- these are both perfectly valid ways to define "contribution." Which one makes sense is entirely up to you. Or more likely, entirely up to which method makes sense to use given your data. Or are you wondering whether one will produce estimates that are in some sense "more valid" than the other?
The first answer is incorrect, as Matthew Crumley's answer shows.  You need to compute $1 - 39/52\\times 38/51 \\times \\cdots \\times 33/46\\approx 0.885.$
Could you tell us anything more specific about the data?  *E.g.*, are the values constrained at all (to be nonnegative, or not to exceed some limit perhaps)?  Could you say a little about why you are interested in the SD and how you intend the chart's readers to interpret it?  Are you open to alternative visualizations (which would likely be significant improvements, since color really is not appropriate for conveying information about a standard deviation)?
If you identify the variance of the latent variable by constraining it to zero, you can then compare the error variances.  I'll try to write an answer later, if you don't get one.
Cheers, thank you for the detailed and informative response. Here is a subset of the data: 800 24 point time series. I've also included some labels here, but they are unnecessary for your purpose. The exact time series above are not in the dataset provided, but many similar ones are.

https://www.dropbox.com/sh/p1od9e4vx8ky66a/igt2OkNDbQ
Would the model with time be $Y_{ijkl} = \\mu + A_{i} +B_{j} + A_{i}B_{j}+ T_{k} + E_{ijkl}$?
+1 Peter. One situation that comes to mind is, for example, doing an online market research survey and being paid some nominal amount (there are websites you can do this on) and the more surveys you take, the more you are compensated (one could make **maybe** $100 a month doing this if they are pretty active). But, this dynamic would encourage just randomly filling in answers as quickly as possible, maybe not even reading the question. If you're using (2) from your list - do you just throw out the survey if there are contradictory answers? Are there other ways to detect "random answering"?
Dear Peter,
thank you for your swift reply. This helps! I am still a bit confused by the article, though. It uses the same regression example (and indeed logs the dependent wage variable - thanks for pointing it out). After applying the restrictions to avoid the dummy trap (coefficients of E3, R3 and F = 0), the article states: "These particular restrictions imply that the control group is a female with only a high school education who is single. This forms the comparison for all the other cases and thus confuses the interpretations of the restricted coefficients"
Sounds a bit counterintuitive
Do more variables always help? I thought there was a curse of dimensionality and adding more degrees of freedom is not always good. This question I have is can you know what the optimal result is beforehand, and then decide to stop say when you are close enough to optimal? Is there a global optimum?
If you're already ok with ban, then you should also be fine with bit. (i.e., no need for the full justification I gave below). Going from ban to bits is just a change of units, log base 2 instead of base 10. (Similarly, go with "nats" if you like base e).
1) In your bivariate Pearson formula, if "E" (mean in your code) implies division by _n_ then st. deviations must also be based on _n_ (not n-1). 2) Let all three variables be the same variable. In this case, we expect correlation to be 1 (as in bivariate case), but alas...
If this is for some course, please add the `self-study` tag. If it isn't self-study, please explain the context in which your question arises.
Hint: look at the coefficient of $r^x$ (for a fixed $r$) and note that $b r^{-m} = \\text{constant}$ has a one-dimensional family of solutions $(b,m)$ with $b =  r^m \\cdot \\text{constant}$.
Thanks a million, that's great!
Ah, a lot of my cases came from Africa, since ethnic conflicts are rampant there.  Do you da geographic study?

Would it be a huge problem to account for time? I found it really useful, especially due to the fact that certain variables are changing with time (political system, cold war etc.)
I think I do not well understand your first point. I do not think there is any regularization term in EM algorithm. Furthermore I launch the algorithm with multiple initialization points and take the result with the best likelihood. As shown in my edit, your second point does not apply here because the sizer of the sample tests in constant. For you third point, I think it is unlikely to caused by precision errors but I can investigate it. Finally, the datasets are all created in the same and do not look very different...
Thanks a lot for this very simple and intuitive answer. +1
I tend to think about this completely the other way around. If your variables were completely *in*-dependent of each other, PCA would be worthless. The reason for doing PCA is *because* your variables are "inherently *inter*-dependent".
@Gary - a p-value doesn't tell you to reject or not any more than a Z-score.  They are just numbers.  It is only the decision rule which determine accepting or rejecting.  This decision rule could equally well be defined in terms of a Z-score (e.g the $2\\sigma$ or $3\\sigma$ rule)
Thank you for this clarification. You are absolutely right.