tttjjj/elt_bench huggingface dataset:
One of the awkward features looks to be that not all ids are ranked in each time-specific ranking exercise, is that correct?  Also, in your extension to several island data sets, are the same things (a, b and c) being ranked on each island?
Awesome thanks Caracal. Off I go and do it that way.
Hi Melory, I agree with the answer by @gung although I disagree with analysing `BMI` as continuous. There is possibly a step function with `BMI` and the publication standard is to use `BMI` categories.
@kwak -- remove those "V1 V2..." from the file before doing anything. as.numeric won't work having unconvertable elements in vector.
Is it this? http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/247
just what I needed! in the end I decided just to calculate the entropy, multiply it by 10 and leave it without scaling it. it's good enogh like that. thank you very much :)
I assume GWAS is a genome-wide association study, but what is MAF?
Thank you!  Yes, I have the exact sample size and standard deviation.
Any thoughts on this? Or if anyone could explain AdaBoost in simple English, that will be helpful also. Thanks.
I know, I was very uncertain whether to put it here or stats.SE. I im truly sorry if I was wrong.
I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that *if feature selection uses the response* then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.
ARIMA is mainly used for modeling? I was advised that I could compare ARIMA coefficients. But I guess that I was pointed into the wrong direction. Do you by any chance know of any chance happen to know any useful descriptive statistics to examine such as standard deviation, median, and so on. Thank you for your help.
As a slight aside, what is the value to you of fitting a smooth distribution as opposed to using nonparametric or semiparametric methods?  And did you try more than one distribution?  If so you will see model uncertainty that makes the final estimate no better than the empirical cumulative distribution function estimator that you check against.
@Matthias: I think if your students are coming/aiming  from/for a social sciences field, teaching them R as a first step in Statistics is an overkill. Most of them will have problems with the concept of console, commands, syntax etc. and you'll spend more time going through "programming concepts" (what is 'function', 'loop' etc.) than "statistics".
I base this on prior experience, when I did tutorials for a Stats 101 in a Soc.Science department; people missed the point of the lectures cause they focused more in getting R to work for them than actually exploring their data.
I don't think there is a particular problem with having a sample size of 1 is the issue, the problem is that the test has no statistical power (i.e. the test will never reject the null hypothesis when it is false).  However, this reveals a problem with the "null ritual" being lampooned in the article, which ignores the issue of statistical power (and what H1 actually is, or prior information relevant to the problem).
Also, commercially available packages such as SPSS, SAS, and autobox does this automatically for you. http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_arima_sect045.htm http://www.autobox.com/cms/index.php/afs-university/autobox-examples/modeling-with-autobox#_Toc509396187 http://pic.dhe.ibm.com/infocenter/spssstat/v21r0m0/index.jsp?topic=%2Fcom.ibm.spss.statistics.help%2Falg_tsmodel_outliers.htm
Thanks @mbq. Got it. I was just thinking about performing "rough" RFE (using for example 100%, 90%, ..., 50% of features) with 10-fold cross-validation (CV) prior the 'mtry' adjustment. I just thought that as long as the elimination criteria are based on Gini-index, this should be relatively unbiased given appropriate number of trees and CV. But I see your point... Thanks for the suggestion!
@DikranMarsupial: Right, power does not depend on the observations. But it does depend on sample size and variability. And it can't be computed without defining a clear alternative hypothesis. So power is the characteristic of a test, given a sample size & SD, and a defined alternative hypothesis.
Similarity between two documents is meaningless. It's only interesting to ask if two documents are more similar to each other than to other documents. If the size of your corpus is greater than 2, then LSA may well produce more useful similarity measures between any pair of documents (or it may not).
Hmm, I'm sure I'm missing something but I tried this and it gave me the same coefficients as a normal lm() regression
Why use a fitted model?  With over 100K values, it's possible that *any* fit will disagree significantly with the data. What's the matter with just finding the $.05$ (or $.95$) quantiles of the data (presuming that's what you mean by "p-value," which otherwise makes no sense in this context)?
Although adding the noise will make `lm` run, your attempted "bootstrap" may then give answers that depend *arbitrarily* on the noise. The `NA` values are a clear sign that the particular bootstrap you are attempting may be invalid or give misleading results. Please consider rethinking your analytical strategy rather than papering over its deficiencies with this artificial fix-up.  Perhaps you could instead ask a question in which you describe your data and what you are trying to learn about them, point out the problem with the bootstrap, and ask about alternative approaches.
Taking a wild guess:  If zero correlation, you may as well conduct separate anovas and thus simplify your task. If very high correlation, you may as well conduct anova on just one of the Y variables since results will be largely the same for all the others.
You're probably right -- I've made a few combined box plots and it seems to get the point across better than an aggregate number would. Thanks!
Thanks Glen.  I will take a look at HoltWinters and Time Series.  I appreciate the input.
@Against, that's *sort* of the way to do it: summarizing the distribution with a putative outlier removed works just fine when there is only one outlier. Life gets considerably more complicated when there might be more than one outlier, because the presence of one can "mask" the others. This is why the methods suggested by user603 are appealing: they do not require you to determine beforehand how many outliers there might be.
Assuming your interested in the relationship between click count and total time, put them on a scatter-plot. You can then use other techniques to distinguish between the tasks and/or testers, i.e. facets in a small multiple graph or color/use different symbols for the points.
Here are some of the questions that I remembered seeing on math.SE, but none of them are quite what you ask: (**1**) http://math.stackexchange.com/questions/64028 (**2**) http://math.stackexchange.com/questions/66777 (**3**) http://math.stackexchange.com/questions/101692 (**4**) http://math.stackexchange.com/questions/50775
This is a very general question.  I would suggest looking at the economic literature as a starting point rather than thinking of it as a purely statistical problem.
I have tested it a bit and it either performs a little better or a little worse. I have never seen the "great improvement" they talk about. How does it work anyway? can anyone point me to some references? I have yet to find a description of how it is performed.
It would be nice if you give the source of the citation.
I don't think there's a simple yes-no answer to this; it depends on the data, the model, and the purpose of the investigation. However, the log of a ratio (such as inflation, which is just a ratio*100) is the difference of the logs. i.e.
$log(p2/p1) = log(p2) - log(p1)$
Great, thanks Remi.b.
Thanks, at the moment I have two answers with no upvotes and I'm not versed enough in statistics to decide on the correct one. Can you a) find a reference, and b) tell me, were I to run a between-subjects study with equal or greater power and still found no result, while I do in the within-one, would my assumptions above be substantiated by that result?
Economists have traditionally used low frequency data (like monthly or quarterly). There has been a lot of work on developing higher frequency data to improve forecasts. So you might look into mixed frequency techniques (Midas or Kalman Filters might be a start).
@BlainWaan The EM algoriithm is a method for finding the maximum likelihood estimate when the the simple method of setting partial derivatives to zero cannot be applied. So it is wrong to say that you are using EM instead of maximum likelihood. McLachlan's books on mixture models and the EM algorithm are very helpful in this regard.  By the way, what does your comment have to do with the OPs question?
Removal of "insignificant" variables is very problematic.
[This](http://stats.stackexchange.com/a/31634/26080) may be helpful.
In addition, your notation suggests that your data are time series and you are plotting successive values. That alone suggests that regression here is not the main way forward; it is to think what drives the time series.
What does c index?
I think you have a good answer here: http://stats.stackexchange.com/questions/73944/what-are-the-limitations-of-kernel-methods-and-when-to-use-kernel-methods/73945#73945. That depends on the size and characteristics of the data you refer to
Yes, GEE always estimates the marginal effects, even when a correlation structure has been specified (unlike the mixed model). Regular linear regression is not necessarily at the population level. The conditional and marginal effects are the same for linear models, but for correlated data, the mixed model estimates intracluster correlations for weighted LS, and this gives you individual level SE estimates (GEE has population level SE estimates which are generally bigger).
From `?train`: `"The combination with the optimal resampling statistic is chosen as the final model and the entire training set is used to fit a final model."`. Often useful to read the help page.
The implication that linear regression and OLS are synonyms is rather odd -- it conflates a model and a method of estimation -- but I guess your meaning is clear.
You mention death as competing risk. Are there a lot of deaths? Censoring the AE event at death distorts the probabilty of having such AE (if you run a Cox-model.)
A _proof_ that the joint CDF has no closed-form expression is long but if you are willing to accept the better-known result that there is no closed-form expression for the _univariate_ Gaussian CDF, then you can argue that a closed-form expression for the joint CDF implies a closed-form expression for the univariate CDF since $F_X(x) = \\lim_{y\\to \\infty} F_{X,Y}(x,y)$. Read about _integration in finite terms_ as in for example [this paper](http://www.cs.ru.nl/~freek/courses/mfocs-2012/risch/1995313.pdf) by Risch from Amer. Math. Monthly.
I intentionally avoided referencing the [Wikipedia article on the noncentral chi distribution](http://en.wikipedia.org/wiki/Noncentral_chi_distribution) because it uses a different parameterization!  (Its $\\lambda$ is the square root of the $\\lambda$ used here.)
A probability, too, is a "mathematical construction" and not "directly observable." Does this mean that probability does not exist?  Overall, this interesting reply seems based on an untenable premise that data should be characterized by the values they *do* have rather than by the values a mathematical model allows them to have. The latter is the crucial characteristic, not the former.  This all suggests that what matters in the continuous/discrete distinction is *how we think about the data* (that is, how we model them).
I don't get your "the mean and variance suffice to recover the whole distribution".
Suppose that my dependent variable is BMI and I am interested in making inference on individuals at the tail of its distribution, how can I exactly use the ordinary regression methods?
I'd add that you might want to log transform the outcome before modeling it, will likely make things easier.
qWeightNormal <- quantile(weight_Normals, probs=seq(0,1,0.01))
So, I can know the percentiles of weight_Normals. Then, how do I know the mapping of each weight of patient with disease to the percentile of noemal patient?
We never like to say that we accept the null hypothesis.  It is more appropriate to say that you can't reject it.  Your statement that your company is at least as profitable as the industry standard is too strong.  I would say that you don't have sufficient evidence to conclude that you are less profitable.
I think you should augment your question with some examplar images. Picture says a thousand words... and it would give anyone who would like to help you some test data to work with.
If the desired frequencies are really probabilities, it would be easier to use the `prob` argument for `sample()`: `sample(LETTERS[1:4], 10000, replace=TRUE, prob=c(0.1, 0.2, 0.65, 0.05))`
Backward selection is not a good method, as has been discussed here a lot.

The rest of your question seems more appropriate to either StackOverflow, an R-list, or the developers of the package.
I'm sorry about not being clear, I can see now that the English statement was a bit ambiguous.  In fact you are correct, I want to test all pair-wise comparisons.
Thanks. So you mean that the shape of the response distribution (in particular, its variance, or maybe its spread) differs depending on the level of your independent variable? However, you end up with a 3-level discrete response variable (taking values in {0,1,2}): what would a variance test, usually applied on a continuous measure, mean in that case? What about your independent variable: Is it continuous or discrete like your outcome?
I'm continually amazed at how overloaded most of `R`'s operators are :)
(+1) This is a very nice paper, and I've only skimmed it, but we need to be **a little careful** here. Some remarks: (**1**) The results are for the regime $n/p \\to \\gamma \\in (0,\\infty)$, so the dimension of the vectors has to be growing roughly proportional to the number of vectors under consideration for these results to hold. (**2**) Even in this case, the results do not hold for "any" distribution; indeed, the conditions in the paper require that the random variables be "almost exponentially bounded" ones in the sense that we essentially require the 30th moment to be finite! (cont.)
Ingrid, sorry to be so long getting back to you.  The method you want is described in this paper: http://www.ce.utexas.edu/prof/kockelman/public_html/TRB08DSOPmodel.pdf.  I don't think there is a nice, canned routine for estimating the model, however.
Yes. You can even use the same for() loop for both.
I found an article that gives the answer to this, Shrout and Newman, Biometrica 45, 1989       I tried to copy their answer and explanation into this comment but the formatting got all screwed up. It is along the lines of Alex319's answer
@David, our actual model is based on natural logarithm of tree heights dependent on the inverse of DBH (1/DBH). We work with fast-growing even-aged plantations. The fit is already pretty good (R² > 0.9). Regardless of the fit, I wonder if the measured trees needs to be predicted with the model, or if I should hold the measurements and only use the model to predict non measured heights. I was already advised to do both ways, but never could get a reasonable explanation supporting either approaches.
If you have the variables then yes its dead easy. But I was indeed searching for the exact mathematical expression.
So the value that `var.test()` reports for the lower limit is the value of v1/v2 that makes that last equation true? Very nice, thank you.
I could imagine a one-way ANOVA with just three observations in each group, the "groups" being the three algorithms, BUT the three "errors" in one group don't seem really independent since the same three corpora are used all three times.  And the corpora may not be a random sample from any population.  So corpora are "crossed" with algorithms, rather than "nested" within algorithms.  I'd also want to know how big the samples were: is the 80% four out of five, or maybe 80 out of 100?  So maybe just shoving this into a standard algorithm isn't the best thing to do.
Nagelkerke is a pseudo $R^2$ and not the $R^2$ in ordinary least square, for that reason, the statement "28% of the variability is explained" does not apply. The statistics is likely more useful if you have more than one model to compare against.
Correct. We performed measurements ourself (with a constant valueable as outcomes (quotient)) and had patients fill out questionnaires (both ordinal outcomes and nominal outcomes).

I have measurements from three time-points (pre-, post- and long-time-post-intervention).

Main outcome is the difference in mean of the quotient between pre-intervention and long-term-post-intervention. (if the intervention has a significant effect on the quotient on the long term). Same goes For the other outcomes. 

Secondary I want to know if there is a sig. effect between short- and long-term-post-intervention
@dmitrij - That is what was implied (or what I inferred) from the link - it is about the robustness properties of a statistic.  Influence functions are slightly more general than 1 data point - you can redefine the delta function to be a sum of them (so many observations).  I would think of it as a "cheap Jacknife" to some degree - because you don't require re-fitting of the model.
I like the simplicity of this one... Results are close to the bootstrap method.
If you are interested in general discussion about Data Mining vs. Statistics, there are already several threads of interest on this site. E.g., [Data Mining and Statistical Analysis](http://stats.stackexchange.com/q/1521/930); [New revolutionary way of data mining?](http://stats.stackexchange.com/q/31513/930)
Agree completely. Yes, at the moment I am using these parallel samplers to try and get some idea/hunch about convergence but I am trying to investigate more into it to see if there are oither things I can do.
+1 for the second reference. A really nice piece.
A little strongly worded.  Not all fields are at peace with mixed effect models.  Many are far from consensus on how to evaluate effects in regards to NHST.  Sticking a bunch of non-orthogonal contrasts into a single model and then interpreting the results in a NHST framework is certain to lead to tears.  (Of course there is plenty of room to argue the NHST framework is also outdated... however it still is pretty dominant in some fields).
http://www.rseek.org/?cx=010923144343702598753%3Aboaz1reyxd4&newwindow=1&q=wavelet&sa=Search+functions%2C+lists%2C+and+more&cof=FORID%3A11&siteurl=rseek.org%2F#956
Are those mutually exclusive events? :-)
Even though Granger causality test may be manually carried out for non-panel data, there are some packages that perform that test (e.g. Package `lmtest`, function `grangertest`). Regarding panel data, it seems to be not that simple (e.g. [...Short Panel Granger Causality Tests...](http://www.uvic.ca/socialsciences/economics/assets/docs/econometrics/ewp1204.pdf)). There are several additional considerations to take into account. However, it just may be the fact that I'm overwhelmed and I'm unable to properly code the procedures.
Thats a wonderful article, written in Cohen's lucid and conversational style.
@Micheal Good answer. Is it possible to extract an index from an estimated survival curve in order to use it to separate deaths from survivals?
Any distance matrix or correlation matrix should work.
I think this is a good idea. You just lose the interpretation of the $R^2$ as the "proportion of variability explained by the model".
1) Do you mean Wishart or Inverse Wishart?
My educated guess: the tick marks on the x-axis represent the position of the cases with respect of age.
(ctd)... If you want to compare with a sample sd across several subjects, the proper comparison would be with the uncorrected (n-divisor) sd, since they're all on the same footing then.
Yes, it is. You should use $\\mu_i$ and $r_i$ computed on the training set to normalize the values of a new record. P.s. you might also use the standard deviation of your feature $x_i$ rather than $r_i$.
Right! You have 1 transition out of $A$ every 20 minutes, on average, and 80% go to H, so ON AVERAGE 1 every 25 minutes, and 15% to M, so ON AVERAGE 3 every 400 minutes, and 5% to S, son ON AVERAGE 1 every 400 minutes.
$Y = m^ \\prime - {\\frac {\\sigma^\\prime} \\sigma}(X - m)$, i.e. $Y = 2m^ \\prime - Z$, has the same mean and standard deviation
@tarrash Nah, this fourth point is rather too fresh to be tested enough. And as I wrote, normalization is either in aligning or in getting into spectra.
First you need to settle on a measure of model performance (e.g., predictive discrimination) that is not sample size dependent (this rules out AIC,for example).  Two possibilities are $R^2$ (or generalized $R^2$) or rank correlation measures ($c$-index or generalized ROC area; Spearmans $\\rho$). Then you can use the procedure I outlined.  You have to be sure to repeat all feature selection and other $Y$-data-driven steps for each bootstrap resample.
The C-index is too insensitive to be used to compare different models, in general.  I would typically use the generalized $R^2$ or other deviance-based measures including AIC.  And note that AIC is *not* related to classification/cutpoints.
You might want to read http://stats.stackexchange.com/questions/27313/how-would-you-fit-arima-model-with-lots-of-autocorrelations/27338#27338 and other posts that I have made on the subject of daily data.
+1: good to point out the approximating formula ;-)
If you look, say [*here*](http://www.graphpad.com/www/data-analysis-resource-center/blog/testing-for-equivalence/) it suggests looking at confidence intervals and avoiding p-values altogether. That sounds like sensible advice to me.
Thanks for noting that.  Unfortunately it did not improve the situation.
CV is certainly a good idea; anyway, note that there may be several "best" algorithms or parameter sets (in a sense that they give the same, or statistically indistinguishable accuracy from the best case) -- that's also not a problem. Even better, it means that you are getting stable results -- sharp optimum may suggest that you have overfitted the model by cherry-picking parameters.
@micans The algorithms that you suggested can handle string clustering right?
What you said about the euclidean space is interesting. Can you please explain more.

Thank you.
You should not combine the groups. See [this](http://stats.stackexchange.com/questions/93651/if-im-not-interested-in-the-interaction-is-there-any-reason-to-run-a-two-way-a), which might count as a sort of duplicate. Instead you should incorporate the group effect into the model before testing the effect of interest for the reasons indicated in the linked answers.
What are $h(x)$ and $f(x)$. Please add details
It is hard to understand what is being asked here.  Could you perhaps provide an excerpt of the data and some description of what these measurements are of? What is the purpose of computing the standard deviation?
That seems like a more clear cut way to do the question. Thank-you.
You can do this but it won't mean the same scientifically or practically or (in your case) economically. Here is a simple analogy. Suppose you have data on your height and weight as a double time series. Now model that. Do you think that will capture the cross-sectional relationship between height and weight across _any_ population of people?
Adding an independent bivariate Normal variable (say with zero correlation) having sufficiently small variances to the second example will preserve the trimodality of the distribution and the unimodality of its marginals.  The resulting PDF (whose contour plot will be a "Gaussian blur" of the original plot) will be *continuous*.