tttjjj/elt_bench huggingface dataset:
If the pairings are pretest-posttest and you want to compute the SD of the test *differences* (which is what is relevant when you're interested in testing for a nonzero effect), then the answer is that you cannot get that answer from the data given.
Well, the regression's $R^2$ changes meaning because the denominator (necessarily) changes. In particular, $R^2$ for a regression through the origin is no longer the square of the correlation. The denominator is different from the usual $R^2$, and some packages even refuse to give an $R^2$ at all.
Cramér's $V$ can be obtained as the root-mean-square non-trivial canonical correlation of the dummy-coded row and column categories, without referring to chi-square. (I believe Fisher showed this, but I don't have the reference.)
@Gilles Thank you for catching the typo--I fixed it.  The initial exponent of $\\theta_0+\\theta_1$ needed to be $\\theta_1+\\theta_2$; the $e^{-\\theta_0}$ within the parentheses is correct.
@DavidHeffernan, yes if scaled by standard deviations then covariance tells us about the strength of the relationship. However,  covariance measure by it self doesn't tell us that.
@Robin: (1) Each participant did 6 tasks/searches, 3 in each approach/condition. Were it the same 3 tasks? I guess not, so 6 different tasks? (2) Were it the same 6 tasks for each participant? If so, was the assignment of tasks to condition random for each participant? (3) If all of the above is correct, then your tasks and conditions are not separate factors! So you cannot simply apply repeated measures ANOVA, maybe "nested ANOVA" or something, but I am not an expert here. ANCOVA will not work for the same reason. (4) I don't understand Moose's point, as I explained in the comments above.
Gamma prior on the precision is what I want to use! My question (and answer) was about how to specify that gamma so that the resulting prior on the precision corresponded to a prior on the SD (1/sqrt(precision)) with a mean SD and SD of the SD that I specified my self. It's a shame that we could not meet IRL, communicating is so much easier face to face :)
@Dave31415 For $n$ and $m$ even, the integral can be explicitly computed for positive integral values of $n$ and $m$. It will equal a linear combination of exponentials and error functions with coefficients that are polynomials in $\\sqrt{t}$. The evaluation can be performed via the substitution $x=t-u^2$. For instance, with $n=2,m=4$ we obtain $\\frac{1}{4}e^{-\\frac{1}{8}(2\\sqrt{t}+1)^2}(e^{\\frac{\\sqrt{t}}{2}}(\\sqrt{2\\pi} (4t+3)(\\text{erfi}(\\frac{2\\sqrt{t}-1}{2\\sqrt{2}})+\\text{erfi}(\\frac{1}{2\\sqrt{2}}))+4e^\\frac{1}{8})-4 e^{\\frac{t}{2}+\\frac{1}{8}}(2\\sqrt{t}+1))$.
The first step in stepwise selection is equivalent to univariable screening, which the literature declares to be a disaster.  You are selecting a predictor because it has a low $P$-value (exactly equivalent to low AIC).  A low $P$ will result from a large true effect $\\beta$, a large estimated effect $\\hat{\\beta}$, or a low standard error.  If $\\beta$ or standard error is mis-estimated this will cause that feature to look stronger than it actually is.  This is also called publication bias, and is nothing more than classic regression to the mean.
Interesting question. I don't see a way to do it from the data you have. Is it possible to do more than one such treatment (at different levels from the first) in each so the assumptions used in the calculation might be checked on the second set?
And why do you think that the permutation test should be different? (if both groups have a similarly non-normal distribution)
In the first two examples, you can tune all parameters independently, so your degree of freedom is the number of parameters.  In the third (regularized) example, you have the additional restriction that the parameters are small on average (what small means depends on the regularization).  So you're not free to change all parameters independently: If you set one parameter to a high value, you implicitly force other parameters to be smaller.  Your degrees of freedom thus are smaller than without the restriction, this is quantified by VC or effective parameters.
+1 While I *love* simulation as a method, it's good to know exactly what you're doing.
4000 lines of data is not big.  Try doing analysis on a few million rows!
I don't know specific machine learning stack exchange site. But in this you can find a lot of "machine learning" people (for example me), as statistics and machine learning are really very-very connected.
I looked it up here: http://www.tandfonline.com/doi/abs/10.1080/00273170701836653 and found following in the abstract: "n some cases, most of the variation in the response variable is explained by clustering the objects, with little additional benefit provided by the within-cluster regression models. Accordingly, there is tremendous potential for overfitting with clusterwise regression". Doesn't really seem promising.
In your first line don't you mean instead to write "... results *originally* with $p\\lt 0.05$ but subsequently with $p\\ge 0.05$ after validation..."? A result with $p$ greater than the threshold $\\alpha$ otherwise is termed a *negative* result. Even after your edits, your characterization of the interpretation of $p$ is incorrect, so I would like to suggest you take a few moments to review some of [our posts on interpreting p-values](http://stats.stackexchange.com/search?q=p-value+interpretation) and reconsider what you want to ask.
FYI what I do now is that I generate the graphviz code with R and I read it in a mediawiki using the mediawiki graphviz extension. (The Rgraphviz package is not easy to install and doesnt work with the last version of R)
In the last sentence of the interpretation, I realized I made a mistake, it should be the increase in growth of $y_{t}$ would be $\\beta_{1}$
Are you aware of papers establishing consistency results for the lasso? Knight & Fu (2000), Yu & Zhao (2006), and various papers by Meinshausen.
That makes a lot of sense to me...thanks for the alternative perspective!
Ah, I didn't explain myself well.  It is intensive in its manipulation of datasets, which in SAS, means too much time spent in I/O.  My initial suggestion was Fortran, but the PI is interested in us switching to R, so he wanted me to check it out.  Thanks!
See http://robjhyndman.com/hyndsight/forecasting-within-limits/
hope it was helpful! maybe taking a look at this paper would help to understand the subject deeper
http://research.microsoft.com/en-us/um/people/minka/papers/minka-multinomial.pdf
It certainly does look like a mixture of two normal distributions of different scale, excellent observation.
Please specify some details about your problem, i.e. a) what actual problem you want to solve in the end? b) what are the types of variables? c) do you need supervised or unsupervised dimension reduction? Linear or nonlinear? If you answer this questions I will be able to help you!
I am not sure what you mean - "residuals" are the "left-overs" of the estimation procedure. We do not make assumptions on residuals, since their properties are a consequence of the assumptions made related to the initial specification. Do you by any chance meant "error terms"?
@Can't Tell. I am not sure to get what you exactly want. But take two **independent** standard normal random variables. Square them and add them. This results in a chi-square random variable whose parameter is 2. If you had had 3 normal variables, the parameter would have been 3, etc.
I personally use the `nnet` package frequently and have had solid luck with it.  The obvious difference is instead of backpropogation, it uses the "BFGS" quasi-Newton method.  As for your general modeling questions I'd ask what sort of cross-validation framework you have created and what hyperparameters you are using to move along the bias-variance curve.  For `nnet`, you can implement weight decay which is my preferred way of tuning for generalization.
@Tom: a better approach could be reading through the answers of *Books for learning the R language* question on SO: http://stackoverflow.com/q/192369/564164
I just realized that this RMSE understates the disadvantage of using SD line. Unlike with the regular regression, SD line estimate's conditional RMSE varies with $X$, and could get quite a bit higher than the unconditional value. Say, with $\\rho=0.6$, the regression RMSE equals $0.8$, the unconditional RMSE of SD line equals $0.89$, but the conditional RMSE of SD line at the $2\\sigma$ outlier ($X=2$) equals $1.13$ - worse than even ignoring $X$ entirely and estimating $Y = 0$.
Wondering if anyone can shed light on the proper way to estimate the impact of an intervention - generally, if the procedure I demonstrated is in fact correct?
I would suggest checking out the [UCLA tutorial on logistic regression in R](http://www.ats.ucla.edu/stat/r/dae/logit.htm) to see how to do the software code. For further R resources check out this [meta thread](http://meta.stats.stackexchange.com/questions/793/internet-support-for-statistics-software/795#795). You could learn alot for the interpretation and other aspects of logistic regression by perusing the other questions with the [logistic tag](http://stats.stackexchange.com/questions/tagged/logistic-regression) on this site.
@Neil Why is it so? The OP specified "but ideally which would extend to programming...". R has been plugged in almost all commercial (e.g., SPSS, JMP, Statistica) or free (Knime, Rapidminer) statistical software and Rserve can be used as a backend to communicate with R (e.g., with the plink software for genetic studies), and it is used for this purpose. The Free Trial version of JMP doesn't let you work with external datasets, am I wrong? (Interestingly, all of us who answered R, *and an alternative software*, were downvoted.)
Is that different from the reality check described by White in [his 2000 *Econometrica* paper](http://weber.ucsd.edu/~hwhite/pub_files/hwcv-077.pdf)?
@NickCox in the picture posted by him it says 95% and it seems, that this is a beginner question, so I think they use the normal distribution and not the t distribution?
(+1) Thanks for the references!
Thanks for your answer. I don't think that's the problem but you're right in pointing out this issue. I will rewrite my attempt of proof more carefully.
Here's some additional information on the FDA's thinking with regards to R: http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html
They're neither constant variance nor independent, so you may want to consider those issues.
Actually, $\\mathbb{E}(\\log(Y) | X) \\neq \\log(\\mathbb{E}(Y | X))$ **ever** unless $\\mathbb{P}(Y = f(X) | X ) = 1$ for some $\\sigma(X)$-measurable function $f$, i.e., $Y$ is fully determined by $X$.
This [post](http://terrytao.wordpress.com/2010/01/05/254a-notes-2-the-central-limit-theorem/) by Terence Tao might be of interest.
You may see that the formula is almost the same as the second one in [wiki](http://en.wikipedia.org/wiki/Bayesian_information_criterion). There, LL is based on error variance $\\sigma_e^2$ which is `Vc` in my notation (the pooled within-cluster variance). `Vc+V` is used instead of `Vc` simply against the case of `Vc=0` when a cluster has one object
25 lags for example 2 would be enough? I have looked on Overlapping Windows plot. It can identify the number of lags?
Sorry, I modified the question so that it's not so easily Google-able…
Ordinal predictors? You may try for example: A) Use the predictors as polynomial sets. B) Quantify ordinal into interval via CATREG (categorical regression). (_But_: are you really sure your variables must be treated as ordinal and not interval?)
Yes it is relative, and also there are *a lot* of problems, so there are still plenty of hard ones out there! And I think the boundary between ML and human learning is always blurry ... if you're creating some super-fancy probabilistic model you're doing the same thing. The good old [NFLT](http://en.wikipedia.org/wiki/No_free_lunch_theorem) tells us that one method can't solve all problems, however intricate that method is, so we'll always need humans to design models/kernels/algorithms or whatever is needed to get more out of your data.
+1. In other words, SPSS is actually behaving as per @Druss2k's initial expectations -- and the way this information is conveyed in the parameter estimates is with a zero estimate for beta, with no associated standard error. Other software sometimes does similar things in this respect (e.g. Proc Genmod in SAS, from memory)
This problem is discussed in http://stats.stackexchange.com/questions/56837/can-hazard-ratio-be-translated-into-ratio-of-medians-of-survival-time . The key message is that HR equals the (inverse) median ratio only if the group times are exponentially distributed.
I will remember that! Thanks
You can find many plausible answers--both theoretical and in software packages--by following the [tag:change-point] link.
What class are your variables? For example, if these are factor (categorical) variables, is there any reason why you wouldn't simply concatenate the two variable into a new variable and add your new variable to the decision tree input? If they're numeric you could add/multiply/whatever to create a new variable, then add that as input. . .  adding extraneous info to a random forest should be ok.
What is the response? Is it a measurement? A count? A binary variable? Ordered categories?
Yes they would be different if one refers to samples from different populations.
+1 For offering a new idea.  Perusing the introduction and conclusions of the Shenstone & Hyndman paper, though, suggests Croston's method is generally not very good: the paper focuses on trying to justify and understand a popular procedure that turns out to be limited; the best the authors can say is that despite this, "forecasts ... may still be useful."  Also, it looks like this model couldn't accommodate the additional data on "current supply levels" as requested by the OP.
The problem is that moon phase data are not linear. The response variables are strongly skewed towards new and full moon.
@John This lecture highlights interesting point about REML vs. ML and AIC (and points to what you said, John), http://j.mp/bhUVNt. Bolker's review of GLMM is also worth to take a look at: http://j.mp/cAepqA.
I'm missing the relevance of unequal group sizes.  As to your question, the proportional odds model is a semiparametric rank-based regression model that generalizes Wilcoxon-Mann-Whitney-Kruskal-Wallis.  I'd recommend this model for your situation, based on what you wrote.
Edwin: Some more information would help people answer your question.  What is the level of measurement of your dependent variable?  What are the level(s) of measurement of your independent variable(s).  How many of each I and D vars are there?  What software are you using?
The answer [here](http://stats.stackexchange.com/questions/9358/creating-an-index-of-quality-from-multiple-variables-to-enable-rank-ordering/9361#9361) may be of interest.
@B_Miner It's kind of hidden, but SVD is one of the common methods of obtaining principal components. Since you're using R, have a look at `?prcomp`.
+1 for [Francis T. Marchese, Periodicity, visualization, and design](http://dx.doi.org/10.1007/s10698-012-9154-0) - exactly what I need!
AFAIR in Tufte's books periodic table is mentioned asone of the hallmarks of data visualization, but certainly not going into its history (I've skimmed 3 books by him).
I have merged your duplicate accounts.  Because you have multiple questions, please register and use only that account.
An identical example in the manual page for `nls` does not require the use of `I`, nor do any of the other examples there.  The code as given in the question works.
+1 Thanks for introducing me to the [UCI datasets](http://archive.ics.uci.edu/ml/datasets.html)
Nevermind - I didn't think that all the way through.
It seems to me that without further information, any quasi-likelihood that was constructed could result in biased estimates of (some subset of) the parameters, even asymptotically.
I up-voted your answer and I agree that my question could have been accompanied with a better title.
@gung I believe the objective is to identify an affine linear subspace of the plane which minimizes the sum of squared distances from the touchpoints to that line. To put the question into a more familiar language, the fitting is done by PCA (by means of a singular value decomposition of the covariance matrix of the centered coordinates, as is usual).  The question asks how to compute the proportion of variance explained by the first principal component.
There are a variety of possible distributional models for compositional data.

When considering two components, or when focusing on one of the components, some people use beta regression, for example. (With multiple components, there's Dirichlet models.)

If you want to use a standard GLM package, you might consider a quasi-binomial, but it will restrict you to a particular form of relationship between variance and mean.

A search on `compositional data` here should turn up a number of questions.
Is this homework? Then it should be flagged as such. My answer below assumes that it is and therefore does not spell out the answer.
Hi Ben, sorry for the delay. Thanks for that hint! I am very glad that the GEV is already coded in R. Telling from the destribution functions on [link](http://www.statmethods.net/advgraphs/probability.html) I thought it wasn't. I will use this one.
http://cran.r-project.org/web/packages/Formula/vignettes/Formula.pdf
@Andy Thank you for the reference. Generalizing Juho Kokkala, when the essential range of $X$ is in $[0,1]$ and $E[X]=c$, then $\\text{Var}(X)=E(X^2)-c^2\\le c(1-c)$ because $E(X^2)\\le E(X)=c$. Equality can hold iff $X^2=X$ a.e., implying $X$ is Bernoulli. The general result for bounded $X$ follows immediately by rescaling $[0,1]$ to the range of $X$.
+1 and I am now inclined to agree that in fact my original question is not that important given these other issues
I think this means the gradient at the initial parameter value is non-finite or NaN. Try changing the initial values.
Thanks for the response. Could you comment briefly on the consequences of using the multinomial method instead of a bivariate method in such a situation? (Would the estimates be biased in any way, or would their precision be under/overestimated?)
These answers touche on the issue of the constant term, http://stats.stackexchange.com/questions/111544/removing-the-intercept-term-in-a-dynamic-regression-justified/111571#111571, and http://stats.stackexchange.com/questions/80790/deliberately-fitting-a-model-without-intercept/80798#80798although as @Glen_b notes, time series have their own aspects on the matter (such as differencing, that validly eliminates the constant).
Also, the excellent book ['Elements of statistical learning'](http://www-stat.stanford.edu/~tibs/ElemStatLearn/)  discusses these methods in great depth.
(+1) Good suggestions! Thanks for taking the time to put these together. Kallenberg as a first encounter with probability theory, even for someone with background in measure theory, may be a tad on the ambitious side. Having Dudley (or any of several other texts) at hand would be sufficient, and, perhaps, necessary.
I __love__ the UCLA site.  I tell my students taking SAS to go there if they've seen/used a different piece of software, but I didn't know they had compiled as extensive of a section on R.  Splendid!
I'm going to migrate this question to the statistics.SE site. There will be a link that appears below the question here that you can follow to the new location of your question. If you need help associating an account on statistics.SE, you can flag your question for moderator attention, and someone over there will help out.
I feel this is hard to answer without getting a feel for your data. We would expect the variance varying with the mean over the different seasons. There is probably autocorrelation in the series. So modelling this would be the way to go; time series and/or mixed models seem a good bet. A paired (paired by day/month) t-test would probably give an acceptable answer as well.
@IrishStat , I doubt that Matlab does this automatically, though I do not personally use Matlab. It is most used engineering software in some industries.
(See above comment) I think I can reduce (a), the number of juries. Sampling error is a function of the number of samples. With 1,000 juries per model, I have a million samples total. Sampling error associated with 10^6 samples is ~0.1%. If I use only 35 juries per model, I'd  have 3.5*10^4 samples, and ~0.5% sampling error. This sampling error is much smaller than my measurement error of ~5.0%. Therefore, I should be able to use 35 juries per model and just use the measurement error to estimate my confidence interval.
Thanks for pointing me toward logistic regression, I think that's exactly what I was looking for. Reading about it now.
@test - I'm unsure of what you are ultimately trying to do.  You wrote, "My objective is to identify the curve elbow."  But isn't it plain to see--among your sample data at least?  Maybe what you really want to do is to generalize from your sample to the larger population.  If that's the case, loess smoothing won't help you as it's totally dependent on the data at hand--it will provide no equation with which to predict where the sharp drop in tb$V4 will occur with out-of-sample data.  While regression splines is not a technique I can explain, you may want to look into it.
Why not present *both* plots?
No, it _is_ the (inferred) distribution for the true mean.
This is not a very well-worded question, but it is not your fault! The key thing is whether weight is rounded. If it is, then I think your solution is sound. If not, then think about what exactly a "continuous" distribution means.
Funny side note: this site's spelling checker finds the word frequentist to be worthy of curly underlining. Is this site secretly mastered by bayesians? Oh, I guess it's not, since bayesians have their own curly underlining :-)
@Andy, can you clarify your question a little bit? Do you have hundreds of thousands of predictors for each response? Or hundreds of thousands of responses? Or perhaps, a data matrix with hundreds of thousands of entries? Or maybe you meant "hundreds ***or*** thousands"? The answer to this will provide some guidance. Also, how many class categories do you have? What sort of computing resources are available? Are your features sparse or dense?
Hi Mark, thanks for your help.
Another excellent resource for learning about and understanding survival analysis is _Applied Longitudinal Data Analysis_ by Singer and Willett. The also give [example code/output for all their models using every stats package under the sun](http://www.ats.ucla.edu/stat/examples/alda.htm).
Could you expand a little on what you mean by "account" for "all" the variance?  As it stands, the vague terminology means this question does not have a clear interpretation; I'm concerned different readers will have different understandings of what is meant.
Probably I can find some hints here:  http://stats.stackexchange.com/questions/43216/different-ancova-results-in-r-using-different-functions?rq=1
But the distance is not $d = x + y + z$ (unless you actually do mean Taxi-Cab distance?), so the errors won't add in quadrature will they?  If we assume normality then $d^2 = x^2 + y^2 + z^2$ will have a non-central Chi-squared distribution on 3 degrees of freedom.  I think the distribution of distance is going to start getting messy without some simplifying approximations.
As @Peter says in his response, if your focus is really on the relationships, then your sample is probably fine.  In that case you are not interested in managers that don't supervise working mothers, so that undercoverage is not a problem (as long as you focus on the relationship and don't branch into manager characteristics).  For the co-workers just make sure you have a clear way to define how you are getting them (or you could get a list of candidates and randomly select from that list).
Thanks again for your input and your help!