tttjjj/elt_bench huggingface dataset:
First off: is this a supervised or an unsupervised learning problem? I get the sense it's the former, as you mentioned multinomial regression and the 5 categorical levels. Are they 5 distinct levels, or ordinal factors? If your goal is to reduce redundant variables, have you considered using the lasso for log-linear multinomial logit model? The book, Elements of Statistical Learning by Tibshirani and Hastie is online and free.
To put your question in a context. The question you ask is proved in any introductory textbook about empirical processes.
This term is defined (in mathematical notation, which therefore should be perfectly clear) just before equation (14) two pages previously.
Impressive; I didn't understand the question at all, but you certainly seem to have answered it.
That's what I was thinking, but there's still the $n-k$ to worry about. Since $Var(s^2|X)=Var(e'e/(n-k)|X)$ and $n-k$ doesn't depend on $X$ you can take it out of the parentheses - but that would give $(n-k)^{-2}Var(e'e|X)$, wouldn't it? Then if the variance of $e$ is $\\sigma^2$ the exponents aren't going to match the expression in the answer. I know I'm missing something, but I can't figure out what it is.
the answer _doesn't_ say $b$ has to be on. Check your understanding of + (OR). Hth.
+1 with the edit
`f <- function(n, ntrials=1) {
  xy <- array(runif(ntrials*n*2), dim=c(n, 2, ntrials));
  apply(xy, 3, function(z) cor(z[,1], z[,2]))
};
par(mfrow=c(3,3));
tmp <- sapply(c(3:6,8,10,12,15,20), 
      function(n) hist(f(n, 10^3), xlab="Pearson rho", main=sprintf("Sample size %d", n)))`
How does this add to the existing answers?
This seems like a better fit for Stats.SE; voting to migrate there.  You may want to add more details though, since the question's rather vague as it stands.
In that case I would start, as suggested, with logistic regression. There's plenty of tutorials and guides online. [This one](http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf) looks good
How did you code (score) female ? are u talking about dichotomous - male versus female?
The techniques in the previous question apply directly once you observe that the number of points within distance $r$ of the origin must be proportional to $r^3$.  Thus if you generate an independent uniform variate $u$ in $[0,1]$ along with a point $w$ on the surface of the sphere, scaling $w$ by $u^{1/3}$ does the trick.
Sorry for the late answer. The first expression you write embodies the assumption that the components of the $x$ vector are independent from each other, so the constraint does not apply.
@whuber, I added a non-fluids/environmental example to the description that might alleviate some of the confusion.
A recent posting of John Myles White on the optim command in R may also be of [interest](http://www.johnmyleswhite.com/notebook/2011/06/18/speeding-up-mle-code-in-r/).
(+1) for the 2nd
A little more precision in the main statement would help.  Where you write $\\Pr[XY]$, what is that exactly? A probability attaches to a *measurable set* (an event), not to a random variable.  Do you perhaps mean to state something about $\\Pr_{(X,Y)}[A\\times B]$ relative to $\\Pr_X[A]\\Pr_Y[B]$ for all $X$-measurable sets $A$ and $Y$-measurable sets $B$?
Hi Zach - In short - I would like to compare and get a P value
One of the major conceptual points behind the Bonferroni correction is that you *should* be counting all the tests you do, whether or not you publish their results. Otherwise your research is no better than the [green jelly bean scientists](http://xkcd.com/882/).
This isn't a "test" per se, but a boxplot (or even scatter plot) might be useful for visualizing the distribution of scores.
Could you give some more details in your post? Where possible, answers (and questions) should retain their value even if the link dies (link rot is a common problem, to which answers should be relatively immune).
Thanks John for your reply. I'm not familiar with logistic regression and lme4, but I spent some time playing with the nlme package, and the lmH2 model corresponds to the Machines example of Pinheiro and Bates book (2000, pp. 24). Having subject and condition nested within subject as a random effects accounts for the random interaction between subject and condition. We don't expect the outcome for a given subject to deviate from the fixed effect model with the exact same magnitude for all conditions. I don't really get yet why the other models that I mentioned would be incorrect.
Do you mean normalize to (0,1)? Please explain clearly what you are asking.
Thanks for your response.  Yes, everything above is correct.  It is a simple sum, with scores ranging from 4-19.  However, I do not necessarily need to give each score equal weight.  Asked differently, is there a way to find the optimal weight for each variable given the possible points available across each of the 4 options?  Bascially, the first index was created, pretty much, at random without regard to weight and the impact on the final score.  If needed (which I believe it is) I have an outcome variable that can help define/optimize the weights for the 4 variables.  Thanks in advance.
Thank you very much for the answer! Yes, 1) is my research question. Thank you for 2) too, now I take a look into the suggested the literature.
Three ideas: (1) How about comparing two groups (aka conditions) at a time, omitting the low-low group: gene 1 low risk vs gene 1 high risk (considering only the gene 2 high risk conditions), and gene 2 low risk vs gene 2 high risk (considering only the gene 1 high risk conditions)?  You will not be able to make a statement about interaction, but you knew that.
@Karl - i think you mean $var(\\epsilon)=\\sigma^2\\bf{}I$
@Aksakal `runif(1,-Inf, Inf)` gives a warning and the documentation says that the limits must be finite. I prefer this to an error and this is consistend behavior in R.
Please note that [the numerical accuracy of spreadsheets is very limited](http://www.jstatsoft.org/v34/i04/paper).
This could be a valuable approach but doesn't explain the specific problem faced by the OP.
@gung: Hoping it to be consistent.
To help you clarify your thinking, please consider this: why do you consider $E$ to be "random"? In your considerations, also ponder whether the set $E$ itself  matters in any of the calculations or simulation, or whether all that matters is the measure of $E$.
I would like to thank all respondents for the enlightening answers.
What would happen if $X$ were a *standard normal*? (Ie, $X\\sim\\mathcal N(0,1)$.) What would happen then?
"Meaningful" differences are, as you suspected, context dependent. A t-test can examine the difference in means and whether it is significant. Only a subject matter expert can tell if it is meaningful.
The reason why data normalization can speed up gradient descent, I guess, is that without normalization the rss has elliptical contours, so given fixed learning rate, it might need more iterations for gradient descent to converge. Whereas with scaling, rss has circle contours (symmetric), so gradient descent converges fast. Am I right?
It rather does not change your knowledge about distribution of other variable. Definition of independence: P(Y|X)=P(Y). We know value of other variable only in rare cases of perfect prediction (correlation equals 1 or -1).
I thought the blocks in the regression commands just ran sequential models. Is there any reason why running the models separately will not suffice?
I wanted to add that the Mann-Whitney test is essentially the same as the t-test but assumes that the Y variable is ordinal, rather than interval.  In practice the results will be the same - the difference between the groups will be significant at the .05 level.
I am a big fan of the Lasso in the [BigData](http://cran.r-project.org/web/packages/bigdata/bigdata.pdf) R package.
How about the covariance matrix of the predictions made in each of the two dimensions? I've never done/seen this but it's just something that occurred to me that you may look into. Also, you may check that the predictions are unbiased in each dimension
1. [Simply no.](http://bodowinter.com/courses/papers/jaeger_2008.pdf) The other question is then obsolete.
This answer seems to forget that the $F_i$ are *functions*: by relying solely on the CDFs of the $F_i(t)$ it examines only the unordered values $F_i(t)$ and ignores the times $t$ at which they were obtained.  It cannot succeed.
Either a very short distance to the hyper plan (it cannot differentiate a melon from banana or apple) or a huge distance in either directions (in which case the classifier thinks the melon is a hell of a banana or a magnificent apple)
"*whether there is a general rule for how many significant figures one should quote for the value of the χ2 and its resulting P-value*" - such things are likely to be specific to the journal. The defaults you suggest (1dp for χ2, 2 for *p*) seem fine to me.
Put box 1 on the left, and it will be clearer. Are there any correlations? Mediation isn't hard in AMOS (I think). If it wasn't done, I'd question why.
The flippant answer is "yes". But, what are you trying to learn? And, are the components (implicitly) independent here or not? :-)
why downvoting?
+1 (clever use of a mixture indeed). But what "Gaussian-only approach" do you refer to? The one that divides a Gaussian by an independent chi-squared variate also works for nonintegral DF and is actually a little simpler to implement on many platforms.
Since you're new to R, I recommend checking out this [post](http://stats.stackexchange.com/questions/138/resources-for-learning-r) to find a good list of resources to help get you up and running doing some analysis.
@aix It depends what you actually call few variables, and what kind of variables you are dealing with. But I think a ridge approach is to be preferred in your case. You can also look at [Boosting Ridge Regression](http://www.stat.uni-muenchen.de/sfb386/papers/dsp/paper418.pdf) (Tutz & Binder, 2005). Penalized ML estimation was also proposed as a built-in method to prevent from overfitting; see e.g., Penalized Maximum Likelihood Estimation to predict binary outcomes: Moons KG, Donders AR, Steyerberg EW, Harrell FE. *J. Clin. Epidemiol.* 2004, 57(12): 1262–70.
@trev under the right sets of assumptions, estimates of location-differences are pretty robust (for particular forms of robustness, at least). But the sort of data I deal with doesn't satisfy the necessary assumptions, and things like variance estimates are more dependent on the distribution. I see in the second paper (1st one is gated, sadly) you link there, Wilcox is using the bootstrap with trimmed means (which for the sort of application there should work okay if sample sizes are not too small), but would work better with a smoother estimator (why not just go to M-estimation, for example?)
oh-- I just added King cite to my edited response. The article does indeed demolish R^2 mania (still endemic to econometrics) even where the statistic has a meaning -- for OLS regression. King notes too that that pseudo R^2 is gibberish that was manufactured to extend the thoughtlessness associated with "variance explained."
I think you should be using NB or SVM instead.
In a sense, yes. ARMA(25,25) uses 50 lagged variables, ARMA(1,1) only 2. I have never seen a time series that needed ARMA(25,25), the model is far too complex for any real world data. (Of course you need to treat seasonality and trend correctly, but ARIMA models routinely do so.)
@palm, please register your account.
I'll add it to the answer. Take a look.
Sounds like you might be looking for [multinomial logistic regression](http://en.wikipedia.org/wiki/Multinomial_logistic_regression).
The duplicate showed up in a site search for ["standard error" + mean](http://stats.stackexchange.com/search?q=%22standard+error%22+mean).  Many more related posts can be found in the same way.  Please note that "$E[X]$" usually refers to a *number*--the expectation of $X$--and that your formula is the *sample mean*, which is a realization of a *random variable.* It is often written $\\bar{X}$ and its realization (as given by the formula) as $\\bar{x}$.  This question seeks information about computing the standard error of the sample mean *qua* estimator of the expectation.
Love every part of the document. Thanks for the valuable reference.
@Cyan working through one of the examples in the BUGS documentation should be sufficient
@Michael: Perhaps. Having been on all sides of similar situations, including in Prof. Neal's position, on many occasions, my anecdotal observations are that paper rejection carries very, very little information content in most cases, as do many acceptances. Peer review is orders of magnitude more noisy than people care to admit and, often, as may be the case here, there are *partial* and *interested* (i.e., not independent) parties and interests at play. That said, I didn't intend my original comment to take us so far afield of the topic at hand. thanks sharing your thoughts on the matter.
Do you seek the chance of one or more successes in $n$ trials? Hint: That is $1 - $ the chance of zero successes in $n$ trials, and no loop is needed for calculation.
@LittleBobbyTables Do you mean, why would one use the PMF to calculate the probability of random Poisson-generated numbers? Generally, randomly generated numbers are used in simulation; PMFs are used in many contexts, including p-values and computing posterior distributions. Offhand, I can't think of a context in which one would randomly sample, only to immediately calculate the probability of the randomly generated data.
This doesn't seem to work all that well for moderate to large data sets.
Hi Charlie. Thanks a lot for your remarks. So do I understand that I really don't have to cluster with 2 periods? Bertrand et al say that collapsing improves the rejection rate, but I am not sure whether this implies that clustered SE are no better than standard robust SE in this situation. So: what IF I do cluster even though I have only two periods? Will the estimator of my SE be biased then? Thanks!
I am actually comparing different features that have their own scale. For instance I am comparing GC content which has a range from about 0.3 to 0.5 which may seem to be small but the difference is quite important; some other features have broader ranges, some others range in very small scales.
Your answer looks to be correct for large samples (which would be the case for the particular numbers here). It would be good if at least an intuitive explanation for the statement in your answer was added.
You could consider an interaction term rather than some transformation
If I cannot expect the residuals to be white noise, then how do I do model-comparison and model-checking? How do I know if, for the mean-model, ARFIMA(1, 0.5, 1) is better than ARFIMA(2, 0, 2), etc.? Thanks again!
Thanks. SPSS gives partial eta for the GLM. I have converted that to eta squared because it is analogous to r.
So in a table header, something like "Student's $T$" would be used for the statistic and "$P$" for the significance? I'm a bit worried that nobody will get this subtlety and people will just assume a stupid typo ...
"*the fact that the big picture ... will inherit the characteristics of the instances*" -- this sounds like the [fallacy of composition](http://en.wikipedia.org/wiki/Fallacy_of_composition) ("what's true of the part is true of the whole"), but in any case it doesn't work like that. "*this is just a feeling*" isn't a sufficient basis on which to expect it to be so (results in probability don't generally accord with intuition), and in this case it isn't so - even in the super-simple case of independent, univariate t, with equal coefficients in the sum, and constant d.f., it doesn't hold.
By "assume a functional form," do you mean like my idealistic normal distribution?
Please take some time to learn the [way to property format your posts](http://stats.stackexchange.com/editing-help). You can't attach files at this site, but you can insert a link to a file posted somewhere else (e.g. dropbox link).
The loss of accuracy in return for only a 4% speedup hardly seems worthwhile.  As you suggest, perhaps on other platforms this approach would have some merit.
There is only one random variable on the right hand side and it is not indexed by i. There is no need to discuss independence here. I would approach this problem as finding moments/distribution of a function (here, $z$) of a single random variable (here, $x$).
-1, the stepwise idea is very poor advice. The fact that you end up selecting only a few variables is unrelated to whether it will yield an overfit model. This strategy is very likely to overfit. The PCA strategy is a good idea, +1.
I added code to illustrate.
If there is no interest in the statistical aspect of fitting survival data, this question should probably be posted to Stack Overflow. We can migrate this question for you if this is the case.
1. Your function does not mimic Excel's `percentrank`-function, which is good (+1) since the latter gives "strange" results (see my [comparison](https://gist.github.com/1026879)). 2. I wouldn't name the data frame `df`, because `df` is an R function (the density of the F distribution, see `?df`).
@Kevin: this sounds to me as if you need to find proper consulting. Your links state that there is no procedure to define the applicability domain for QSAR models that is universally agreed upon. In other words, you need to think what properties your application needs and from that conclude how to verify that your QSAR model is fit for purpose (= valid) and which definition(s) of the applicability domain can be used. My guess is that this is much less a problem of coding than of thoroughly understanding the application. But that's my 2 ct, and I don't have any first hand experience in QSAR.
I was thinking it was something along the lines of the second equation but I could not find a definitive source. Thank you so much!
I suspected something like this, but the differences I observe are much larger than 10^-10. They are in the magnitude of 10^-3, which is considerable for a statistic that is bound between 0 and 1.
@Glen_b See my second paragraph. :)
Hi, Please take a look at the discussion of homework questions in the faq [the lower part under this heading](http://stats.stackexchange.com/faq#questions) and also the [self-study tag wiki](http://stats.stackexchange.com/tags/self-study/info). With that information in mind... what have you tried so far?
It would help if you can write out the model equation(s). Also, how are you fitting this model?
"In NN, you make classification, assign the input to a predetermined class." Not necessarily, NNs can be used for curve fitting too. Specifically, the matlab function fitnet, which Serg uses, fits curves.
@Kurt, gung didn't want to mean that this is not an answer, but rather that your answer does not fit to this site (and especially for the present question, actually)
How have you used PCA to initially developing the questionnaire? Have you used the PCA to choose the better items and discard the worst?
@evan54 - if you make the random flip *after* you choose which envelope to give me, then it's equivalent to problem 1; if you choose both amounts of money, and then make a random flip on which envelope you give me, then it's the situation described above; they're different situations.
What exactly do you mean by "not nearest to any point", exactly? It would seem that by definition, if there are centroids living in a real or other metric space with points, a given point must necessarily be nearest to some centroid.
Yes, I now remember looking at one or two of these videos a few years back. They are quite mathematical.
Hmm I'm not so sure about that. If I understand it correctly, my precision values should also be the same as my accuracy and recall values. But it's only accuracy and precision that are the same. Precision is most often way higher.
take a look at the `blme` package ... and http://glmm.wikidot.com/faq#fixed_vs_random
Are you saying you cannot draw a sample that includes every element in the population, or that you cannot draw a sample such that every element in the population has equal probability of making it into the sample?
The OLS _residuals_ are by construction orthogonal to $X$. They also have zero mean, again by construction. So "residuals correlated with X" makes no sense. Perhaps "errors correlated with $X$"?
Chapter [7 Clustering](http://infolab.stanford.edu/~ullman/mmds/ch7.pdf) from the free book [Mining of Massive Datasets](http://infolab.stanford.edu/~ullman/mmds.html), and 7.6 Clustering for Streams and Parallelism might interest you.
@Maarten You did; it does: sorry I didn't read carefully enough.
I tried to fix up your grammar so that it is more understandable, but perhaps I failed. What is your dependent variable? What are your independent variables?
@jlovegren Five of the six variables are counts.
@whuber: I did tell Mike @ m.SE that he needed to have *some* idea about how things are distributed in his application, and that assuming things are "normal" doesn't always work... oh well.
Your explanation is quite good, in practice, though I would add a slight caveat. ARIMA models can be implemented as State Space models (R's `arima` does this, under the hood), also known as Dynamic Linear models (DLMs). DLMs are also extensions from regression (in a different way than Mixed Effects), so I'd guess that there is a deep-down relationship between ARIMA and Mixed-effect models. That doesn't change the differences *in practice*, which you summarize well.
At time T do you know what was in the basket at time T-1? In other words do you get any sort of feedback as to the accuracy of your previous prediction?
In what way are you measuring error now? What does GEP mean?
@MichaelLugo +1 on your irony detection -1 on your helpfulness.
You talk about manifold learning so the following blogpost could be of help:      http://normaldeviate.wordpress.com/2012/09/08/hunting-for-manifolds/
If $\\lambda x$ is small, you can Taylor expand $e^{\\lambda x}\\approx 1+\\lambda x$ and solve the second degree equation to get an approximate MLE $\\hat{\\lambda}=n/x$.
The logistic model is a direct probability model.  When estimated probabilities are available, accuracy scores should use the estimated probabilities and not whether the risks exceed some arbitrary threshold.  The log-likelihood and measures derived from it (e.g., generalized $R^2$) are the gold standards.  Proper scoring rules are also good (e.g., Brier score).
It sounds rather like it might be too advanced for your lecturer :-).  Why don't you just ask him where the factors of $1/9$ come from, when they do not appear in the definition of $F$?
In order to draw a random sample you need to specify the distribution from which these samples need to be drawn, e.g. a normal distribution, gamma distribution, Laplace distribution, ... So, which distribution do you want to use?
Do you mean to say that $x$ is an actual or observed value and $y$ is it's prediction. Further, are you interested in how well $y$ predicts $x$? Otherwise, I can't conceive of why you'd aggregate $x$ over values of $y$ and not the other way around.
@BlainWaan I forgot to say:  Always try it and see.  The fine details of the imputation strategy may be washed out and give you much the same results whatever you do.  Some imputation rather than none is often the biggest difference in practice.
I'm assuming that the dependent variable in each condition is numeric and is measured the same way in each condition?
Thank you for clearing that up!
My main point is that the output we get from GLMs, which its easy to justify via parametric arguments, can also be interpreted as the output from a semi-parametric analysis. This is important because the latter analysis, while numerically identical, requires many fewer assumptions.
Asked and answered on MSE at http://math.stackexchange.com/q/763364/6179.
Depends on application... Any more details? #s of movies and users? Where do you want to use this dissimilarity?
The man's name is Wilks, not Wilk. Wilks' lambda is the proportion of within-groups sum of squares to total sum of squares. Take these SS from ANOVA table for discriminant scores.