tttjjj/elt_bench huggingface dataset:
I do not understand what such adjusted data will be used for, it will not provide the solution that maximizes tonnes in every wagon? Residuals, as GaBorgulya suggested, shows just the deviations from the average. Well it is not what you are actually looking for, isn;t it? I think, you have to play with some mean value. For example $\\max(tonnes)$ will be just intercept term of your regression `113.92`, but it is hardly reachable. You may try to fill the gap between `mean(df$tonnes)` and this value by some technically reasonable proportion.
That's possible, although it looks like you are not using nearly enough trees. Typically you need thousands. Note that number of trees is not a parameter to be tuned in the RF algorithm, more is always better, but once you have 'enough' (to be determined empirically) the OOB error doesn't improve with more trees. Even for small simple data sets, anything less than 500 trees is not nearly enough.
I bet it is this, as the "Introduction to" part is small and dimmed on the actual cover.
@Glen_b I will fix the grammar exactly according to your first edit.
The R function `step` does [stepwise regression](http://en.wikipedia.org/wiki/Stepwise_regression), not a [step function](http://en.wikipedia.org/wiki/Step_function). (Search on `stepwise` here or for the `[stepwise-regression]` tag for lots of discussion of the problems with this.)
Which F-statistic? There's more than one. Do you mean the one in ANOVA?
I think this works: http://en.wikipedia.org/wiki/Multivariate_normal_distribution the mode of a multivariate normal is $\\mu$
First, correlation requires the number of observations (n) to be the same for each subgroup. If that is not the case, then it will take minimum n (in Stata) and do correlation which is not good. In the example above, n1=24, n2=50, and n3=126. So, correlation is obtained using only 24 observations which you may not want.
Could anyone answer this question
Your question is somewhat meaningless. Correlation is a number, while SVM is a classifer (it's also a regression algorithm), i.e. a function. A better question perhaps might be: what is the relationship between an SVM classifier and a, say logistic regression classifier. Or, maybe, SVM-regression and linear regression.
Some remarks: (**1**) The OLS estimates are *not* unique when the matrix of predictors is not full rank. (**2**) Consistency is an asymptotic concept and so requires a *sequence* of estimators. This means you need to define the type of sequence that you are considering, and the type of growth you are interested in *does* matter. (**3**) There are multiple types of consistency and understanding the differences among them can be illustrative. The [Zhao & Yu (2006)](http://dl.acm.org/citation.cfm?id=1248637) paper has a nice discussion. (**4**) Unbiasedness is overrated.
@Stephen Kolassa: It is the transcendental number e.
FWIW: I voted against closure: It's a little mathy, but the original paper is about dimensionality reduction which is squarely on-topic here.
Do you want to close this Q at this point? Alternatively, you can answer your own Q on CV (although this may seem weird), if you think the Q & A might be valuable to someone in the future. Since you're new here, info like this is listed on our [FAQ](http://stats.stackexchange.com/faq).
@Memming Somebody at stackOF commented that my question will be more appropriate here. I don't know why. But my question received very little attention there, so I guess he was right.
@whuber: sorry, but I think your point has completely eluded me. I don't really get what you mean by "draw a picture ... *of the process* ..."?
Ha, I had posted this for a few other questions and then found this Q&A.  I second this as a great resource.  It helps explain things very well.  Also there are many other sections that would probably help to understand other aspects of machine learning.
First, arima() does not fit a transfer function model of the kind in this question. You would need the arimax function from the TSA package for that.

The example you give should give an error because the differenced x variable is a constant, and the arima model already has a constant. The fact that R happily estimates the model is a problem. In any case, the coefficients are only equal to the first three decimal places.
Then you can simply do this for each category separately (e.g. black vs. non-black).
@MichaelHardy I've updated the question to make it a bit more clear.
Well, in the sense of 2. in your question -- via a different (implicit) definition of multiscalar. I would place e.g. OPTICS between your idea and standard hierarchical clustering. More generally though, any clustering algorithm with control over granularity is in some sense 'multiscalar'. I get that you are after more explicit multi-scale approaches, but it is worth bearing in mind.
**For the random variables constructed in my answer,** $$P\\{|X-\\mu| \\geq a\\sigma\\} = P\\{|X-\\mu| = a\\sigma\\} + P\\{|X-\\mu| > a\\sigma\\} = P\\{|X-\\mu| = a\\sigma\\} + 0 = P\\{|X-\\mu| = a\\sigma\\} = \\frac{1}{a^2}.$$ Chebyshev's inequality says that $$P\\{|X-\\mu| \\geq a\\sigma\\} \\leq \\frac{1}{a^2}.$$  As pointed out elsewhere, for _most_ random variables, the $\\leq$ sign in Chebyshev's inequality can be replaced by $<$. The random variable $X$ described in my answer is one for which such a replacement is _not_ possible: Chebyshev's inequality _does_ apply, but _equality_ holds, the $=$ part of $\\leq$.
Since these samples are the same standard inference assuming independent random samples form a given population is violated. What is the inference that you want to make?  Are you wanting to test whether or not the data indicate that the incidence for B is greater than for A in the population that the 100 patients were sampled from?
@Sarah You could perhaps accept Peter's answer by clicking on the little check mark next to it if it solved your problem.
@emanuele, there is more than one one to measure forecast performance. The simplest one is to calculate the generalisation error (error produced on the data that the model was not exposed to during training).
@JeromyAnglim - Mostly civil engineers and planners that need to understand the basic statistic concepts.
Thanks for the detailed explanation. It's not for any specific analysis that I am doing. It's more for me to get a better sense of the potential problems that contaminated normal distributions can cause. I added additional info to my original post, just in case my original question wasn't clear enough. Thanks!
That's a pretty vague question which is likely to elicit generic answers without much interest for future readers. Would you mind giving more information on the context of your study or application? Also, do you mean Fisher, linear, quadratic discriminant analysis?
(+1) There's also some discussion of the overfitting issue with nominal scaling, which is the least restrictive level, in Meulman's papers, [Prediction Accuracy of Regression with Optimal Scaling Transformations: The .632 Bootstrap with CATREG](https://openaccess.leidenuniv.nl/bitstream/handle/1887/12096/03.pdf?sequence=19) (p. 55), and [PCA with nonlinear optimal scaling transformations for ordinal and nominal data](http://bit.ly/HZhe6U).
(ctd) ... *ANY* attempt to model or even describe these data needs to (explicitly or implicitly) be aware of, and preferably take this structure into account; failure to do so can lead to difficulties which are already well understood and have some good approaches to solutions. Your argument would be like claiming that a bank can't use a Poisson process to model the number of loan defaults because *biostatisticians* use it. I've made no particular attempt to introduce any particular methods yet, but any of the areas I have raised do use methods that may be relevant ... (ctd)
Exactly, and that is, surprisingly for me, a flat line! And, since you don't gain anything with lowering recall, all the baselines in that curve are trivially dominated by the classifier where $p=1$.
I assume that the mean of the sample responding to a Likert item is generally a meaningful summary of the group's position on the underlying dimension.
It's interesting to think about when the meaning of a Likert item would vary systematically between groups. Of course, this issue extends beyond just Likert items, probably to any subjective measurement procedure.
(+1) The key here is to recognize that $\\{X+Y \\leq x\\} \\subset \\{X \\leq x, Y \\leq x\\}$. A good follow-up is to come up with a (very!) simple counterexample to show that the result fails when $X$ can take on negative values.
Could you please [merge your two accounts](http://stats.stackexchange.com/help/merging-accounts)? A moderator will convert this non-answer as an edit to your question, unless you want to do it yourself.
What exactly are you wanting to predict, whether or not the bone breaks, whether it's loose, medium, or tight, or the amplitude of the frequency? Furthermore, you ask about predictive modeling, are you actually going to use this model to make predictions in the future to guess at the output value when you have the input values, or do you want to understand why a given result happens?
@GregSnow, thanks for the tip, I didn't know that. I'll delete my suggestion.
This might turn into a really deep answer if you tell more; in particularly, decide to compare @Gottfried's proposal (PCA with oblique rotation) with your proposal (ICA), - what are differences and shortcomings of the two.
@Chad Thorson It may be in your case that the p-values don't change much but that will not always be the case.  I suspected it would be in your case since you said that practically all the variable fit well to a normal.  There is a difference between rejecting normality and thus suspecting that the assumptions for the parametric F tests do not hold versus rejecting normality and thus expect that the results will be sensitive to it.  Comparing the p-values for Friedman's test to the p-value for the parametric repeated measures F test is a way to do sensitivity analysis.
@Glen_b  doesn't the F-test represent how the variables nitrogen,depth and nitrogen*depth all taken together describe the variation in Y?Not their individual influence right?
Completely confused!
I was assuming that uou can use a function for generation of random Poisson deviates, such as `rpois` in R. Your book has a more elementary approach. If you let $F(x) = 1 - e^{-\\lambda x}$ be the cdf of the exponential distribution $\\mathcal E(\\lambda)$, then you have $Y = F^{-1}(U)$, from which you should be able to prove that, $U$ being uniform, $Y \\sim \\mathcal E(\\lambda)$. Then the successive values of $T$ along the iterations of steps 2 and 3 are $T_1$, $T_1+ T_2$, etc. You should be able to conclude.
First, you should be using regression, not correlation. Second, the *assumption* is that the residuals are normal. But if there are outliers, leverage points etc. then the regression may not be telling you what you want to know.
We welcome questions like this, @Louise, but we treat them differently (see our [help page](http://stats.stackexchange.com/help/on-topic)). Please tell us what you understand & have tried already, & we'll provide some hints to help get you unstuck. In addition, you are going to need to tell us what the point of this question is.
Nice, I think I got it now. $(X')^{-1}$ can exist when we use the maximal model, i.e. $X = I, b = y$, and in that case, the residuals **are** indeed all zeros.
Thanks Macro, this is super helpful.
Tks a lot, I didn't know these websites !
@aewrwaer, I updated my answer there to include the actual values. It may be more helpful now. Eg, high leverage does not *necessarily* mean low abs(standardized residual). In addition, although I didn't include DFBETAS, you can think of it similarly to Cook's distance; the reason the predicted values, y-hats, will change is b/c the betas change.
Indeed, it was just that.
michelle, also take a look at the faq for some guidance on how to ask questions here.
Well, I expected a test statistic with a given distribution in which the null hypothesis is $OR=1$, instead of Fisher's exact test. Obviously, you can use a contingency table to obtain the odds ratio, but that's not necessary.
It doesn't generally make sense to talk about the relative power of tests against very broad alternatives without specifying which kinds of alternative you want power against.
For each cluster there are 10 variables for which the probability is 0.1. I didn't use the real data only because I wanted to know 'true patterns', and this pattern is rather simple. I found a method that can deal with this even with higher noise. I've randomly permutated values in all columns and then I used random forest to predict whether a row is from orginal matrix or permutated. EVERY single variable from var1-var100 had positive importance and all noise variables had negative importance. After the noise is removed it is fairly easy do cluster this.
This is a little problematic to me, see:-
@J Dav: This is not a software problem and I explain what I know about this Problem
could you pls give the details of your data? possibly, dput(data_from_r)
Please explain to us what *two* variables are being correlated. A "rating themselves below average" seems to be only one variable.
Thank you Gavin Simpson for your comment. Two questions: if I do not use the argument `x.points`, how exactly are the points chosen? If I add `x.points` then the returned `y` values correspond to the `x.points`. After all, why is it necessary to order my input `x`. In my notes there are other examples about `ksmooth` where we do not order `x`. Should it be done generally using `ksmooth`? Again, thanks for your help
How on earth did you determine that $E[T(X)]$ has value $40$? It should have value $40\\sigma^2=160$ as used in the handwritten solution.
The covariance can be negative, so it is not bounded at 0. It is also unclear to me what you mean by your last sentence, `The units of COV(X,Y) are the units of X times the units of Y.`, care to elaborate?
seems off-topic
As I said using `date`, having only two levels, as a random effect is a bit "excessive" as you can't really see it is a Gaussian; if you use it as a fixed effect it will be more sound.  I think you'll also find `m0 <- lmer(Y1 ~ X + date + (1|Site)...` being better in terms of LR tests. (As I said, whether or not standard log-ratio tests are applicable for this sample size is another question). Good luck!
@daniel: I don't really know of a good way for that (I would have for the other problem :)) But my guess is that rank-based methods are probably the way to go. OTOH: not knowing the specifics, could it be that participants would have chosen even more if they were able to?
@FranzBrentano Yes, you are correct. I would also check the normality assumption of the samples. For this purpose, you can use shapiro.test() in R or a graphical tool like qqnorm() and qqline().
@Dimitriy V. Masterov I supposed edit distance really only works for variant spellings and misspellings of the country name.  You could also include abbreviations on your list of known countries.
I published an illustrated explanation of convolution on the Web at http://www.directionsmag.com/features/convolution/129753, http://www.directionsmag.com/features/convolution-part-2-of-3/129751, and http://www.directionsmag.com/features/convolution-part-3-of-3/129750.
I tried it but it did not provide me with good forecast. I first used dshw which gave me a pretty good model with very small errors. but using bats, it gave me totally off forecasts.
@ Roland. I would like to use F-stats so I can obtain identical results to SPSS which uses F-stats to test contrasts within subjects in repeated measures anova. Additionally the results from t-stats I obtained using R appear to have different statistical significance than those from SPSS, so I am trying to figure out how am I wrong in R.
ADF test can't reject possibility of unit root in residuals for levels, so - no conclusion. I'm pretty sure residuals from diff'ed regression will not have unit root. But I don't think this is a spurious relation - in my case it has some economic meaning and visually series look following each other closely. I'd rather believe in @Dave's suggestion above that correlations at different time-scales can be different and high-frequency noise affects this particular realization of sample correlation between diff'ed series.
What I really want is to simulate/generate the 1-sec data, nothing about solving. Sorry if I expressed myself uncorrectly.
This minet looked nice, but it needs to have some external adapter to make appropriate input from general data :-(
The procedure is illustrated at http://stats.stackexchange.com/questions/68984, as well as in many more of the thousands of posts found by searching [sampling distribution](http://stats.stackexchange.com/search?q=sampling+distribution). Most of them focus on the sample mean; a few on the sample correlation; and probably none of them describe how to compute the sampling distribution of the SD because (with few exceptions) it is an intractable calculation.
A Bernoulli variable (or a binomial w/ n=1) has a 0 or 1 as the only possible outputs. However there are lots of different distributions (the normal is quite common to work w/), and you can use a GLiM to work w/ any response distribution in the exponential family. That includes the normal, the binomial, & (among others) the Poisson. [The Poisson distribution](http://en.wikipedia.org/wiki/Poisson_distribution) is the basic distribution for *counts*. If your response variable is a count, you can use a GLiM w/ a Poisson as the response distribution (& the typical link would be the log).
(necessarily free Online PDF) ?
as far as I see when the outcomes of simulation is evaluated by any pass criteria, it is possible to find out required number of simulation for any confidence level, but in my case I want to find out mean and variance of entire result with specific confidence level with any finite number of iterations. Hence for any n samples, variance is used to define interval of mean, but indeed I need variance as well to find  any value that may represents CPDF of 0.975. thanks for any comment
http://stats.stackexchange.com/questions/84012/choosing-optimal-alpha-in-elastic-net-logistic-regression seems very related
@Dan I really have no idea. I am a physicist and did an experiment. In the first situation I am getting $n_1\\times \\pi_1$ occurences for $n_1$ trials, in the second situation I am getting $n_2\\times \\pi_2$ occurences for $n_2$ trials. I am looking for the probability of the two situations differing by a given amount. - Does this make any sense at all?
Just to check I understand.  First, you say style 1 is a ratio scale, but in your text you talk about "people with style 1" which makes it sound as though it is a categorical variable they can have.  Do you mean "people with high scores on style 1?" or is it really an on/off variable?  Second, are you interested in the effect of style 1 and style 2 in general (regardless of IQ), as well as the combinations you say, or only in combination with those particular levels of IQ? (this makes a difference to the models you test).
You need to rethink this. What about, for example, $v=-255$?
I do not have a specific graph in mind. I'm just lost on how one can create a single joint probability distribution from a joint and conditional probability distribution.
Thanks! I will look into that. Of course I do realize reasons might be many (including wrong theory) but this was the one I did not know how to deal with (or if I can deal with).
When you say "compared to the others", do you mean "compared to prosthesisA, prosthesisC and other", or "compared to other"?
Interesting! Minute point: the axis title or label "proportion" does not match the units of %.
I like the "Not guilty" example.  Going one step further, re-opening cases based on DNA evidence that we did not know how to use in the past and having some convictions overturned is a perfect example of how adding more data may be all that's needed to have enough evidence.
Do the analysis remotely and send back the results.
Do you have in mind any particular application, that is do you seek for general advices about how many EVs we need to consider apart from any application (i.e. on a pure mathematical side) or should it apply to a specific context (e.g. factor analysis, PCA, etc.)?
Do you mean for the last term to be (1+min_in_class+min_hw|student_id) or for it to be (1+min_in_class+min_hw|class)?
@Forinstance, `why SPSS doesn't give me the FI(x) value for 2x2 tables`. I'm not quite sure. I think just because different programs compute it. For 2X2 table, computation is done by the core Crosstabs routine. For greater table, computation is done by Exact tests module.
@FMZ: This is potentially an interesting question. I suggest you search for duplicates and, if you don't find one, ask this as a separate question. Cheers.
Actually this method does not properly handle survey weighting estimation -- it simply treats a weight of e.g. 100 as though it were 100 cases, but doesn't correctly estimate variances (and hence standard errors). See my answer to another question on using SPSS for weighted data [here](http://stats.stackexchange.com/a/65492/16974) for more detail.
I don't know if I'm really understanding your question correctly, but either you display the number of observations (i.e., 5), or you tabulate your data. In the latter case, you probably want to display the mean across all five observations per cell replacing $y_{11}$ with $\\bar{y}_{11}$
@fabee (2) I did not really understand this comment. Could you perhaps clarify it?
@msh210 I've put a bit more in about possible definitions and computations of effect size.  While you can also look these up in a book, I think it's helpful to sample instead because this method will generalize.  Also, the R code should run when pasted in now.
@ScanSally I've edited your question, please inspect now.
For large enough $n$ (as long as the typical $p_i$'s are not *too* small) the standardized version of a random variable with this distribution ($Z = (S - \\mu_S)/\\sigma_S$) can be well approximated as a standard normal (though I'd use a continuity correction!).
Actually combining p_values can be a way to address multiplicity - at least for strictly independent p_values. This is a very technical paper on combining p-values http://www-stat.stanford.edu/~owen/reports/AOS697.pdf - believe combining dependent p_values is still poorly understood.
"Predictive" and "dimension reduction" are different concepts and have no necessary association with each other. It is possible to retain 99+% of the variance in a set of features and yet completely destroy their ability to predict another related feature.
Are you wondering how such correlations are calculated (ie, how to arrive at the number), how they are tested (ie, if they could have come from a population w/o any correlation), or what they mean?
I understand most of what this code is doing but, as I'm not fluent in R, there's a few lines that I'm unlcear on. Would you mind adding a few comments to explain the gist of what you're doing? Thanks.
Is there a way to infer the score from likelihood in your model?
If the only two categories are "spam" and "not spam" then their probabilities must add to 1.  So this means you have $p(spam|D)=1-p(non-spam|D)$, now you have $log(\\frac{p(spam|D)}{1-p(spam|D)})>C$.  This can be re-arranged to give $p(spam|D)>\\frac{exp(C)}{1+exp(C)} $ (assuming your log is natural log, otherwise replace $exp(C)$ with $b^C$ where b is base of the log).  The two big numbers are basically 1 (excel can't tell the difference) - and the $1.68025$ gets a probability of about $84$%.
I like most of this answer but not the 4th sentence; are you sure you meant that?
Thank you! That is much appreciated. I would also like to hear more about software options if someone has some knowledge in the area.
@whuber - you're right on both counts; I should have put in "integer".  The "permutation" clarification went up after my answer post.  Maybe I'll delete my answer as irrelevant...
What hypothesis are you testing there?