tttjjj/elt_bench huggingface dataset:
Does't removing the infinities affect which parameter is being bounded by which value?
If the variables aren't all on the same scale, then the $\\beta$'s will be difficult to represent all on one plot. If the variables are all standardized, for example, you could make a plot with side-by-side dots for the point estimate and error bars around the bars - the numerical values on the $y$-axis and the variable names on the $x$-axis
I will try this solution, and compare its efficiency and space requirements to that of the polar coordinates.
Could you clarify what exactly you are trying to do?  It sounds like you want Weka to run on a server and for your client to be able to call clustering algorithms on that server.  Is that right?
@Glen_b Thank you very much for your help, I was assuming that they are independent and that there are no errors in the sample parameters (since the error appears negligible). That wikipedia article will be a great asset, I did not know that there was a name for this specific interval!
This answer begins with a useful and relevant observation but then appears to conflate a "squared residual" with variance.  The two differ; a squared residual can be a lot larger than its variance even for influential data.
Just as a note: no matter what visualization you use, these won't end up looking like 'typical' data, b/c counts are distributed as a Poisson, $\\mathcal{P}(\\lambda)$, not as a Gaussian, $\\mathcal{N}(\\mu,\\sigma^2)$. In addition, you have zero-inflation & overdispersion. If you're not familiar w/ these issues, [this](http://www2.sas.com/proceedings/forum2008/322-2008.pdf) provides a quick overview using SAS.
+1: this looks right, but I don't think you should say a mixture of a normal and a gamma distribution, but rather a normal-gamma–normal compound distribution and motivate this construction by saying that the normal-gamma distribution is the conjugate prior of the normal distribution (parametrized by mean and precision).
@BabakP The common variance is unknown. How do you put a Wishart prior on a variance matrix with equal diagonal entries ?
Short answer is no. The measure is based on a difference and then a ratio. Different differences and/or ratios are compatible with the same value of d.
@whuber I cannot figure out how to place paragraph spacing in the comments and apparently there is a character limit... I will just edit the question at the bottom to answer you better as far as my understanding.
I have no idea. What is the theoretical question you are trying to answer?
You might be interested in [this](http://www.personal.psu.edu/users/f/k/fkv/2000-06-moment-as.pdf).
A recent analysis of accuracy of spreadsheet software for statistical calculations appears in [Kellie B. Keeling and Robert J. Pavur (2011): Statistical Accuracy of Spreadsheet Software, The American Statistician, 65:4, 265-273] (currently a free download at http://amstat.tandfonline.com/doi/pdf/10.1198/tas.2011.09076).  The results are mixed and perhaps a little surprising. Notable is the huge improvement in distribution calculations between Excel 2007 and Excel 2010 (which appears to be more accurate than `R` or SAS).
Not all distributions have means, e.g. Breit-Wigner. I also would not characterize mean as a parameter of distribution. Some r.v.s have maximums.
"Beginner Stats Book: Statistics in Plain English" There are two different books with that same same title by different authors. Is the one referred to here by Harvey J. Brightman or Timothy C. Urda?
@cbeleites I have already done regression based analysis using the equivalence package in R.  I believe both of my error matrices are independent--I provided an example of an error matrix between max likelihood classification and reference data.  I also have another error matrix that is not displayed which is between random forest classification and reference data.
Thank you for your reply and welcome to our site!  (For the reasons why I removed the closing remarks in your reply, please visit our [FAQ](http://stats.stackexchange.com/faq).)
What is it that you actually need help with? It's hard to tell what your actual question is here. 
If its a code question, can you provide a toy data frame to work with? We don't know what your `Masters` data looks like. 
An example of output that you're expecting to see would also be helpful (if it differs from the actual output).
If its a more general "hey am I using this statistical technique correctly" question, consider adding the [statistics], or more relevant tags.
This looks like the answer! I'll update you once I've had a chance to further examine this phenomenon.
Excellent exposition.
@Sveltely Nothing
@Death As explained in the text immediately preceding the figure, the gray data are the original stream of data.  Its running mean is the black curve.  The colored curves are based on the algorithm.  Vertical deviations of the colored curves relative to the black curve are due to the randomness in the sampling.  The expected amount of deviation at any index is proportional to the standard deviation of the gray values preceding that index and inversely proportional to the square root of the sample size (taken as 50 in this example).
Could it be because of the data itself?
My prior leads me to believe this probably can be done. Can you add more detail?
@Tal the dist option: each variable has a vector of absolute correlations. Variables with more similar vectors would have smaller euclidean distances: http://en.wikipedia.org/wiki/Euclidean_distance
The interpretation of an independent variable in logit regression is a $\\beta$ unit change in the log odds of $y$ for a one-unit increase in $x$ ($\\sqrt{age}$ in your case). Out of curiosity, why are you bothering to transform an independent variable? More specifically: what does the normality of an independent variable get one in a logit regression?
A more interesting example if you can find the data was in a paper in The American Statistician many years ago, where a binary logistic model was used to estimate the probability of a successful field goal as a function of the distance attempted, in American football.
Who is the audience?  What is the key message that you want the audience to get from the graphic?  Why are the numbers, their magnitude and significant digits the right ones to use?
(+1) for the link.
The standard test in such a setting is McNemar's test. (It is equivalent to an exact version of Wilcoxon's signed test)
Onestop: This is certainly the spirit of the question I was aiming for. And thank you for the clarification between criteria of evaluating estimators and methods for deriving them!
I've seen quantile regression suggested for this (when false positive should cost more than false negative etc.), see [Asymmetric Loss Functions for Forecasting in Criminal Justice Settings by Richard Berk (2011)](http://dx.doi.org/10.1007/s10940-010-9098-2). Pre-print [here](http://www-stat.wharton.upenn.edu/~berkr/loss%20copy.pdf).
You are getting what you should. Look at this [CrossValidated thread](http://stats.stackexchange.com/questions/10613/why-are-p-values-uniformly-distributed-under-the-null-hypothesis)
Could you please elaborate on what specifically it means to "use" a regression model in a "live system"?
@AlexWilliams Yes
The usual approach would be to do a chi-squared test of homogeneity, which involves looking at it as a $2\\times k$ table - thus treating both sets of empirical observations as, well, *observations*.
Is your question really nothing more than 'what lines of I code do I write to make program R do procedure X'? Then it's off topic here. It *might* qualify as a programming question. If there's a statistical aspect to your question (like 'does this make any sense to do?') then you should clarify and emphasize those aspects of it
I believe that for models of equal complexity R^2, AIC, and BIC should all give the same answers.  There are some rather delicate issues with R^2 for nonlinear models ...
Julian; you'll need to review the notes again, there are important clues to the approach you'll be expected to use there. Some may not agree with my answers (though my answers may be more useful beyond the assignment)
I'm not sure whether this question is a similar one in essence:
http://stackoverflow.com/questions/4052225/how-to-equidistant-resample-a-line-or-curve
Thanks. I am using a similar approach! I think there is a typo in your post: there shouldn't be a square on $(x_i-x_j)$.
No - you keep drawing samples until you get one less than 5, so if temp_x<5 you break the loop...
I think that part of the reason that this may be difficult is that you would need sets of related data. Using automated methods, I would think it would be quite difficult for an engine to determine that that are indeed two different Roku, 2 and 3 (and perhaps the original).
Thanks Zhubarb for your thoughts! To answer your questions: 1) The predictor variable is nominal so ordering is not important. In fact, it was actually chosen in a arbitrary manner. 2) This is an interesting point that I will look more into. 
3+4) The data has no temporal effect and is not incomplete . We have sufficient statistics in each class so I believe the learning algo is able to learn what each class represents.
@drknexus: If only there were a feature request & issues submission page for ez... https://github.com/mike-lawrence/ez/issues :)
Are the data really that coarse?  Are there really only 3 states, or are there, for example, instances when I > II and then instances when I >> II?
Thank you. Are you sure the result is a binomial distribution? I simulated this on a million random sequences, each 100,000 letters long, and the resulting curve did not fit the binomial distribution. It was indeed bell-shaped (central limit theorem) and the mean was at the place where the binomial would put it but it was shorter and wider.
you don t want to use a generic bootstrap package to bootstrap robust estimates. There are enormously more efficient ways to get bootstrapped CI's (computer wize) for robust estimated. Have a look at the frb package and [this joss paper](http://www.jstatsoft.org/v53/i03/)
We should be careful with what we are referring to as the *empirical entropy* here. Note that the plug-in estimator is always biased low for all sample sizes, though the bias will decrease as the sample size increases. It's not only *difficult* to get unbiased estimators for the entropy, but rather *impossible* in the general case. There has been fairly intense research in this area over the last several years, particularly in the neuroscience literature. Lots of negative results exist, in fact.
@Ben I am not sure whether you are addressing your question to me or Alexander.  But the point you are making now is exactly the point I thought you made with your initial question. I think that although you may feel that the results are crazy, this is new territory for statistics an area we used to say should not be touched.  Statistical dogma always said to model n must be a lot bigger than p.  Now we are accepting that something rational can be done when p is a lot larger than n.  20th Century statiticians would be rolling over in their graves!
It should be stated or at the very least implied by every regression text. The converse is that if you add a new predictor, expect coefficients to change unless the new predictor is uncorrelated with all the others. Try it with data.... (You seem to be trying to learn statistics by searching the internet. You need to work through reliable textbooks, not browse for what you think you want.)
One easy approach: do normal inference on the logs; the inference on the parameters (which in the usual parameterization are the same for both) should be the same as long as the priors correspond. Jeffrey's priors have the nice feature of making the inference not impacted by the parameterization.
@ Alecos Papadopoulos; Thanks for your comprehensive answer.
Well, so there’s a difference...
+1. In other words, change the manual calculations to `mean(sample) + qt(c(.025, .975), length(sample) - 1) * se`.
Adding this explanation in the help may be the best way to avoid the confusion.
user779747 did note in his cross posting to Rhelp that this had been posted here first. I haven't seen a specific request to post such a notice in SO, but some (most?)  of us cross-helpeRs expect it because that is the stated expectation in the R Mailing Lists.
A tool that can generate either would be fine although the latter would be better.
Imo this question is offtopic on this site as it not goes about statistics itself.
In probability theory, superscripts $\\!^c$ often denote complements: $A^c$ is the complement of $A$ in the sense that $A^c$ occurs exactly when $A$ _does not_ occur. Here you are using $c$ as a indexing variable in your product as well as what seems to be complements, or maybe not.  Please clarify.
That was my first inclination. Hard to tell that to a reviewer though.
This is a "big" reference book! Thank you for suggestion!
@gregory_britten For the moment, lets say that they're iid, though I would be interested in answers in both circumstances.
To the best of my knowledge 'no'. All the methods I have seen are for means rather than medians.
Sorry, I just edited the post. You asked me about the other variable. Thanks again
Thanks for your answer, however, I have read many articles and other stuff that the out of bag error is the real measure of the CCR in random forest. The k-fold cross validation method may not be suitable.
If the assumptions of ANOVA are met, it is slightly more powerful; it is also more familiar to most scientific audiences, which doesn't make it statistically better but is nevertheless useful. Also, the parameter estimates are easy to interpret; the W statistic is not intuitively meaningful.
At first glance, the curve seems like it might be the *complementary log log* (see my answer here: [difference between logit and probit models](http://stats.stackexchange.com/questions/20523//30909#30909) for a picture). There are other sigmoid curves that can be used as well; I list some in my answer here: [Is the logit function always the best for regression modeling of binary data?](http://stats.stackexchange.com/questions/48072//48137#48137)
Done. Please see at http://stats.stackexchange.com/questions/59946/how-do-we-get-log-likelihood-out-of-knn
**Quick reminder.** A suggestion is generally made to move long series of threads to dedicated chat rooms. The system is supposed to handle this automatically. When this is not the case, you can create one (you only need 100 rep) or use [Ten Fold](http://chat.stackexchange.com/rooms/18/ten-fold) directly. This allows you to continue the discussion at your leisure.
Thank you for your reply. Well i guess i need a supervised techniques for classifications. I have thought about using decision trees but as i said i dont have an already  known set of examples. However based on a deep study regarding Introvert/Extrovert i have extracted some rules that can help me in the classification. 

My question is how can we create a decision tree from a set of rules not from already known sets ( I have read some papers but couldnt fully understand the steps). Thank you in advance for your time.
@EMS This is exactly where massive parallelization techniques come to play, there is a nice recent book entitled "Scaling up Machine Learning: Parallel and Distributed Approaches" which deals with this problem
$w$ is a vector with its first d coordinates being $\\sum_j\\alpha_j x_j$ and the d+1 coordinate being $b$. Here, d is the dimension of the feature vector.
This doesn't really sound like a programming question. If you're not sure how certain statistical values are calculated, that's really a general statistical question and not specific to R or any programming language. So perhaps you could edit your question to make it a more specific programming question, or consider posting to [stats.se] instead.
If you set $C$ to zero, you are saying you want to totally ignore violation of the margin constraint. If you actually want to do that, you shouldn't be using the soft margin form in the first place.
Does it have good tutorials or examples that are easy to follow for beginners?
What kind of answer are you looking for?  Even in the case $N=1$ there is no closed form solution (except possibly for special values of $r/\\sigma$).
A negative binomial count model might help - it's a little more flexible than the Poisson model. http://ehs.sph.berkeley.edu/hubbard/longdata/webfiles/poissonmarch2005.pdf
Excellent answer. I was confused about how to calculate the t value in your example. However, Wikipedia explains why: http://en.wikipedia.org/wiki/Student's_t-test#Slope_of_a_regression_line. In R code: `denom <- sqrt((1/(length(x)-2))*sum(resid(lm1)^2)) / sqrt(sum((x-mean(x))^2))`; `coef(lm1)['x']/ denom`. This is equal to `69.1599`
@Glen_b Thanks for the advice. I'm actually thinking of going the fixed effects route and allow for random intercepts for participants. I'm just surprised that almost everyone keeps running repeated-measures ANOVA's on what are potentially problematic data and don't report any information apart from some semi-cryptic error bars in bar graphs.
of course you will! You will lose all information from  sample 2-4. What if 1 and 5 are outliers?
@suncoolsu, I just added further elaboration on the model.
Thanks for the advice!  Can you elaborate on how to select between logit vs probit?  In particular: (1) How do I tell when you are concerned with the tail part of the curve?  (2) How do I select a model by looking at likelihood, log likelihood, or AIC?  What specifically should I look at, and how should this influence my decision about which model to use?
Also: what exactly do you mean by 3/3 bootstrap?
@Abrham D Flaxman, Thank you.
So, in sum, the real and imaginary parts are not independent...but they are multivariate normal in four dimensions, yes? I believe that this is okay for the model-based clustering algorithm I am using (Mclust in R), but will check with one of the package's creators in a meeting on Monday. Thanks again for your patience (this goes to you, too, cardinal). This is some advanced stuff for me.
The PDF of a reciprocal normal is $\\exp(-(1/x-\\mu)^2/(2\\sigma^2))/(\\sqrt{2\\pi}x^2\\sigma)dx$.  At 0 all derivatives equal 0; finding critical points of its logarithm identifies a positive and negative mode (easily computed in terms of $\\sigma$ and $\\mu/\\sigma$); the integral of $|x|$ diverges like the integral of $|x|/x^2=1/|x|$.  The problem with infinite first moments attaches to the reciprocal of *any* random variable having positive probability density at 0, which includes all normals.
@JoeKing, I've edited your title to be more descriptive and perhaps draw in more users who can help you. I've also found that fitting `lmer()` models to large data sets can take quite a while, especially if you need to do it many times. An answer to your question may lie in parallel computing although other users (e.g. @DirkEddelbuettel) would be much more helpful than me with this. There's also a chance that you may get better answers on stackoverflow.
yes but what does that have to do with ruling out obviously poor predictors??
You could try bootstrapping.
Just add them as extra columns in the X matrix (z in your notation?) and use standard Bayesian linear regression.
What exactly are you asking? Of course, you can still calculate the $z$-score, but what are you trying to use it for? The number of standard deviations from the mean (which is what the $z$-score is) may not be a particularly useful statistic in some situations.
Readers here will also be interested in the following thread: [Essential data checking tests](http://stats.stackexchange.com/questions/11659/).
@B.Miller Yes, if you mean the similarity of normalizations: a z score normalizes the distance from mean (to the standard deviation); correlation coefficients similarly normalize the covariance (to the product of the two standard deviations).  But don't try to see the correlation coefficient as a product of the Z scores of the two variables -- the Z scores are for each variable individually; the covariance and correlation coefficient explicitly describe the interaction between the variables.
Just found tonight at Schwartz, E. S. (1997). The Stochastic Behavior of Commodity Prices: Implications for Valuation and Hedging. The Journal of Finance, 52(3), 923–973. doi:10.1111/j.1540-6261.1997.tb02721.x
@MichaelChernick: Reference for your claims? (Both are false.) The sample mean is not even a function of the sufficient statistic!
+1, very clearly explained; I notice you've been doing a lot of that
Could you clarify or provide an example of what you mean by "items in a strong association rule"?
It's hard to imagine what more could be inferred from the information you've provided than you already have inferred. FWIW, heteroskedasticity can be an interesting result in its own right (see my answer [here](http://stats.stackexchange.com/q/87416/32036), or [the tag wiki](http://stats.stackexchange.com/tags/heteroscedasticity/info)).
Poisson random variables are zero with positive probability!
**Hint**: If $y = a + b(x-c)$, and you know $y,a,b$, and $c$, can you solve for $x$?
Why don't you have the data? If the samples are really large non-parametric tests should work great, but it sounds like you're trying to run a test from the summary statistics. Is that right?
@owensmartin As I mentioned in the answer, there are many ways to take this kind of model ensemble average - the various methods you mentioned are all approaches, some more sophisticated than others, and all with their own pros and cons. Do you have a particular example?
If you allow me, Neil G, I would just point out that the empirical distribution you describe at the beginning of your answer has no density (pdf). Sorry to nitpick with the terminology, but a student reading this may be confused.
**What does "best" mean in the context of your question?**
Depending on field and discipline there are _many_ rules, or rules of thumb, for which PCs are to be "kept", used or taken seriously.
Nice set of lectures indeed, thanks!
@Henry The variance of any distribution $F$ can be defined without reference to the mean as $\\lim_{a,b,c,d\\to\\infty}\\mathbb{E}(I_{(-a,b)}(X)I_{(-c,d)}(Y)(X-Y)^2)/2$ for iid $X,Y\\sim F$. For the Cauchy, this is the limit of $\\frac{1}{2} \\left(\\frac{1}{2} \\log\\left(\\frac{a^2+1}{b^2+1}\\right) \\log
\\left(\\frac{d^2+1}{c^2+1}\\right)+\\left(\\tan ^{-1}(a)+\\tan
^{-1}(b)\\right) \\left(c-2 \\tan ^{-1}(c)+d-2 \\tan ^{-1}(d)\\right)+(a+b)\\left(\\tan ^{-1}(c)+\\tan ^{-1}(d)\\right)\\right)$. Since all such limits independently grow without bound, the Cauchy variance diverges ("is infinite").
What happens under approximate collinearity between the explanatory variables? In the conventional forward search algorithm in regression analysis, we are often faced with the situation where two variables x1 and x2 have similar explanatory power. If x1 is in the model, then there is no need to include x2; conversely, if x2 is in the model there is no need to include x1. If I understand your procedure correctly, you will tend to include x1 half the time and x2 half the time, leading to stability probabilities of about 50% each. If so, you might falsely conclude that neither variable is needed.
@Placidia as noted in my question, I would like to Forecast this type of data which has complex multiple seasonality.
Discussions of the Pearson system of distributions usually don't reference an excellent, characteristically concise and original, exposition within Jeffreys, H. 1939/1948/1961. _Theory of Probability._ Oxford: Oxford University Press. I just checked and the Wikipedia entry is another example of oversight. (Jeffreys' book is not so much probability theory, but more data analysis with a Bayesian twist.)
An overall ranking is what I'm looking for; it's something to see "candidate a is better than candidate b is better than candidate c" and so on.
@GaBorgulya This is absolutely true, but if Ionuț needs a quick answer he might be willing to make the tradeoff in accuracy.
Thanks for the tip. Boost looks like kind of a big hammer for my little nail, but mature and maintained.