tttjjj/elt_bench huggingface dataset:
Are you asking for an `R` package? Are you wondering about anything other than the name of a package that would do this for you? If not, this question is probably best asked on the r-help-listserve; it would be off-topic here (please see our [FAQ](http://stats.stackexchange.com/faq)).
Forgot I was only analyzing in a given time range, sorry about this.
@S_H. You're right. I edited my answer.
Hmm, I'm definitely not using the model out of sample. And I'd argue that I also don't use the estimated vola out of sample. I rather augment the in-sample-period with some add-ons (e.g. using a mixture distribution which was not part of the GARCH estimation and some adaptation of the vola and mean level close to regime switching points for smoother progression). So I'm using the in-sample-period a bit out of context. I'm still not sure if this is a classical "out-of-sample context". One might call it an "out-of-context-sample" :)
Summing the diff and the raw values gives you information when the ones are sparse and when they are dense.
Computing ROC curves is not a computationally heavy task. Whatever performance measure you intend to use will require you to make predictions with all classifiers. In ROC analysis you just use the decision values instead of labels. ROC analysis is still useful when your classes are skewed, but working with area under the [Precision-Recall curve](https://lirias.kuleuven.be/bitstream/123456789/295592/1/d.) may be better.
You could simply look at the three proportions and endow each of them with a binomial confidence interval.
Welcome to the site, number such and such. Post your code, your results, and why you judged a high level of heteroskedasticity.
(please don't shout) What Answer? What update?
Can you clarify this a little bit? EG, have you developed a new type of analysis for this situation, &  want to compare how it behaves relative to standard methods on a known (perhaps classic) dataset?
Are you allowed to use it if you don't have an eighties haircut and a Hawaiian shirt?
Maybe sparse cells would make the chi square test invalid.  Do you have enough information about the proposed table to back up your assertion that the two methods would give similar results?
I doubt that this completely explains the phenomenon you observe, but you should not be using the test set for anything other than testing. You should not be using the test set to normalize the training set. The model you produce from the first $90\\%$ should not depend on the test set.
I would have given almost the same answer but here it is and well stated, too.  Let me add just two things based on experience.  First, I have found it's almost always worthwhile re-running any analysis I possibly can: it serves to check my understanding and, more often than one might expect, it exposes errors in the paper itself.  Second, it's essential to locate the key references and to find references of your own by searching the Web for phrases in the paper.  A substantial number of contributions recently are (auto-)plagiarisms or bald attempts to get another paper out of old work.
Which package is 'effect' in?
Somewhere somehow someone must have stated something about how F and N are derived. Did you have the original documentation about this model? And if so, post it here or we may not be able to give any concrete answer.
I don't think Bayesians own "evidence" (the concept), though they have certainly formalized it. I would argue that a small p-value is evidence of something.
...if X and Y are independent random variables
`Var(XY) = E[(XY)^2] - E[XY]^2`;  
        `Var(XY) = Var(X) Var(Y) + Var(X) E[Y]^2 + E[X]^2 Var(Y)`
Are you talking about sufficient statistics?
You know that the failure rate $h_A$ of A is smaller than the failure rate $h_B$ of B. Since the failures of B are exponentially distributed with mean $n_0$, we have that $h_B=1/n_0$. Since X is also exponentially distributed,$$h_A=\\frac{1}{E[X]}=\\frac{1}{m}<h_B=\\frac{1}{n_0},$$
that is, $m>n_0$. So if $n_0$ is a known quantity, then the null hypothesis $m\\leq m_0$ is trivially false if $m_0\\leq n_0$ and untestable if $n_0<m_0$ since samples of B have no information about A. If $n_0$ is not known, test null hypothesis $n_0\\leq m_0$ and if you reject it, then reject $m\\leq m_0$ also.
Thanks! I will look into those, too
One point worth addressing is the potential confusion of data with random variables. If the right hand side is called a "likelihood," then "$y$" must refer to data (that is, a realization), not to a random variable.  Furthermore, if we accept that calling the RHS a likelihood was intentional, then we must emphasize its dependence on $f$ rather than dismiss it.
while I think fitting to known parametric families has its place, but what does that tell you in this case? From the asymmetry we know Cauchy, normal and $t$ can be ruled out right off of the bat. Can you think of any parametric families that could possibly capture this combination of asymmetry and long-tailedness? It seems to me that a mixture distribution (like the one I simulated from) may be needed, or a non-parametric estimate, as @whuber alludes to in his comment to the main question.
See [here](http://pymc-devs.github.com/pymc/modelchecking.html#goodness-of-fit) for an example where the above data-vis technique works.
Because as it stands the question is, imho, overly broad: it covers so many distinctly different applications that there cannot possibly be a single answer.  It is unclear what the question really seeks in the nature of an answer.  This is *not* a discussion site (even though the comments sometimes make it appear so...).  Please read our FAQ and refine your question following its guidelines.
Thanks Adam - inverse variance weighting was what I had planned to use so it's good to have confirmation that this is a good idea!
Nice references, Larry Wasserman also recently had a shorter post describing the technique in less detail, [The Amazing Mean Shift Algorithm](http://normaldeviate.wordpress.com/2012/07/20/the-amazing-mean-shift-algorithm/).
"In Bayesian statistics, for instance, you typically assume that there is no true underlying parameter, and that instead it itself is a random variable".  This is not correct.  *Uncertainty* about the value of a parameter (like uncertainty over almost everything else) is modeled as a random variable and updated in the light of data. The parameter itself is whatever the *generative model for the data* says it is.  This is independent of the estimation strategy.
Thanks for your help - from some googling with your suggestion and the one provided by suncoolsu, it seems that his answer is the right one; I hope I'm not overlooking something here ;-).
No problem. I am curious if you were able to solve the problem after the answers given
Very nice! And yes I meant to say arima (1,1,0). Your expanded answer was a nice bonus!
Please give a data sample and one subplot if that is possible.
@inc42 I've revised it to be a little more coherent. I'm playing around with the R package `circular`, and I'll let you know if I manage to do anything helpful.
I won't attempt to give a definition of exactly what the confidence interval means as this is formally tricky and I'll probably get it wrong.
@vonjd: I forgot to mention that R also has plenty of differential equation solvers.  I have used several packages and I can say that it is a lot easier to use R than typical C/Fortran/Basic versions of the same methods.   Here's soda http://svitsrv25.epfl.ch/R-doc/library/odesolve/html/lsoda.html
I agree with @Peter with respect to your IV and whether or not it is actually binary. It seems to be a very subjective question dealing with subject matter too complex to be boiled down to a binary yes/no. However, perhaps you have some theoretical basis for doing this. Why don't you do regressions and control for level of crime?
+1 Normally a reply like this should just be posted as a comment, but the advice is *so* important here it benefits from every possible emphasis.  PCA results in particular will likely be uninformative until variables are appropriately re-expressed.
True comment. But as an experienced programmer I feel dirty every time I use Matlab/Octave, which is a horribly-designed (if it's designed at all) language. Of course, I also cringe at SAS, which was obviously designed for punched cards.
I am assuming very little prior knowledge of statistics apart from the definitions of mean and standard deviation. Calculus wise integration by substitution is fine to use. A proof using a detour via exponential distributions will be fine.
1. Isn't this a question solely about statistics?  Would you like to explain why this is appropriate/suitable for CS.SE?  (You might like to refer to http://meta.cs.stackexchange.com/q/704/755 for site policy, or at least some viewpoints on this.)  2. Have you tried using maximum likelihood methods?
One measure of association between an ordinal and a nominal is called "Freeman's $\\theta$". Unfortunately, I don't have any open access reference at hand.
I don't think your answer for (A) can be correct. The probability that A and B are teammates in round 2 isn't 1/14. There are 42 slots. One is occupied by A. Of the 41 other slots only two of them are in the same group as A.
You've put 165 variables in a single model! This is a recipe for certain disaster. AUC will hardly be the only worry you should have. 90% or more of your variables have insignificant effects for which you would have to do rounds of feature selection. Consider splitting your data and a possible use of PCA for dimensionality reduction.
@Keith  No argument over the definitions, but you do have to be a Frequentist to consider them to be more than interesting and perhaps handy bits of mathematics.  That is, *if* you think sampling theoretic properties are vital for statistical inference then you will (or should) be *equally* keen on confidence intervals and hypothesis tests since, as we agree, they have this symmetry.  Mine was a response to the questioners contrast between 'good' CIs and 'bad' HTs. By lumping them together I wanted to refocus on the contrasts brought up in other answers.
I would not have a problem w/ claiming that there were differences among a, b, c, & d, under these circumstances. I would not use the phrase "*partially* accepted".
My point is that listwise and pairwise deletion methods lead to different results (based on the available literature, and also when I perform the bivariate regression in SPSS). I know that pairwise deletion has its drawbacks. However, I would just like to know if and how to perform pairwise deletion in R.
Since links can rot, care to give an account of his explanation?
A great movie, but not particularly centered on maths.
See chapters on making two-way and multi-way analyses in Tukey's [*EDA*](http://books.google.com/books/about/Exploratory_Data_Analysis.html?id=Fz5DnwEACAAJ).
@Stat you can see your email address on your profile, but the rest of us can't - email and real name entries are private. If you want people to see your email address you'll need to put it in your About Me.
I don't understand what "sub-features" are, as opposed to you simply have some relevant variables. In addition, I don't see why you need to use MDS at all. Generally, you use MDS b/c all you have is a dissimilarity matrix & you want to build a kind of approximation of what the underlying feature space might look like (eg, its minimum dimensionality). In your case, you already know what the features are, so you can use them directly. If you want to visualize the locations of the issues in the (eg) 2D space w/ minimal distortion for some reason, you can do PCA & plot in the 1st 2 PCs.
The question about the form of the water is still unanswered. "Randomly dispersed" is ill-defined. I can think of at least one interpretation of "randomly dispersed" where the probability is always 1.
Thank you for that explanation.  So what I gather is that when the software returns the pilot parameters, they can be plugged directly into the journal formulas, except for C.  The software returns gamma = exp(C).  To translate into the journal article's equation, take log(gamma) to get C.  ** You've helped a lot, thanks again **
The Black Swann by Nassim Nicholas Taleb explains why simple models have been used in the financial world and the dangers this has led to. A particular fault is equating very-low probabilities with zero and blindly applying the normal distribution in risk management!
... and it will be a 2x2 matrix.
@abaumann: As far as I understand things, there's no magic in randomization: it is just the most objective way we have for creating samples that are reflective of the population. That's why we may use randomization within strata, or use stratification and weighting to attempt to compensate for not-so-great randomization.
I think you cannot compare the fits that come from 2 different loss functions, because they are answers to different questions. Once you decide that a given loss function is the appropriate one for your situation, the fit follows from that decision. You cannot fold it back to validate the choice of loss function without this becoming circular. If you have some other criterion that both loss functions can be understood to be encompassed by, you could use that, but you need to have defined that in advance.
@user4673 There are many ways to determine "match", based on ytour scales. Some scales *are* matched - validation studies or the original design made them so. You could also look at the distirubtion of your continuous score to see if there are two clear "humps". With that, you're essentially calculating the odds ratio of heading from the bottom of the distribution to the top (which is similar, but not identical, to going from 0 to 1 in the binary score). Honestly, more specific advice beyond that is probably best left to one who knows your data - any chance you have access to a stats type?
The interpretations look okay to me.
@PeterFlom is right. Although written in a different context, I discuss categorizing continuous variables here: [how-to-choose-between-anova-and-ancova-in-a-designed-experiment](http://stats.stackexchange.com/questions/24077//24080#24080), especially after the update. It may help you to read it.
(+1) Ah, I didn't catch that meaning in the link: thanks for pointing it out.  I took "joint region" to refer (perhaps) to a joint confidence region for the regression parameters, whereas in the link it appears to represent some region of applicability of the model.  (Why it is drawn as an ellipse is a mystery to me, though, further adding to my confusion.)
I feel like this question should just be closed. We have collected several recommendations on various related threads, see e.g. this query: [books +introduction +statistics](http://bit.ly/RzRvnI).
I believe in #4 you intended to refer to the eigenvalues of $X'X$ rather than of the design matrix $X$ itself.  ($X$ often has imaginary or negative eigenvalues.)
If by "rolling regression" you mean regressions within moving windows, then your question has been asked and answered at http://stats.stackexchange.com/questions/6920/efficient-online-linear-regression .  If this does not provide a full answer, please clarify what you need.
It is a matter of some topological interest that no subspace (or even differentiable submanifold) of dimension less than $D-1$ can separate $D$ dimensional space into two or more connected components. Of greater import is your second question. You might enjoy reading some of our higher-voted threads on SVMs, such as http://stats.stackexchange.com/questions/3947.
Since you're only interested in the discrete outcome of 0 events, `dpois()` (density function) is what you're after --- the probability of exactly 0 events for each of the chromosomes.  So in this case you don't need the cumulative distribution function `ppois()` since ppois and dpois agree for the case of 0 events.
$I(x \\leq \\theta) = 1$ when $x \\leq \\theta$, $0$ otherwise.  Since $x \\leq \\theta$ all the time (the pdf = 0 otherwise), it's always equal to 1 for any observable values of $x$.  (More generally, $I(x \\in A) = 1$ if $x \\in A$, $0$ otherwise.)
(+6) Nice thorough exposition and well illustrated.  It is interesting that attempts to confirm your chart through simulation will be doomed when $\\sigma$ is much larger than $3$ because the *sample* correlation coefficient is extremely variable (due to the chance of getting one extremely high value of $X_2$, which will have high leverage). That places a higher value than usual on a solid theoretical analysis.
Actually, I know of several hundred consultants who charge nothing for their work :-).
Q-Q plots are far better tools for incisive comparison of empirical distributions.  Using them avoids the binning problem altogether.
It seems that classical XOR 2-1 net is good example, but I would appreciate some theoretical reasoning.
I am using the Support Vector Machine. But I have a data that contains multiple classes not only one.
Good to know, it also might be helpful. Thanks!
@MichaelChernick: ISBNs in both links are the same.
Hi Peter "I think it depends on what sort of data you have (in each dimension) and what you want to emphasize in the plots". I suppose these are the guidelines I'm looking for. You mention a small number of discrete categories for color as the 3rd dim, so I guess that's one. By lattice graphs, do you mean plotting different combinations of `x,y` as a panel, or am I misinterpreting?
Thanks JKP, I knew it had to be something obvious I was overlooking; excluding cases with missing data seems to bring everything back in line. Much appreciated!
Don't bother to use clustering at all then. Most clustering methods are designed for continuous values. When you want to analyze your categorial data in a classification context, why not analyze your decision tree or random forest? It's actually a best practise for analyzing variable importance.
Poker would be a strange game indeed if the distribution of win rates were normal: this obviously is a contra-factual assumption.
@RRs_Ghost : suggested reading Alan Washburn, Moshe Kress. Combat modeling. Springer 2009. ISBN 978-1-4419-0789-9.
I'm reacting to this, Macro, because in practice it does not work well to be so inflexible. The risk is that the client will ignore your advice and turn for help from someone who does not know any better (but claims sufficient expertise in statistics). The OP's scenario is a familiar and common one: it's best seen as an opportunity to inform and educate the client as well as offer them alternatives (along with a frank discussion of their pros and cons). We need to say "yes, the CI can be made smaller, but here are some of the consequences of doing that" rather than "no, you're screwed."
Are you trying to compare proportions?
If you have unbalanced clustered data, a mixed effects model will not only try to provide better standard errors but also better coefficients (and thus better predictions).
It's not big, but the sample is large enough to make it clear it's different from zero.
Our policy is not to just give people answers to homework problems. I would be upset if my students were simply getting their answers off the internet. These are hints that are intended to help you get past the point where you are stuck. Think about how you would do this physically & what that amounts to logically. Then you just need either of the two R functions to do that. They are not that hard to find; in fact, you are already using one of them.
I looked through that exact link last night actually, which was what prompted me to convert my bugs output to CODA, but from what I can see, they aren't calling WinBUGS, they are using a built in function in R's MCMC package.
Thanks, that helps me understand (as does @fosgen's answer).  It's nothing to do with your question, but in passing, I strongly recommend avoiding relying on red-green contrasts, because of the color blindness problem.
In my case, I cannot assume that my distribution is normal or is not, and the outlier detection should work for all cases. In fact, I only have an idea about what data set to run the outlier detection on just after the user interaction. The variance test was the most general idea that doesn't require me to have previous knowledge of the distribution of my data. What I understand from your explanation is that I should replace comparing the distance with comparing the statistic, is that what you mean? and what is "med" and "mad"
There are random network models 
that account for geographic proximity, 
e.g., the [Kleinberg model](http://en.wikipedia.org/wiki/Small-world_routing#The_Kleinberg_model).
But if you want your results to be valid for real data,
you should probably use real data: for instance, 
there is a "place" field in twitter data, with coordinates.
Good idea! Zanran search engine is particularly interesting for economic data searches. It is uniquely suited to numeric data **even if already embedded within spreadsheets, charts, databases** and not otherwise possible (or very difficult) to search for and find online.
If I understand your question correctly, a chi square test may be for you. http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test
Some comments: the code is wrong. Also, avoid ever using lowcase l as a vector name because lines like this: `1 / (l * l)` are unreadable in monospace. It also helps to use the `traceback()` function to debug your errors. This points to your `Calculate_K_plus` function. Also, are you aware you have `NAs` in your matrices? `A` and `B` have dim 3x3 whereas diag(num_unique) is 1000x1000.
What are you trying to learn about, Tim? You mention "two population samples." This suggests you might not be interested in deciding whether two particular schools differ (obviously there are some huge differences even in your small example) but perhaps you would like to know whether the two populations have significantly different distributions of passing proportions?
@J. M., thanks! This was first book I read on this topic, but I forgot the authors and the exact title. Very good book. It also has Fortran code examples, which was very useful for me too.
@smwikipedia because R has a heritage that includes S-Plus and S and that latter language evolved over time. Also, there is a need to differentiate between the objects users can create and the base object types.
Offhand, I'd call them outliers. But, are those observations outliers only wrt this variable, or others as well?
Thank you! I will try the last remark
What is ECE? Electrical and computer engineering?
Perhaps you could quote the relevant part anonymously.  If there is a possibility of multiple interpretations, we can suggest other ways to make sense of it, and if there is an error in the manuscript, you can share that with its author, who in either case should be grateful for the opportunity to clarify or correct their work.
I'm wondering why you want to use probit. Logistic regression is standard for relating covariates to a discrete binary response. Should you be interested, I wrote a good deal about all that here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523/30909#30909).
Nice list, it will be very useful.
I am not an expert on time series, but I believe differencing *is* a method of detrending.
Alright, so Calinski/Harabasz pseudo-F and Duda/Hart Je(2)/Je(1) index are the two only stopping criteria available in Stata by default, I guess. I was hoping there are more for agglomerative cluster algorithms.
I update my post with code :)
@ttnphns: Good point - that was wrong. I should have written (and correct my answer to): "with large sample sizes". Usually one would use a permutation test for rho in small samples; my idea was to put both Pearson and Spearman coefficients onto the same metric to make them comparable.
I would not say either of those things.