tttjjj/elt_bench huggingface dataset:
You can handle the corelations between the factors by including _interaction_ terms in your model when doing regression. This is done automatically in ANOVA. If you run multiple comparisons on the same data set, you need to correct for multiple hypothesis testing. There's a wiki page for it: http://en.wikipedia.org/wiki/Multiple_comparisons but, the simple answer is just to multiply your p-values by the number of tests you do.
(+1) This is quite clever. It should be noted that to guarantee a valid transform pair, we need an integrability condition on $X(f)$. Namely, $\\int_{-\\infty}^\\infty X(f) \\,\\mathrm{d}f < \\infty$ will guarantee that the stated inversion will recover the appropriate density. In a sense you employ such a condition later on. (I've already assumed the nonnegativity constraint on $X(f)$ has been imposed, so it needs no modulus.)
That's nice, and in other contexts would be great advice, but using a glm is not going to fix the problems identified in my answer.
I bought Max Kuhn's Applied Predictive Modeling Book a couple weeks ago and now am trying things like creating a train and test set and looking at ROC curve to pick different cutoff point (trade off between sensitivity and specificity), etc. But I am finding between trying all these different methods & trying the different number of years of records to use to train with, looking at variables and derived variables that this is really really time consuming. I know this is isn't suppose to be easy but I thought I could get at least get some solid advice regarding how to structure the data.
Since the problem is resolved, I've cleared this comment thread.
perhaps you can summarize what the link is about and howvit relatescto the question. As it is now, your answer is better as a comment.
In a big city at summer, murder rate positively correlates with rate of ice-cream consumption.
Dear @whuber I really want to implement this in R so I have posted a [question](http://stackoverflow.com/q/8408760/168139) in stackoverflow. You may be interested. If you use R I would love you to answer it otherwise you may just want to sit back and whatch
I've also seen it called "set validation".
I like the clarity of your answer.
For a simple example let $x=(2,20,20)$.
The central limit theorem says *nothing* about n=30.
I don't know of a specific paper about this, but a decent book on linear models should contain the theory needed to justify it.
Well, you can't, that's the point. You must manually add this datum attributes, train the forest, get importance and do the test on original versus permuted.
likert-type scales are not ranked, they are rated. What are the characters that people have to rank? Are they the scale levels itself or some different objects? Give more detais please.
By the way, you certainly cannot use AIC in generalized estimating equations models, as the estimates produced are not maximum likelihood estimators.
There were similar questions already, some with very good answers. http://stats.stackexchange.com/q/15958/3277, http://stats.stackexchange.com/q/53421/3277. Did you browse it?
+1, but a minor point is that using `lmer()` with `family='binomial'` gives a warning in the latest version of **lme4**. The note in the [release guide](https://github.com/lme4/lme4/blob/e89cb416011a94410f35d64f8aaaff2fba895e8a/misc/notes/release_notes.md#lme4-release-notes-v-10x): "In the past, lmer automatically called `glmer` when family was specified. It still does so, but now warns the user that they should preferably use `glmer` directly."
If your data really are that you observed 26 distinct classes out of a sample of 26, then you have essentially no evidence against the hypothesis that all 127 classes have equal probability. This can be assessed with a multinomial distribution calculation.
Thank you!! Great answer!
@onestop  I guess it is an unfortunate artifact of my age.  It is not an understatement to say that Feller's books revolutionized how probability was done -  worldwide. A large part of our modern notation is due to him. For decades, his books were **the** probability books to study. Maybe they still should be.  BTW: I've added the title for those that haven't heard of his books.
possible duplicate of [Difference between logit and probit models](http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models)
What particularly do you want to do *with* SAS. It's an extremely expansive language.
Why not take it at face value? Weak predictors are getting their coefficients shrunken away to nothing (for a given value of the shrinkage parameter).
This is a nice start, but please re-check your series coefficients: the signs seem to be incorrect. After all, $x \\log(x)$ is convex and so must have a positive second derivative.
The data is uniformly independently distributed at random across all the nodes. I meant optimum for the expected value. Information can be sent from one node to another at unit cost
@Joris The df is the correct value to use, not the nominal number of parameters.  The former has mathematical and statistical meaning whereas the latter does not.  If you use more parameters than there are df, then one or more of those parameters will not be identifiable.
@Glen_b: I have edited the question accordingly.
Also look at answers [here](http://stats.stackexchange.com/questions/26450/why-does-a-95-ci-not-imply-a-95-chance-of-containing-the-mean), [here](http://stats.stackexchange.com/questions/11609/clarification-on-interpreting-confidence-intervals), [here](http://stats.stackexchange.com/questions/3911/when-are-confidence-intervals-useful), & [here](http://stats.stackexchange.com/questions/79998/what-should-everyone-know-about-confidence-interval) until you get tired.
Please, [don't cross-post](http://math.stackexchange.com/q/618608) on multiple SE sites. There are various formulae to construct QQ-plot, i.e. expected probability normal probability (usually plotted on the y-axis). Unless I'm wrong, this formula corresponds to Blom transformation for normal scores.
(1) I would like to be able to open and close the data file as easily as I can open an Rdata, Excel, or SPSS data file. Spending time walking through a wizard works, but it's not quite the simple and reliable workflow that I'd ideally like. (2) Yep, I agree about using an irregular delimiter. In general Tab is sufficient for me most of the time; (3) I don't have huge problems with CSV/TSV. I have occasional problems that are easily resolved. However, I'd like not to have to think about the issues of delimiters and format conversion.
Sorry, I forgot to add the crucial part of this question: What if there can be 20 winners? What are the chances of a person with n tickets winning then?
@gung It collapses a larger table to a bunch of 2x2 chi-square tests (the cells [i,j]; [(i),j]; [i,(j)] and [(i),(j)]) - nothing more. (If you already have a 2x2 table, it's simply the ordinary 2x2 chisquare test repeated four times - about as pointless as things get)
@mbq Also FYI there is a pyramid chart capabilities in the [HH package](http://www.inside-r.org/packages/cran/HH/docs/as.pyramidLikert). This [question on the site gives](http://stats.stackexchange.com/a/25156/1036) examples for Likert type data, which I imagine can be hacked to produce similar bar charts. In ggplot2 one would need to display reflected counts, an easy way is to just have one bar be negative and the other be positive (not sure if ggplot2 supports mirrored dimensions like pyramid plots though).
If $x$ is uniformly distributed, the normalized one is is uniformly distributed on the unit hyperball.
your solution would require `data$V1` to be pre-sorted. More generally, `sort(data$V1)[.95*length(data$V1)]`, would be the approximation you want. However, if `.95*length(data$V1)` is not an integer it would just round down to the nearest integer when indexing `sort(data$V1)`, so this approximation would always underestimate in that case.
Your question would benefit greatly from including the actual plot...
@JohnSalvatier, I see your point about avoiding the explicit inversion for very large covariance matrices. The reference you found seems good.
@myself:
downsampling in `randomForest` package happens much easier with:
`sampszize=rep(length(minorityClass,2)` (in `randomForest`)
The rest of the questions remain unanswered.
You don't "lose data" at all, in the sense that all your data points are preserved. You lose some information about how the original points are located relative to each other. To assess losses, you could check how much euclidean distances between the points are underestimated under the reduction done. This is essentially what Stumpy Joe has proposed.
+1, this is a nice way of putting it. It didn't occur to me to extend the discussion beyond a squared term.
Also consider what it means if the population covariance matrix is singular. Then the data sit in a proper subspace of the 64 dimensional Euclidean space.  So there is no 64-dimensional distance that can be defined!  If all the classes have the same singular covariance matrix then you can use a pooled estimate of the covariance and you can think of the problem in a lower dimensional space where everything lives and the lower dimensional covariance matric is nonsingular.
And the same stands for calculating correlation coefficients, I guess?
See [the self-study tag wiki](http://stats.stackexchange.com/tags/self-study/info) if applicable.
@Moritz : In reply to your comment about EM being a heuristic, EM does have strong theoretical foundations and it is known to provably converge to a local maximum.
the question of choosing `lm(log(Y) ~ x1 + x2)' versus the original exponential model is down to how you think the *error* behaves. If, in your data, you see that the noise increases as `x1' and `x2' increase, then you have good reason to use the log model, but if you see that it's constant, then the latter is more appropriate - my argument depends upon the lm function assuming constant variance of noise - i would have a look at http://stats.stackexchange.com/questions/47063/how-to-perform-exponential-regression-with-more-than-one-predictors-using-r?rq=1
I don't think there is an "acceptable" and "unacceptable" range.
Check out [Khan Academy](https://www.khanacademy.org/).
For the normal it gives a divisor of $n+1$, but one problem is you don't actually know what distribution you really have. Yet the $n-1$ form is unbiased for every distribution. I often just use ML, but I'm generally as happy with $n-1$, and not averse to $n+1$, even though I rarely use it. It's only a hard choice when $n$ is small.
A "date," by definition, is an amount of time elapsed from a fixed origin, such as Dec 31 1899, January 1 1970, or January 1 CE. As such it is a *numerical quantity*. A fractional date of course is a fractional amount of time; for example, right now it's about 3/4 of the way through the day here (almost 6:00 pm).  Where are the difficulties, then, in computing statistics for those numbers?  @zbicyclist In Excel a date is a *floating point number* representing days elapsed since the end of the year 1899. This allows Excel to represent times of day as well as days.
(+1) I agree - the direction(s) of causality is not at all clear.
+1, I just implemented one by choosing existing points as initial centroids, and yes that's more robust. Kmeans++ might also be good.
(+1) I remind me of `plot.design()` (but yours in an enhanced version :-)
@ Jonathan Thiele: Thank's! That looks fine. I will try it.
I am currently reading The Elements of Graphing Data, http://books.google.com/books?id=KMsZAQAAIAAJ&q. If you are familiar with it, do you know what the differences between these two books are (I can't seem to find a good description of either online)?
Speaking of CIs when the number of successes is 0 (out of N trials), I've always enjoyed the simplicity of the _"rule of three"_ http://www.jstor.org/discover/10.2307/2685405?uid=3738984&uid=2129&uid=2&uid=70&uid=4&sid=21100794941751
Glad to hear that. If you like a particular answer, upvoting it is the best way to that. If you think that an answer is 'the answer to your questions', you can accept it as an answer.
Good god why are you doing this in Excel??
Thank you for your contribution, Prof. Sims. Do you agree with my answer bellow? P.S. Now we have Nobel Prizes posting on SE. How about that? http://www.nobelprize.org/nobel_prizes/economics/laureates/2011/sims.html
Thanks for the help.  For expedience, I ended up using Excel's correlation function and that got seems to have gotten what I need.  I found some VBA to implement Kendall's Tau in Excel and will experiment more with that later.  Also, tried R again and remembered why I usually stick with Excel.
Actually, you can probably approximate NNs with appropriately transformed regressors in a linear regression as closely as you want (and vice versa). Better practice than quadratics and cubics are splines, though - I heartily recommend Harrell's textbook "Regression Modeling Strategies".
Thanks, this is exactly what I was thinking of.
Is $a_{12}$ correct, by any chance?
Is this a real experiment or an exercise? If it's the latter, please consider adding the [tag:self-study] tag.
Does *y* include all inbound `pcs` (divided by total number of hours of operation I'm guessing?) or just those that are handled by the end of the day?
@alis Excel's `=CHIINV(0.308364587, 4)` function gives the correct answer, 4.800701.
There is more than one way to define a residual.  See http://data.princeton.edu/wws509/notes/c3s8.html for instance, which lists three.
+1 for the references.
I'm still waiting for that feeling.
For anyone who wants to know more about the use of negative correlation in Monte Carlo simulation, try googling "antithetic variates".  More info in course notes [here](http://www.columbia.edu/~ks20/4703-Sigman/4703-07-Notes-ATV.pdf) or [here](http://www.math.kent.edu/~oana/math60093/10lecture5.pdf).
Some alternative comments. First, expectation is a linear operator, so you can pull $x_i$s through, and be left with $E k_i$. Second, you are sampling from a finite population $x_1, \\ldots, x_n$, so a good sampling book with sufficient explanation of the finite population sampling and inference will have all the necessary results, like expectations and covariances of the multinomial variables. Finally, the problem is not clearly formulated. There are two probability spaces, that of the normal $x$s, and that of the bootstrap samples. Correlation wrt to which field are they asking about?
I think this is an instance of the well-known Blazing Saddles problem.
Aniko,thank you for your invaluable suggestion,I will try later.
Another possible source of confusion is that the pdf of a _discrete_ random variable (also called pmf - probability mass function) indeed cannot exceed 1.
+1.  The course on Jaynes' theory (to which you link) contains this remarkable quotation, which I think automatically settles the matter of Jaynes' credentials for any mathematician trained after the 19th century: "There are no really trustworthy standards of rigor in a mathematics that embraced the theory of infinite sets."  (The course notes address some further mathematical solecisms: pp 8-10.)
Thanks, i did come across the difference of poisson variables, but it was difficult to make the leap to the binomial distribution of to consider the final number of equal elements.
If you have the relevant sample sizes ...
Actually I think nowadays the skew in a lot of countries is more towards seniors, not tots.
Najat, I think you've got unstandardized and standardized mixed up. Also, the difference shouldn't matter for model selection. I'll try to see if I can offer another answer here soon...
I'm pretty sure the topic is here: http://stats.stackexchange.com/questions/12842/covariance-and-independence and the answer by Greg (Snow) is here: http://stats.stackexchange.com/a/12898/2073
@harold Well, it is not so easy, given I am not an expert on the analysis of frequency domain data. Maybe you can standardize your input values by dividing them by the integral (=sum) of all 128 values? There are some books from Bishop and Ripley about Pattern Recognition and NN - however I did not read them, so better taking first a look into before deciding. Or consult a Data Scientist :)
@Error404 That's skewed, but not what I'd called _immensely_ skewed. Many applications involve probabilities of $10^{-3}$ or less: rare diseases, insurance risk, credit risk, etc.
I was thinking of one exemplar question, such as "Do you approve of the way that the Management Office handles complaints?". It seems from textbook examples, that if you had two actual (independent, as you point out) polling results which were gathered closely in time (so that time can be ignored) you wouldn't need to model anything... Or maybe I'm just fuzzy and in fact you have to model your uncertainty otherwise you end up with two spikes for answers -- I'm thinking of modeling in the sense of a linear regression with multiple predictors, etc.
@Aksakal- I agree that mscohere() in matlab does exactly this, but I don't have access to Matlab and was unable to find an R package that performed the same functionality as plainly.
I'm not sure that the "partial effect" is as meaningful as the estimate of the difference in $y$ between two chosen values of $X$.  By default I use the inter-quartile-range effect of $X$.
The gam model starts with knots set at 10, which you do not have enough data for, however you have densities of 5 and 6 groups - you can set your knots 1 fewer and the model should run. Something like    model4<-gam(rates~s(pred,k=5)+s(prey,k=4),data=data)
I think StackOverflow will give you better answers.
Some example data can be useful, so answerers could test their ideas.
What happens if, after losing a gamble & refusing to reduce his bet, he has $d\\ <\\ ^M\\!\\!/_N$ left over? Does he stop betting & that is his final winnings?
Have to agree with @PeterFlom on this one.  Don't get too complicated if your audience isn't statistically knowledgeable.  You could also do comparative/parallel box-plots with an overlay of butterfly plots on top.  That way the box-plot summary is visible as well as all the data.
Not always.  For instance, let $f(x)=e^{-x}$, a non-negative function defined on all the real numbers: there is no normalizing constant. But when you are told, as in your question statement, that "so-and-so is the PDF for such-and-such a distribution" then there is nothing whatsoever to verify: by *definition* it integrates to unity.
Robert, I believe that is some misunderstading of what generalization error means vs. complexity of the network, i.e. the ability to model more complex functions. There are a number of papers describing the different terms, like http://cbcl.mit.edu/projects/cbcl/publications/ps/notices-ams2003refs.pdf.
@slhck These data suffice to compute the *range* of the desired confidence interval, *but they do not determine the mean of y.* To see this, consider what would change if you were to add a constant $c$ to all the $x_i$ and simultaneously add $-0.00024 c$ to all the $y_i$: the answer is, *nothing.*  Consequently you cannot estimate the mean of $y$.  Thus, if the original question is answerable at all, understanding the vague "upper and lower 95% and 99%" is crucial. I would guess these would be confidence limits for the mean of $x$. (See the replies for why I'm wrong. :-)
Your research question is not clear to me nor the theoretical interrelationships in the data.  If you were to block on a variable (i.e., do a repeated measures analysis) would you block on rows or patients or both?  Would you like to do one test to see if there is any statistical difference(s) in the data or do you have specific questions in mind?  Are you interested only in central tendency or is dispersion of interest to you also?  With a little more detail, perhaps someone here can be of help.
Thanks for the answer. $\\Vert X_{1} \\Vert^{2} = {}^t X_{1} X_{1}$ is actually a quadratic form in the random vector $X_{1}$. I'll look into that !
Thank you sir! :)
Which command takes hours? If this is `seqefsub`, try setting maxK=3 (4, or 5). It limits the length of the subsequences we are looking for (we only search for subsequences of length 3, 4 or 5). Most of the time, longuer subsequences are not of the main interest.
The residuals vs fitted plot is usually more for identifying non-linearity: the *scale-location* plot is slightly better tuned for diagnosing heteroscedasticity, although these two plots are admittedly pretty similar.
Presumably the OP wants the covariance structures here: http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mixed_sect019.htm ... these can in principle be constructed via the `corClasses` facility in `nlme`, but it's not trivial and as far as I know hasn't been done ...
Oops, you're using agglomerative methods, so the "1 or 2 clusters" part doesn't make sense -- the "inverse" applies, you don't want n-1 or n-2 singletons, etc, i.e. letting clustering work for a bit before applying validity criteria shouldn't be problematic.
I'm a big fan of simulations. They can be a good way to explore your data & help you think about the possibilities that lay behind them. I would use them to plan a *future* experiment rather than to draw conclusions about *this* experiment, though. EG, they could guide you towards where to sample data from (in the $(J, Re)$ space) in your next experiment to best differentiate between the different possibilities. OADN, I'm not suggesting that you can't learn anything from these data, only that the p-values won't be very meaningful.
Fixed the link to TestU01
This may get a better response on the statistics SE site, which is more appropriate (SO is primarily for programming, algorithms, and the like).  If you flag your question and ask for it to be moved, it will likely be accommodated.
I'm not sure about R. I would switch to Python. The script I linked can be used with a csv actually, by calling `python gl_tools.py csv_name.csv out.png -a <alpha>` but you need some libraries in python installed.
Is your goal primarily prediction or some form of inference?
(+1) Thanks, that is helpful. I will look into it.
Questions that are *only* about how to get R to give you something are off-topic for CV. You may want to edit your question to highlight how goodness of fit measures can be understood / calculated when using multiple imputation, & ask about R code / examples as a sideline.
Wow! Fantastic answer!  Thank you!
I think you might benefit from reading a little bit about mixed-effects models first, perhaps especially in R. (Look up the package `lme4`.) In particular, the sentence "controll[ing] for the difference between time 1 and 2 [in] group 2" does not make sense to me.
@Alexis But the binary matrix that I have is of dimensions 20000 X 300 and it primarily populated with zeros! I calculated the sparsity it is around 99%.

And sorry there was a typo, the sample size is 20000.
[Any use?](https://www.ideals.illinois.edu/bitstream/handle/2142/11103/Random%20Number%20Generation%20Using%20a%20Biased%20Source.pdf?sequence=2)
In this algorithm you cannot specify the number of clusters, you can change them varying parameter 't'. For R implementations just Google it, there is at least one.
Hi Dikran, Thanks very much for your reply. It is really helpful. I am wondering if you have a reference for this theory you mentioned please? A.
"...but perhaps there is an effect (or: a bigger effect) of _today's_ svi and tweets on _tomorrow's_ tradevol" :-)
Could you be more specific and expand on your question? Which tree algoritm for example? What do you mean with "bivariate" here?
@Henry: No, the expectation is not conditional on a black ball being observed. So to be clear: the desired value is the expected number of white balls observed until **either** a black ball is observed, or $j$ picks happened, whichever comes earlier.
My internal debate seems without end!
OK, thank you @whuber, I'll do as you suggest from now on.  I agree with you, even though I ask how to do it in Stata, the core of the question is what is the correct way to do it, because I know there are lots of ways of filling in gaps, but I would like to know what is the best way to do it.