tttjjj/elt_bench huggingface dataset:
I agree with the database suggestion and the plan is that the data will reside in a database eventually but the method of data collection must be in Excel. In terms of database management this is the easiest method and I will write a short script to transpose the data from storage method #2 to #1. Thanks
Nope! Look at how complicated the negbinom fitter is in the `glm.nb` code, the $\\theta$s are estimated using the EM algorithm. `drop1` has no default method for negative binomial. However, the `logLik` function *does* (see `getS3method('logLik', 'negbin'`). I have fixed the OP's code and shown that doing this gives inference that's almost identical to the Poisson model under the null (which still has a significant calibration issue due to other problems).
I think you should ask this as a new question. In brief you should probably use an ordinal regression technique and might want to include a dummy variable for each country. (Probably not two separate models. But as @Conjugate Prior's answer suggests, the model used depends on whether you have variation in environmental variables *within* countries or only *between* them.)
The integrals do not appear to be "intractable"; indeed, they appear to be solvable with elementary functions.  The [mean distance from a point to a line segment](http://gis.stackexchange.com/questions/7440/transform-polygon-to-equivalent-circle-qgis) involves linear expressions times quadratic expressions and quadratic expressions times logarithms of linear plus quadratic expressions.  Integrating the latter by parts gives rational functions of quadratic algebraic expressions.  Those all can be done by completing the square and trig substitutions.
thank you arne, that is very helpful. Any suggestions for implementing the z-test in R?
Hi Alfred, thanks for joining the site, this comes off as a bit promotionalist though. Perhaps you have an example similar to the OP's dataset (14 variables and 345,000 observations) that you can provide a picture and describe/demonstrate how parallel coordinates could be useful? Many of the static parallel coordinates charts I have seen with so many observations tend to look like spaghetti on a plate, I suspect you have better insight how to make sense of such big N data though.
Adding `survplot(myfit, loglog=T, logt=T)` could make the graph even more interpretable. As always UCLA have a lot of info on this, including other aspects of checking the proportional hazard assumption:[http://www.ats.ucla.edu/stat/examples/asa/test_proportionality.htm]
I just saw your blog article ... and remembering this question, came here to point to your article, which I thought would be a good response. Looks like it was because it was your response. :)
Instead of averaging over daily batting averages why not calculate batting averages using the data from every day?  For example, if a player went 10-10 one day, then 1-1 the next day then 5-10 the next day then their batting average is 16/21 = 0.762 rather than (1.000+1.1000+0.500)/3 = 0.833?
There is not a single tree... But see @Shane's response for plotting one of them for illustrative purpose.
Is there some reason (e.g. very small/large proportions) you can't use the usual 2-sample $z$-test (http://www.statisticslectures.com/topics/ztestproportions/)?
I think if you only eliminate a few vectors, then you don't change the amount of information you can extract by much. Quantifying this accurately may be messy -- there are naive bounds which are probably too pessimistic.
Because "smooth" in mathematics (as an adjective) means something that the loess curve is *not*, it might be clearer for some readers to use "smooth" as a noun, as in "the curve is a loess smooth."
@whuber Oops, correct about the variance (I treated $n$ as a constant). As for the version of WLLN, well, actually I didn't try to use any, this was an exploratory exposition. We have non-identical r.v.'s with finite but asymptotically diverging variance, and I don't recall any such WLLN version (I am not saying that it doesn't exist). So I analyzed the problem and then tried to check directly the probability limit and whether the average sum converges in probability to the common expected value.
Excuse me, but I think that using "artificial intelligence" and "neural networks" without any context hardy counts as explanation. And I cannot see how one could use neural net for this kind of thing - what would be the output function, for example?
In the bottom third of the (long) page at http://www.quantdec.com/envstats/notes/class_06/properties.htm, I provide a graphical demonstration of Chebyshev's Theorem. It ought to make it immediately apparent where the probability must be concentrated when equality holds (thereby supporting @Glen_b's comment).  This is discussed under "Comments" at the very bottom.
@JuhoKokkala can you look at my edit, please?
For a classic, systematic approach to such questions in a least squares regression context, please see [*Regression Diagnostics* (Belsley, Kuh, & Welsch, 1980)](http://books.google.com/books/about/Regression_Diagnostics.html?id=GECBEUJVNe0C).
This is a standard question as might be found in a textbook, and should be tagged as self-study. As I was already editing your question for clarity, I've taken the liberty of adding it. Please check the [self-study tag wiki info](http://stats.stackexchange.com/tags/self-study/info). In light of that, what do you already understand about the question, and what have you tried? When your question says 'plot it', are you supposed to use software (and if so, which), or - more likely - draw it by hand?
Btw, I am a bit unsure about the $M_t$ multiplicative factor, which I asked about in a separate question: http://stats.stackexchange.com/questions/83849/interpreting-good-figures-of-merit-in-xorshift-context?lq=1. Would be really nice if you could take a look at it at some point.
I would really like to hear more about the "look into the use of evolutionary algorithms to design, prune and optimise architectures" part of the answer. Maybe I will ask a question about it!
Surely that's not what homogeneity is - sounds like heterogeneity.
@Glen_b please, verify if my last addition helps you.
Generally speaking, I would say that you should keep variables that are theoretically important or that have been significant in prior studies, even though your data do not support their effect. That being said, to get a more specific answer, I think you should add a couple of lines to explain your model and its purpose (e.g. identifying risk factors, making prediction, ...).
Thanks a lot for a great answer tobip. The above question was definitely posed during my "brain-scramble to understand basic model validation topics." Fortunately, it's all straightened out at this point. Will do my best to stick to fewer questions next time.
@Kyle Clarification: The effects of removing skew and turning multiplicative relationships to additive ones only apply when you take logs of the *dependent* variable - neither effect applies to the specific details of your question, which had logs only on independent variables. However, it's important to know about those effects.
I think to get an answer you're going to have to define what you mean by associative.
See also: http://stats.stackexchange.com/questions/3947/help-me-understand-support-vector-machines
That is absolutely delightful!
@whuber - thanks for the comments. I edited the original post per your suggestion. I hope it is more clear. The four metrics are related to maintaining the spatial smoothing of the variability. Consider a single point sensor that measures irradience. The recorded values on a partly cloudy day are variable. However, the average irradience value over a square mile would be less variable. The four metrics I calculate are quantifying the areas that the actual sensors represent - larger areas probably have more smoothing. The sensor density is just a clustering metric.
Mods: This is cross-posted from http://quant.stackexchange.com/questions/11183/whats-the-practical-difference-between-the-johansen-vs-engle-granger-tests-for?noredirect=1#comment17628_11183 as per the suggestion in the comments there. If you think one or the other should be deleted please let me know.
Think of a user, using a machine. He sees the data (w) but does not see y and does not care what it is. It is hidden from him and is hence a hidden variable. y is a construct you impose to model the system. Hope this helps.
That is a trick question. Try to think through what you would expect the correlation between the `residuals` and the `fitted values` to be.
I didn't really follow your explanation in further detail, sorry, perhaps you can explain in more detail in an edit to your question. Where you say - "*surely if I've got n number of results for non-irradiated with an equal n number of control*" - you can compare them only if you know which is which. It's possible I misunderstood something, I suppose. Do you have some of each group where you do know which treatment group they're from?
See any probability textbook. The distribution function of sum of iid variables is [convolution](http://en.wikipedia.org/wiki/Convolution) of individual distribution functions.
I *think* that the waist-to-hip ratio was actually considered by Quetelet or his contemporaries, but also got rejected because it wasn't normally distributed either. How far we've come....
I would like to preface the question by saying, I know I should have gone to r-help for questions like these. However, I know that some of the gurus are members of this forum as well. Also, I fear total destruction on r-help and r-dlevel for asking simple questions like these.
.subscript in $Y_{n\\in \\mathbb N}$ means stopping time.
By "$j = \\pm 2i$" do you perhaps mean $j = i \\pm 1 \\mod 10$?
I add my reponse in the post. Please take a look when you have time.
+1, However, I wouldn't call "a variable you include" an "omitted variable". Would you mind tweaking that?
They are substituting an automatic procedure for thinking.  A good index of predictive discrimination is Somers' $D_{xy}$ rank correlation between predicted probability and observed $Y$.  This is $2\\times (c - \\frac{1}{2})$ where $c$ is the concordance probability or ROC area.
Thanks for your reply.

If I use hypothesis tests to show there are no difference between means and variances of those data sets, does it mean that they have the same distribution?
thanks..I am not much concerned about missing value imputations..nor can I do many analysis and then average them..Basically as I said, I have 5 channels. 1 channel out of those have right values..for the rest 4 channels, I need to fill missing in those variables. My concern is , how to tell SAS that dont delete these variables, these are created deliberately by me as actually it doesnt make any business sense to have agent characteristics for those 4 channels..Hope you got my point..or any setting in which I can tell SAS that dont delete missing values.
Hint: the relation ${\\rm E}(u_i u_j) = {\\rm E}(u_i){\\rm E}(u_j)$ holds for all $j$ except one. Which one is it?
I agree it is important not to get technical unless absolutely necessary. My only comment was that, once you've already chosen to introduce the concept of consistency (a technical concept), there's no use in mis-characterizing it in the name of making the answer easier to understand. I think your edit does address my comments though.
Sorry...I meant 10 training data, I edited it in the Question. These data arise from 2 classes A and B ; so the first 5 from class label A and next 5 from class label B; I have also added these info.
Draw a histogram of some of your data to see what they look like.
+1 Nice clear explanation. The data set I just added in my answer supplements it, and will hopefully help focus future discussion of the question.
Thanks! I'm a little bit ashamed, it was a simple mathematical issue...
Did anyone else notice that the tour guide changes colors between the second and third frames?
@vinux has a nice answer, and provides an informative link for understanding the role of normality in correlation.  I just wanted to point to this question: http://stats.stackexchange.com/questions/298/ which is very good for understanding what logs do in regression.
Does anyone know if R's glmnet package supports group LASSO? If not, would anyone suggest another one that does that in combination with logistic regression? Several options mentioning group LASSO can be found in CRAN repository, any suggestions of the most appropriate for my case? Maybe SGL?
Note that $e^{5.53} = 232.75$ That's why the Poisson and NB give different intercepts: They aren't on the same scale. The Poisson model is $\\log(E(Y|X)) = \\theta'X$. The intercepts are very close.
@KapelNick, so which parts don't you understand or need more explanation about?
Adding irrelevant variable increases SSE. I am begining level. So I have No enough knowledge to answer your questions properly. @Stat
2nd paragraph:  Shouldn't you divide by 3! as well, yielding 220 combinations (not permutations)?
"Help me brainstorm an open-ended question" would seem to come rather close to the problem discussed [here](http://stats.stackexchange.com/help/dont-ask) "*If your motivation for asking the question is “I would like to participate in a discussion about ______”, then you should not be asking here.*" --- it's probably a more suitable topic for chat. If you can rephrase it into questions that have clearer answers, it it might be more obviously on topic.
Thanks, but I was hoping an idea of a general model taking all variables into account. Here is an example of a 3x2 table, mine is a 2x4 table: http://stats.stackexchange.com/a/9721/29337
I disagree - that one is closely related to machine learning, not statistics *per se*!
Hi @steffen. Thanks for your suggestion. I added the definition.
What would it mean here to "make a conclusion about the male / female general interest"? Are you wondering how much data you need to get a 'significant' result that men & women differ if the true proportions are 4/1k & 1/1k?
Our goal is to make sales predictions based on zip codes. Our database contains only last year data.  Thanks for the suggestions, I will do some research on that... Do you have a link to C5 trees in R?
The Tobit model generally also has normally distributed error terms, I'll check out GLM. Suppose our response variable was test score that ranged from 100 to 300, which model would be appropriate here?
Maybe have a look here: http://stats.stackexchange.com/questions/17488/test-to-prove-two-means-do-not-have-a-difference?rq=1
A note on notation: this is an interesting combination of rank statistics and order statistics. Unfortunately the numbering conventions in these two areas is directly opposite: in rank statistics, "1" indicates "highest", while in order statistics "1" indicates "lowest". For example, in the question's example $O_1 =1 \\Rightarrow x_1 = x_{(4)}$, etc.
This question appears to be off-topic because it is about asking for R packages / code.
Any chance of matlab implementation of CEEMD?
Good night, Stephane!
Can you give a concrete example of what you're interested in? "Signed likelihood ratio test" does not appear to be very standard terminology.
As for currently, this question is not about clustering. It is about text mining or semantic analysis and, maybe, implies topics such as stemmatization. Before you can cluster you must get the data in the unambiguous form.
Followup question: if I used a custom step method for sampling one of my variables, then how do I even define the variable in PyMC given that I don't want to bother with a log likelihood function (since I'm defining my own sampler)?
@gmacfalane: i'm not sure what is your claim. Are you are stating that there can't be outliers in the regression settings when the responses are multinomial? If this is your claim, it very simple to construct a counter example using the fact --stated in that second section-- that observations outlying only in the design space will have an unbounded influence on the estimated parameters of any M-estimators (a class to which multivariate logit belongs).
A solution of type 1; have a look at the literature on "loss estimation", in which one makes an additional decision that depends on how much (expected) loss there was for a primary decision - say a point estimate.
Fine. If you have any doubts, just ask.
This is the answer I have been hoping for. Thank you very much! I do think I should be able to calculate the numeric results in Matlab :)
@Andre Silva: thanks! that's useful
@DavidR Well, the idea is to decide of a cut-off value (read, "subject at risk above a certain value"), which comes with a lot of caveats from a statistical perspective but most clinicians are used to or prefer to work this way. (Sorry about not having noticed your comment earlier!)
chl's answer to [Comparing test-retest reliabilities](http://stats.stackexchange.com/questions/3855/comparing-test-retest-reliabilities) addresses this. You may also want to see [Inter-rater reliability for ordinal or interval data](http://stats.stackexchange.com/q/3539/32036).
1. Checking the kurtosis to check the presence of bimodality can be quite tricky; probably a $strong$ negative number might indicate that but still it would be a stretch to be simply based on this. Kurtosis is sensitive to outliers so be weary. 2. Using two distribution will almost certainly give you `samples that most resemble(s) the original data` in comparison with a single distribution so that does not say much. 3. Check the following thread: https://stats.stackexchange.com/questions/95015/is-it-valid-to-use-hartigans-dip-test-to-reject-uni-modal-null-hypothesis-with it will probably help.
If it's not a big secret, what are the values? Measurements of something? Times? Ratios of things? Compositional data/proportions? Why did you try to transform it? If there are zeroes, why would you take logs?
From A, I separate out values less than and equal to 60 in A1 and other values more than 60 was put in A2 so that number of samples in A1+A2=A. Similarily, splitting was done in case of dataset B. Then we compare mean of A to B, A1 to B1 and A2 to B2.
+1 Beautiful answer.
Some magic words to get you started are *Marchenko-Pastur* and *Wigner semicircle distribution*. The classical theory takes as a basic point of departure random matrices with mean-zero entries.
It seems to be standard in education, e.g. http://books.google.co.uk/books?id=3LhPwUhrVIcC&pg=PA397 or http://www.umich.edu/~exphysio/MVS250/SpearmanRankCorr.doc
Are all the features important? How many samples do you have? If I understand the problem correctly, you can try to do some variant of boosting - repeatedly select subset of samples and fit all variables to them and see which ones pop-up more often.
(Corrected my p-value typo.) Good points here. So collapsing levels, provided it is based on some real-world and logical reason justifiable in the context of the study (that might also happen to parse them out along the significance break) is reasonable, but not just lumping them arbitrarily based on their significance. Got it.
This [article](http://onlinelibrary.wiley.com/doi/10.1002/bimj.200410103/abstract) will certainly clarify matters for you. Unfortunately access is restricted.
Thanks you for the mathematical explanations.
@whuber thanks for pointing that out. The paper was pre-$\\LaTeX$ and the Greek letters are just a bit smaller than the Latin ones.
Thanks, Jeromy. Actually this is even better than what I asked.
Could you explain what you mean by eigenvector "orientation"? As far as I know, there is no such thing that is *intrinsic to the data.* (That's one reason why different software will produce different normalized eigenvectors.) So it sounds like you're asking for something that does not exist and is meaningless.
Do you mean 50 of the 100 subjects? And significantly different from what? Do you want to know if the part is different from the whole?...the remaining part?
The question is a tad vague.  Can you focus it a bit more?
I seriously doubt *anybody*, except a few pathological souls, would be so meticulous, statisticians least of all (because they know better).  Think about it: if you need to be recording these times and dates, then you're likely having sex a lot more than once a month (it happens to a few lucky couples :-), in which case the time of conception is still indeterminate, so what's the use in having the record?  Statisticians would also be aware that gestation is variable, so that a full term might be 39 weeks for some women, 41 for others. In light of that a precise record is superfluous.
Most undergraduate texts on theoretical statistics or probability theory have a chapter on distribution theory that covers these questions. But how much mathematical background would you want to assume?
(+1) This gives a good first start. Combining this with linearity of expectation leads to an economical and elegant solution.
@Macro, Okay, I must not be using the term linear model correctly. I need to evaluate the set of 3-space datapoints to see if they fall any line whatsoever. In school we were given an equation for linearity factor R, as described here http://stattrek.com/statistics/correlation.aspx. If a correlation exists, I need to find best-fit linear model as shown here http://mathworld.wolfram.com/LeastSquaresFitting.html. Finally, I need to see  if it is parallel to any of the axes.
A little niggle: I don't think you need a phd in statistics to deal with random effects correlated with regressors by adding averages of individual level covariates as predictors for group level intercepts. This fixes assumption ii) across levels, and if you don't like it within level then you ought to think twice about regression models altogether.
[Masson and Loftus (2003)](http://web.uvic.ca/psyc/masson/ML03.pdf) is an oft given reference. They show example plots in which they panel the CI of the difference in a second panel of the same graph.  Whether that will make the everyone in the circus happy I do not know (I feel for you though!)
I assume you mean to say "*without* the aid of computers". The answer is - perhaps unsurprisingly - by the use of algebra. The derivation is quite straightforward and in many places it's common for statistics students to derive it as an exercise/learn it as undergrads.
I can probably only perform the logistic regression if I have the raw data, correct?
@Ioannis: that is true and you cannot standardise a Cauchy distribution to have a mean of $0$ and standard deviation of $1$.  My second paragraph was aimed more at the common interpretation of the Central Limit Theorem as suggesting a Gaussian approximation with dispersion $1/\\sqrt{n}$ times the original
Why can't you use dummy categories? I would have thought that almost all statistical software will do this for you. e.g. in R you could just have _race_ and _state_ as two factors, with each level being a string (letters), e.g. _white_, _black_, _purple_; _texas_, _virginia_ etc, and as long as the software recognises these are factors (usually automatic if you've got strings in the data) it will automatically convert them into dummy variables in the model?
Did you have a particular method in mind and if so could you post a link?
Although these statements make no sense as given, they make perfect sense--and lead to an intriguing question--provided "probability distribution" is replaced by "mean" throughout.  Is this perhaps what you intend to ask?  Also, is this "certain numerical range" specified in advance of the analysis or are you asking how to construct an interval that has a given chance of enclosing at least one of the [means]?
(+1) Welcome to the site, Jan. This site supports $\\LaTeX$; I've tried to typeset the math. Please feel free to make edits if I've introduced errors.
@whuber. good point. Being a stats consultant in psychology, I sometimes get questions by email that are fairly elementary. My ideal situation is to encourage such students to post directly here. In general, I prefer to answer these questions on this site rather than send an email response to the student. That way, my response can be an ongoing resource for the internet, and others may come up with an even better response.