tttjjj/elt_bench huggingface dataset:
Don't think that would help. I gave those examples because I know people with those jobs doing statistical data analysis.
I don't see where the DKW inequality is limited to continuous distributions based on the wikipedia article http://en.wikipedia.org/wiki/Dvoretzky%E2%80%93Kiefer%E2%80%93Wolfowitz_inequality
No, i meant steps 1 and 2 in the order I wrote. After all, the distribution of y is specified conditional on X, so you must generate an X before Y. As for Gibbs sampling, that is a more complicated solution meant for more complicated problems. Yours, as you describe it, is pretty straighforward, IMHO.
The attributes you would use would depend on what your goals for the metric are. However, I only meant that it is a better *analogy* for the CoV. NB that the 3rd quartile is the median of your data that are above the median, & the 1st q is the median of those below, so IQR/2 in the long run will equal MADM (nb, they aren't guaranteed to be equal in a given sample). The IQR will vary further, on ave, from it's true value in the pop, but I'm not sure what, if any, implications that would have, & the stand. err. of IQR/2 should be the same as SE of MADM.
+1 (actually, I upvoted this a month ago & just happen to be back rereading it now) at any rate, it occurs to me that you recommend checking the contrast type by `options()`. You will have to scroll through a lot of junk to find what you need.  You might try `options()$contrasts`, which will only output what you want.
An example is to take $A=X$, $B=Y$, and $C=X+Y$.  We can take $X$ and $Y$ to be independent, yet both $A$ and $B$ are correlated (positively, Pearson) with $C$.
Thank you very much! That was helpful. I do have another question, though. How do I use zero-inflated count models for surveys (specifically zero-inflated negative binomial), and survey logistic models?
@ Fergus -- thank you for the pointer.
Yeah, but you use the residuals to test your assumption about the unobservable error.
If you can deal with a static image, you can select the range, then copy and paste special as a metafile or an image into Powerpoint/Word.  Otherwise, probably not doable.
Can you please call the tests out by name? By a test of individual unit roots, presumably you mean the Im, Pesaran and Shin test, and by common unit root test, you mean the Lin, Levin and Chu type tests?
I agree.  The population is often a construct representing abstractly an infinite number of in your case two populations of students, one that is treated with alcohol and the other that is treated with marijuana. Statistical inference is made by assuming a parametric or nonparametric model for this population and the test subjects are viewed as though they are a randomly selected sample from this population.  In other situations such as with survey sampling, the population is a finite sampling from from which an actual random sample is selected.
angela, if you want to be sure @gung will see your comment, you need to comment under his answer rather than your question, or put "@" in front of his name. Either will notify him. This comment should do it.
I think you mean "prediction interval". If v and b are known, then all you need to find is the residual variance. If v and b are unknown, you can use ordinary least squares as the equation is linear in the coefficients.
I think that what you are doing is a form of "Coordinate descent". If you google it you should find quite a lot of literature on the topic, as it is a widely used class optimization of algorithms.
Thanks. It sounds like using a delimited file format with appropriate care may be the best option.
That seems fine, what is your question?
That was very informative. Thank You!
Your question doesn't really make sense. Power is the probability of rejecting the null hypothesis given that the null is not true. I think what you're asking is more like "To achieve the same power, does one require a greater sample size when testing an interaction than when testing a main effect?".
Sure, I'm not so much concerned with what the proper name for these attributes would be. My goal is, for example, to find a list of product scores which, if a new merchant started using them for deciding which products to promote, would minimize the expected mistake. The score shouldn't reflect any actual observable quality, just something which makes it possible to distinguish between winning and losing products.
I know how to identify some of the data as potential user user or non potential. The problem is my data has about a million records. How many records that I need to  identify manually in order to build classifier? Is it still reliable to do classification method in this case? Which method that I should take if I have some defined parameters to identify the class?
Do you know mvrnorm() from the MASS library in R (http://stat.ethz.ch/R-manual/R-devel/library/MASS/html/mvrnorm.html)? It enables to generate a vector of values from a multivariate normal distribution. Then, you can proceed as for simple linear regression, conditional on the random effects.
Machine learning is a big field, and people are successful doing all sorts of things. You can work with datasets that easily fit in memory and happily make use of R and CRAN all day, or you can do deep learning neural networks using six months of time on a giant cluster at Google burning cycles as fast as possible with C++ or a similar language. I know of at least one well-known GP package written in Clojure. I think it depends on your goals. If you have a problem to solve, I'd stand on the shoulders of giants. For learning and fun or novel research, writing your own can be useful.
For the connections between DD and DP and Polya urns and Chinese restaurants and sticks: http://blog.echen.me/2012/03/20/infinite-mixture-models-with-nonparametric-bayes-and-the-dirichlet-process/. I don't know of a programmatic exposition of the Gibbs sampler for it. There are a lot of tricks and things to keep straight in a Gibbs sampler for DP, so each implementation can be different, and any such document will probably fail to cover many features of other implementations. http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm.html#example-mixture-plot-gmm-py is one implementation.
All the conditions of the experiment you describe remind me of A/B-Tests ... coincidence ? :)
@wcampbell thanks for your comment. But I'm interested in finding an appropriate bandwidth by eye
The preceeding ANOVA usually has more power to detect that there is a statistically significant difference among groups. It often happens that you perform ANOVA and find a p-value < 0.05 and then when you perform the multiple comparisons all p-values are non-significant
God must bring data too.
Welcome to the site. You should use a title that is more descriptive. This is a statistics site; just about every question could be called "Help: Statistics"
Both you and the reviewer have a defensible case. The reviewer is treating the term narrowly and you are treating it very broadly. I have been on both sides of the relationship here, and a journal editor too, and advise treating this as needing clarification, not confrontation. But as you ask for specific advice: (1) I wouldn't use the term "correlational" for multiple regression; (2) I would make my point in other terms when one choice of terminology is clearly an irritant.
@Eupraxis1981 Yes, I see what you mean. Check the updated answer.
Welcome to our site! Could you please elaborate on how your answer addresses the question about "accuracy"? How, exactly, is that related to percent of variance explained?
Can you explain this a bit more? At first thought, it seems like any method that claims to objectively translate ordinal data to interval data would be magical, since it seems like you could always give not-so-pathological counterexamples that would "break" any method, but I could be missing something important. It also seems like it would require some _a priori_ notion of the distance between ordinal categories - where does that come from?
Nice paper and suggestion -- thanks.
Any chance of some references to where 'it has been reported' or this stuff about training images? I'm not really sure what you are on about here.
I didn't say it is "bad". But suppose that you have good prior information about the smoothness of the random density. You can't use this prior information if you are modelling with the plain DP. That's the kind of thing that I have in mind.
I think you are right that I need something nested.

I just want to check that I understand the approach you linked to:
Could the dummy variable be an additional independent variable which is 1 if the participant has answered correctly and 0 if the participant was incorrect? With a dependent variable that still has 14 levels ranging from correct answer plus highest confidence rating to incorrect answer plus highest confidence rating?
Or do I misunderstand?
Thanks very much! (And thanks for the recommendation of Train's book! It's great!)
Your interpretation seems right. The model allowing for different correlations between the three task pairs does not fit significantly better strictly speaking ($p = .07$), but its AIC is lower, which suggests that it may be a better approximation to the true model (then again, its BIC is higher). I would actually tentatively conclude that there may be evidence that the three task pairs correlate differently. I'll get to your follow-up question later (this part is more tricky).
...to continue, we have previously explored a number of correlation measures, with varying degrees of success. The problem has been finding a method that can cope with the non stationarity and aperiodocity of our data. For instance, I used [this local correlation tracking](http://www.cs.cmu.edu/~spapadim/pdf/loco_icdm06.pdf) algorithm, although nothing meaningful could be inferred. The tool that this question concerns is just a simplified way to measure a particular process we are studying.
That was my understanding and like I say, the prop.test function of R gives the same direction of results. I just wondered if there was something that I was doing wrong to not get the z=.08.
An alternative to {E|C}FA is the [multi-trait multi-method approach](http://www.davidakenny.net/cm/mtmm.htm). This is basically founded on a correlational approach--with its pros and cons (wrt. latent trait)--and it has been discussed on the following threads, among others: http://stats.stackexchange.com/a/9944/930; http://stats.stackexchange.com/q/24418/930.
Were the singular values different? Or just all 1?
(+1) It is clear from the very beginning that this is a well considered, informative explanation.
It sounds like you might be looking for something in the realm of a sequential design/analysis.
@gung Your comment confounds the `logSigma` used as an argument to `summary` with the `logSigma` reported in the output: although you describe the former, this question is about the latter. I believe that all the values printed after the estimate of 2.10986 are merely artifacts of the vectorized code used to produce the t values and p values. After all, it scarcely is of interest to test whether a log variance equals zero! See the manual page or the code for `summary.maxLik`, which is what actually produces this output.
I have several discrete indicators  - some of these are for a covariates that I'm adjusting for. eg I have 30,000 observations of X=1, and 1,000 observations for X=0. This is true of predictor I'm making inferences on as well (available observational data is unbalanced).
(+1) A nice step-by-step illustration.
+1 for R and ggplot2
Beyond what you have said, what do you think of a *p*-value "adjustment" that turns .0733 into .1?
@PeterFlom: I edited my description, is it clear now?
Interesting. What examples are there of the _unusual_ behaviour of a negatively skewed distribution where the mean is greater than the median?
REGRESSION procedure in SPSS uses Type III SS only. Use GLM (=UNIANOVA) procedure instead if you want to choose among types of SS.
@lakesh An ungated copy of the paper is available [here](http://public.econ.duke.edu/~boller/Published_Papers/joe_86.pdf).
You mean what the folks over on math.SE call the _freshman's dream:_ $(x+y)^2 = x^2 + y^2$ is _not_ true? :-)
If you don't have a lot more data than dimensions it is hard to find a direction that removes most of the variability whihc is what the first principal component is suppose to do.  In general there is the curse of dimensionality. Data tends to move away from the center in high dimensions.  Bellman wrote about this in the 1960s.
Are you speaking just of one-factor random data (random spheroid)?
@ Andy W: yes, you're right. Maybe this was why I was uncomfortable with the idea of time varying covariates.
This is normal situation. [Similar question](http://stats.stackexchange.com/q/51583/3277) (you may ignore that the talk there is about stepwise selection)
I don't think this is really a "sin" as the results obtained are not wrong. However, it does throw away a lot of useful information so is not good practice.
(1) not sure if f is correct (2) for b, why don't you substitute x (150 miles) into the equation you have (y=101.695+71.186x) to get predicted value of y?
The use of jargon ("CPI" and "ONS") and vague phrases (such as "throw in") make this question difficult to understand and problematic to answer. Please consider editing it to clarify what you would like to know.
You should consider editing the question to reflect this.  For example, take another look at the t-test results.  I'm sure people will also soon be pointing out the need to adjust for multiple comparisons, so you might want to get the jump on that by searching this site on [tag:multiple-comparisons] or "Bonferroni".  Your data are interesting in that this adjustment makes a substantial difference: it's not merely of theoretical concern.
Part of this question looks cut off. Is there some content missing?
No, this is interesting too. I was hoping to think about this question beyond simply `1/365`.
@GaBorguyla, thanks for the pointer. My roommate is actually one of William Stein's students...so I should probably have a chat with him :)
Hi!
Thanks! Yes, I think I am using the Wald test hehe :) Confusion on my part.

Your point seems obvious and clear, but I then I do not understand my results. I posted the test and some of the output above as an edit.

I might be way off in my choice of test and/or my interpretation, but doesn't that correspond to a Wald test rejecting equality of the coefficients?

Thanks again!
For a long list of approaches to finding outliers in multidimensional datasets, please read http://stats.stackexchange.com/questions/213/.  Your question is specialized (to 2D, small numbers of points, and has prior information about typical amounts of variation) and therefore is amenable to some special solutions not appearing in that list, but it might give you some useful ideas to get started.
& $\\Pr(X_i > 0)=$ ?
You are kind of asking, "I'm a dentist, and one of my patients has a brain tumor that I need to remove. What is the best way for me to operate?" In order to generate and interpret a predictive model, you will either need to learn some statistics yourself or consult with a statistician.  There are many reasons to not use Excel (e.g. www.coventry.ac.uk/ec/~nhunt/pottel.pdf), and SQL's strength is managing data, not analyzing it. I'd recommend looking into R and the various introductions to statistics available here: http://cran.r-project.org/other-docs.html. Knowing C++, R shouldn't be hard.
This is a good idea. However I actually have a few of these plots from different studies that I want to put on the same page for easy comparison. I think having those together is more important than splitting them up as this approach would require. I would, however, have my version of your lower charts at the same location in the layout of a different page than the upper charts, which I think has the same effect. I do like to look at the data in as many ways as possible as well to search for structure in the noise.
Can you clarify your notations ? In particular, you didn't defined Z which appears in your posterior p(f|Y,Z).
@whuber I think I understand. Since the above relation must hold for any choice of PDFs $f_A(x)$ and $f_B(x)$, choose $f_A$ and $f_B$ so that $\\sigma_A=\\sigma_B=0$. Then the relation must still hold, and we get the desired result.
I would add the regression line to them.
[This search](http://stats.stackexchange.com/search?q=likert+ordinal+interval) showed 52 posts that relate to this question. If your question isn't answered in any of them, please let us know why not.  In short, there is no easy answer here.
The distribution of times between successive runs of gc depends on the distribution of times between the process runs.  If the process runs every $k$ clock ticks, then the Exponential will be a reasonable approximation to the actual (discrete) distribution in this case, otherwise, maybe not.  (A counterexample: imagine the process runs at 1 minute past the hour, then 999 times between 55 minutes and 59 minutes past the hour.  The inter-run gaps of $gc$ would be far from exponential.)
Thanks a lot! And may I ask if I am ristricted to use the Gaussian function and it just falls too fast (I mean that if I do not consider the $\\epsilon$ but just use like $\\epsilon = 1$ that is $e^{-t^2}$) and the matrix when solving least square is nearly sigular, then how I could choose a $\\epsilon$ to make it better? I have read this one: http://stats.stackexchange.com/questions/17708/optimal-basis-for-regression-problem, but still not that clear. Sorry if the question has been too amateur.
You can search this site for c-index and likely find links to what you need.
very crude but simple approach: if computing time is not a major factor (i.e. you don't have to do this a ton of times) you could fit the new data in `lm()` and extract the model matrix from the fit, and apply it to the model coefficients from the previous fit.
Sorry, do you mean ANCOVA?
This is very helpful, thanks a lot.
I am basically deriving as many estimations as possible from the same data set by considering each time a progressively bigger part of my data set. For example, consider that I have a data set of 10 numbers and each estimation requires at least 2 numbers. Then, estimation 1 is derived by the first 2 numbers, estimation 2 by the first 4 numbers, etc. I am finding the cdf and ecdf because I want to find the probability that a particular estimation will exceed. Therefore, the CI of the ecdf will have to give me an interval where the true value of the statistic lies.
Invertibility conditions do not involve AR coefficients. In backshift notation the model is: $(1-1.5314  B + 0.55 B^{2}) (z_{t} - \\mu) = (1 + 0.1261 B - 0.032 B^{2})$. The *invertibility* conditions for this model are: $|\\theta_{2}| <1$, $\\theta_{2} - \\theta_{1} <1$ , $\\theta_{2} + \\theta_{1}<1$, and these are satisfied. The *stationarity* conditions are: $|\\phi_{2}| <1$, $\\phi_{2} - \\phi_{1} <1$ , $\\phi_{2} + \\phi_{1}<1$. These are also satisfied, however,  $\\phi_{2} + \\phi_{1} = 0.9814 \\approx 1$ does raise concern about *stationarity*, not *invertibility*.
A [question posted just a few hours ago](http://stats.stackexchange.com/questions/98971/mle-of-a-cauchy-distribution) provides a counterexample (well, it comes close enough: it concerns a distribution without a mean, but it's easily modified to give a counterexample with a mean).
What do you mean by intervention points; the series may affected by an incident and change the behavior of the time series after that point? If interventions are expected can I divide the data set into subsamples considering the intervention points and model separately? (because I need to compare the accuracy of two techniques)
Why do you want to use the multivariate CLT and the delta method here? You can get by with a simpler analysis. Is your aim just to practice the techniques you've mentioned or are you more interested in simply knowing an efficient and economical means of obtaining the stated result?
(1) The best approach to imputation depends on the proportion & pattern of missing values, the relationships between the variables, & what assumptions you're prepared to make about the reasons for missing values. (2) Any single imputation method can be used to provide input to LASSO; the difficulty's in assessing how imputation affects the results. I don't know how to combine multiple imputation with LASSO (doubtless someone does), but an informal comparison of results from different imputation runs (are the same predictors usually selected?) could still be informative.
@elisa You might want to edit your question then.  What you're asking for is currently not clear to me.
Thank you gung.
You can avoid the z-scaling issue and juxtaposition of the two subplots using `lattice` and its `wireframe()` function which let you write formula like `y ~ x1 * x2 | x3` to get a trellis display.
Have you looked at repeated measures ANOVA? Hypotheses 2 and 3 should be amenable to using contrasts. The big caveat on all of this is the confounding.
OK so you can then take a Gaussian assumption. Take the mean of your measurements to get a single value per user then do a two sample t-test (ie student's). I am sure people will comment about if you should weight by the error on your mean but this will get you started. Because you have so few measurements there could be cause for a lot of debate.....
Thanks for this answer! It definitely helps.
Sorry it has not been clear. I will have a think about this and edit the question this evening. Thanks you for pointing it out.
Surely that statement of the confidence interval interpretation is wrong: it should be 'For 95% of the samples that we could generate using rpois(), using a sample size of 100,000, and a desired mean of 10, the true event rate will fall within the calculated confidence interval.' You can say nothing about the particular sample confidence interval.
Thanks for your help. This is obviously not my strong point. The DSR is calculated from a model specifically constraining constant survival. In an AIC comparison, this was the best model, though there was uncertainty - this is for a gov't report where they just want avgs. though, so it is a best approximation for comparing yearly estimates. The SE was calculated internally via standard mark-recapture style survival estimation. SD was simply calculated as SE*sqrt(effective n). This is all standard procedure. I have to shoehorn reality into a fixed estimate of 38 day survival from the daily est.
@Thomas, if it is the same data for all three models, then I do not suggest averaging. The common practice you mention is used  to show that there is no omited variable problem. But since the same data is used to estimate all three models, then you cannot assume independence. If there is no omitted variable bias, then the average will be unbiased, but the standard error of such average cannot be calculated easily, since the coefficients will be clearly correlated.
Do you get the same answer if you use the same algorithm (ie. use "L-BFGS-B" with -Inf and Inf bounds)?
No, in general I can't assume that. The normal is what I was using and it is quite robust, but I was looking for something more flexible.
Questions that rely on understanding a particular coding language and reading through code tend to be unsuccessful because (a) they limit your audience to a (very) small fraction and (b) any errors that might exist in the code will get translated into errors in the answers.  Therefore, please restate your question in terms of standard English and/or mathematical notation, telling us what the inputs will be, what the intended output is, and how it depends on the inputs.
Thanks for the reply!  My data is pretty skewed, which is why I'm considering the Shlosser estimator.  Though I may end up not using Shlosser, nevertheless I want to make sure my Shlosser implementation is correct, which is why I'm looking for example sets.