tttjjj/elt_bench huggingface dataset:
A nice example of a need to make inferences about the means, no matter what, is afforded by some environmental risk assessments. To simplify greatly, imagine you are planning to develop land into a park. You test the soils for some compound of concern and, as is often the case, find its concentration is approximately lognormally distributed.  Nevertheless, people using the park--who might directly become exposed to these soils--will effectively "sample" the soils uniformly at random as they move around. Their exposure over time will be the arithmetic mean concentration, not its geometric mean.
Actually, I think the previous question is more a subset of this one. EAMann was asking for a datasets with some particular characteristics (although I'm not sure anybody is paying attention to those criteria); this question is wide-open.  For example, I feel comfortable voting up many of these answers, because they are, in fact, datasets, but not any of the answers on the previous question, because I haven't opened them up to see if they suit EAMann's requests.
can you give me an example with R? Thank you so much
Because you can always recode values, why is this an issue? Use any codes you like but be sure to distinguish no answer from "don't know" so that you don't lose this fundamental distinction. Is your question perhaps really about coding for some particular analysis? If so, what analysis are you contemplating?
I think you're misunderstanding two uses of the term "rank". Rho and Tau refer to calculating a monotonic relationship between two variables. The other two metrics are used for determining the effectiveness of a statistical model. In this sense, they may "rank" models according to their predictive power, and provide information about the models, however they are not measures of rank correlation.
What do you mean by "predicted negatively"? Do you mean that a high value of V1 suggested a low value of V3?
+1. Most likely, we are talking about different words spelled "statistics". Website usage statistics that can be easily pulled from logs using Python or Ruby is a different animal than say having to program Cox model or the variance of the ratio estimator of a total based on a two-stage stratified survey using Ruby. (One would have to be absolutely desperate along the lines of your answers 2, 3 and 4 to even think about Ruby as the language of choice.) One can supplement their learning statistics with simulations in whichever language is the most convenient, and then 1 is a valid answer.
I see your point - this is an odd thing to do. I'm testing output from a simulation under a very large number of possible configurations, so I expect failures at 1% significance level. I thought of combining output from every configuration to get a single AD* number. I didn't do this because it wouldn't give me any hint as to which individual configuration was buggy, and because  buggy configurations have (so far) failed this test spectacularly. I can see that by using a 0.1% significance level, I will admit some bugs. Perhaps I need to look the distribution of the individual AD* results?
+1 Thanks for sharing.  (Sorry I missed the question when it originally appeared.)  Did you consider plotting cross-variograms by look-ahead time? The most effective ones would not be the traditional smoothed directional variogram clouds; instead, use two-dimensional plots of the variogram cloud densities.  You can then construct cross-variograms of *those* to explore the temporal relationships.  Your propagation results ought to pop out of such an analysis automatically.
I do not see how adding a dimension could be of any help. $dF_{Z|Y}$ and $E[X|y,z]$ appear as difficult as $E[X|y]$, if not more so.
I understand for the power link with a GLM, we will try it. About using other error families, we tried Gamma regression but we didn't know anything about residuals analysis of this regression (and find few bibliography). In our Gamma regression model, the residuals are distributed like a Gamma distribution and not like a gaussian distribution. Is that correct ? Do you know more, or have references about residuals diagnosis on Gamma regression ?
Wow, I never heard of that, thanks :)
@NickSabbe: Hi Nick, sorry for my unclear question, I'm not good at math... :). My data come from a mixture of normal distributions (the number of components is known), I have used EM algorithm to estimate the proportion, mean and variance of each component. Suppose that the dataset is big enough, what I want to know is how to find the confident interval for each estimated proportion, mean and variance.
Independence then becomes the probability of a specific joint probability being equal to the product of two individual probabilities. It is mathematically equavalent to saying that the proportion of agreement for each pair of supervisors is the same as for any other pair.
(+1) This is a cogent reply, well worth the read.  It clearly explains how and why people can so emphatically defend different answers.  Thank you!
I think I have seen people doing some sort of CV within CV, where they first take k of G groups out, and with the remaining data take k2 of N data out from each of k groups. But in my projects I simply do a CV on the group level, that's it. So if G is not very large, I do a G-fold CV. Otherwise k fold CV where k<G. I wonder what other people do.
The abbreviations iV and dV are fairly easily decoded, but (minor point) IV usually means "instrumental variable" to many economists and some others (major point) perpetuate the awful, if still popular, terminology of dependent and independent variables. "Independent" variables usually aren't. At an elementary level, many people still get confused about which is which.
It may help to observe that $H = (X^tX)^{-1}X^t$ is considered a constant matrix in the regression setting and that $y$ is the realization of a (vector valued) random variable.  That bit about the CLT, though, is not quite correct: it is relying either on $H$ having a certain structure, which sometimes does not actually occur even with huge datasets, or else on $y$ itself being multivariate normal (but then there's no need to invoke the CLT).
Of course: you apply the logarithm function to the percentages. But nobody would come here to ask that, so you must have something else in mind. Could you please clarify what you are looking for?
"One error observation alone does not have variance." True, but it can be thought of as having been drawn from a distribution that does have a variance.
What has this to do with accuracy as asked by th OP?
Is there a particular algorithm you inquire about?
Not sure about `predictSurvProb` but you can do this with similar commands using the R `rms` package; see its `survest` function.
There is zero-inflated log-normal regression, which seems to fit your needs. See [this paper](http://smm.sagepub.com/content/20/3/175)
If you're looking for a way to test whether or not a time-series is non-linear, yes, you can do that. There are numerous tests and one popular test is called the BDS test. A good reference is [Modelling Nonlinear Economic Time Series by Timo Teräsvirta, Dag Tjøstheim, and Clive W. J. Granger](http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780199587148.001.0001/acprof-9780199587148). By the way, time does matter, in the sense that the *order* of the data is important anyway.
(+1) Also called [no free lunch theorem](http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization). I do not completely agree with the parameter sensitiveness comparison though (Single Decision Tree is one of the most sensitive approaches IMHO), but we should not discuss about that here :).
+1, but could you explain why exactly you would "pick the most parsimoneous model (least variables) that is within one SE of the best value"?
+1 for good answer, but it's essentially just the first sentence plus a link ... even though it's a standard result I think it would be better to give the values in an answer, perhaps with a link to a question which gives more details, [like this one](http://stats.stackexchange.com/questions/30588/deriving-the-conditional-distributions-of-a-multivariate-normal-distribution)
See http://xkcd.com/882/ for an illustrated example of Type I errors in a "real time scenario." Perhaps after reading that you could come up with an analogous example of Type II errors.
That's very interesting approach..
Thanks Peter. I will ask a new question to clarify the statistical significance versus substantive meaning' part.
@MichaelChernick, yes but in least squares regression (which I think the OP is talking about based on the linked slides), the mean of the predicted values equals the mean of the $Y$s, so you can just call it the sample variance of the predictions.
Thanks. I know the papers your mention. Their setting is however different from the one I describe.
If R is only tag for your question then this question is better fit on StackOverflow.com
Have you heard of [Halton sequences](http://stats.stackexchange.com/search?q=Halton+)? For "too evenly," people (beginning with Fisher's investigation of Mendel's pea experiment results) have referred to the (usual) chi-squared statistic to the *lower* tail of a chi-squared distribution.
Thanks for the reply. I wasn't aware of how "open" an area of research this was. To your point, I care about BOTH the conditional mean vector and the conditional covariance matrices. The benefit of the two metrics I mention is that they allow for comparisons of both the mean and covariance. Other statistics, like AIC/BIC, are problematic since I don't plan on performing ML jointly for everything.
I find dividing $\\alpha$ by $n$ algebraically simpler than calculating $1-(1-\\alpha)^{1/n}$, but I am lazy. Also Bonferroni does not assume indenpence hence it is "simpler" in the sense of assuming less. But you pay the price of it being more conservative.
I did not test for the significant of the model, I only replicate them. They are based on http://ir.lib.sfu.ca/retrieve/4269/etd2703.pdf the model on page 44 and the result on page 47. The author did not explain anything about the R^2, but I was required to explain about the R^2. Since the result are weird, I dont know how to interpret them. I got only 1 more week to submit my result. my friend told me maybe the model was bad, should I use other model? (i will include p-value n adj R^2 in my question)
It would be useful to know more about the specifics of your Gibbs sampler. I believe since the stick-breaking prior is intractable you would have to use an approximation..?
Nobody has an idea ? Thanks.
I am afraid I only have time for a quick comment, but if you search for "compositional data analysis" you should quicky find some useful info. In short, if I have 3 variables that sum to 1, as soon as I know the value of 2 of them, I must then also know the value of the third...thus trying to estimate 3 independent "effects" is nonsensical. Also, please see this related question http://stats.stackexchange.com/questions/68944/analysing-data-measured-as-proportional-composition
Edited to disambiguate 'above.'
Proportion of what? Is this binomial data? What's B? Why do you want a smaller N than 580? Are you sampling from the original 580 or is it a new random sample? What are you trying to find out from the sample?
Is [inter-rater reliability](https://en.wikipedia.org/wiki/Inter-rater_reliability) relevant here?
@whuber, I sacrificed a bit of readability to the scalability. Function $L$ has 5 parameters and 7 constants. If we increase $n$ to 10 we will have 12 parameters and 21 constants, but my fitting function will still require 4 parameters, since constants are separated to two vectors and 1 constant. I fail to see how using symbolic environment could be advantageous for $n=10$, since typing 33 individual variables and keeping track of them should be tedious. However my experience with symbolic environments is very limited, so I would be glad if I am proved wrong.
I wonder if there is a self fulfilling profecy here.  We get
Why are you confident that separation isn't occurring? In [the `bayesglm` paper](http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf), they argue that separation is "a common problem, even when
the sample size is large and the number of predictors is small"
I'd bet that the threshold would be a design parameter set by the client.  Basically it will come down to which is more important for a specific client Type I or Type II error (probably Type II - Falsely accepting someone who should've been denied access).  With a knowledge of what the client wants, you could then state that under a specific threshold, the probability of falsely admitting someone who should be denied access is 1 / 10,000,000 or something.
@ZevChonoles
Thanks Michael and Zev. I have done more investigation. Although I can find the theoretical explaination and replicate the curve in the blog link I gave, I found that the result does not match simulation experiment, as Michael pointed out the asymptoptic expression for sample distribution for say, a median estimator, is only valid for large sample size
To be more specific: are you interested in the magnitudes and directions of the coefficients, or making accurate predictions, or both?
In the jargon of web marketing, this probability is called the 'conversion rate'. Not my field, but the Wikipedia articles may give you some leads: http://en.wikipedia.org/wiki/Conversion_rate
Not an answer to your question but why do you think the sufficient statistics has to have 4 dimensions?
I don't understand why you are doing any of this. On an ordinal measure, if you don't like the arithmetic mean, try the median; I can see no reason to use geometric mean (which also assumes equal distances). I also don't know why you want to cut off 20%. My blog post [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) may help.
Hello Steve, I have indeed read quite some literature and also run GLMMs before that have been checked by statisticians. Would you mind telling me, at least briefly, why you think the equation is wrong??
+1 for the nice counter example.  But this would be clearer if you started with "No" rather than "Yes".
Related: http://stats.stackexchange.com/questions/9427/interpreting-gaussian-probabilities-greater-than-1 (also closed as a duplicate).  A duplicate Bayesian question is at http://stats.stackexchange.com/questions/13275/bayesian-probability-1-is-it-possible.
I believe this answer of mine may be of some use to you: http://stats.stackexchange.com/questions/80805/averaging-multiple-distributions/80826#80826 , especially regarding variances.
What do you mean by the Kolmogorov distribution here?
Your design is an _unbalanced proportional_ design: the two main effects are independent (indeed, chi^2 of your table = 0) but their interaction term is not independent of them. With such design SS type I is the same no matter A or B is specified as first or second term. However sequence `A A*B B` or `A*B A B` will change results.
@Procrastinator I am not arguing with you about the properties of estimators.  I am just address the question of how to fit a multivariate lognormal. Of these various ways ,one would probably prefer to use the one that gives the best fit.  Then we would be talking about the accuracy of the estimates of the lognormal parameters.
I agree; no significant difference.  I use this approach for constructing posterior predictive distributions; just put the predictive values of the right hand side variables in along with the past values, and NAs for the target variable "observations" corresponding to the predictive values.
@Firefeather, yes I'm aware of the need to control the familywise confidence level. Would Scheffe be more appropriate here than, say, Bonferroni?
possible duplicate of [What is the meaning of p values and t values in statistical tests?](http://stats.stackexchange.com/questions/31/what-is-the-meaning-of-p-values-and-t-values-in-statistical-tests)
You can try [logit](https://en.wikipedia.org/wiki/Logit) or [probit](https://en.wikipedia.org/wiki/Probit) models for this. Based on the regression coefficients, you can find the actual impact of time on probability of selling.
+1 I agree this sounds much better for your problem
What? "Score in the subject" is quite plainly the DV. I think your biggest problem is dealing with inference issues like alternative explanations for the differences than teaching method.
Dimitriy, I would say that these are different enough models to diverge in small samples ($n=200$ ain't a terrific sample size). `nl` is assumes that the errors are homoskedastic around the mean (given by the probit), while `probit` is smarter and knows that they aren't. I am actually more worried about the standard errors here -- I would argue that you need to use `nl, robust` standard errors here (and in pretty much any other `nl` analysis, I dare say).
I would describe the algorithm Iam looking for in the following steps. 1) Check for all user how they step during the Website depending on time (it should look like a tree). 2) Summerise if there are some main patterns, means if a number of people shows the same behavior.
@StumpyJoePete Sample mean and sample median are statistics and as such don't estimate something unless you spell out what "something" is. I think you're thinking about "good" estimators. And indeed *sample mean* might be a rather bad estimator of the *distribution mean* for asymmetric distributions. But it can be treated as such an estimator in any case and you can compute its error and other properties of estimators.
Thanks for your help it was really helpful.
[Classifiers that just barely beat random guessing can be extremely useful in some situations.](http://en.wikipedia.org/wiki/Boosting_%28machine_learning%29) Thus, having some test that quantifies confidence in a classifier being better than chance is also useful.
Can you clarify what you meant by writing $Y-X\\beta_{neg}$? You had regressed $Y^{-}$ on the $X$'s corresponding to the negative $Y$'s if I understand you correctly.
@GraceCarroll, you might look at McNemar's test if the data is paired (you have the same injury on the same pig at both time periods).
@NickStauner, Same
@gregr: Thanks! That helps a lot.
Nice, that's pretty much what I had in mind. I'll allow for some time for other answers, just in case, then I'll accept an answer. Btw, could you outline the proof for this? Is it straightforward algebra?
I'm not sure what you're asking ... can you give examples of attribute values for those unfamiliar with the game or the website you mention?
+1 This is a very good point, if you teach students something that is incorrect, you encourage them to construct a model of how statistics work that is faulty, and may well cause them to misunderstand other elements of statistics that are on the syllabus (e.g. what a confidence interval - if you encourage students to think a frequentist probability can be attached to a hypothesis, then why can't it be applied to the hypothesis that the true value lies in a particular interval).  Understanding is the true aim of education, and this requires accuracy.
I believe that statistic inference is destroyed more than implied above.  And non-normality of residuals is sometimes indicative of not having the best transformation of $Y$.
+1 I am giving an upvote because although a little off the point of the question it is related and interesting and did get 3 upvotes. Keep it in.
*How to use linear regression for missing data?* is clearly on-topic for CV. Perhaps you should edit your question to clarify that this is your real question. The only question I see, as currently written, is "Is there a Python package for data imputation?", which is an SO question, not a CV question.
Ah, perhaps I misunderstood. Were all products presented simultaneously?  Could they choose more than one, or just one?
I need to know more... did you use the `categorical` button to assign a variable to be categorical or did you make a series of dichotomous indicator by yourself and feed them into the model? If it's the earlier, pay attention after assigning the categorical variable, you can actually set your reference group to either the first or last of the coding scheme (if your desirable ref. group is at the middle then you'll need to recode). If it's the latter, then you should not feed a group of dichotomous indicators into stepwise... bad idea. use `block` button to test the whole categorical variable.
I thought you would call cut using apply for each Vi of the dataframe and then stick them together.
Obesity, as a clinical artificial category is a binary variable, but it makes no biological sense to use it as an outcome. Your real outcome is BMI or weight.
I agree with Michelle, the best way to prevent bad interviewers is through project management techniques, rather than analysis techniques. Standard techniques include requiring supervisors to listen in on X% of calls, and re-contact Y% of calls to verify key datapoints. The client should listen in on both interviewer calls and supervisor calls.
In overfitting, we also have smaller sums of squares, but it doesn't indicate a good fit, right?
Can you tell us what the textbook is?
@Glen_b: If you could name or describe these approaches, I think that would be a good answer to the question.
"When should I prefer this or that estimator" is like asking "In this recipe, when should I prefer a higher proportion of choc chips and when should I prefer more frosting?". Ultimately, it depends on your loss function / equivalently the utility of the various properties you're choosing between. How much do you like unbiasedness? How much mean square error? How much do you like the nice invariance properties that come with MLEs? etc etc
@Patrick: I think it's distinguished from the other question by its focus on a specific case.
ctd ... Similarly, tests can't even tell you that the distributions are identical, since they can differ in trivial ways. You may want to search on 'equivalence test' or 'equivalence testing', on which you should get quite a few hits here, or on google.
(+1) Helpful answer, and for your data I'm getting things to work fine, but after extensive trials I'm still finding these various commands to work inconsistently. loess, lowess, and scatter.smooth may or may not work on any given trial.  Different levels of "f" or "span" may fail to yield different degrees of smoothing (or else may fail to produce lines at all).  It's enough to make a person yearn for SPSS :-)
This is an amazingly ambitious list!  I can't imagine you covering all of this in three hours - I don't think I'd even be able to say the words to cover all of this in three hours, much less answer questions.  There are a few concepts I'd skip entirely: workspaces (I prefer not to use them - regenerating everything from script makes code more robust), attach (also problematic, especially for new users who don't understand environments), transposing (trivial to look up when you need it), and cbind (I almost never use this).
Check the list of related questions to the right -> and perhaps try a search. If you can't find another question with an answer or answers that deal with your question, please go ahead and answer it yourself. If there is another question/answer that does answer your question, it would be better to delete.
Right, I was hoping I could say a little more than that though. I suppose if you accepted everything (e.g. T = infinity), the stationary distribution is just the proposal distribution. Then I guess the most you could say is that a constant T controls the mixture between the density defined by $K(\\theta|X)$ and the proposal.
I could very easily know my RVs are nonnegative but still want a KDE.
+1, this is a nice layout of the options.
Maybe you can try the total least squares approach. But I'm really not sure this is what you're looking for, I don't know this topic. I asked [a question about it some time ago](http://stats.stackexchange.com/questions/43832/two-stage-least-squares-approach-to-deming-regression), and I have never made some progress then.