tttjjj/elt_bench huggingface dataset:
"High" presumably means no more than "challenging" to one or more of the hardware, software, programmer or user. I agree with Peter's implication that there is not an agreed technical definition.
It sounds to me like your company already has a model (that you don't understand) and that you want to use that model to make predictions and see how good the predictions are.  I take it you have not just expected spending, but actual spending for some users?  I don't think you'll get a helpful answer until you can describe the model and data in more detail.
well... there is a fine line between inane platitude and profound wisdom. I like the quote for it's poetic quality. Any insight is of secondary importance to me.
Do you know anything about the group sizes? Are the given bounds confidence limits or just sample quantiles?
Great, thank you very much. Setting C to different values makes sense and I'll read the paper. (I don't really care about the probabilities, I just thought I could somehow use them as a last resort.) (I'm not an expert, only a software engineer who still remembers a little calculus and statistics barely enough to struggle through his way in academic papers)
I'm not even sure..in theory everything is more or less clear now but i have some results which i can't understand as i should interpret. A collegue performed a lda on one data set with 3 predictors X1,X2,X3 and 2 classes. All i got now are the 3 coefficients  of the discriminant function and the mean for each predictor for each group. How am i supposed to combine this elements to predict a new observation?
More information about the data would be welcome: what *type* they have (which is all you have revealed so far) often is of less consequence than what they represent, how they are measured, what their distributions could be, and how the analysis is to be interpreted.
As mentioned here (https://stat.ethz.ch/pipermail/r-help/2008-December/182575.html), maybe marginally different.
"I expected that main axe of ellipse will have the same direction with linear regression line, but it hasn't."  This is a fundamental (but surprising) property of linear regression, called "regression to the mean."  (Do a search on that phrase. :-)  Unfortunately, I am unable to find any Web page containing a suitable illustration (in the form of a bivariate scatterplot)!  Introductory stats textbooks do, however, explain this phenomenon copiously.
What is the statistical model you are trying to fit? Note that you do not state the objective functions, you state the equations.
@Viktor Why is that information not in your question? I'd suggest beta regression as a possibility.
@MartinMächler It seems like you're arguing against a claim I haven't made there. If you'd like to put up an answer which also contains a comparison of other high-breakdown robust estimators, especially ones that are roughly as simple to understand for someone at the level of the OP, I'd look forward to reading it.
@srikant thanks! i'll take a look at it.
You could say this is a somewhat philosophic point though, as even the math mechanics of OLS don't really work even theoretically to include all this data. So the argument above is technically reversed: The model is correct because we just assume it, NOT because we assume it CAN include all variability. The logical conclusion is of course the same.
It's a reasonable question but the sense of 'confidence band' needs to be made more precise.  What properties do you need it to have?  E.g., should there be $1-\\alpha$ confidence that *every* $p_i$ lies within $[l_i,u_i]$?  Or maybe that *at least one* $p_i$ lies in its interval?  Or perhaps should each $[l_i,u_i]$ be a $1-\\alpha$ CI for $p_i$?  Or maybe you would prefer the bounding box for a $1-\\alpha$ simultaneous CI for all the $p_i$?
Isn't it actually a programming-in-MATLAB problem? If so, it should rather go to SO... If you agree, I'll migrate it.
thanks! will take a look
Why are you treating temperature as a discrete random variable? Surely it can take on any value greater than 0 Kelvin. Similar arguments can be made about humidity.
@AndyW If I do that, then how do I handle the probability distributions (which will be different) for every `videoId`/`topicId`?  How do I aggregate those together?
@Glen_b: related but not exactly the same. For example, I didn't ask if all distribution free statistics are nonparametric statistics there.
@whuber I re-wrote the question a bit, hopefully clarifying what i'm trying to do.
No real strong reason.  I was originally going to say 5 but with 28 points 7 divides it evenly.  I think the idea is to take enough to avoid the 0 in numerator or denominator problem and not more.Just scanning through your data it seemed that 7 might be about right for that (but could be 6 or maybe a little more than 7).
If you don't fancy a model specifically designed for regressing proportions on things, then you should at least transform them.  In the absence of zeros, *logits* will behave much better in a regular linear regression setup, whilst not being much harder to interpret.
Can you explain the part about probability coverage?
Also, what sort of software are you using for your analysis? If you're using **R** you can check out this "*vignette*": http://cran.r-project.org/web/packages/gmm/vignettes/gmm_with_R.pdf
I think the question was badly formulated (I've edited it and included the link to the video which is a short one from Khan Academy) since I meant sampling distribution of means.
Sample estimates for skewness and kurtosis have upper bounds. In some cases, this can lead to samples denying their own parentage. A tutorial review is freely available at http://www.stata-journal.com/sjpdf.html?articlenum=st0204
Is there some reason you do not suggest multinomial logistic regression for the second problem?  BTW, this is really two separate questions about two different models for two different responses, so there's no question about preferring one over the other: the O.P. seeks advice about each separately.
Thanks for the link! If i read Hastie et al. right they show a correspondence between the non-parametric boostrap and the Bayesian bootstrap and claims that the former approximates the latter. They don't write much about why the bootstrap (bayesian or not) results in sensible inferences in the first place. What I was hoping for was something like: "Under [some general circumstances] the bootstrap approximates the true posterior distribution of the parameter/statistics with an error that is [something] and that depends on [this and that]".
What different conditions have you seen?
A quick comment on your third paragraph - Monte Carlo sampling of parameters, not the data, would be Bayesian. Bayesians are very aggressive with applications of probability; they're able to make probabilistic statements about quantities that classical statisticians would consider fixed, unknown truths.
i dont think there is an example of efa in lavaan package but look at efa in a cfa framework example at http://www.amarkos.gr/research/lavaan/ for how to implement. Wish i had noticed this before.
We can answer your implicit question without any simulation: although the Pearson correlation coefficient is a biased estimator of the correlation, [the bias is usually small](http://www.uv.es/revispsi/articulos1.03/9.ZUMBO.pdf), so the center of the sampling distribution you propose will be close to the correlation coefficient of the data from which you are sampling.
(+1) great advice, especially regarding the ML-class, actual knowledge/work > certificate and the blog.
The prediction interval should be of the form you stated except for the degrees of freedom. A little more though should be given to it.
+1 Good finds.  The Goovaerts chapter briefly reviews almost all the software listed in the AI-Geostats site.
Any random variable, by definition, has a distribution, which might be discrete or continuous based on the set of values it admits and independent copies mean independent draws from the distribution of $X$, i.e. $X_1,X_2 \\sim F_X$. Stable distributions are called so because a sum such as the one you described, leaves the shape of the distribution unchanged. Also, the example with Normal is correct. 
i.e. $ X \\sim N(\\mu,\\sigma^2) \\Rightarrow aX_1+bX_2 \\sim N((a+b)\\mu, (a^2+b^2)\\sigma^2)$ and $ cX+d \\sim N(c\\mu+d, c^2)$, so, the definition holds for $c^2 = a^2+b^2$ and $d = (a+b-c)\\mu$.
THANKS. I corrected it.
@SashikanthDareddy, What you will be getting from what you describe is definitely not a prediction interval.  A prediction interval is determined by more than just being wider.  Yes the individual trees form a bootstrap, but the bootstrap estimates parameters, not individual values.  The prediction interval is very dependent on the distribution of the individual points.  The fact that your method gives an interval for the proportions with a categorical outcome instead of the categories shows this.  See my example in the added answer.
My suspicion is that the 10 is recorded successes, not failures, and is really a 0.  Probably not checkable, but it is the obvious explanation.
...For the dissertation, I generated a time series of the number of people moving at any given moment (above a minimum threshold). I then made simple probability calculations: e.g. if no one is currently moving, what is the likelihood that if someone starts moving that others will move also? In general I found that about 90% of the time, others will move too. For publication, I'd like to do more sophisticated stats and these references are really helpful.
Please take a look at the following from Tsay, it is not covering all cases, but was quite helpful for me: http://faculty.chicagobooth.edu/ruey.tsay/teaching/uts/lec8-08.pdf
1) Can you include the full model and the data to make this reproducible?  2) If `isref` is continuous in SAS why do you make it a factor when using `lmer`?
A method that does not use the numerical nature of the variable should be avoided at any rate.
Good point about forgetting the initial state, I completely glossed over this in my answer.
I think the simplest way to achieve this would be to test each pair. Your composite null is kind of tricky.
Well, no, that's not right at all. (i) Note that a p-value is a conditional probability (see the first sentence [here](http://en.wikipedia.org/wiki/P-value)), and (ii) You should take care [not to over-interpret a p-value](http://stats.stackexchange.com/questions/94974/is-the-exact-value-of-a-p-value-meaningless). In particular, since you just tried to compare two p-values, check [this out](http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf)
The inputs are assumed to be iid Gaussian, implying there is no direction of maximal variance.  But perhaps I have misinterpreted this assumption?
The [caret package](http://caret.r-forge.r-project.org/) comes with a series of vignettes (and a JSS paper) that cover most of your questions. Could you indicate what precisely you mean by "derive some inference on the effect of particular variables?"
There is no unique or best answer to this question.  Similar questions have been debated at least since the 1200's and were simply insoluble until a definite probabilistic meaning was given to the scores (in a famous series of letters between Fermat and Pascal in the mid-17th century).  To make progress and not be completely arbitrary, you *must* stipulate something additional about how those scores arise (as one of the [math replies](http://math.stackexchange.com/a/97738/1489) indicates).  In particular, there's no basis to recommend softmax or anything else.
Peter I think that paper used an improper scoring rule.  Different results may be obtained when using proper scoring rules.  Also, the paper did not address "volatility" of the analysis.  With small total sample sizes considered there, repeating the process using a different random split will result in much different models and much different accuracy when compared to the first split.  I see that is very undesirable.
In the question you mean "sensitivity" not "sensibility".
Can you please elaborate? This question makes no sense in the current form.
Please see [this section of the help](http://stats.stackexchange.com/help/on-topic) which says "*Please note, however, that cross-posting is not encouraged on SE sites. Choose one best location to post your question. Later, if it proves better suited on another site, it can be migrated.*"
Please be more precise in what you wnat to do and what you have done.
Put it into words and see if that helps. E.g. P(C|U) means "probability that a shirt is bought with cash, given that it costs more than $200". Any simple P(-|-) formula can be translated into English this way.
Sorry I forgot to mention that I am using regularized logistic regression which should be like adding a prior.
@Max I am very familiar with the backshift operator as used by Box and Jenkins in their book.  But standard notation would be BX$_t$=X$_t$$_-$$_1$ and not with b coming after the variable it is operated on.  So I thought it might be something else like a constant or a time independent variable. It is just not standard notation. Also if you are going to apply the operator to the Xs why not apply it to the Ys as well?
@Macro Don't you think your remark sounds a little harsh given that you are addressing a newcomer?
More to the point: why is there any reason to suppose such data would give reliable information about course usage of software?  There are so many possible confounders that the results could at best be suggestive, at worst be woefully wrong.  This sounds more like a *management* problem to me.  For instance, one could collect these data, fire a summary back to the professors, and threaten to charge them license fees unless they confirm non-usage.  *That* might get some responses!
If you have time, then you almost certainly do have dependent data; did you measure he same people more than once? As I said, colliner results are unstable, don't trust them.
@AVB The example suggests using ANOVA to test the hypothesis that all means are equal, which takes into account the different variances. The grouping, however, are arbitrary within the test itself.
+1 "**guidance**" : the process of controlling the flight of something (such as a [pigeon]) *(from m-w.com)*
... line of discussion as there was no indication of optimization in the question (which is where this becomes really important).
You might find the material and references here useful: http://lingpipe-blog.com/2011/06/23/steyvers-lee-miller-hemmer-2009-wisdom-of-crowds-recollection-order-information/
Not an answer, but you could probably calculate the kalman filter steps analytically for this simple model, as it would only involve small matrices.  And which "Kalman Filter" value are you comparing: the smoothed value, 1-step ahead prediction,..?
Generally, you are _not_ obliged to add in one scale (=construct, = composite score) items that correlate. Correlations says that the union is statistically justified because the items, to an extent, measure the same thing. But if you want the construct to be "apple and oranges", and sometimes in psychometrics they admit it, then just add items whatever you want, and without weightings.
Do you know the numbers of females and males that produced these ratios?
Red-green colour blindness is common, meaning that graphs requiring separation of red and green are better avoided.
If it weren't CW it would have to be closed as subjective and argumentative!
Yes great. Thanks for pointing out that $X_i$ has two states. That's definitely right. Very clear explanation thanks for your help.
Good comment, I at first missed the fact that the time series must be stationary.
There are multi-class SVMs as well ( http://svmlight.joachims.org/svm_multiclass.html ). For that matter, from the name of the file, it seems they are using one vs all multiclass SVMs - http://stats.stackexchange.com/questions/21465/best-way-to-perform-multiclass-svm
You should ask another question on the new issue instead of editing the existing one.
Instead of using fancy methods, have you tried just computing the _numerical_ values of the success rates of the four cases, that is, $9/(9+4), 8/(8+4), 4/(4+3), 12/(12+7)$ to see if anything is glaringly obvious? Hint: the first decimal digit might be looked at....
I am confused by this question.  It starts by telling us you are sampling from a definite distribution with a variance of 100 (or maybe 100^2?), but goes on to ask about how to "reduce its variance."  What is the proposed mechanism for the variance reduction and how is that related to this initial information about the variance?
(+1) **Step 1**: Plot your data.
Hi, thanks for your answer. It also seems to me that they are reporting something different than SD of the means. However, in defense of Aigner et al., they do not apply Fischers’exact here but Student’s t test.
For some reason (perhaps the context given by your comments) it doesn't confuse me any more, so I now feel I'm being pedantic. Nevertheless the null hypothesis, whether for an F-test or for the equivalent t-test (you're right), is just that the true value of coefficient for that predictor is zero - e.g. $H_0: \\beta_3=0$. Even if you're using the test to decide whether or not to include a parameter in a final model, that *hypothesis* doesn't change.
(+1) I like that book, too, and it is a good starting point. A nice addition would be a picture or other demonstration of what slowly varying means. The OP seems to be aiming for some intuition. Then regularly variation can just be seen as a generalization.
Fun how you share the same question and the same surname, and the same country lol :)

Not yet, I had an statistician tell me that they should be cross-sectional but not enough details for me to reason over it. So the question still remains.
Is there any reason you suspect the absolute value of the obvious estimator for $E[X-Y]$ would be biased?
An **infinite** $p$-value seems to hint at some problem with the numerics.
Thanks -- I browsed through the package documentation but it did not seem to support random effects.
All you seem to be asking is what happens when you use a clock set for a different time zone to record the times. Isn't it perfectly obvious that the process doesn't change? Am I missing something?
But if you do not include other determinants for quality of life, for which there are a lot, then your statistical procedure is not even asymptotically correct? Turning this around - without all determinants known you can not simulate (ie. forecast) new datapoints.
@cardinal Yes, this is my first Applied Stochastic Processes class so any assumptions I'm not being explicit stem from my inexperience with this subject. W/r/t your specific question, $|\\mathcal{S}| = |\\Bbb N|$ so $\\mathcal{S}$ is indeed countable.
Equivalently the model is still ${\\cal N}(\\mu,\\sigma^2)$ but with a prior for $\\sigma^2$ having support contained in  $[\\sigma_0^2,\\infty[$. If you take an inverse Gamma prior truncated to $[\\sigma_0^2,\\infty[$ then you get a truncated inverse Gamma posterior with the same conjugacy relation than the case when there's no truncation.
Post your warning text in full.  Can you display a snapshot of the data set (Probably just a few observations).  Post the output of statsmodel that shows the $R^2R
You may be interested in reading about the [negative binomial distribution](http://en.wikipedia.org/wiki/Negative_binomial_distribution).
The important thing here is to understand that $\\mathrm{E}[X]$ is a real number, while $\\mathrm{E}[X\\mid Y]$ is a random variable. Take alook at this question: http://stats.stackexchange.com/questions/38700/conditional-expectation-of-multivariate-distributions
Thanks for geting back to me. What would be considered large? The break down of data is as follows 

23.19%
13.45%
4.60%
8.58%
10.92%
13.58%
11.48%
4.44%
9.76%
I get the following error with your code:

>> [L, p] = cholincsp(mydata, 10^-10);
Index exceeds matrix dimensions.

Error in cholincsp (line 78)
   L(diagonal(i:m)) = X(diagonal(p(i:m))) - sum(L(i:m,1:i-1).^2,2)';
@PeterFlom. The # of negatives may be as many as 10,50 or a 100. The # of positives is 1, or at most 2 or 3.
Could you explain this answer in more detail or provide a reference that explains it?
@AndyW, I wasn't saying there was no point, only that the point wasn't apparent to me (which is why I asked what motivated this inquiry). You've made some useful comments, but I'm still not quite sure what the utility would be. 	
I've seen people try to use power analysis after the fact to rationalize why their results weren't statistically significant (I believe this is called post-hoc power analysis) but I've also read a paper (I can't place it right now) whose message what basically that post-hoc power analysis has some very bad properties and essentially should never be done.
@Stephan Alternatively, we could take the appearance of this question on CV as a request for information about checking for multicollinearity among regressors (regardless of the computing platform), in which case your comment would be the start of good answer :-).
Or see [here](http://stats.stackexchange.com/questions/5235/what-is-the-expected-correlation-between-residual-and-the-dependent-variable?).
Thank you. But actually this is not my question. My question is how to sample data by "rfcv"?
@Max The idea is that removing a few values at both ends of the data is unlikely to affect your estimate of where most of the values lie, whether or not what you throw out is an outlier.  I'm just pushing that point a little, knowing how bad Web data can be, and suggesting an even simpler approach of using the middle value to estimate your typical price: that's the same as "tossing out" all but a single value!  You could also look into a closely related statistic, the [Winsorized mean](http://en.wikipedia.org/wiki/Winsorized_mean)
You might have more success if you narrow down what it is that you are asking. Is the MLE procedure challenging, or do you have questions about how to show that the MLE of $\\theta$ is unbiased and consistent? Are you uncertain about confidence intervals? The [FAQ](http://stats.stackexchange.com/faq) provides guidance for asking good questions. As it stands, it looks like you're asking us to do your homework.
Thank you @Druss2k, unfortunately I don't read German. Maybe there's a translation available?
Thank you, I think the core of my problem lies in the range of the data. Two of the countries have values in Var1 that are much larger than 3rd place and due to the vast difference I think they are being clustered together despite (what I think is the right answer) belonging in separate groups based on Var2
You might want to check Brownian correlation http://en.wikipedia.org/wiki/Distance_correlation. It taps many but not all nonlinear "associations/shapes".
I might be missing something, but colorred is the reference group. This is fundamental concept in categorical variables and should be covered by wikipedia (dummy variables) or http://www.ats.ucla.edu/stat/r/library/contrast_coding.htm
I do not agree that in general one can perform the inverse transformation on the CI of of a transformed estimate in order to obtain the CI on the untransformed estimate. For the reasons I have already written. That said, there *are* transformations of the variance of a function of $x$ that can provide more or less accurate estimates of the variance of $x$ (from there to standard errors to CIs). If you read Altman and Bland's third paragraph, they are pointing this out.
The path of least resistance: take out the ANOVA table. Report F-ratios for the tests that *you* care about, not the ones that fill out an ANOVA table.
Sure: Kingman, J. F. C. (1993). Poisson Processes. Oxford University Press, USA.
@ Russell: "median(x)-(IQR(x)*1.5)" No, you want Q1-1.5*IQR, not median.
What rules do you know that might enable you to compute the expectation and variance of a sum of random variables or a constant multiple of a random variable?  (You can look up the expectation and variance of a Beta distribution: Wikipedia lists them, for example.)
You're asking several different questions here.