tttjjj/elt_bench huggingface dataset:
It is correct. Compare this to the median of the service time (50% of the customers), which is 70sec*ln(2) = 48.5 secs.
could you please mention how you calculate $u$ and $\\hat d$
Depending on what you need it for, in specific cases you may be able to do numerical convolution, or simulation.
@vzn many people choose models for which the optimal solution can be found. This is because the use convex loss functions, as SVMs do. Finding the true optimum here means "finding the optimal solution in your search space", so that has nothing to do with how the search space looks like.
As jb said, for general loss functions, finding the true optimum is usually impossible / infeasible.
@chl ("Integral" is the adjectival form of "integer".) With more than two items you can of course still rank them in pairs this way.  If you have intransitivity you can resolve it in some *ad hoc* ways, such as computing the percentage of time one distribution is the "winner" among a group of three or more distributions.  As you know, the subject of comparing distributions gets complex because it includes most of the paradoxes of voting theory.  But I like this approach because it forces us to confront such inherent difficulties rather than letting us stay in ignorance of them.
@Luca said: "for example for a univariate Gaussian when we actually observe that the variable takes on a certain value, shouldn't the conditional distribution be a point mass with zero variance?". No. The observed value is just one of the possible realizations of the r.v. It doesn't determine its distribution.
@StéphaneLaurent 
I think in the near future I will learn this definition
Note that your expression for $\\hat{\\beta}$ in the line starting with "The solution to..." is incorrect; you need to post-multiply by $y$.
But, $Y$ and $Z$ here are also realizations of $X$. So, the distance between $Y$ and $Z$ is a number
@PeterFlom Yes. It is an ordinary least squares regression
Apologies, should have mentioned. The number in X, can vary as there are a number of X's being tested, but they range from 500 - 10,000. The non-zero values in each X could 1 - 5 ish. I will edit the question so this info is above also
**REALLY** Useful.  Great tutorial.  Thank You!
@RStudent, thanks I updated the answer.
So: how about you edit your post (the edit link is at the lower left) to ask an actual question (you originally put question marks at the end of declarative statements). This is a question and answer site: can you improve by asking an actual question in the body of your text?
+1 for this.  Question didn't say a CI with minimal coverage, and in fact implies that this might be acceptable through its curious wording, "a confidence interval with a confidence level of *at least* 99%."
@bgbg - You're right, for p-values to be exactly correct, you need a normal distribution. With your case and its "outliers," the distribution is slightly-to-moderately skewed. I think you could make a convincing argument that if p is nowhere near your alpha and is, say, .5, then it would not fall below your alpha under a normal distribution either. I was trying to hint earlier that you really don't need to run a test, since the lack of outliers is so apparent.
Actually, despite the title, that book has virtually nothing to do with R.
One thing I do not understand is why it is $\\exp(i^2/100) - 1$ and not $\\exp(i/100) - 1$? Thank you otherwise for this beautiful answer, including the R code!
Related: [Equivalence tests for non-normal data?](http://stats.stackexchange.com/q/52897/32036)
Do you have multiple observations for each ID at each level of z1, z2,and z1:z2? See output of `xtabs(~ID+z1+z2)`.  It is unusual for this to be the case unless there is crossing of random effects, in which case your model is overspecified, even according to Barr et al.
Talking about this a bit more in the office here at ScraperWiki, it seems in some disciplines there is a different assumption. e.g. Physics uses 1 sigma (68%) only quite often. Would love good in answer to this in an answer too!
Actually, this was the first thing I tried. I tried this before posting the question. Finding a list of courses wasn't hard, but finding information about which books actually got used for those courses and what sections of those books were covered was much more difficult.
@probabilityislogic, would it be possible for you to demonstrate your idea in your answer for naive bootstrap estimate of $\\rho$ in AR(1) model $Y_t=\\rho Y_{t-1}+u_t$? I do not think that it is possible, hence the basic reason for downvote. I would be glad to get proven wrong.
Right on both counts (my apologies). It shows you understood despite my lapses. (I carelessly made a late search/replace that effectively negated most of the meanings of "interaction" in the text!)  I have made the corrections.
I think your first example refers to Bertrand's paradox. Very nice illustration of the different ways to define a probabilistic space!
Would there be any benefit or could you plot them on a 3-d scatter plot?
@PeterFlom Hierarchical clustering is incredibly expensive and slow.
hey Thanks for your suggestion. But there could be a way to check this in SPSS as well rite?
This question has a credit of 3 scores when I open my account and this score is not exhibited here. Is there someone to tell me the reason for it.
@gung That's an unnecessarily complicated way to look at it. It's a vanilla geometric distribution problem
Kudos, terrific answer, except I think you're underestimating how much work it will take to master logistic regression, in R no less.  Just getting the data into R and then making sure variables are appropriately treated as continuous or categorical can take a nontrivial amount of time to learn.  Then there are all the differences between logistic and linear regression in the way results are interpreted, the way the model/solution is evaluated, etc. This may be one of those occasions when hiring a statistical consultant is the best bet.
Can you explain more clearly what you intend by the second sentence? I am not sure I correctly understand what you mean.
Is this homework? Please add the self-study tag and read our policy on that http://meta.stackexchange.com/questions/10811/how-do-i-ask-and-answer-homework-questions/10812#10812
A note to add to gui11aume's answer: There is a "d, p, q, r" syntax for distribution related functions in R. For example, `dnorm`, `pnorm`, `qnorm`, and `rnorm` are the density, cumulative distribution function (CDF), inverse CDF, and random variate generator functions for the Normal distribution, respectively. See the [probability distribution task view](http://cran.r-project.org/web/views/Distributions.html) for a comprehensive list of available distributions.
Hint (for the edited version): For a covariance matrix $\\mathbb{A}$, the first principal component $\\mathbb{x}$ maximizes the norm of $\\mathbb{Ax}$ over all unit vectors $\\mathbb{x}$. Suppose two components of $\\mathbb{x}$ had different signs: could you then demonstrate that $\\mathbb{A x}$ cannot have maximal norm?  (This is where you're likely to need the assumption of positive-definiteness of $\\mathbb{A}$.)
As far as I can see it still discusses models where the standard deviations of the groups are not modelled, like in the first model sketch in my question. Or perhaps I'm missing something?
+1 on both SO and CV for answering without complaint :)
Have you checked missing data?
R for statistical computing
Thanks a lot. Although at the end it seems inappropriate to use it, I'll definitely keep the code you wrote.
"This estimator of the sparsity function is sensitive to the iid assumption. Alternately, Koenker and Machado (1999) considered the non-iid case. By assuming local linearity <..>, they proposed a local estimator of the density function by using the difference quotient. A Huber sandwich estimate of the covariance <..> used to construct the confidence intervals." This is from SAS QUANTREG manual section. So as I understand there are two (or more) ways to estimate this sparsity function and one is for iid residuals and another for non-iid residuals (Huber sand). Question - what is the difference?
@Charles, I think whuber is doing a basic variance of a discrete random variable. See http://www.stat.yale.edu/Courses/1997-98/101/rvmnvar.htm
+1, the graphics are a very nice feature! How did you draw them?
Thanks for your take on this.
I am measuring distance-to-event data rather than time-to-event data on seabirds and their reactions to cruise ships. I used a laser range finder to get measured distances but in some instances I had to estimate distance. Based on measured vs. estimated distances I was on average off +or- 25 meters when estimating. I would be happy presenting results in 50 meter intervals.
Is there any underlying theory about measurement? For example, are certain items meant to measure the same latent construct? Or are all Likert items "independent" of each other? Judging from your wanting to calculate mean and variance, the former is the case and there are a number of procedures you could eventually use (factor analysis with polychoric correlation or item response theory for example) Would you care to share more information so we can provide a more tailored answer?
@Macro, I can calculate the proportions fall number easily, but I am wondering why for a good calibrated Gaussian Process must be at least 68% ?!
I am not hunting for points :) Glad it could somehow help!
@Bradford: Yes it is, but that "estimate" is for that player, against the AVERAGE opponent.  So in your example, the regression model thinks that that player has an estimated mean points of -2.7648, but they played against easier than average opponents, which is the reason their actually average points is 2.975.  In other words, the regression model corrects for the fact that each player has played a different set of opponents, and some of those opponents might have been easier (or harder) than average.
You can't really assess the suitability of the regression assumptions from this output; try [`plot(reg1)`](http://stats.stackexchange.com/a/65864/805) (that's a clickable link, by the way) as a first step. As for what a p-value means, try [this](http://stats.stackexchange.com/questions/14928/meaning-of-p-values-in-regression) (the first sentence [here](http://en.wikipedia.org/wiki/P_value) has a basic definition).
+1. I would probably say *half*-decent, not even necessarily decent textbook. To that extend a very nice free e-book I came across lately was [OpenIntro Statistics](http://www.openintro.org/stat/textbook.php), it includes some comments about $p$-values on linear analysis (and a general treatment of $p$-values in Hypothesis testing obviously).
Also, the heteroskedasticity wouldn't bother me really because given the distribution of the resposne, that's almost certainly bound to happen.
What do you mean by a "rising trend" - B's scores have been falling steadily over time. I also agree with @KarlBroman - omitting data because it doesn't give you what is, as far as I can tell, the "results you want" is a flawed way to go about any analysis.
Well... you have n=39 split up across 16 variables, for one thing.
@Andrei But what is your research question? It is difficult to answer without knowing what kind of relationships you're after.
+1 for a great question. My 1st guess is that neither model is really better than the other--there is some ambiguity given your data. If you want predictions, you could try model averaging.
I also asked the question on stackoverflow [http://stackoverflow.com/questions/13986981/how-to-preform-1nn-with-single-centroid-per-class-in-sas]
Hmm. So yeah, if you need $1-P(X_1\\leq x_1,X_2\\leq x_2, ...,X_k\\leq x_k)$, you kind of do need the cdf. Zen's idea of simulation is certainly a way to do it (and the higher the number of dimensions, the better it starts to look), but if you do that, use one of the packages with built-in implementations of `rdirichlet`. If it's only 3-variate or possibly 4-variate (the last component, of course, being redundant) it may be worth trying numerical quadrature.
if $X$ is bounded away from zero the ratio has moments. The only problem is with $X$ which have non-zero probability of being close to zero. For example with body mass index, height is never  zero (or close to it), so there are no moment problems.
I think Marat is close - there could be something with the `ifelse`. I would recommend to simplify your model to smallest version that doesn't work! This could show you the way.
Technically speaking, a histogram is used to show summary information about a frequency distribution. I think you want to ask "Is a bar chart satisfactory?", to which I would answer Yes. If there are repeated measures on the categorical variables, then you can capture variability with a box plot, as mentioned in csgillespie's comment.
Cool. Do you mind elaborating just a bit? I know a little math, but I'm a total rookie when it comes to stats.
They are independent. But, I have colleagues that are looking for what to do if they are dependent. Do you have a recommendation for that? Even thought it is an ICC, the data is not hierarchical.
No offense taken at any point! A formal clarification helps me too understand what I "think I know".
@Scratch did you ever get a python implementation of the gap statistic to work on the Iris data set? I am struggling with the same issue.
It think you are referring to the [Birthday problem/paradox](http://en.wikipedia.org/wiki/Birthday_problem).
Without a knowledge of the S it looks like it is not possible.
I think I've found a way to do it, but I don't know if I'm right. I've edited the post to show it.
@xan 1 within subject variable with 4 values and 1 between subject variable with 3 conditions
Thanks Karl. Quick question: how did you determine your alpha and beta? I used the variance and the sample mean to get alpha and beta, but I think I may have confused the sample mean with the population mean so I am not sure I have gone about it the right way... see Glen_b 's comment above.
I think you clearly stated the problem yourself. Your OLS results will not make any sense.
If you can't solve the ML because the derivatives are too complicated, then EM is a good alternative. If $u$'s are observations and follow a gaussian distribution, then you need to include latent variables in that distribution. For example, what is the probability of a signal $u_{t}$ given some other variable $z_{t}$ provided you can calculate $p(z_{t}|\\theta)$. If you can do that, follow the procedure I described and it will give you an estimate of $\\theta$.
@mbq, doesn't make coffee? That's a deal-breaker right there.
I think that method is precisely a post-hoc power analysis.  I suppose one alternative method might be the use of Bayesian methods instead of Pearsonian hypothesis tests, but in my field (psychology), Pearsonian hypothesis testing is still the dominant statistical paradigm.
@ Frank Harrell Thanks a lot Frank, very informative as always, I am indebted :-)
If an IV is nominal, you should code it as a set of contrast variables (see Nick Stauner's answer). If an IV is ordinal (and you don't want to think back and treat it as nominal or interval) then, well, you might a) rank its values first, b) use optimal scaling categorical regression, c) use decision tree. See a related topic on correlating [nominal and ordinal variables](http://stats.stackexchange.com/a/73118/3277).
Your data do not follow a Gamma distribution.  At a minimum, you would need a third parameter to shift it.  To produce a peaked version, consult the [Wikipedia reference](http://en.wikipedia.org/wiki/Gamma_distribution) and note that the kurtosis ("peakedness") is inversely proportional to the `shape` parameter.  Choosing a value slightly greater than $1$ will do the trick.  But *why* do you need to fit a Gamma distribution to these data in the first place?
Looking at the code for `msExtrema`, it's a simple wrapper for `peaks` from the `splus2R` package, which you'd be better off using directly if you only want the local maxima and not the local minima. I can't see why using the default `span=3` wouldn't find all the local maxima. And 2^15=32768 shouldn't be large enough for efficiency to be a big worry.
Aaah, I think after reading your interpretation of the sentence, I just misunderstood it. The thing is that my colleague argued that once additional edges are added from y variables to the x variables (as you typically have in CRFs), you would no longer have a MRF. He pointed me to this document for an explanation, but I wasn't really satisfied as his statement contradicted what I believed to know about MRFs. Thank you a lot for this answer!
@MichaelMayer But, what you say about the validity of the model? To make predictions, to make decisions based on it?
I'm not really sure what you mean by any of that. "Calculating the next event" is pretty vague, as is "use gaussian to calculate probability...".
Glad to help! Let me know if anything's unclear. For kicks, I ran the numbers. You'll need 7044234255469980229683302646164 guesses; like I said--a lot!
there is great info here to digest. firstly, i'm very sorry that i didn't gave the exact numbers as i haven't expected that you will get into numbers so detailed and i also wanted to focus on just one topic. anyway, i have 73 training & 58 test data. these are not statistically separated, they were just taken by me in different times. Training samples are firstly taken into feature selection algorithm (wrapper fs method also has internal 5-fold cv) and then 5-fold'ed and best accuracy is taken and model is saved. and then that model is used for calculating test accuracy.
Glad it was useful to you. If you so eager to reward me, can't you mark my answer as "accepted"? (although, really, I do not participate in this forum in order to collect reputation points).
@SabreWolfy: Regression models can produce coefficients for continuous, categorical or binary variables. In the case of a binary variable, the results usually provide a coefficient for only one of the values (e.g. $Female = True$) with the other value ($Female = FALSE$) implicitly used as the baseline of comparison.
What is your problem?  I don't see how you can compare two-class solutions to three classes solutions if the objective is to classify as correctly as possible an object with a given feature vector into the correct class when three are specified.  Clearly a classifier using only information from classes 1 and 2 which may be able to correctly classify objects into class 1 and class 2 has no capability of clasifying correctly objects into class 3 as it has no decision boundary for class three.  The rule is not equipped to classify into a category it has not created a region for.
**\\$\\varepsilon\\$** ... see [here](http://www.artofproblemsolving.com/Wiki/index.php/LaTeX:Symbols#Greek_Letters) for more. Someone else has inserted $\\epsilon$ (**\\$\\epsilon\\$**) instead of $\\varepsilon$.
"*I did not want to enforce any prior believes on it*" -- then you can't assume it's smooth, or even continuous (those would be prior beliefs). In which case the ecdf is about your only recourse.
You correctly said that without a time component your residuals cannot be serially correlated. So a test for serial correlation in this case is not valid. The most common concerns in cross-section data are heteroskedasticity or spatial correlation (e.g. crime rate in city A affects crime rate in city B), but both are easily corrected for with the robust and cluster options in Stata.
@EMS: What makes prediction markets superior? The whole point of a scoring rule is that the score IS independent of the competitors (though admittedly they are not often implemented that way in practice: i.e. all the money is given to the person with the highest score)
You are confused,  $\\mathcal{N}_2(x)$ is not the probability of $x$ in ..., it is the probability density of $\\mathcal{N}_2(x)$ in ....  Your first proposed notation could be OK, given that you expålain it!
Also, thanks for the references thus far. The Hastie et al book is pretty close, actually. It's been a while since I had it off the shelf; thanks for the prompt :)
Hi all - OP here. Thanks for the discussion. 

I had finally groped my way towards the Pearson's formulation too. Alexis has a point, however, that each tissue has a different overall concordance rate. I've calculated the Jaccard distance between pairs of samples and in each tissue it is different (likely reflecting sample, rather than assay, quality).

I think the assumption of iid observations hold - each sample is a biological replicate prepared separately rather than a true paired technical replicate.
Great link.  I was unaware of this textbook.
(+1) I believe that's the overwhelmingly most common situation. Clustered SEs are *possible* in random-effect models though, and have been considered in the context of meta-analysis / meta-regression. See [Hedges, Tipton & Johnson *Research Synthesis Methods* 2010](http://dx.doi.org/10.1002/jrsm.5).
What you want is a logistic regression. Cross Validated is the better stack overflow forum for this
No problem, it's always nice to have another way to do it, even if it's just a different version of the same. +1
A curve, by definition, portrays how one quantity (the expected log odds in this case) depends on a *single* additional numerical value.  To depict fully the variation of a dependent variable on $p$ independent variables requires a $p$-dimensional surface embedded in a $p+1$-dimensional space.
This is a bit unusual as a standard deviation generally wouldn't be normal distributed since it is always positiv.
It is true that any triangular distribution with its mode at one end of the range has a linear density. However, the general linear density doesn't necessarily have $f(x)=0$ as either end. Think instead of a mixture of a mode-at-one-end triangular density with a uniform (making, in effect, a trapezoid with the two parallel sides parallel to the y-axis). There's some discussion of the issue [here](http://stats.stackexchange.com/questions/113759/how-to-test-if-some-data-points-is-drawn-from-a-distribution-with-linear-pdf)
Thanks. No code necessary, its just Excel. I'll add some info about the setup when I'm back at my computer.
Ok, it is hard to tell without the full knowledge of what constraints do you have, but it still holds that it is better to use the constraint to simplify the fitness function than to limit the search to some crazy subspace. Even if this is impossible, there is still an option to make a "smooth constraint", i.e. add something like $A\\cdot C(\\mathbf{p})^2$ to your fitness, where $C(\\mathbf{p})=0$ is your problematic constraint, $\\mathbf{p}$ are the parameters and $A$ is something that makes this term larger than the fitness deviations near the optimum.
Are you sure you reproduced the homework as asked? It seems to me that you should be asked to prove that $(Y_n)$ converges to $X$ in distribution. Which is obvious.
Thanks DL. On a mathematical level, I understand and agree.  What I'm interested in is a (justified) general statement about the correlation I can make to someone who knows very little statistics.  For example, is it correct to say "among all categories c, the average correlation of x with y is b_1", and if so, which b_1  should I use here?  The one in model 1, 2, or 3?
I recommend the GPML book for you. http://www.gaussianprocess.org/gpml/
My point, Ariel, is that you need to clarify what you mean by "chance of success." The "overall" chance of success is 100% because you are assured of success after the 20th roll. Your chance of a success within the first four rolls is $2093/5000$, which is less than 50%, and your chance of a success within the first five is $11279/20000$, which is greater than 50%, so at no stage is your cumulative success rate 50/50, either.
Question that is a hint: How many standard deviations does a 95% CI of a standard bell curve (aka a normal distribution) cover?
@tony, I see. Perhaps these two websites can give you some inspiration: using [R visreg package](http://myweb.uiowa.edu/pbreheny/publications/visreg.pdf) and [error bar plot](http://www.r-statistics.com/2010/07/visualization-of-regression-coefficients-in-r/) to visualize regression models.