tttjjj/elt_bench huggingface dataset:
The actual cdf exists everywhere, but for $(-\\infty,0)$ the cdf will be zero; the functional form you gave only applies for $x\\geq 0$ (this is amenable to simple reasoning; what's $P(D^+<0)$?
Can you say more about your situation? Are your variables your groups? Do you have different numbers of measurements for each study unit? Are length & width 2 different response variables? You may find it helpful to read [this blog post](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) in formulating your question.
Why not simply the fraction of devices that crash + average/median number of crashes for those that crash?
Aghila, thank you for feedback, I agree; unfortunately in our case time constraint imposed by financing istitution doesn't allow to do both the studies.
see the `lines` function
It's difficult to know what you are seeking here. If $X$ is insignificant (at whatever level), it is insignificant. $R^2$ quantifies a different question. Change in $R^2$ has nothing to do with a significance calculation; the model doesn't remember what you did previously.
Could you explain where the 82/15/3 values come from?  You appear to be using numbers that you haven't yet shared with us, such as "the monthly CDDs" (whatever those may be) and the "total predictions for the year" (whatever those may be).
1. Why are you squaring your putative cosine distance; 2. why the arccos?
Do the columns have a natural ordering to them? Or do you want to find a tridiagonal approximation under some ("optimal") permutation of the columns? I'm assuming that when you say $A = \\mathrm{Cov}(X)$ you're speaking of the covariance structure of the features. Can you confirm this?
I have encountered one last problem with regarding to the Metropolis-Hastings algorithm. I know that ergodicity is needed in the algorithm to imply convergence to a unique stationary distribution. But how is ergodicity proved in the M-H algorithm? Thank you for your help @David Marx
Can you post the error which you get? How does MAP choke?
Do you mean $Q = \\frac {X}{Y} {\\boldsymbol 1}_{\\{X\\leq Y\\}}$ ? (defining a random variable conditionnally on something makes no sense, you could only define its distribution in this way)
Thank you for your answer. Could you please explain yourself a little more - what do you mean by "maps x to all powers of x"?
I'm perplexed as to why you selected this answer to a question that asks "what can cause".  I answered what can cause.  What can one do instead is a different question.

Collapsing the data is traditional in psychology (and many other fields) because it yields data you can run an ANOVA on.  The approach you decided on is the mixed model approach that Horst and I mentioned above (lmer is the R version of PROC MIXED).  I suspect your next question might be "where are the p-values". Rather than get into why that is a bad idea, here is a link: http://cran.r-project.org/web/packages/afex/index.html
Restriction (3) cannot possibly be binding for your beta distributions--it's a strict inequality.  The extremum *must* be reached when the derivative is identically at its limiting value of constantly zero: that's a uniform distribution (or scaled Beta$(1,1)$, if you like).  I leave it to you to decide whether this gives a maximum or minimum solution :-).
Optimal in the sense that the weight vector minimizes some objective function. For example consider the "minimize variance" objective function: Arg Min: weight vector(transposed) * covariance matrix * weight vector, where weight is the choice of the optimizer, and covariance matrix is estimated by repeated draws from the predictive distribution
Make sure you also apply the same shift. PCA usually involves centering the data first.
Thanks, logistic regression looks interesting its definitely a possibility.
@chl thanks for edits. Quite useful.
That's written using LaTeX, a common way of writing mathematics in documents. When editing, click on the gold-diamond with the question mark at the top right and you can go through to some (very) brief help on formatting, including a tiny bit on LaTeX. One easy way to see how to do something is to find a post that does it, right click, `Show Math As -> TeX Commands`, select, copy, paste between \\$ signs, edit. There are many documents online (web pages and pdfs) with lots of details on mathematics in LaTeX, much of which works here. [Here's one](http://en.wikibooks.org/wiki/LaTeX/Mathematics)
@Glen_b Yes. I am learning to distinguish graphics that need to be clear and precise from those that benefit from some "friendly" simplification. This, evidently, is one of the latter.
There's probably a better way of doing this, someone who does it all the time will point it out I hope.
Bingo! Thanks a lot. This is something I was looking for. Just one question, Does Dixon test can detect more than 1 outlier?
@Macro Sorry.  Peter Ellis points to "so-called p-values"  because p-values are computed for each biomarker.  The ordering of the p-values keeping biomarkers for further consideration from those with the lowest p-values is a common first stage method to filter out the biomarkers that have no effect.  But the p-values ar eonly used as a screening device and not to be taken literally about how significant the marker id.
I agree with the statement that "Randomness is not the same as independence". By your reference to pearson's coefficient, i'm assuming you are talking about "independence of variables" right? The question is with respect to "independence of observations".
@shabbychef: foregoing referees, yes, but Wasserman isn't really arguing for getting rid of peer-review generally, just opening it up. Also, he notes that such a solution isn't necessarily good for medicine, I would argue that it's also potentially a bad idea for controversial sciences like climate science (where there's a political incentive to write misleading papers).
Hi. I see you have posted a lot of questions about improving some forecasts with Kalman filter, without getting answers. To me it is a bit unclear what you are trying to do. To my knowledge, Kalman filter is used to generate the state estimates (and forecasts) from measurements based on some state-space model. Apparently you are trying to apply the Kalman filter somehow on top of ARIMA results? If this is correct, could you add some references / more explanation about your approach. This might help in getting answers.
Some points of clarification: K-S considers differences between sample distributions, period.  It has nothing (explicit) to do with location, variance, or skewness.  It is truly non-parametric, too.  Its implicit definition of "similarity" is "equal a.e.".  Note, too, that "rank" is not a property of a distribution: the ranks (order statistics) are sample statistics.  Finally, it's not usual to consider the M-W test to be a form of ANOVA, although it can be cast into that framework.  It's simpler to consider it a nonparametric form of the Student t test.
I don't have a problem with going the density estimation route. My only concern is speed. A place to start might be this journal article: http://alumni.cs.ucr.edu/~wli/publications/deosd.pdf. I might be inclined to do a small scale speed test here to evaluate the performance differences.
Thanks Playitagain! This one is very useful information. Even ur name is interesting  :)
What is `UID` and its significance in your analysis?
This is backwards, as the measure concerned would be 1 whenever the residual and total variation were identical, which would correspond to a useless regression. But yes, there is a good reason why r square is so named and this is covered in many places. See e.g. http://stats.stackexchange.com/questions/65779/why-is-coefficient-of-determination-used-to-assess-fit-of-a-least-squares-line or
replace `str` with `dput()` and you'll get much better answers.
Dilip's end result $f(z) = \\frac{1}{\\pi}\\sqrt{2-z^2}$ is known as the semi-circle distribution with domain of support on $(-\\sqrt2, \\sqrt2)$. Neat!
+1 for your great response. I will probably accept the answer at the end. However, the reason why I asked this is because we have a small sample size problem. N<10, if I don't try to increase it. In this case, asymptotic don't apply. I believe n(X^TX)^-1 will decrease with increasing n quite dramatically in this range of n. What would your response be in light of this?
The example actual/fit/forecast upperlower limit plot given in the 2nd answer looks similar to the discrete SUM time series I am looking for. How can I obtain it for a layman like myself?
Smoothing _is_ a big part of functional data analysis, and it can be converted into a vector mean estimation problem by projection onto an appropriate basis (e.g. Fourier or wavelet), but there are other problems in functional data analysis depending on the functional structure that don’t translate as easily.  Take for example functional regression where you are interested in predicting a functional response from covariates.
It seems that the table already gives the answer: because it says that $2.24\\%$ of all filers with incomes of 50K or more are audited, that's the chance that a randomly chosen filer in that income group will be audited. Because that is far from the answer provided you, I would guess the question differs in some important--but perhaps subtle--way from how you have phrased it here.  Could you tell us *exactly* what the question asks?
Thanks for your answer Sergio. It really helped me. Unfortunately I can only mark one answer as my accepted answer.
For clarity, did  the same two raters see all 18 participants for all four time points?  Did all 18 participants experience the same four-conditions? Finally, what is your research question: rater difference, condition difference, or time difference?
I agree with your "MORE INFO" paragraph, if they are the same 6 hospital before and after, I'd suggest looking into [Wilcoxon Signed-Rank test](http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test), which is the non-parametric version of paired sample t-test.
I never said anything of a uniform distribution
Do you have a three measurements on one individual or three measurements on three individuals?
Dear @Glen_b: From a mathematical point of view, the Taylor series doesn't work, regardless of the coefficient of variation. Since this is a question about a mathematical property, I think it's important to point this out. Regarding "practicalities", I think that, in certain circumstances, one could easily argue the opposite---the outlined approach could be highly dangerous precisely because it *appears* like it might work the vast majority of the time. But, when it doesn't, the error is catastrophic. (The relevance of such an argument would depend on the application, of course.)
That's true--but unfortunately logistic regression is a *generalized* linear model and it is *not* linear in the parameters.
@Felix, alpha is still considered among measures of reliability. Internal consistency is a term better to avoid, because some take it as inter-item consistency (a facet of reliability) and some as item-total consistency (a facet of validity). Melissa, you are free to choose to use your items as one scale or not. "Apples" and "Oranges" are quite dissimilar, but both represent construct "fruits".
According to  https://en.wikipedia.org/wiki/Earth_mover%27s_distance   the EMD is the same as the Mallows or Wasserstein distance, so you can try googlin that.
Fixed that, wasn't it.
Following on from gung's point - the lack of upvotes and accepts suggests to potential answerers of your questions that you don't think any past answers have helped you - not even once in more than 20 questions. This would not encourage anyone to think that answering this question would be likely to help you now. If no past answer has been of any value to you, you should consider whether asking more questions is futile. On the other hand if some previous answers *have* helped, you should consider changing the impression your lack of upvotes and accepts gives.
It seems to me there are two questions: producing *neat* graphics in an *automatic manner*. I would add that those graphics should be meaningful and not lead to subjective or perceptual biases (I can see four [pie charts](http://bit.ly/sz5l6o) out of the six graphics that come up with the given URL). As you tagged this question with `interactive-visualization`, what do you have in mind? Something like [protovis](http://mbostock.github.com/protovis/) or its successor [d3.js](http://mbostock.github.com/d3/)? Easy to set up with csv or JSON data, but what about your data?
[This page](http://www.car-accidents.com/pages/stats.html) states that there are ~40000 deaths/year due to car accidents in the US, which has a population of ~300000000.
+1 for a good, thorough answer.
$\\phi_i$'s and $\\phi_{ii}$'s are related.
Actually, you'll have a single model that includes the ridership, the student population, and an indicator for the fare change (easy to do in X13 with LS2011.FEB) and you see if the model with the indicator fits better than one without. This isn't specific to X13, you'd do the same thing in R. I've edited my answer.
(+1) I think it is hard to find this proof in standard categorical data analysis texts like Agresti, A. (2002). Categorical Data Analysis. John-Wiley.
Something to think about: What if $X_1 = X_2$, or, even $\\mathcal C(X_1) = \\mathcal C(X_2)$ where $\\mathcal C(\\cdot)$ denotes the column space?
As the one who plotted that (albeit not in `r`) I may say: if there is even no ready program, it can be written. The plot is actually a scatterplot (with spikes to group centroids) with 5 groups: 3 classes + 2 discriminant scores clouds which are straight chains when drawn in the space defined by original variables V1, V2. So, the only trick necessary to plot them is to compute coordinates of the discr. scores (which you get from LDA) onto V1, V2. The coordinate is the `score * corresponding eigenvector element`; eigenvectors of the LDA.
You can try a permutation test.
For an introductory stats class I think this example works well pedagogically - it is something students are likely to have real-life experience of, can reason about intuitively, and can confirm against widely available data sets.
I'll nitpick your answer I basically agree with.  @svadali said: "If you know that the model is incorrect then obviously you would not use it to compute the confidence interval."  But to paraphrase George Box: all models are incorrect, sometimes computing confidence intervals is useful anyway.
Not sure I did justice to the last sentence; please [edit](http://stats.stackexchange.com/posts/108395/edit) if the meaning can be clarified. Welcome to Cross Validated BTW!
PCA is not a classification method?!
As far as I know a DecisionStump is different from 1Rule. A Decision Stump is always a binary 1-level tree (for both nominal and numeric attributes). 1Rule can have more than 2 children (for both nominal and numeric) and for numeric attributes have a more complex test than binary split by a value. Also, in WEKA there are 2 different implementations: DecisionStump and OneR.
See http://en.wikipedia.org/wiki/Binomial_test . Talking as a moderator now: so can you please make a new, good question? You'll get a better answer and the wiki spirit of this site will be held.
You may be interested in this [paper](http://www.crim.upenn.edu/faculty/papers/berk/regression.pdf).
According to [this page](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f_oneway.html), `scipy.stats.f_oneway` only returns an $F$ statistic and its $p$ value. **Neither** of those things is an effect size (as the term is normally conceived at least), though $F$ is certainly related to effect size. What makes you call it an effect size?
By looking at the histogram. If you get a near symmetric bell curve, then the mean is a good representative measure. If you have a skewed or a zero-inflated distribution, then the mean is less relevant.
@Philipp, latest versions of SPSS has multiple imputation procedure, it is very universal, although a bit messy to use, perhaps. Also, to imput quantitative data, SPSS has EM and regression imputations in Missing Value Analysis procedure. For a hot-deck imputation macros, please visit my web-page.
@user77404, error distribution may or may not depend on data. many nice properties of least squares, such as small sample properties like BLUE, require that error distribution does not depend on data - this s called _exogeneity_, read about it. unfortunately, in time series this is too strong of a requirement. but something tells me that it's not what you meant in your question. so you may want to clarify it.
Thanks for a great explanation. One minor thing I'm not fully clear on is your point re: the location shift and whether 0 is inside the interval. I remember reading that an interval which contains 0 is not statistically significant. In this one-tailed case we've only got one actual limit, can we use an unbounded interval? So (-72 < 0 < +Inf) thus it's not significant (as 0 is between the limits). If, for example, the range was (22 < +Inf), this interval is fully positive and 0 isn't within it. Is that correct?
That was it. Thank you!
@javlacalle you are exactly right on my frustration, there does not seem to be a good way that I can see to weight the more recent results to cause the projection to continue to shift upwards.. the seasonal pattern  seems to overwhelm the year to year drift upwards..
What is the purpose of the regression?  Prediction, exploration, model fitting, something else?
unless you explain what you mean by 'via a linear operation' your question is impossible to answer. If you 'rescale' the entries your correlation matrix from [-1,1] to [0,1] the results ain't a correlation matrix no more. I'd advise you to edit the last 3 § of your question to make this clear by calling your 'rescaled' matrix something else. Also, 'rescaled' may not be the corrrect term here unless you somehow go from [-1,1] to [0,1] only by multiplying those entries.
I think there's some value in the question, from two points of view: (a) illustrating that it's not at all hard to construct samples with mean=variance by hand and (b) illustrating that even though for a Poisson mean and and variance are the same on average, that having that be true in a sample isn't an indication of its Poisson-ness (by giving an example which is pretty clearly not Poisson). There's a misconception underlying the question which is worth dealing with, I think. So - while I understand the impulse to close, I want to keep the question.
(+1) This issue recently came up in discussions of two questions related to finding the moments of $X^{-1}$. It pays to take additional care with such matters. :-)
@PhD: done. It's more "quick and dirty" than it is "at a high level" though, I think. I'm new to equivalence testing myself, and as you'll see, I'm not exactly sold on it.
Much better question but always test `p < 0.05` not `<=`.
2/2 I don't think your statement about the CLT is wrong. I am just saying that I think the CLT is an inevitable consequence of the standard space we're working in. In other words, if you can't say *anything* about the distribution of the X's, then I don't see how you can make any inference about a function of their distribution. You have to have *some* knowledge about the behavior of the X's to be able to say something about a function. The only scenario where this wouldn't be true is if you're working in some weird space. E.g it's ok for a sequence have 2 different limits.
That makes sense now, thanks! I don't know why I was so sure that he was using Bayes' theorem...
@AdamRyczkowski: I linked one paper, maybe that gives you a start.
The Weibull is defined for non-negative argument, so there are no guarantees that finite samples will be returned only in the interval you specify. In any case what particular samples look like is, literally, "the luck of the draw" within the definition used for a distribution. You have to choose what to sacrifice. If you believe your variable is bounded as you state, the Weibull is a poor model. Perhaps you should explain why the [1, 52] interval should be honoured. Wanting a model that mimics idiosyncratic details of present data is, for example, usually an unsound idea.
Actually my experiment got one predictor at 4 levels and I did a repeated measurement.  Therefore, I use random intercept and slope for each subject. Levene's test on the predictor variable is not significant, however, for the day is significant.  Do I need to have homogeneity of variance for the day as well? I thought I had modeled the variance by AR(1)/Unstructure covariance matrix already.  Thank you so much for your generous help.
Andrew Gelman provides some insight on this problem on his [blog](http://andrewgelman.com/2011/10/25/how-do-you-interpret-standard-errors-from-a-regression-fit-to-the-entire-population/).  Would still be interested in other thoughts.
(+1) Thank you @Dilip, I am unpacking and digesting what you wrote, though I believe I understand it. I upvoted this post and the one you linked to on math.se. However, i) I now understand how the selection of $z = \\sigma \\sqrt{2 log N}$, will make the above limit go to 1. But why the 2 inside or the sqrt? Anything particularly unique about them besides their nice/required behavior in the limit? ii) Is there a particular name for the tight bound that you derived/referenced? Thank you so much!
An SVM outputs a real decision value, namely the signed distance to the separating hyperplane in feature space. In classification the label is assigned based on the sign of this decision value. As such, SVMs *do* output more than just a binary value, their output is just binarized as a post-processing step in classification.
Weren't the Bayesians before the Frequentists?
Then the distribution will depend on the parameters - $\\omega$s and $\\phi$s and $A$s as well as the variance of $z$
@Jack Tanner Feel free to accept an answer or request clarification.
Of the 160 in the Procset not in the Resultset, none of them were statistically significant?
Each sample is a single x, y point.
You have repeated measures, but neither Friedman nor Wilcoxon pays any attention to the structure implied. The main answer to your post is that different questions get different answers; that is not a contradiction.
@Emilie: you could take your age into consideration when estimating your chances. It might be that couples formed 50+ years ago have a very different age-difference distribution from couples formed now, and so your coincidence is becoming more common over time and your chances are better than your grandparents's chances were.
Welcome to the site, @Martin. Please don't sign your posts. Note that your flair & a link to your userpage is automatically added to your posts. Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about), which contains info for new users.
I suggest that you do it, it will help clear away the false impression. Assume the $x$'s are identical for each $i=1,..,n$. Then

$$\\prod_{i=1}^{n}E\\left(P_i[\\lambda(x,\\gamma)]\\right) = \\Big[E\\big(\\exp^{-\\lambda(x, \\gamma)}\\big)\\Big]^n$$

while

$$E\\left[\\prod_{i=1}^{n}P_i[\\lambda(x,\\gamma)]\\right] = E\\big(\\exp^{-n\\lambda(x, \\gamma)}\\big)$$

If you set $z \\equiv \\exp^{-\\lambda(x, \\gamma)}$ and $f(z) \\equiv z^n$ then this is a classic case of Jensen's Inequality, $f(E[z])$ against $E[f(z)]$
Pardon my ignorance; but what was the evidence for overfitting that you refer to?
Hello Harlan, can your details be translated in mathematical notation? It would be very helpful to understand the details (for me).
Hi @Weka and welcome to the site. First, I would discourage you from excluding 5% of the data just that the rest looks like a normal distribution. If these 5% data were correctly measured, you should include them in the analysis. Maybe a [transformation](http://stats.stackexchange.com/questions/61217/transforming-variables-for-multiple-regression-in-r/61226#61226) would be applicable in your case, but remember: it is a common misunderstanding that the original variables in a linear regression must be normally distributed. The *residuals,* though, should be approximately normal.