tttjjj/elt_bench huggingface dataset:
(1) the Jarque-Bera test is based on the asymptotic distribution of the test statistic, and is not suitable for small or even not-so-small sample sizes  (2) it is not a good idea to use goodness of fit tests in this situation.
"*For example, if the median is larger than the mean, the distribution must be left-skewed.*" -- that depends on [what your definition of skewed is](http://stats.stackexchange.com/a/89383/805).

Further the relative weight that is given to observations is different, which makes determining the relative behaviour in all circumstances difficult (e.g. if there are outliers in just the right places, and the actual relationship is curved, and the error distribution has just the right shape, might the expected relative positions be reversed? ) It may be difficult to answer in general.
$\\mu_0$ and $\\mu_1$ are _column vectors_ or $n\\times 1$ matrices if you will.  So the product that you are computing  is the product of a $n\\times 1$  matrix and a $1\\times n$ matrix and is thus a $n\\times n$ matrix.
Do you just want to delete class A? Or do you want something else? (And what are these classes?) Also, it's multiple linear regression, not multivariate (I edited this for you). What do you mean by "automated"? I wrote [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) that might help you ask this better.
....Though now that I read your great explanation below, maybe using those "templates" could help a lot of us clear up these sorts of problems in understanding.
Excellent. Will read, thank you.
@gung "lose power" would assume that some particular alternative was available, as if one could choose to reallocate the available sample sizes in some other proportion. If that's the case, then yes, uneven allocation will have lower power than an even allocation.
2)  d1=e(1); dt=μ+d(t−1)−θ*e1(t−1)+e1(t) is correct for simulation purpose.
Btw, can you explain to me why the additive property is of relevance here? We are nowhere adding up random variables when checking for best-fitting distribution.
Thanks, this is correct. However I am not getting exactly when to use which test! Any light?
Thank you so much...Its really helpful
I am sure that the data process is not stationary. Is there a better way to update my prior to account for this?
@TomMinka they are vehicles, so they both have Length and Width. For example, L = 5m and W = 1.7m, but it is okay to assume that they are circular with radius L/2 to be able to discard the orientation.
About what you said... I'm not sure if I fully understand you either.  In  any case, one of my points was: even if we fail to reject the null hypothesis that b2 + b3 * mean (x) == 0, it means only that we can't say that the average effect of W, conditional on average values of x, isn't zero. However, the whole point of an interaction term is because we want to condition the effect of W not only on the average x, but on the whole distritubion of x!
How is increasing the chance of failing to pick up a difference doing it "correctly"? If you think that the behavior of a significance test in a large sample doesn't have desirable behavior, it's a clear signal that your actual question of interest isn't answered by a hypothesis test.
I'd probably call it a "predictive model," but that language certainly doesn't exclude classifiers.
Do you know any good Matlab packages that can get the job done?
Have you thought about linear mixed models or marginal models?
Ohh. $((k - 1)T < X \\leq kT - u) < ((k - 1)T < X \\leq kT)$ so regardless of $(k - 1)T < X \\leq kT$ happening, $(k - 1)T < X \\leq kT$ will happen?
@Sam I edited my post for you.
"Flipping the pulse and moving it to $-\\infty$" is a dreadful way to begin. Just "flip it" and note that this is the position for zero offset, that is, if $h = f*g$, then $$h(0) = \\int_{-\\infty}^{\\infty} f(x)g(-x)\\,\\mathrm dx.$$ To calculate $h(\\tau)$ for positive values of $\\tau$, move $g(-x)$, the "flipped over" pulse $\\tau$ units to the right; for negative values of $\\tau$, move it to the left.
Some coclusions may help. $\\sum_{i=1}^{N}l_i(x_0)=1$, $\\sum_{i=1}^{N}l_i(x_0)(x_i-x_0)^k=0, k=1 \\, ,\\cdots, \\, d$,
Thanks for your answer. Does the approach of asking these kind of questions have a name or is it simply "asking the right questions"?

The reason I am asking is that the examples in the literature that I have found all go the route of measuring A and B, and I have yet to find an example where the relationship between those variables itself is "measured". That's why I was assuming that I am either overlooking important issues, or looking in the wrong spot.
@Emily I know that you did the experiment with 34 participants and got the results you described.  I was addressing the question as to how you might have initially set up the problem so as to detect only differences that you would consider meaningful.  This might have led you to choose fewer than 34 subjects for the experiment.
As in my comment above, "removing them from the data set" is too strong here. Trimming or Winsorizing just means what it does, ignoring or replacing as may be, for a certain calculation. You are not _obliged_ to remove the tail values from the dataset, as if you were throwing out rotten fruit. For example, faced with possible outliers, you might do an analysis of the data as they come and an analysis based on trimming and see what difference it makes.
Because this proposal collapses each time series of 60 values into one mean, it seems likely it will lose a tremendous amount of potentially useful information. For instance, one would guess that *changes* or *trends* in the explanatory variables $a,b,c,$ and $d$ might be related to the endpoints, but taking the means loses all such information.
Are you certain you have an intercept term? That could affect the likelihood. This is something I often overlook.
Hi Daniel, welcome to the site. You don't have to code it yourself. The Yeo-Johnson transformations are implemented in the [`car` package](http://cran.r-project.org/web/packages/car/car.pdf) with the function `yjPower`. So just use `yjPower(datc$plot, lambda=lambda.max, jacobian.adjusted=FALSE)`. I think that should work.
$b_3x_3$ will be incorporated into the residuals either way, but *if* it is uncorrelated in the population, then your other $b$s won't be biased by the absence of $x_3$, but if it's not uncorrelated, then they will be.
I see no connection between having an ordinal response and a logistic distribution (which is a continuous univariate distribution).
I think you should take a look at this question http://stats.stackexchange.com/questions/665/whats-the-difference-between-probability-and-statistics/675#675 because Likelihood is for statistic purpose and probability for probability.
Yes, Euclidian distances have been used, see [here](http://stackoverflow.com/a/16010600/1036500) for an example and further reference.
Just FYI, this question seems to be on the border between a question about statistical issues, & a question about the implementation of procedures in R. As such, I think it's OK here, but also may be appropriate on [Stack Overflow](http://stackoverflow.com/). *Please don't cross-post* (SE strongly discourages this), however, if you don't get a satisfactory answer here after a reasonable period of time, you can ask to have it migrated.
Predictions are often more accurate in the center of $X$ than outside its range. I.e. you could consider the distance from the center as some sort of very crude figure (look for Mahalanobis distance/leverage).
Although your intentions are noble, this strikes me as being a spectacularly misguided use of data types: usually people use a type indication as a hint to the software about how to treat the data.  Data do not have any inherent type: it depends on the purpose of the analysis and what the data mean.  Software (at least to date) knows neither of those.  Asking it to guess the type seems tantamount to having the software automatically limit its own capabilities!  Please visit the thread at http://stats.stackexchange.com/q/25776 for a little more about this.
Thanks for your response, that is a big help!
+1. Small notes: I think the percentage of missed should be $1473/10000 = 0.1473$. I just re-ran your Monte-Carlo simulation with the same parameters in R and I consistently get different results. My average lower bound is around 5, while the average upper bound is around 18. Only 0.83% (83) of my simulated values did not include the population variance of 12. The average length of my CI was around 13. Could you maybe post your simulation code? I could also post my R-script for a comparison. Sorry for the trouble.
@PeterEllis I actually just made a new spreadsheet with the same data; the original was poorly defined and had some unnecessary information. Updated the link.
thanks for your answer. i'll try the modifications you suggest. :)
So your dependent variable (defect density) is right skew, right? That is, it has a long right tail? Have you tried taking logs? Or, if there are some true 0's, taking square roots?
+1 For bringing in new ideas (with references).
thank you, i really appreciate your efforts and time here and in your blog.
Not clear what you are asking...what do you want to add another market to?
Hint: have you ever noticed the "residual standard error" part of `lm`'s summary output? It is *not* the square root of the variance of the residuals. It's computed with the command `resvar <- rss/(n - p)` in `summary.lm`, where `p` is the "degrees of freedom" reported, rather than as `rss/(n - 1)`, which is what `var` is doing.
+1:  This is what I was trying to get at in my comment above.
@adparker "Nothing to do with" seems a little too strong to me. An equivalent formulation of a Poisson point process is that the points are independent and have a uniform distribution over the region.  It is *called* a Poisson process for the reason you give.  (PS I edited your comment to make it fit the space available. If any of those changes are objectionable, just flag it for moderator attention and we can fix or delete it.)
Does the same intuition work for m = cov(xy)/var(x)?
(+1) Thank you for offering the useful illustrations, Sean!
I edited my post, with some clarification and answers to your questions.
A big part of whether anything is significant is how much data you have. N=400 seems like a lot, but it really depends on how big of an effect is 'big', which is a subject-matter judgement. 1st, ignore the p-value and get some descriptive stats & effect size measures--do you care about the effect you see? On a different topic, chi-squared is not a 2-tailed test (see here: [chi-squared-always-a-one-sided-test](http://stats.stackexchange.com/questions/22347//22350#22350)).
Can you say a little bit more about this index & the "similar measures" to explain how they will solve the OP's question? (This post is being automatically flagged as low quality because it is so short; it is more of a comment than an answer in its current form.)
@whuber This is exactly what I am thinking of when I am questioning the usefulness of these charts. They may accurately represent the pattern of data. But I question the use when they are so often hard to read and act on (without considerable cognitive effort).
I'd say your answers to 2 and 4 involve a little handwaving.  For 2, I'd go back to $\\bar{X}_1 = (1/T)\\sum X_i$ and work from there.  For 4, I'd do the same for $\\bar{X}_3$. It seems to me you know the stuff, you're just not doing the proofs at a sufficient level of detail.
@guest Feel free to put this explanation in an answer rather than comments.  It's more visible there and the questioner can accept it.
Could you be more specific about the background of your question (something must have prompted it)? To be a bit tongue in cheek, one potential problem is that your PC will blow up, but this is probably not what you had in mind. Do you have some analogy from something else? Are you talking about problems of fitting, assumptions, interpretation, software use, etc. Currently, the question is too broad in my opinion.
Sorry about that tone - it really wasn't intended that way. The problem with probability paradoxes is that there are undefinable terms, multiple paths to solution, and simple shortcuts that are often taken without proper justification. The upshot is that, to convince a proponent of the "wrong" answer that yours is "rigorous," you have to both demonstrate yours with no room for objection, and find an inescapable hole in the opposing argument. I think my attempts to point out that hole are what you found "testy."
The [penalized](http://cran.r-project.org/web/packages/penalized/index.html) R package offers similar option (see argument `unpenalized=`), although it deals with L1/L2 penalization schemes, and not FS *per se*.
In a way, the measurement you proposed is widely used in case of error (model quality) analysis -- then it is called MAE, "mean absolute error".
If you have 3 ordered categories as your response variable, you probably want to use [ordinal logistic regression](http://en.wikipedia.org/wiki/Ordered_logit). Note that this is more advanced than standard linear regression, so you will either need to do a bit of work to get up to speed, or you should work w/ a statistical consultant.
If you use a regression, it gives its own results and the different types of correlation aren't relevant.
Good answer under the assumption. Really sorry but I had to pick the answer which solved it without the assumption. Still good insight though!
looks like fanning out at the higher $Claimed$ values
Sorry. It's a leftover from previous badly worded version of my question. I'll edit it accordingly.
Are these predictors truly binary or are you dichotomizing them?
@ probabilityislogic  I see what you mean, thanks.
Thank you it worked thank you very much
Thanks everyone for looking at the question. Here is a more concrete example:  Imagine a web service being served by N server instances. Each server constructs a CDF of the response times of a request. Now I would like to aggregate or merge the CDFs to find say the 95th percentile response time of the web service.
Hmm, I'm still a little stuck on how to go from the SVM result, to knowing what scores to give the different tests (I need to find the weights of the different features right?) Is e1701 suitable for this? If not do you know of any other R packages that R, and any documentation?
+1 Nelsen is quite readable.  I bought a copy a few years ago even after going through a lot of the online materials.
A clearer notation would be $E[X|X=t]$, then integrate over the area where $X=t$ (a point in this case).
Of course, if all 3 terms, COVxy, SDx, SDy, are true population parameters, then COVxy/(SDx*SDy) gives true population r, because "population" and "sample" coincide here.
@jbowman, following Henry, perhaps you should convert your comment to an answer. Then the OP can accept it, as it does answer the question & she has acknowledged that (& I can upvote it ;-) ).
$\\max$ is short for 'maximum'; it's the converse of $\\min$, and gives the largest, rather than the smallest value.
Answer too long for a comment. I'll edit my answer.
I hadn't realised that $\\sum_{i=1}^{n}(x_{i}-\\bar{x}) = 0$ until I saw your answer.
Thanks a million!
Thanks, two questions: 1. Do you know of any procedure in r, that might work well for me in debugging this? 2. I can see, that when I run the exact same model on the exact same data with glm() instead of glmrob() I don't get any errors - does that make sense, or does it imply an error in the glmrob() function?
This is an introductory book for people who have a decent amount of statistical background already.
@Corone I resample the original data and then trim.
Another possibility is that you have a small sample size
Peter's right.  Actually, the set could be quite interesting in itself: conditioned on needing to operate on *both* eyes, were outcomes any worse?  The reason we advocate not assuming independence is that there are a lot of reasons why this *could* be wrong.  If there's a large enough sample, test for independence.  The insight could be very interesting & practically useful.
Is this a structural equation model?
@Deathkill14 correct i read the paper, information-theoretic measures are mentioned (in fact as good alternatives)
You could look at elements on the diagonal, those are your correct predictions, off-diagonal elements are wrong predictions. That's a start.
Hi @Tiffany, could you give a little bit more explanation in your answer? Also, the OP seemed to wonder how to include covariates in the model.
You have changed the description of your approach I, and my comment no longer applies.
You should explain *exactly* what you intend by "normalize", since it seems to get used in (at least) three different senses. Precisely what did you do to normalize your data?
@whuber the dummies are 0/1 for no/yes. They come from a survey that has no/yes questions, which ask about a few concepts (the factors) in various ways. Sorry if I'm being dense, but it hasn't answered itself.
+1 for the attempt
At the same time you are showing how good linear regression can be, please also show your audience how it fails in situations where the relationships are not well described by straight lines: `require(mlbench) ; cor( mlbench.smiley()$x );  plot(mlbench.smiley()$x)`
@learner, if you don't mind answering one more doubt that I have - don't the rows of stochastic matrix have to sum to 1? if i have one row of almost zeros, would that even be a proper transition matrix?
Hi Al-Ahmadgaid thanks for your responce. I have looked at the answer you provided previously, I just wish to confirm the following:


I have read that back-forecasting is only used in large cases of t, other wise the error term is considered zero? Is this the case?
Actually I don't have information about how many times they visited the advisers but I have the number of hours they spent with the advisers. In 2012 we did not have advisers but we collected data on demographic variables of the students. In 2013 we introduced advisers and also collected information about the demographic variables. Now based on the similar demographic variables in the 2013 data, we want to make a comparison with the 2012 data on retention of the students. We also want to see if introducing advisers in 2013 had any effect on retention.
its nice, but somewhat tricky to export to LaTeX IME.
Thanks!.That sounds like what I was looking for.
This is essentially what the **Bradley-Terry model** was made for. You'll have to make some adjustments to handle the possible outcome "about the same for both". There are, however, [whole books](http://www.amazon.com/Method-Paired-Comparisons-Statistical-Monograph/dp/0852640137) written on paired comparisons.
thanks. Poisson is a good approximation for my case. By the way I don't have an analitical form of $f$.
@high No; the trapezoidal rule is working fine.  The problem is that it is working with an incorrect discretization of the integrand.  You can't possibly have "a lot of narrow spikes corresponding to the data points" when there are 10,000 data points and only 512 values in the density array!
I'm not sure what you mean when you ask about a standard, but here is a link to an R implementation of the diffusion model (written by Andrew Heathcote of Newcastle University): https://gist.github.com/cf6a07d42f70f9c4a6b7
From the resources already mentioned probably your best bet will probably be the FDA book by Ramsay & Silverman, Chapt. 7 on the "registration and display of functional data". A very good resource is also the book : "Inference for Functional Data with Applications" by Horváth and Kokoszka; Part III "Dependent functional data" touches upon the issue of normalization mostly because ones cares about change point detection.
Inserting shameless plug for Rcpp :)
In Gibbs sampling, if the conditional distribution is non-standard normal, then that's what gets sampled. It's not clear to me exactly what "the conditionals of various parameters are product of two non-standard normal distributions" means (post the math!), but if it means that the conditional density is the product of two non-standard normal densities, then you need the following product: $\\exp\\bigg(-\\frac{\\tau_1}{2}(x-\\mu_1)^2\\bigg) \\cdot \\exp\\bigg(-\\frac{\\tau_2}{2}(x-\\mu_2)^2\\bigg) \\propto \\exp\\bigg(-\\frac{\\tau_1 + \\tau_2}{2}(x-\\frac{\\tau_1\\mu_1 + \\tau_2\\mu_2}{\\tau_1 + \\tau_2})^2\\bigg)$
The formula defines a confidence _interval_: it has two ends. One is obtained by using the + and one is by using the -. The lower end (the lower bound of your title) is obtained by using the -.
I believe it's the characteristic function that more resembles the Fourier transform. The moment generating function is a Laplace transform.
Ahh yes, I see now. The problem says that only 69% of workers have saved for retirement. I suck at statistics. Now I need to figure out how to calculate the probability for this.
[This post](http://stats.stackexchange.com/questions/69842/anova-with-mean-sd-and-group-size?rq=1) points to the `rpsychi` package, which does at least some of what you want, with the `ind.oneway.second` function for the ANOVA.
Can you clarify what you mean by confidence interval. It sounds to me like you have a confidence interval for the fitted model (the regression line), which would not include on average 95% of the observations or anything close to that value. There is a *prediction interval* for the OLS model which may be what you were expecting?
@whuber I think the following minor correction applies to your comment because $v_\\max$ is the maximum absolute value, not the maximum value: $$P\\{v_\\max < z\\} = \\prod_{i=1}^N P\\{-z < v_i < z\\} = [2\\Phi_\\sigma(z) - 1]^N$$
Thanks!  I don't know $\\sigma^2$ and was going to estimate it from the pooled sample variances (assuming the underlying variances are equal, which I think is a pretty good assumption).
@endy_c Does that help?
The concavity of the *function* $h$ doesn't depend on the distribution of the values of the arguments you apply it to. Please clarify your question.
Well, not to be snide, but I think maybe you need to consider hiring an expert. You have a combination of spatial data and longitudinal data - each area is complex. The intersection is more complex.
Probably you wrote it that way for a reason, but anyway, the Python code could be [simplified somewhat](http://codepad.org/on8azo8W) or [totally rewritten without the nesting](http://codepad.org/EiKMHKB3)
Could you clarify in what sense your model is 'not linear'? Or, are you just working with time series data?
I'm not sure what that is supposed to mean, @luciano. If that is what he says, he's wrong, although there may be some ambiguity in the way he put something & he didn't quite mean that. You can see in the examples what changes & what doesn't, & under what circumstances. All those examples were simply run in R & cut&pasted here. You can run them yourself & get the same results.
Luca, if these are still the distributions you were discussing in your previous question, the independence of $w,\\phi$ and $\\lambda$ gives you the marginals immediately - the $q(w)$, $q(\\phi)$ and $q(\\lambda)$ you had in the previous question.
This looks great. I will see if this answer deserves the 50 points :) Thank you for your response
Thx for your answer! Could you please explain why I would "do better making predictions"?
For the case E(XY|X=x) how do we go about calculating thing? is it the same as xE(Y|X=x)?
The symmetric example is clear and compelling. I am actually more interested in the skewed case. If the Pareto estimator you describe is the MLE then it is consistent. But why should it be unbiased?
Interesting.  I hadn't heard of this before.  I found the zoomable stuff on the canvas page, http://gnuplot.sourceforge.net/demo_canvas_4.6/, and had to use Firefox to see it (Chrome didn't work for me).