tttjjj/elt_bench huggingface dataset:
In addition to the link above, this question: [when-should-you-center-your-data-when-should-you-standardize](http://stats.stackexchange.com/questions/29781/) will be of interest.
There's nothing unfortunate about being French.  :)  Regretfully, I don't entirely know what Gary was talking about (hence the question).  I might assume that he meant "loss of information", but would be happy to get guidance in that respect.  I would say that this is correct english, but questionable statistics.
Can you give a more concrete example, especially with dummy data and or code, especially if you would prefer an [tag:r] related answer? At the moment I don't have a good feel for what it is you are asking, so a tangible example would help.
@whuber: sorry, I made some mistake in my question. It should be before applying the inverse Fourier transform: In the equation for $f(t)$,
$(it)^r$ is inside the exponential and $\\psi(t)$ multiplies with the exponential, so how can $(it)^r$ and $\\psi(t)$ multiply together to get
$(it)^r\\psi(t)$?
If this is concerning the homework assignment, then please use self-study tag. We encourage such questions in this site, but you must be specific. I.e. you must explain the assignment, tell us what you did and where have you stuck. You do not need to tell all the specifics however. Giving a good question is not easy, but the effort usualy pays off.
@gung I interpreted the same discussion as a heuristic tool - an easy way for the reader to understand the approach, rather than a way to reduce the work of the designer.
67% here is presumably an echo of the 68.3% or so of cases being within 1 SD of the mean for a normal distribution; but that's not a universal result. The key point is that means and SDs are themselves typically inflated by outliers, so do not provide resistant criteria, which are needed here.
Checking against a Exponential E(x) of 1/lambda = 1/(exp(2.18+0.66)) yields a similar estimate of 0.058.
At this point you need to consider how the data were generated and you need to more formally specify your study objectives.  The Kruskal-Wallis test, as I recall, looks for a shift of location but assumes homogeneous variances, which obviously is not the case here (compare V4 to V6).  The data-generation process may suggest useful analyses (e.g., these data might be rescaled counts or they could be autocorrelated or the low values might represent measurement noise only).
I am still in the middle of learning it all, and I do study pretty hard.  Most days I feel reasonably on top of it.  I am definitely patching it all together, but consider: in any statistically oriented field besides actual Statistics, none of your classmates will have a stunning mastery either, and your ability to push through hard stuff, weird notation, and mental gymnastics will serve you well.

Yes, I am in America.  I also think it helped that I was honest upfront with my advisor about my pure-to-the-point-of-esoteric math background.
Yes, thank you. It is more the suitability of HMMs for the task that I am unsure about.
@John Hear hear!
I made an error in the last sentence of this comment but couldn't edit. Remade the comment instead.  	
Re comment #2: The PgvnD slope moves to 18.152 when Asym==1 because the interaction was estimated 26.618 (26.618 - 8.466 == 18.152). When Asym==0 (the reference), then PgvnD parameter is just as it was estimated -8.466. This is because the contribution to TotalPayoff from the `Asym:PgvnD` parameter is effectively 0 when `Asym:PgvnD` is 0 (i.e. Beta * PgvnD * 0 = 0)
The mean of the pairwise *squared* distances is related to the variance.
Here is what I tried to do:                             >data_ts<-ts(data[,1],start=1959.1,freq=12)
     >t<-seq(1959.1,2012.1,length=length(data_ts))
        >model=arima(data_ts,order=c(1,1,1),xreg=cbind(t),seasonal=list(order=c(0,1,1),period=12))
  > predict(model)
  Error in predict.Arima(model) : 
  'xreg' and 'newxreg' have different numbers of columns: 1 != 0
      How can I do to to supply new values of my regressor (the time variable)? Because if I create a time variable longer than my dataset I have problems in fitting, so I need to provide them expressly for the forecast. Thank you
It was a pleasure to help. =)
+1, very elaborative. Take $h_1$ as the example, We have drawn many different samples $D_i$, each with 2 points, so we trained many models each with different model parameter estimate $(\\hat a_i,\\hat b_i)$, right? And for a specific input $x_0$, different $(\\hat a_i,\\hat b_i)$ gives different prediction $\\hat y_0$, so the predicted target value $\\hat y_0$ is a random variable varying with the model parameter estimate $(\\hat a,\\hat b)$. And the bias and variance in your fig are computed for each $x$'s predicted $\\hat y$, right? Am I correct?
I worry about any statistical method that uses binning.
Thanks, @Matt.Vn this was exactly my reasoning. I don't want to loose too much power whilst still being able to express rejection of the $H_0$ in terms of location parameters as the PI is a challenging thing for people to deal with.
Likelihood is only a function of the model and the data. If those don't change, the likelihood is the same.
There is a bidding market for individual consumer information ???
Sure.  Maybe think about it like this: it's the *parameter* that's fixed and it may represent something in the world that's 'really' constant, or it may not e.g. if it represents the average of something that actually varies. The question is: what inferential difference does it make to put a fixed effect in rather than something else.  In the causal inference case the question is: does assuming fixed effects *decrease* confounding more than the small variations left uncaptured by the parameter *increase* confounding.
I think I managed to repair my answer now so that it fits your situation.
I found a paper may have some interest for some of you. https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&cad=rja&ved=0CG8QFjAJ&url=http%3A%2F%2Fwww2.unil.ch%2Fbiomapper%2FDownload%2FLobo-GloEcoBioGeo-2007.pdf&ei=RWAkUtfkC8iW0AXu_oCgAg&usg=AFQjCNEvmOPt92quGwJ1earYdTprclWLuA&sig2=p6i_EMbre0ZNtYWley8wAQ
(+1) One way to justify the symmetric Normal CI is that it minimizes expected length.  Ultimately that just pushes back the subjectivity to the choice of length as a loss function in a decision procedure: but that's arguably a "good" kind of subjectivity (because it exposes the role of our analytical objectives in our choice of statistical procedure) rather than "bad" subjectivity, which sounds merely like some pejorative epithet.
One of your doubts: $Y_n=\\max \\{X_1,\\dots,X_n\\}$. Also, be careful with "Let $\\theta$ be an **observed** value of the random variable $\\Theta$." This sentence makes no sense, since we don't observe the values of parameters. Tip: write the indicators explicitly. Check this question: http://stats.stackexchange.com/questions/52531/how-to-find-mle-when-samples-depend-on-the-estimated-parameter/52558#comment102571_52558
Thanks for your answer.  Did you read Jaynes' defence of the principle of indifference?  E. Jaynes, “Where do we stand on Maximum Entropy?,” R. Levine and M. Tribus, Eds. The MIT Press, 1979, pp. 15–118.
+1. For the record, the details of your demonstration via the c.f. are short and sweet: the [standard Cauchy c.f.](http://en.wikipedia.org/wiki/Cauchy_distribution) is $\\exp(-|t|)$, whence the c.f. for the sum of $n$ iid values is $$\\exp(-|t|)^n=\\exp(-n|t|)=\\exp(-|nt|),$$ immediately implying the mean has the same c.f. as the original distribution. The same result obviously holds for the location-scale family of Cauchy distributions, which includes the location-shifted version in the question.
yes, thank you. a small trick, but a big difference. great! I prefer not using T3 to draw plot for I have more than one hundred traits' results to draw, and I haven't take my time to learn how to use command parameters of T3 to do this batchly. that's why I just use the results of T3 and use R to draw plots batchly.
+1 Nice explanation. There are some additional problems with this code. It is unnecessary to compute `p2` at all; the smaller of `p1` and `p2` corresponds to the smaller of `x` and `y`, respectively--that's an inefficiency. A possible bug is that the second branch of the conditional fails to compute `p2` at all and uses only `p1`. I am also suspicious that the code might be entirely erroneous, because it does not appear to compute a p-value: it is just one-half a binomial probability and perhaps ought to be a *tail* probability. Why not just use `pbinom`/`dbinom` and be done with it?
The answers on the SO thread do not help you because there you did not explain that the values in the arrays will exhibit strong spatial correlation. There is one answer there that correctly addresses the question you *did* ask: it is the one that points out the need to adjust for multiple testing.
I mean more in theoretical terms. I can find the formula in any book, but i'm interested more in the explanation in words. The expectation of the sigma is unbiased and we can transform the estimate into the expectation?
thanks for the online calculator Harvey, I'm a typical biologist with insufficient background in statistics and your calculator was exactly what I needed.
In your case, the last clause `(family$link == "log" && any(y <= 0))` would evaluate to `FALSE` now that you rescaled the data and that would mean the entire set of clauses evaluates to `FALSE` and hence the error passes. This clearly seems like a bug so I'll email Simon and refer to this Q&A.
Relevant [question](http://stats.stackexchange.com/q/55805/7071).
@whuber Oh... right. LOL, sorry it's been a long day. Will revise.
@Ran: If I am not mistaken, FLD *is* the same as LDA *under the assumption of normality*. Under normality assumptions, there is a close connection between LDA and logistic regression where the latter can be viewed as a conditional version of the former. As ttnphns alludes to, [Efron showed](http://www.jstor.org/stable/2285453) that the ARE of logistic regression with respect to LDA is about 70% under the normality assumption. When normality breaks down the LDA procedure is still the same as FLD, but logistic regression can dominate performance-wise.
can the use of multiple iid walkers be used in aggregation to approximate that state. The search is for testing the locality of a discrete distribution's consistency for the iid walkers
You should probably not say "natural parameter", which are mean divided by variance, and 1 divided by variance: http://en.wikipedia.org/wiki/Natural_parameter
@Momo 'solve' is too strong; 'reduce' is better.  	
If parameters are estimated from the same observations you're testing, then - unless you account for that effect - the deviations of the sample from the distribution will be 'too small' - the rejection rate goes *waay* down. Using another sample removes that effect. The parameter values resulting from estimating from a second sample still suffer from sampling error. That will have some impact on the test (pushes up the type I error rate), but won't have the dramatic biasing effect that using the same data for both does.
Analysis of covariance is to lines as analysis of variance is to points.  I said "a form of" because the authors of the study constrained their model to a common intercept.
@Glen_b I've reformulated the question and posted here
OK, I think I understand the meaning of C now. :)
EM is _not_ a heuristic, it is an algortihm for maximizing likelihood with known properties. The error you are looking for probably has nothing to do with EM, but rather the efficiency of the maximum likelihood estimate. This will surely depend on the unknown parameters, and not just the sample size.
Thanks :) Sadly, I seem to have run aground on convergence issues, but I'll tinker with those.
Have you noticed $P(|X_{(1)} -\\theta| > \\epsilon) = P(X_{(1)} -\\theta > \\epsilon)$ ?
What is the purpose of the metric you seek? For what is the Frobenius metric inappropriate?
I see no reason to think so.
Didn't you ask the wrong question then? Maybe the FDA approval process should specify a larger gain vs placebo (perhaps related to costs of the drug, including its adverse effects) instead of just requiring statistical significance? Because there can very well be a real difference, albeit very small, and that difference was shown to be statistically significant however small it is.
Perhaps [this](http://stats.stackexchange.com/questions/32405/how-is-poisson-distribution-different-to-normal-distribution?rq=1) is helpful.
Pardon me for asking, but what exactly do you mean with non-negative lasso? To keep all coefficients > 0 or only allow positive predictions? Googling didn't enlighten me but it sounds like something I'd like to know about.
What kind of limiting process are you using to define the integral?  The answer will depend on this.
Haven't you find anything relevant by looking at our questions with the [tag:multiple-comparisons] tag? If not, could you clarify where you are having difficulties?
Accepted style tends to be affected by which area you're working in - and sometimes, by what a particular journal expects.
Note that a non-significant p-value doesn't justify a conclusion of "no difference between groups".
@IrishStat What values did you use for the following dates: March 11, 2011/ March 25, 2011/ June 24, 2011/December 02, 2011/ December 30, 2011/ March 30, 2012/ April 20, 2012/? I ask because values for these dates are not given by the OP. The OP also refers to the data on these dates as outliers, but if I've understood the question correctly, they're actually *missing values*; not *outliers*. Has there been a misunderstanding regarding terminology here or am I in the wrong?
As a matter of interest - how big? Sample size, range of data.
If the standard deviation is zero, then the column is populated by one value. So if your goal is to prepare the data for regression, you can throw the column out, since it will contribute nothing to the regression. Adding small noise will only give you more problems.
Chi sq does have some assumptions.  Are all your observations in one list independent of all observations in all other lists?  Are all observations within one list independent of all other observations within that list?  The answer to that question probably depends on the source or nature of the lists.
"Log Y and X - a one unit increase in X would lead to a β∗100 % increase/decrease in Y": I think this applies only when β is small so that exp(β) ≈ 1 + β
In this case, x represents the rms prediction errors over 5 repeated out of sample folds, across 3 different models. The corresponding model labels are represented on the y axis.
From [@dominic](http://stats.stackexchange.com/users/36535/dominic): Can you provide more information about the feature vectors X and Y, what their relationship is, what kind of data - ordinal, binary, categorical etc.
If there's any suspicion of non-constant variance, for the primary analysis you might well be better off not testing this sort of assumption at all, and just using an unequal-variance analysis. See e.g. http://biometrie.bfh-inst2.de/images/content/dateien/do_not_use_the_two_samplettest_nay_m.pdf ... permutation tests also cope with unequal variance.
Not for model selection. For model checking and diagnostics.
What is the population about which you wish to make inferences? (It looks to me like you might even already have the population rather than a sample from it, but it depends on the type of statement you wish to make)
Thanks for the reference to Yann LeCun's talk. I look forward to watching it.
This sounds like a classic MCAR/MAR/MNAR example. Have you looked at the relevant literature? I don't know a conventional wisdom rule but given you analyzed "*everything*" without being concerned for missing values, it feels a bit odd to start do this now.
The econometrics lecture notes was very helpful in addition. Thanks.
It all depends on what varies in your data.  Consider the case of linear supply and demand.  You observe the equilibrium price and quantity.  Now, suppose the supply curve shifts.  You observe another point.  Connect the two observations, and you have traced the demand curve.  What if instead, the demand curve had shifted?  Connect the two observations and you have the supply curve.  If both curves shift by an equal amount, you will have a horizontal line that is neither the supply curve, nor the demand curve.
@Steve Yes, the numbers seem right. However, I don't understand your issue with the warranty. The length of the warranty or the duration do not figure in the computation of the confidence limits at all. You observe the proportion of failed products under some conditions and make some inference on that proportion. It happens to be a proportion of products that failed after a one-year simulation but the exact same technique can be used for proportions that have nothing to do with on-going processes.
Are these standard $t$, or location-scale family $t$?
Linear algebra, at least some basic calculus, at least a basic course on probability, linear algebra, a little computer simulation, some statistical theory, and maybe some linear algebra. While not critical, some basic programming would be an asset. Actually the questions generated here by students tend to suggest a lot of the kind of background needed.
**Factor analysis** on **binary data** is a special theme. Try to search on this site for different opinions on the topic.
@cardinal: right, for purposes of compactness, one often stores the (scaled) Householder transformations for forming $\\mathbf Q$ where the zeroes of the triangular matrix ought to be.
I can't really follow your argument here, & I'm not sure if it's correct. E.g., the surface area of a sphere is [$4\\pi r^2$](http://en.wikipedia.org/wiki/Sphere#Surface_area_of_a_sphere). Beyond that, I'm not sure how this relates to the question. The key issue is whether or not the omitted variable is correlated w/ variables that are in the model. I'm not sure how what you are saying addresses that issue. (For clarity, I demonstrate this with a simple R simulation.)
Do you know of any published research that looks at the relative merits of using likert scales as intervals vs ordinal data? Perhaps, the flaws of treating them as interval level scales are not serious enough to warrant a complex approach. If that is the case then your approach may simply be a wild goose chase.
Hy Kjetil. I'm trying to code my own UKF, but I can't find a practical example showing clearly the way that such a filter is running.
What does it mean when you say that **probabilities aren't correlated?**
Welcome to the site, @helen. Providing references would have helped. My gut feeling is that you are referring to some highly specific forms of the bootstrap developed in a small niche of the bootstrap literature, so more guidance may be needed for simple folks like us. Michael Chernick might be able to answer this off the top of his head, but I would personally like to see more of the explanation of the "approximately unbiased" bootstrap does (as well as your understanding of what the "regular" bootstrap does, as to me, there is no such thing as the regular bootstrap.)
$AIC = n \\cdot ln \\left( RSS/n \\right) + 2 \\cdot k$ where $n$ is number of data points, $RSS$ is residual sum of squares, and $K$ is the number of parameters in the model.  This assumes that the inequality $n/K < 40 $ is true, otherwise you want to use corrected AIC, or $AIC_c$.  You can get RSS from ($R^2$) if you know the residual sum of squares to the mean, as well as to your fit function.  $R^2=1-\\frac {RSS_{fit}} {RSS_{mean}}$.  Many fit programs provide RSS directly.
Thank you for your reply. I used step wise regression to prove the argument that weight predicts most of the variance in total energy expenditure. So are you saying that a simple multiple regression would do? If so, what about the interaction factor?
The batches are the artifact of the reporting, in fact every trial is independent (the coin is put back to bag after one toss).
One natural way to cope with the situation in the quotation is to recognize that all data are actually *intervals,* because few measurements are absolutely perfect. Thus in reality the likelihood of $x$ would not be $f(x|\\theta)$ but rather $F(x+\\epsilon_{+}|\\theta)-F(x-\\epsilon_{-}|\\theta)$ for appropriately small $\\epsilon_{+,-}$. One could approach the solution of the original (non-interval) problem by studying the limiting MLE as a function of all these $\\epsilon$: there is a limit as they approach zero and it equals $\\max(x_1,\\ldots,x_n)$.
Great! Thanks. This is exactly what I was looking for. I wasn't aware of the rms package
What precisely is the "problem"? Is it that you don't understand how indicator variables work? Given so many indicator (dummy) variables, one is redundant. That's why a single indicator suffices to handle a binary predictor (e.g. male or female). You can control which variable is omitted in Stata, but that falls outside the scope of CV. It is well documented, so just read up about handling factor variables.
I should also add that when one includes a constant the hypothesis being tested is "does the variable have a significant effect above and beyond the mean". If you don't have a constant in the model the null hypothesis is "does the variable have a significant effect above and beyond 0.0" . Thus if one has monthly data and excludes a constant and includes 12 monthly indicators , the test of each indicator's coefficient i.e the seasonal effect of a particular month is whether of not this coefficent is different from 0.0 rather than does this variable have an effect above and beyond the mean.
+1, this is a good answer, that I didn't think about, but it presupposes a quadratic loss. I don't think this can be generalized to (all) other common loss functions used for classification trees. I suppose that the *typical* $pN\\log(N)$, but according to ESL worst case $pN^2$, behavior comes from a branch-and-bound algorithm, but I haven't found any confirmation of this.
A low degree polynomial definitely will not work, Josh.  You need a local smoother--think of a kernel smooth or certain kinds of splines--and it needs *not* to be a polynomial, which has terrible properties.  (Polynomials can introduce spurious, huge peaks between what otherwise appear to be well-behaved data series.)
I think the first sentence in your answer is a bit misleading. *Most* kernels do not map to infinite dimensions, but the most popular one does (RBF). I think it is more appropriate to say that most kernels map to a higher dimensional space, but usually not infinite dimensional.
Actually, none of the conclusions after the first paragraph are likely to be true.  If the bugs are initially divided equally between the chambers, then there may initially be some trends, but then the system should settle into a dynamic equilibrium and the counts will no longer be "lopsided."  The longer the experiment runs, the closer the average changes during the experiment will be to one another.
A gaussian mixture would be too unflexible (I don`t want to work with more than 2 components at a time due to intuition issues in setting up scenarios). Other distributions easily get even more complicated I think (e.g. skewed distributions). One very important aspect for me is that there should be (implicit) analytical formulas for the Value at Risk and the Expected Shortfall (which I have set up for the t mixture already). So this gives nice opportunities for performance testing of risk measures. But feel free to suggest another alternative. I`m open minded!
Allow me to modify that statement.  The roughness penalty would act as I said, and, if the splines have the same number and location of knots, then the roughness penalty makes the spline smoother.  However, it is likely that the splines that made up your plot were differently parameterized to start with.  The penalty would make the penalized spline smoother than it would otherwise be, but, depending upon the parameterization, it might still be rougher than the other spline.