tttjjj/elt_bench huggingface dataset:
@Rob Hyndman, @Pantera, Rob could actually add some nice ideas here. But as for autoregressive process with trend, I would not go for OLS directly (in fact the parameter estimates will differ if you go for lagged dependent variable regression and estimate $AR(1)$ process). The choice is quite large: maximum likelihood, Yule Walker equations, Burgs (well OLS is among methods for $AR(p)$ process though :)). The autoregressive residuals cures are more for classical regressions, in my opinion.
What can I say, all your points are valid, so it is possible that the authors of the articles in question are doing the wrong thing. Without more context it is impossible to say anything concrete.
If the answer to Macro's question is no then I would be very cautious about trying to analyze the count data.  Large counts for one race relative to the others  might simply mean that the population for that race is much larger than for the others
You may want to check out @user603's answer here: [is there a boxplot variant for poisson distributed data](http://stats.stackexchange.com/questions/13086//13429#13429) for info on how to adjust this rule for skewed data.
Linear model? Every meteorologist will tell you that everything they do is just a small adjustment to "tomorrow will be the same" ;-)
Hi @Fazil - if mpiktas has answered your question well enough for you can you "accept" it by clicking on the tick mark next to the question, so it is marked in the Cross-Validated database as resolved.
+1, this is the best statement of this point of view that I've seen from you on CV.
@Jand: You have a very similar (but slightly different) [question on math.SE](http://math.stackexchange.com/questions/62959/proving-that-one-expectation-is-greater-than-another). Can you clarify whether you are actually interested in both or just one of them and why?
@EMS any of those are meaningful residuals. Just because one is a Bayesian doesn't mean one can't check whether the assumptions are reflected in the data.
Euclidean metrics do not "assume... un-correlation". Euclidean distances work worst in high dimensions with uncorrelated variables. Consider the extreme case: you have very many dimensions that are all perfectly correlated, r=1, now your data are in fact uni-dimensional, & Euclidean distance works fine w/ uni-dimensional data.
Formerly, for your second $H_A$ the null should read $H_0:\\, \\mu\\le 5\\ \\text{ft}\\ 7\\ \\text{inches}$, not $H_0:\\, \\mu = 5\\ \\text{ft}\\ 7\\ \\text{inches}$. See one of @whuber's comments to this question, [Do null and alternative hypotheses have be to exhaustive or not?](http://stats.stackexchange.com/a/19012/930).
@probabilityislogic Do you know of any good implementations of the double pareto penalty, in a language like `r` or `python`?  I'd love to try it out.
Is there any implementation of this theory on python ?
@IrishStat it is *not* generally nonsense to add mixture and higher order terms; only bad when it is done without restraint and without regard to theory
Thanks @gung it's definitely worth pointing out that the daily occurrence of rainfall would perhaps be better seen as Bernoulli rather than Poisson (and then the question's use of 'probability' fits better), so thanks for that. As for 'is it possible to hail without rain', in a practical sense a gauge will normally have water in it even if it was all frozen as it came down. I'm not sure how you could be sure none of it was liquid before it hit the ground rather than after - but perhaps I should leave that issue for people with more subject-expertise.
I know, I'm calculating a weighted mean of F-measures for all clusters later.
A pneumonia scenario is different from a HF scenario, so it seems to me you are test both time and scenario.  I was thinking of summing all item scores and normalizing by number of items.  An improvement would be to normalize by summing by all the maximum possible scores per item.  This is not without danger though.
@whuber @mpiktas Let's see you both volunteer too!
Could you explain why that quantile regression problem is equivalent to my own?
In the Bayesian approach, the models would be weighted by their marginal likelihood (i.e. Bayesian evidence) and any prior placed over the hyper-parameters, so it would be a special case of averaging over an ensemble with a particular method for weighting the models.
Note that as you've given some Stata syntax, adding the actual `xtreg` command you used would be consistent with that and is in fact the single piece of Stata information that is most relevant. That aside: there is a statistical question here, but what is it precisely? Do want to know what collinearity means? If not, what kind of explanation do you seek? We don't have your data to look at correlations or scatter plots but you can do that.
Erik's answer is correct, as inspecting some small cases will demonstrate.  Colithium, your denominators are too small: they should all equal $x$, not $x-j+1$.  E.g., for $x=3,n=2$ Erik gets $2/3$ for the answer.  You compute $(2/3)((2/3)^1 + (1/2)^1)$=$7/9$ but only $6$ of the $9$ equiprobable outcomes $11, 12, 13, 21, 22, 23, 31, 32, 33$ have a unique minimum.
I haven't added an example but I added edits of what I did further.  If you could comment on that I would appreciate it.  I am trying to get a sense of if I am headed in the right direction.  Thanks again
I think this question may be on-topic for CV b/c it seems to be about understanding the ideas rather than help w/ R per se.
Is this true for *any* $X$ and $Y$?  Suppose $X$ and $Z$ are uncorrelated and let $Y=X-Z$.  Then I suspect $X$ will not be correlated with $X-Y$.
"Z score" of what?  (By "distribution" surely you mean *data*, so what you have is an *estimated* odds ratio (and its p-value).) And what is the second row of statistics you show?  A second example?
@MarkusLanthaler How are you going to randomly sample from the 6 billion (actually, it's well over 7) people on the planet? Are you really going to sample Somalia and Brazil and Vietnam in proportion to their populations? From your question this doesn't sound like your target population at all, which sounds more like "people who *could* use facebook but may or may not actually do so".
@Affine the same holds for me! I am sure I heard about it, but I am not into the topic nor do I have good articles or books about it.
When you say it looks like it follows an exponential distribution, are you referring to the time between visits? I may be misreading your question, but it seems like you're saying the number of visits follows an exponential distribution. Typically, this would be modelled as a Poisson distribution, and it is well known that the time between Poisson observations is exponential. Also, for testing the hypothesis of whether it follows a Poisson distribution, it would be necessary to have a null distribution, i.e. what does the pattern of "infrequent flyers" look like?
@Marc Claesen : How can I find probabilities in LIBSVM ?  Thanks.
Before I was using the Cholesky-Decomposition to get the desired weights but I find it much easier to use the Singular Value Decomposition instead. Though this matrix does control the correlation in each group it tends to get larger in the group $u_{1.2},u_{2.2}$ for a certain $\\rho$ if $q[i]$ increases. If $\\rho$ reaches 0.5 the overall correlation does does decrease too if $q[i]$ increases. I know that my descriptions are rather cryptic and im very sorry about that. If someone is really interestend in my simulation study i would really like to explained it in more detial perhaps via chat :)
IMO This is material for crossvalidated.com.
I don't think so; if you compute the LASSO path from the least squares estimates to some point with high shrinkage, if a coefficient is shrunk so much that it would switch sign, at the point at which the sign change takes place, it is removed from the model (i.e. fixed at 0). This is in contrast to ridge regression where shrinkage can switch signs IIRC.
Hi, Thanks alot for the answer!!
Of course there is no perfect situation in econometrics.  So by definition, I am asking for the best possible approximation. In other words, is there a method that would lose less information than the PCA method?
At a fixed sample size, this would be a matter of comparing statistics that measure deviation from normality (whether Shapiro-Wilk, Anderson-Darling, or whatever you feel captures what you want to capture). It would be difficult to compare across different sample sizes without a much more careful identification of what you're trying to achieve, which *might* help you identify a suitable metric. Why is it important to compare how normal they are?
If you make your state include the values at the previous time-step, it will be first order.
Yes @whuber, I agree. To Dail, it is good practice to comb through the related questions on the site. In this instance, just perusing the questions tagged [tag:anova] would turn up a wealth of potential resources and explanation for similar questions.
Is this an answer to your question? If not, it would be better to append this directly to your question.
Is this really $p=0.1500$ (and not, e.g., $p=0.0150$)? Also, could you please tell us what's your understanding of $p$-value and null hypothesis statistical testing?
The 'terms' predictions, I gather, are centered. Do you know how this is done?
They are completely different approaches starting from the way that *Probability* is interpreted (see *e.g.* [link](http://en.wikipedia.org/wiki/Probability_interpretations)). In addition, there is no unique (even less accepted) definition of noninformative prior simply because there is no unique (or accepted) definition of *information*. Even if the estimators are quantitatively the same, the interpretation of a frequentist estimator and a Bayesian estimator are different. As I mentioned in a previous comment "It is like saying that oranges generalise apples."
One wonders just what $f'(x)$ even means in the example.  The usual sense of "noise" is an uncorrelated process, but that's almost surely not differentiable.
I understand. That would solve my problem. But I think that there should not be limitation on the content of folds in cross validation. One fold may contain only examples of one class. The extreme case is Leave one out Cross Validation where the number of folds is equal to the number of examples. Thus each test consists of classifying one example.
While others (@Karl, @ttnphns) have directly answered your question, let me answer a question that you did not ask. As an alternative or supplement to the Dickey-Fuller test, you may consider the KPSS test: http://en.wikipedia.org/wiki/KPSS_test, which has stationarity as its null hypothesis.
Thank you. I will work on #1 and #3, etpoisson is not useful, as I read it, because it does not account for overdispersion or a mass-point at zero. What I'm looking for is something like etzinb.
The MLE cannot be considered Bayesian starting from the interpretation of parameters in both paradigms. From a Bayesian perspective, a parameter is a random variable while in the classical setting is a value to be estimated. The MLE coincides with the MAP (and possibly other point Bayesian estimators) in many cases but the interpretation is completly different.
Actually it does a surprisingly good job. For the matrix B in my example, you can very accurately approximate the 50 singular values using the algorithm with k=20 p=35
@Macro I edited the post to (I hope) answer your question.
For problems I am interested in, analytical tractability is not possible.
I think this was my point in stating that using the quartiles is more robust than using empirical mean and variance.
There is not that much difference, in this context, between interaction and nesting.  Whether `B` is nested in `A` or just interacts with it depends on whether the main effect of `A` is included in the model or not.  If the main effect of `B` is *also* in the model then it's crossed ...
would PCA do what you need?
@walkytalky: whooops, thanks !
(ctd) (When did real - i.e. non-simulated -- data ever actually have a normal distribution?) ... With that QQ plot at that sample size, your inference should not be substantively impacted by the very mild non-normality that seems to be present.
I don't think this is correct: it focuses on regression when nothing in the question signals that; if "usually" means "most or almost all of the time", it is empirically wrong, as I would say that significance testing and causal inference are often, but not usually, mixed.
When I said they're related I should have said they meet the standard of matched pairs. In the textbook I have the paired T Test is also described as a small sample test, this is my concern that the test is not valid for N over 30.  Thanks again.
Not 1 exactly, not for truly continuous data; or, rather, the chance of getting 1 exactly is infinitesimal, but not 0. 

It gets into a question of integrals vs. summation signs. 

You can say things about "1 or less", if you like.
Answer is here: http://stats.stackexchange.com/questions/35634/how-to-prove-that-the-radial-basis-function-is-a-kernel
I'm not sure what you mean by "able to be compared with each other".  ordinal: can be ordered; nominal: no ordering.
(+1) In particular, if you happen to not be using centered predictors and you do have an intercept in the model, you probably don't want to include it in the penalization.
Another similar example: if +1 and -1 are equally likely then the variance is 1, but the variance of the squares is 0; if 0 and 2 are equally likely then the variance is again 1 but the variance of the squares is 4.
Are you looking for probability estimates or just counts?
I've noticed that model prediction is quite a bit better if I don't add all variables. Here is a link to the code and references to papers where it's been used: http://evansmurphy.wix.com/evansspatial#!random-forests-model-select/cksm
Are your IVs binary or just integer valued?
Actually first i extracted factors based on Eigenvalues greater than 1, as a result 9 factors were extracted on which none of the items loaded! so i repeated the FA and this time chose "fixed number of factors" and put '8" as the value for the "factors to extract" (because i have 8 constructs in my research model). Then based on the results I deleted 4 items with low factor loadings and then ran the EFA again and this time in "scores" tab i ticked "save as variables" and chose "regression" method.
This question (especially the second part) is rather abstract, which makes it seem vague: would you be able to give an example to clarify it?
+1 It's a straightforward sampling problem and this is a straightforward solution.
Like this: `APdf <- data.frame(Time=c(time(AirPassengers)), Year=c(floor(time(AirPassengers))), Month=c(cycle(AirPassengers)), Value=c(AirPassengers))
; APdf$Month <- month.abb[APdf$Month] ; ggplot(APdf, aes(x=Time, y=Value)) + facet_wrap(facets='Month') + geom_line(data=APdf[,c(1,4)], colour='gray') + geom_point()`. God damn, I love ggplot2.
@dadrivr: MAD is just a robust measure of dispersion, and you have several options. As Bogdanovist mentions, you have to decide what you're actually measuring and how errors are counted. For example, if you calculate the average absolute difference between players' predictions and outcomes, you're assuming all games count the same. You could also weight some games more than others. Or penalize predictions that are higher than the actual score more than those that are lower. It's not necessarily a cookie-cutter solution.
You compute the $n^\\text{th}$ power of the transition matrix and inspect the relevant entry.  For small $n$ this is elementary, as illustrated at http://stats.stackexchange.com/questions/45920. A standard, more sophisticated method diagonalizes the transition matrix: see http://stats.stackexchange.com/questions/87385.
These errors cannot be reproduced without data. Could you also supply a (small) dataset that creates such errors?
Are these *time* series, where each data value follows after the previous?  If so it would change the way you'd go about fitting them to models.
+1, nice answer. Just to clarify something: by FIML you mean the "vanilla" ML maximization criterion right? At least that is what Maas & Hox mean. (I am asking cause I come across the "FIML" term of the first time)
The first inequallity is also studied in [The Tight Constant in the Dvoretzky-Kiefer-Wolfowitz Inequality](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aop/1176990746).
@Placidia is right: `site` needs not to be random, but I see why you (@RobertF) want to model it as such. But also consider the number of sites you sampled from. If they're too few, maybe using a random effect for `site` might not be a good idea.
It seems to me that picking points uniformly and independently at random does exactly what you're looking for.  It sure has a simple likelihood function :-).
just a quick idea: it seems for this question to belong to the realm of Bayesian approach (so may be there will be such a reply from an expert in that field). From the point of view of a regular $OLS$ user, why not to re-run the same estimation with a new dataset (it is hardly believable that calculations take too long time, though Bayesian tricks could be quicker here)?
The t-test assumes normality and (generally) equal variance so that under the null distributions are identical, and tests against a location shift. The WMW is usually used to test a location shift with equality of distributions under the null (but without the normality assumption). However its sensitive to more general alternatives of the form $P(X>Y)\\neq\\frac{1}{2}$ (e.g. see Conover's *Practical Nonparametric Statistics* which discusses that kind of alternative). K-S also has a null of identical distributions, but its alternatives are completely general.
If it is time series data then it is a time series opportunity ....But that's just my opinion. Since I own a hammer , everything looks like a nail !
@Scortchi Overlap refers to the situation where 'used' units also occur in the sample of 'available' units.  That is, the same sample unit appears in the data twice, coded as both '0' and '1'.  So the situation is not analogous to a case-control application of logistic regression.  Overlap is desirable in my case because I am uncertain that 'unused' units would remain unused if my sampling intensity was greater.
I think your intuition is wrong. The chance level is 20000/3 = 6666 hits. So 6750 hits are 84 hits away from chance, whereas 6800 hits are 134 hits away -- that is a large difference, enough to reduce p-value from 0.1 to 0.02.
See @whuber 's answer
@conjugateprior I must not have been clear. I'm not interpreting no observations as zero, the H matrix does not correspond to observations but encodes the linear model that connects the data and observations. A column being all zero simply means one of the states in the X vector plays no role in explaining any of the measurements in z.
[Wikipedia](http://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2) confirms your supposition that the expectation will be zero. The distribution, as a linear transformation of a ratio of sums of squares, tends to be strongly skewed. You might consider beginning with the case of *no* predictors (except a constant) and consider what happens when you include one independent predictor.
Hi Jamie, thanks for your input. A couple of comments:

1. I'm not sure about that. It was, actually, my first attempt as a solution but both my intuition and simulations do not agree with that. The thing is that I actually *don't observe* $X_t$, I observe $R_{t}$; plus, can you prove (arithmetically) that the conditional mean of the random variable $R_{t}|R_{t-1}=r_{t-1}$ (note that it is not $R_{t}|X_{t-1}=x_{t-1}$) is actually $\\phi \\hat{x}_{t-1}$?  

2. Can you elaborate on the application of the Kalman filter to this particular problem?
Feel free to click the check-mark and/or up-vote my answer if you find it useful. :)
"try it and see" - golden words :)
+1.  But--over 100 values?!  The terms will almost surely underflow well before then, at least in double precision. Because the series alternates, the error does not exceed the absolute value of the last omitted term.  Thus, to compute within $\\epsilon$, find the smallest $n$ for which $2\\exp(-2n^2\\lambda^2)\\le\\epsilon$; i.e., $n=\\lceil\\sqrt{\\log(2/\\epsilon)/(2\\lambda^2)}\\rceil$, and compute the sum for $j=n$ *down* to $j=1$ (to minimize rounding error).
It sounds like you really need only to store one tail accurately.  That could simplify your precalculations somewhat.  After all, there's little practical difference between a p-value of 50% and 25%, but a huge difference between 5% and 0.001%.
+1. As a working example of the equivalent function in Excel (just to show how simple this is), set up an array of cumulative sums of the probabilities (e.g., (0,0.1,0.5,0.7,1.0)), give it a name (e.g., `cumsum`), and use `=MATCH(RAND(),cumsum,1)` to generate values 1,2,3,4 with probabilities 0.1, 0.4, 0.2, and 0.3 respectively.  This clearly shows how weighted sampling is related to array searches.
Apologies, mine is also broken now. The ref was Efron and Tibshirani (1993) 'Introduction to the bootstrap'. In it the authors write "The final estimate of prediction error is the apparent error plus the downward bias in the apparent error", with the downward bias given by the average difference between 1) the error when applying the model fitted on the bootstrap sample to the original data, and 2) the error when applying the model fitted to the bootstrap sample to the bootstrap sample itself. I hope this makes sense, thanks again.
Ah, OK. I'll venture a guess that mixed models are appropriate.  Your primary concern seems to be that the response is fixed, within-subject.  One way of looking at this is that the within-subject variance is zero. I suspect that this fits in with mixed models as a special case, but here is where my knowledge ends.
+1 Thanks.  I should have mentioned this connection.  The hypergeometric distribution classically appears in sampling-resampling experiments.  The 12 specific people (my "red tickets") are like fish that have been caught, marked, and thrown back into the pool; the sample of 232 is like the set of fish that are subsequently caught.  The hypergeometric distribution describes the frequencies of recaptured fish.
Note that there is an [open-access version](http://projecteuclid.org/euclid.aos/1176345637) of the Bickel and Freedman paper at projecteuclid.
Correlation makes no sense if one of your variables is nominal. Do you mean *relationship*?
I just googled for PCA and missing data and found that: 4.2 How does SIMCA cope with missing data?
Put simply the NIPALS algorithm interpolates the missing point using a least squares fit but give the missing data no influence on the model. Successive iterations refine the missing value by simply multiplying the score and the loading for that point. Many different methods exist for missing data, such as estimation but they generally converge to the same solution. Missing data is acceptable if they are randomly distributed. Systematic blocks of missing data are problematic.
Yes, I have about 15 million rows. That was just an example of the data set. Sorry I should have been more clear. What would you recommend?
These are univariate BLUP. But I'm looking for multivariate BLUP.
could you write your formula in LaTeX? Since you have only exterior parens, right now it looks like you mean $\\frac{5*10^6*16}{1024^2*6}$ which would be $\\frac{80*10^6}{6,291,456}$ but I am not sure if that is what you meant.
If you're generating a confidence interval based on Model 2 for data generated under Model 1, this seems to indicate the two models are related and contain some of the same parameters. Can you explain a bit more? Also, when you say "spurious" in your second bullet point do you mean wrong or just unimportant? Larger numbers of simulations shouldn't produce bias but it could reveal a bias that has little practical importance that you wouldn't see with a smaller number, similar to how you can detect (i.e. get statistical significance for) a very tiny effect when you have a very large sample size.