tttjjj/elt_bench huggingface dataset:
My question was: I have a times series which go from 1980 to 1990, and I want to check if my model gives a good out of sample fit. In order to do so I divide my dataset in two parts and starting from January 1980 I calculate 1 month ahead forecast errors, by dlm (f). Than I want to calculate 12 months ahead forecast errors, so once my t is january 1980 then february 1980, and so on. I would like to know if there's  a way to do so?
I've calculated the critical F-Value according to this: http://en.wikipedia.org/wiki/False_discovery_rate#Dependent_tests (from the paper: The control of false discovery rate in multiple testing under dependency by Benjamini and Yekutieli, 2001).
I'm beginning to suspect that I'd best go about this by creating ***n*** dummy variables after all (rather than ***n-1***), in order to compute the simple mean importance score (as an unbiased estimate of overall importance). By doing so I consciously accept the introduction of perfect multicollinearity, to which RFs don't seem to be sensitive anyhow. Any better / theory-backed ideas are still most welcome, though.
Jim, there is currently a discussion on a very similar question [here](http://spssx-discussion.1045642.n5.nabble.com/Repeated-measures-analysis-of-fractions-summing-to-a-constant-tp5719257.html); you might take interest.
(+1) Good exposition.
${X_{i,j}}$ are Bernoulli random outcomes. Sorry for the inaccuracy. So, ${X_{i}}$ are the sum of scores for sporting teams for sequential equal intervals of time. It turns out that after the first goal is scored the probabilities of the next goal in interval will be different.
Is there a language issue here? What lejohn described is *exactly* the same as this.
Thank you for your confirmatory response as well as your advice. I ended up not using this type of analysis for various reasons (time and resources, included) and instead, presented things descriptively only. I was uncomfortable in trying to make any correlation-related conclusions about trend using only two time points, so I went for the safe route. Unfortunately, it means the data doesn't have much to say towards my research goal. Oh well.
Are these any help? http://stats.stackexchange.com/questions/55571/r-differences-between-lm-and-aov and also http://stats.stackexchange.com/questions/53432/can-ancova-disagree-with-multiple-regression/54448#54448
These are credit ratings, I'd like to know the correlation between a particular rating in consecutive periods.
So if I in every node compute ratio of A class items count to count of all items in node then I don't get probability of class A in the node?
No, it does not. Reliability can be assessed based on a single measure on one sample on one test (e.g. split half reliability) or two measures on the same sample on the same test (test retest reliability). Validity cannot.
It is an interesting approach, but a potential problem is that if you have 5 low-skill experts who endorse the same rating versus a high-skill expert who rate differently, then it is possible that the sum of the informativeness scores of the 5 low-skill experts may exceed the high-skill one, and the rating picked by the high-skill expert **is not taken into account in any way** in the final result.
Alas, this question is off topic here.  It *might* be on topic on SO, but you need to improve the question before they will accept it.
Thank you very much for your suggestion I'll look into the Pankratz book. I updated the question with an example of what I want to check.
The prevalence ratio **is** the effect measure in a prevalence study. Another possibile effect measure would be the *prevalence odds ratio*. [This paper](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1247374/pdf/ehp0112-001047.pdf) discusses the two measures. I've found a funnel plots with prevalences [here](http://www.who.int/bulletin/volumes/90/2/11-091850/en/) (Fig. 3) and [here](http://www.sciencedirect.com/science/article/pii/S1438463913000448#). As the prevalence is a proportion, it might be possible to calculate the standard error if the sample size is given.
Hi, Penguin Knight - thanks for your advice.  With all due respect, most textbooks give transformations for right-skewed and left-skewed distributions, and distributions that are exponential or inverse exponential, etc. without reference to the type of data, nor to the specific type of analysis.  That said, your point re: the type of analysis is well-taken; most of the transformations I mentioned in the previous sentence are intended to make the data act more linearly, since a great many parametric statistical techniques are linear "under the hood".  I will update.
Really - deserves far more votes than it has.
@pchalasani - for a more general overview of discrete choice modeling, [Kenneth Train's book](http://www.econ.berkeley.edu/books/choice2.html) is a very good place to start for both theory and practical applications. It's a fairly dense read, but well worth the effort if you are interested.
..continuing...It may also be the case that you have very "good" data (say p-value of 0.5).  BUT the alternative hypothesis may be *better* (or more consistent) with this data (say p-value of 0.99999 when the null and alternative hypothesis are switched around).
Also, I don't think what you've got here is exactly an interaction.... You've concatenated two y variables and two x variables into one y and one x.
Also sometimes '*a priori* contrasts' or 'planned comparisons'.
This looks like standard bookwork (especially given the start '*You conduct...*' - I don't!). If this is for some subject, or even just for your personal study, please add the `self-study` tag and [read its tag wiki info](http://stats.stackexchange.com/tags/self-study/info)
An F-test can be thought of as test of the hypothesis $\\Delta\\rho^2 = 0$. Can that be used to derive the standard error and confidence interval you are looking for?
@soakley Your approach works for $y(0,1)$, but it's not clear how it would apply to other arguments of $y$.
R has outsmarted you :), it treats the two regressions differently.  In the first case, it takes the first factor and re-frames it as a smaller number of factors, each representing the difference between a level and the first level (so there are 2 coefficients to estimate in your case instead of 3.)  It then does the same for the second.  Hence you get an intercept and 3 total coefficients.  In the second case, it knows there isn't an intercept, so it doesn't re-frame the first factor, instead leaving it as is.  It still re-frames the second factor, though, to avoid the multicollinearity.
Hope I didn't waste time then. Not sure if I have much useful to add. I think if N>P you could use any feature selection method you wanted to (from stepwise to lasso) and just choose what is easiest to implement. It shouldn't be an issue. (1) these methods may include weak predictors that are statistically significant. but since most predictors are binary you can compare coefficients and include strongest predictors (2) leap-bound (http://cran.r-project.org/web/packages/leaps/leaps.pdf) is a popular method for this sort of problem. But I haven't used it myself.
For a start you can treat locations the same way you treat patients (i.e. observations are nested within both patients and locations). This would allow you fit random effects for days/years/whatever within each location.
But isn't that the point of having a validation data set? Shouldn't I just pick a model which has low error on the validation (i.e. out of sample) set?
Hi there, if you have a truly chaotic system, i.e. one that meets the definition of a complex system (see http://en.wikipedia.org/wiki/Complex_system) you will not be able to forecast it using time series.
Good, but $N$ does not have constant meaning. The divisors should be $I$, $J$, $IJ$.
Are the men and women *couples*? That is, are you asking whether marital adjustment for both the man and woman in a heterosexual marriage depends on the social role of each spouse?
Hint: the data counts are 24,5,4,1,2,1,1,0,1,1. The *cumulative* counts therefore are 24,29,33,34,36,37,38,38,39,40.  What do you get when you re-express those cumulative counts as percentages of the total (40)?
You did everything right--but you started out wrong, at least if you're doing [Poisson regression with a log link](http://en.wikipedia.org/wiki/Poisson_regression). There the *log* expectation is assumed to be a linear function of $x$: $\\log(\\lambda) = a + b x_i$ should then be your point of departure.
Great reference to an elegant graphical representation, thank you.
This joke falls flat for me because I can't imagine a statistic professor saying this - a student who failed the course, sure.
+1 for mentioning **complexity** which was not directly mentioned by the top answers.  However, the challenge arises when explanatory models are used for interventions.  How does one ensure that the estimated coefficients are not biased which is a common problem resulting from parsimony?
+1, there's a lot of good information here. One question, in trying to think through this, is there / what is the substantive difference b/t a hist of ln(data) & a hist of data plotted on ln(x)? Also, re: difficulty interpreting qq-plots, I like to pair them w/ a plot of the kernel density & possibly the theoretical dist.
This sort of statistic is taught about too, just not in the courses you've had. There are lots of discrete distributions. And polling has a lot of complications.
I expect more details (and a somewhat clearer explanation of what kind of answer you're looking for) will be required.
(+1), allowing for one notable exception: when $\\mu_X=\\mu_Y=0$, the distribution obviously is uniform :-).
Preventing someone from entering a value that they believe to be right can introduce more error than detecting and investigating it.
@Nick Stauner The important point here is that we consider "actual minus target" signed, not the "distance" in absolute values. We keep the sign in order to know whether we are above or below the target. The intuition here is, exactly as you write, that "highly motivated" behavior will push "actual" closer to "target", creating asymmetry.
About using priors, I would rather suggest to look if there are some data available for country with small $n$ at the population level (e.g., through census or national surveys).
R's `pbinom()` tells me that you should expect 50 or fewer hits on about 10.5% of all days. Nothing to really pester you with questions about...
You should define what is `significance` for you.
No. This is not right. The *standard normal* distribution has mean 0 and variance 1, but a normal distribution can have any mean and variance.
If you hardly know about R, how do you want to understand th source code? Otherwise I agree with you.
Do you know of any good user guides? the one provided by the developer is... well... non existing. all these solutions (except google refine) seem to be focused on merging two variables. i just need to compair a cell with a group of accepted answers, got any tips? :) And Thank you!
Thanks, Christophe, for formatting my post!
Yes, there are clear spikes in the CCF plot at lags 28-31, particularly the 30th.
@probabilityislogic that's correct - both names are fairly common, though BIC is perhaps the more common of the two.
A permutation test is one way to implement Fisher's exact test and I think that it's the easiest way to think about just what hypothesis that it's testing (i.e., that all the observations come from the same bag and that assignment to groups is just an arbitrary label). You could also use formulas to calculate the p-values directly without needing to perform the permutations, but I don't think that is as clear.
@whuber I just installed R 2.15.1 on my Windows Vista/32 computer, and got the same `false convergence (8)` warning as you report. Anyway, playing with other models based on the same data set it looks like the issue is that there are 3 groups for the purpose of fitting random slopes (similar data sets with 4 and 5 groups converged fine), so I suppose that my question is going to end up as an academic issue.
It would help if you give us more details. For example, how long is the series, what the frequency is, if there's any obvious seasonality/cyclicality, whether the volatility appears constant over time, and what the correlogram looks like. It would also help if you described what sort of software you are using.
@Alecos although the description of notation is a little off in the post, its meaning is clear: to understand the censoring, we contemplate the (hypothetical) *uncensored* data (giving the second likelihood, where the $z_i$ are uncensored observations governed by the distribution $\\theta$). After censoring, the likelihood equals the first expression. The indexing and naming of variables may be a little problematic here; I prefer to use unique indexes for the variables, as illustrated at http://stats.stackexchange.com/a/49456, because it helps avoid confusion.
That sounds more as if you want to do **classification** with respect to the clusters you obtained earlier. Which is very natural to do with decision trees, actually.
Thanks, that makes total sense. Will definitely have a look at 'tests of trimmed means, medians' etc. The sample will be quite small, so describing the statistical tests with that in mind is probably important. Thanks again.
I am sorry @user11852, yes you are correct, thanks. (BTW, for `m2` it is valid also (which is subject of [another question](http://stats.stackexchange.com/q/65386/5509)).
There are examples in the help files. Please take the time to read the documentation.
@StuartRobbins  The likelihood is just a given function of the parameter(s) using the observed data. You find its maximum which gives the maximum likelihood estimates.  For the Akaike criterion plug in the maximum likelihood estimate(s) inot the likelihood to get the value of AIC.
I agree it's a bit inept to recreate the ADF-test by hand. It was more an exercise for me to get used to the language of ADF-tests. Thanks for the help, very much appreciated.
(+1) But, note that it is not (quite) true that if $X_1$ and $X_2$ are orthogonal, then their individual $R^2$ values will sum when including both in the model. We *also* need them to be orthogonal to the all-ones vector $\\mathbf 1$. Note that you can use $\\LaTeX$ on this site for marking up the math. :)
For illustration, here are some additional papers: http://j.mp/cBJ0hI, http://j.mp/deMrA8, http://j.mp/dhwcrv, http://j.mp/cyVx0m.
In your answer, you mentioned that "The "advantage" of working with a kernel is that you change the usual "Euclidian" geometry so that it fits your own problem. Also, you should really try to understand what is the interest of a kernel for your problem, what is particular to the geometry of your problem." Can you give a few references to start with. Thanks.
That makes sense. Based on which metrics can I identify the "best result"? Would those be the residual sum of squares, or the autocorrelation function of the residuals?
Can you post an example graph?
Just a minor note: I'd rescale with the sum of the values, rather than the first value, if this is a set of counts in disjoint bins (as I inferred in chat).  If it's a true CDF, then the maximum value (as you've done) seems reasonable.
Cells with zero count are not permitted in a chi square test? See my question here: http://stats.stackexchange.com/q/73708/2824
I have added $\\TeX$ for you. You wrote "sigma" but mentioned that these were variances, so I made them sigma-squared. Make sure it still says what you want to ask.
Cramer's phi = 0.1573928, Chi^2 = 45.4823
thanks a lot for your comment. I'm not very fluent in SEM though I can guess how a model like what you suggested might work. I'll have to spend some time on it to see how I can build and evaluate such model.
@ocram, no ONE price and ONE date fro about 700 days.
[Here](http://bayesianthink.blogspot.in/2012/12/the-best-books-to-learn-probability.html#.UldEJdJHJBE) is good list of books for non-statistician scientists. Most of them are probability related and some are readable for ones with a non-statistics background. HTH
@chl .. I agree, I would have would like to give (+2) :-), actually I never thought so deeply about this question until this update.
Remember that the log of a ratio is equal to the difference of the logs: $\\operatorname{log}_{k}(a/b)=\\operatorname{log}_{k}(a)-\\operatorname{log}_{k}(b)$, where $k$ is the base ($k=2$ in your case). Maybe that helps you figuring out if this is what you want to calculate.
$\\gamma (h)= \\text{Cov} (X_{t+h},X_t)$ where $h$ is lag difference. and ACVF stands for Auto Covarianve Function
@Livid, your code simulates a Hidden Markov model, which is very well known. I describe HMMs in my answer.
This is because the assumptions you've mentioned all refer to the error term in the regression model and not the original variables.  The residuals represent the unexplained variability in Y (that is not explained by X), this is the error term (e) in Y = b1 + b0*X + e. When you look at a QQ-plot of residuals (for example), you are comparing this error term against standard normal counterparts.
@Karl Oskar The probability that there is _defect growth_ and the probability that _the defect is detected_.
Consider reviewing what you know about regression. If you were *told* the correlation, as well as given most of the data in the question--$88$, $64$, $8$, $45$, and $5$--could you compute the prorated value of $55$? If not, what additional information do you feel you would need?
Thank you for help.
If I read correctly, it says "linear model [)] in which expected predictive accuracy is independent of within-profile variability." If I understand correctly, the rest of the page deals with predictions made by humans.
@cardinal, I will look into the book. Could you point me specifically to this problem? Thanks.
The first point can be partially addressed by *membership query synthesis*, since the strategy generates the instance from scratch.
@user3671 I agree with you.  The p-value output in the table you display appears to be the usual frequentist p-value and there would be no reason to look at it in that context when doing Bayesian inference.
"Is there any problem?" is too vague. What do you want to do with these data? Your huge kurtoses are associated with huge left skew. Since log(p2/p1) = log p2 - log p1, a huge left skew indicates that there were a few times when this was very low, that is, p1 much higher than p2, compared to the usual case. Could be a company going bankrupt or something like that.
@whuber But this is the OP's work (to check and diagnose), isn't it? We do not posses the data, neither any specific information about the whole situation, the OP does. And my answer explicitly starts with "If we can assume that $X$ and $N$ are independent..."
I also calculate 1/80,000.
I have edited my answer. I hope this clarify those points.
There's no proof in statistics except that proofs in mathematical statistics should follow mathematical standards. However, such proofs are irrelevant to your question. In terms of your example, no-one except an idiot or humorist claims that each American family has 2.4 children. At most, the claim is that, on average, families have 2.4 children; that can be checked by sample surveys or another census, but as the number may fluctuate or change systematically a different result doesn't mean the first result was incorrect; nor does the same result mean the first result was correct.
Might as well use Pearl's word for "directly setting [a variable's] value": an intervention.
You might be better off asking on http://gis.stackexchange.com .
I assume you're familiar with [Stigler's law](http://en.wikipedia.org/wiki/Stigler%27s_law_of_eponymy)? (To be interpreted a bit more broadly than as stated.) :)
You can't judge normality from the QQplot when the residuals exhibit curvature and the variance isn't constant; with those other assumptions violated the normality isn't readily assessed.
The scipy page you link to describes describes *exactly* how to make it two tailed. And the test being two-tailed essentially has nothing to do with whether it explicitly tells you the direction of the effect. You can do that by inspection (e.g. by comparing the U to its expected value under the null)
I feel bad for having missed this earlier, but there are a lot of good answers here: http://stats.stackexchange.com/questions/5235/what-is-the-expected-correlation-between-residual-and-the-dependent-variable?rq=1
No, $R$ did not do anything "wrong" in this instance. This comes down to understanding that the SVD is not unique. The vectors are invariant to common sign flips, and this is precisely what happened here.
About 10 comments appeared in the 20 minutes it took to work out the details and write this down, but I hope it still contributes towards the original question.
If they all have the same mean, the sum of their squares is a scaled non central Chi-Square distribution with the mean as the non-centrality parameter. If they have different means, I know of no nice closed form solution.
It isn't just about how to close a program, but how to deal with a particular issue when analyzing lots of data using a software popular among statisticians. Therefore, although the question is about scripting/programming, it's also about how to use a statistical tool more efficiently.
I see, so i probably should read through this: http://en.wikipedia.org/wiki/Marginal_distribution in my statistics books =)

Thanks
@RomanLuštrik: this is a *very* hesoteric definition of outliers. For example, it is not consistent with the view of outliers used in Glen_b's answer (or for that matter with the definition of outliers used in textbook on the subject such as "Robust Statistics: Theory and Methods"). Care to cite a source to support your definition?
Yes, but that's because the cycles are not strictly periodic. They are almost periodic, and as you go further out in the forecast horizon, it becomes harder to predict which part of the cycle you will be in. The point forecasts are means, and so they will naturally flatten out, indicating this uncertainty.
The issue of standardization (there is much more ways possible than those three) is entirely out of clustering scope. It is the question of how to pre-process the data in order to make the chosen distance measure (yet a separate question, but partially tied with clustering) to reflect just what you want.
Looking at the residuals is good. What I was referring to is verifying that the distribution of the random intercepts are also approximately normal, which you should also check.
What happens when you do the logistic regression with just the dichotomous predictor? If you don't get p=1 then, which variable do you have to add in order to make p=1?
bmx, It looks to me like you have a clear idea of your question and a good awareness of some of the issues. You might be interested to review three closely related threads. http://stats.stackexchange.com/questions/17773 describes prediction intervals in nontechnical terms; http://stats.stackexchange.com/questions/26702 gives a more mathematical description; and in http://stats.stackexchange.com/questions/9131, Rob Hyndman provides the formula you seek. If these don't fully answer your question, at least they may give you a standard notation and vocabulary for clarifying it.
Hi castielle and welcome to the site! The sample variance is defined as $s^2=\\frac{1}{n-1}\\sum_{i=1}^{n}(x_{i}-\\bar{x})^2$. So you have to square the value *after* you subtracted the mean (which equals $100$ in this case). In Excel, you can use the function `VAR.S` to calculate the sample variance. Alternatively, you could make use of the following relationship: $\\mathrm{Var}(X)=\\mathrm{E}(X^2)-\\mathrm{E}(X)^2$.
The lowest AIC/BIC is obtained on the 17 variable model - so this is the best? Shouldn't I do some calculations to check if the 'additional AIC or BIC' is worth an extra variable? Also - would you use LASSO to get several models, subsets regression, or neither? I used Subsets to create 6 models but now I'm wondering if its better to use LASSO? The question also is - is this method even correct for justifying which model is the best 'compromise'?