tttjjj/elt_bench huggingface dataset:
:Andy Exactly !the process is filter/convert X to x where x is white noise.Apply this filter to Y to get y;use cross-corr on these suitably stationary/transformed series to IDENTIFY a suitable model between X and Y.This is done separately for each X in model on the premise that the x's are independent of each other.Careful model RE-IDENTIFICATION enables one to "correct" the form of the transfer between and X and Y. Ultimately one must identifu both the ARMA component in the TF model reflecting omitted Stochastic Series and the Pulses/level shifts/trends refectimg omitted deterministic series.
My opinion is that it is justified to use this version of the DIC, but it would also be desirable to strengthen the conclusions with the use of Bayes factors and perhaps BIC.
@Andy Can you help?
Does this method apply only for Gaussian distributions (as specified in the question), or can it be used for generating correlated variables that follow other distributions? If not, are you aware of a method that could be used in that case?
Completely agree about focusing on the parts of the curve that are sensible for the application, especially when deciding between different models.
@cardinal - good point, but the multivariate generalisation is not polemic, just follows from using a multivariate guassian mixture model, and doing the same "splitting" of the data into "location learning" and "scale learning".  Same with the choice of components.  All the necessary principles of what is being done are there - Jose Bernardo just hasn't done the maths.
I think that was implied by the answer. You perform the post-hoc tests after the signifcant ANOVA.
@Corone 10K results could easily be recorded on one standard sheet of paper using, say, a dot and bar code (as in `||||..|....||.|..||.` etc). This can be compressed using (for instance) hexadecimal. In the previous example, letting `|` be 1 and `.` be 0, the hex representation is `f21a6`. By writing small but visible characters I can easily fit 50 such characters in one line of writing and 50 lines on a sheet, thereby representing a sequence of 50*50*4 = 10K outcomes.
@PeterRom. I have explained my situation in detail. Actually I applied PCA twice so it seemed like the data had mean of 0. The original features in the raw data had different scales
For any reasonable sample size, $1/n$ is negligible compared to $2/\\sqrt{n}$ so it doesn't matter much. I've corresponded with Andrew Metcalfe and he acknowledged the error with respect to R. I guess they haven't updated the errata yet.
I've seen your post on r-blogges. Thanks Michal.
Is the reasoning here that the ratio will no longer properly fit the Poisson distribution?
could you point out where the book discusses time series? I just searched through it and could not find the topic covered.
By construction the interaction of $p\\cdot trt=1$ if both $p=1$ and $trt=1$. The coefficient $B3$ gives you the difference in differences estimate.
Since this is explicitly a standard textbook type problem I've taken the liberty of adding the `self-study` [tag](http://stats.stackexchange.com/tags/self-study/info).
@whuber I agree fully. Considering this is published and maintained by the IEEE, I expected a whole lot better. I'm just wondering if they are handwaving a bunch of stuff because it's a software engineering certification, and they don't want to get in too much depth in other things. But that's no excuse for some of this confusion.
The (unsigned) distance between two points is a nonnegative quantity that may be difficult to model as a Gaussian random variable, and this is supported by the fact that the square of the distance is (at least in the independent case) a non-central chi-square random variable, whose square root is not a Gaussian random variable.
Many thanks! I spaced on this one...
Are you asking how to design an experiment to determine this, or do you already have the data? Do you only want to know how to get SPSS to do this (in which case, the question is off-topic, per our [FAQ](http://stats.stackexchange.com/faq)), or are you wondering about the statistical methods involved in assessing this issue?
@ablimit Your problem will just likely average out =)
It simply means: you should add some small positive constant to the distance matrix. Add, then do double centering to check if its eigenvalues has become all nonnegative. If not, add again, etc. repeat. (Please remember that double centering is done on _squared_ distances. But you add a constant to no-squared distances. Also note that eigenvalues of doubly centered distance matrix always has one zero eigenvalue; you check that there is no negative ones.)
Since you are using a form of regression model, the distributions of your variables *do not matter.*  What matters is the distribution of the *residuals*.  For more on this (extensively discussed) topic, please [search our site](http://stats.stackexchange.com/search?q=regression+distribution+residual).
I don't know if that's really applicable to qq-plotted data (see also my comment on my question as to why this won't work for my specific case). Interesting point though. I might see if I can get it wo work on individual models vs obs.
"The parametric method chooses an origin O (intended to be the mean)". I would said that the mean is more related to the mean of the gaussian from which are drawn the $(y_i)$ and that the value of O is more concerned by the equation of the hyperplane. Am I right ? (maybe it is just a question of vocabulary...)
Are you sure your model is not overfitted? PCA sometimes can generate nice, but purely artificial clusters.
From your comments this is apparently for some subject. Please add the 'self study' tag.
Its simpler than that. $\\hat \\theta$ is a deterministic function of the observed data, so it is not a random variable if you already know what data you have. If all you know is that the statistic has a particular value, then, as you indicated, there are only so many combinations that allows this, one of which will always be the data that you observed and used to calculate the statistic. So yes, you can view the observed data as a subset of the possible data that could have resulted in the observed statistic value.
not long at all, I am comparing different methods, although I was not sure whether correlating with main components does make sense in terms of time series data. On the other hand the major trends are indeed orthogonal hence PCA should give me solid results
Made a few additional tweaks to improve it.
If the two measures differ substantially, then a linear regression might describe the data in a quite unnatural way and you could think of adding a quadratic term or transforming the x variable. Instead of looking at rank correlations, the scatter plot usually tells you the same...
The question was answered in Math Overflow. See the following link for more details http://mathoverflow.net/questions/123875/moments-of-the-kolmogorov-distribution/123895#123895
That is a good point and seems to me superior to the strategy I outlined (as improved/clarified by Stef). Ultimately I followed [Efron (1994)](http://www.jstor.org/discover/10.2307/2290846?uid=3737536&uid=2&uid=4&sid=21102523158061), which is very close to your reiteration of Shao & Sitter (ie imputation comes after bootstrap sampling not before).
(-1) You didn't tell us what you know about sampling distribution, confidence interval and estimation, and you didn't bother reformulating your homework so that it could still be useful for future readers.
You can't take the difference of categorical variables. Please clarify. Do you actually have 163 team pairs in your data?
Thank you for this very comprehensive answer.
Can you explain why you think that entry order is important?
@Learner: Well... I have not read the document, but $\\theta$ is the standard notation for a parameter. Now, a parameter is a fixed quantity, not a random variable, and it does not make sense to compute $P(\\theta)$... Could it be a typo ?
I think these ideas would often be reasonable, but they may not be right in this case. The 'outliers' identified in the Cook's distance plot are contiguous w/ the rest of the data; the model is more likely misspecified than it is that those data come from a different distribution. Adding $x^2$ would be indicated if the res v fitd plot were of a single variable, but may not be w/ 2 vars.
You should be looking for a _likelihood_ of $N$ not a _probability_. That's why there is no PDF. One should be able to easily calculate $L(N|m,p) = P(m|N,p)$.
@gung Agreed. The problem I anticipate is that there may be *too many* statistical questions here, ranging from how to conduct hypothesis tests with various kinds of variables, whether to do that at all for exploration or summary, whether or how to correct for multiple tests, and more. That is why I encourage you, Joe, to clarify and focus this question lest it be so broad as to be unmanageable! You can always ask followup questions, so there's no need to address everything in one go.
I am sorry, this I don't know. I stopped using MINITAB over twenty years ago. Someone else will probably be able to offer some advice.
Parzen window based classifiers are probably what you're looking for.
Let's assume for the sake of simplicity that you're dealing with constant probability of complication. Then the expected number of complications in each group is on the order of 2-4, but the standard deviation of the number of complications is typically more than half that! To show meaningful 'non-inferiority' you'd need a CI for the difference not to be really wide.
@whuber: E.g. $\\mu$ is the mean of all normal distributions, i.e $\\mu\\in R$. Of course answering to this question may bound it. So as far as I know, we can discuss correlation. By "given", I didn't mean they are one number fixed.
@MatteoDeFelice, you are welcome. That's the main attraction of this site: the solutions remain on the internet forever :) BTW thank you for reminding me about this answer, it turns out this is useful for me too.
Thanks a lot for the kind clarification
@EmilioA The answer to that will depend on your data, but you're most likely looking at using some form of linear regression. That's likely best suited to being a separate question.
@Cam.Davidson.Pilon - the OP linked to the article disputing the rule of thumb (http://aje.oxfordjournals.org/content/165/6/710.long) with some bibliography at the end.
I am thinking of the common situation in labs where the triplicates are simply three different test tubs (or wells). The order of the three as presented in the table is arbitrary. There is no connection or correlation between replicate #2 in the first experiment with replicate #2 in the second or third experiments. Each experiment just has three measurements. So not really repeated measures. Right?
Correct, but I meant sort of at intuitive level.

The exception is that for example, if you have the array
[1,100,1,100], the autocorrelation will be negative, because the size jumps up at down, while here I want it to be weakly positive.
Think about the hypotheses behind the p values provided by "summary". Are such comparisons to baseline categories really of key interest to your research question?
"Moments" is a bad name they got stuck with, admittedly. The method in fact works with estimating equations, and I hope you do see some in this example, @whuber. To rephrase, the GMM theory covers anything that can be done with the quadratic loss for estimating equations, including higher order asymptotics and weird dependencies between observations or equations.
No, it should be Bin(12,p) since you are comparing it for 12 new data points. See http://en.m.wikipedia.org/wiki/Binomial_test for more on the binomial test (but make sure not to use the large sample version)!
Great answer, thank you Greg. So assuming you don't know much about the variables and just want to see who correlates with who, and so each cell in the corr matrix is of interest, which method would be most suitable: Bonferroni (e.g. divide by total number of pairwise correlations), or control of Family-wise Error Rate (FWER)? In fact, how would a 'family' be defined in the latter case?
Well, what the technique is and whether or not it's the best technique for the job are two different questions...
@David J. Harris: "Notoriously low power"?  For sample sizes of 650? This is contrary to everything I have read or experienced.  Do you have a citation?
Edit your question to include this explanation. Please explain more about how the preference is collected: is it a forced ranking, just strongest preference, scores for each option, etc.
Depending on the context of your study and the number of variables, I would say that variables A and B 'contribute' to a certain extent to the 2nd dimension of your PCA, or maybe drive that particular component. When set as illustrative variables, the first PCs are linear combinations of other variables that are poorly correlated with either A or B. I would suggest to look at PC2 (with A and B) more carefully to see what this dimension might mean, or share with us a reproducible example or illustrative sample dataset.
I've made it clearer by now
I'm not sure I exactly understand your query. Are you asking me if I am contradicting StatSoft? Their answer is wrong, as Peter has mentioned above.

Edit: The reason the StatSoft definition is wrong is because it is trying to get at the Neymen-Pearson approach to significance, which is different than the Fisherian approach (P-values). The correct, formal definition is the one I indicated above and the one by Wikipedia. (The papers elaborate more on this, definitely worth reading).
Given that you have *data*, could you explain what you mean by the test not being "statistical"?  What kind of test do you have in mind then?
Your description seems unnecessarily complicated.  If I follow correctly, you are describing sampling without replacement from $\\{a,b,c,d\\}$ with the given probabilities. With this simpler characterization in mind, can you identify anything that would cause the chance of any outcome to vary with position? You might consider drawing a probability tree... .  BTW, http://stats.stackexchange.com/questions/20590/how-do-i-sample-without-replacement-using-a-sampling-with-replacement-function looks like a strikingly similar question.
Google "standard normal distribution".
Can you give some more details of your approach? For example, what do you mean by 'overall rating'? What variable are you trying to predict, and what are your independent variables?
A couple of questions: Why do you think that quantiles are inappropriate for normally distributed data? What do you mean by "equivalent of the boxplot for normally distributed data" (i.d. say the equivalent is the boxplot)? Would you care to explain what you mean by "ideally, something based on confidence intervals" (confidence interval for what)?
As indicated in the above comments, choose one SE site and don't cross-post. As this is merely a technical issue (about which I have doubt you get a working solution---if R crashed, you're likely to loose everything unless you write to disk from time to time), I'll suggest keeping it on SO.
Where exactly does this value of 1.04 come from? Is it perhaps the Durbin-Watson statistic?
thank you, great answer!
However if a later step assumes that lambda is estimated without error, then all confidence intervals will be too narrow and P-values too small (i.e., confidence coverages will be far from the claimed value such as 0.95).  A unified "estimate lambda while estimating the betas" approach is needed.
'test' and 'diagnosis' aren't the best labels here, I think. Maybe "Ill" and "Well" or some such?
Given this is a homework question, you should add the `self-study` tag.
I write comments in my role as moderator: where there is some potential for posts to be misinterpreted, my job is to point that out and request clarification. Of course my interpretation could be incorrect, but the important message is that miscommunication is happening. Instead of accusing your readers of ill will or stupidity, please bear in mind that good communication requires honest effort by *both* parties and try to use what I have written to improve your post.
+1 Agree with this interpretation about it looking like DIY Arellano-Bond. NB: I have found Arellano-Bond to be trustworthy only when the number of cross-sectional units is very large---as in, many hundreds.  Arellano hints as much in his articles and textbook by indicating that the consistency is in the number of cross section units, and the convergence rate is not all that fast.
@JoseRamon, I can't answer these questions, you have the data. You can run the statistical tests to see whether the difference is significant
@redracer Looking at your past questions, I see that you never accepted an answer and tend to resist the advice you are given. What you are asking (a univariate test of an inherently multivariate hypothesis) makes little sense to me. Again, why don't you want to consider multivariate tests?
Perhaps something along the lines of a partial residual plot (whether in the form of a boxplot or whatever)?
I was of the understanding that in glmm, the points within a grouping variable (in this case would be subjects) would be expected to have a dependent relationship? I might have a misunderstanding of glmm.
You have started out correctly. Now think of an event 3 - which is the event that you miss work (i.e. that both events 1 and 2 occur). Forget about events 1 and 2, and concentrate only on event 3 which has probability 0.067 as you stated. Each day, you either miss work or not with probability 0.067...Now think of coin tossing probability calculations you may have already done...
Median test is a particular form of Pearson z-test for proportions. It tests H0 that the groups do not differ in respect to the proportion of cases lying above (or below) the value which is the overall median (or another quantile) of the combined sample. http://stats.stackexchange.com/a/16226/3277. So, the asymptotic test has no assumptions besides usual [Chi-square test](http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test) assumptions (such as >=5 expected count in a cell). The exact version of the test has no assumptions.
Mortimer, I am not sure exactly what you are trying to do. But I am suggesting that if you know that "javaisland" + "java_island" represent 2% of your population, you bootstrap sample from your 500 sample, say 10000 times.  You obtain a confidence interval for your "javaisland" + "java_island" term frequency of 1.5% plus/minus 1%.  Since the confidence interval contains the population mean of 2%, you have no reason to suspect that your sample is not representative for the term "java_island".
Hmm.. I get the attributes-consideration-for-finding-results-based-on-clusters part..but The data Ive given isnt a dataSET per se...Each row is a session..And each column represents the number of times that particular entry has occurred in that session. So is this data useful enough to form clusters?
"The $r^2$ is a function of both the slope and the intercept." Maybe I'm missing something but... isn't it just a function of the slope? Maybe you could provide a concrete demonstration?
I can buy that.  Data mining is more exploratory application of statistical techniques.  Though, I don't think that distinction is enough.  When I'm doing EDA on my set of 100 observations from a designed experiment, I don't think anyone would call that data mining, would they?
@MarkDollar: Yep!
I've added the time series tag, and you should check out all the (vast) time series literature because this looks a standard problem.  You will have to deal with the seasonality and the well known problem that correlations between non-stationary time series are often spurious (generally the solution is to transform or take differences or growth rates so the series become stationary).  On another point, just from the plots you've put up here I would have high hopes of building a model with rainfall as a continuous variable, I don't see why you would have to split it into three categories.
Suppose I have $p = 100$ binary variables. Then there will be $2^{100}$ possible combinations, and effectively $2^{100} - 1$ free parameters. Counting is out of the question because I'm not going to have enough data to estimate all $2^{100}$ possible combinations. An appropriate graphical model, however, will drop the dimensionality down substantially (the amount by which depending on how densely connected the graph is).
How can the scores simultaneously be i) normally distributed and ii) between $[0,1]$? Why is the final measure required to be normally distributed?
So, one of the variables would have to be excluded from the regression, as in the solution to the dummy variable trap. The interpretation of the beta coefficients would then also be relative to the excluded variable. That is, if $X_i$ is excluded, $\\beta_1$ measures the change in $Y$ based on a 1 percentage point change in $X_1$ relative to $X_i$. Correct?
BTW, I just came across a paper that made me think of this question...it's very rare for Machine Learning papers to talk about confidence intervals, but here's a notable exception http://www.ncbi.nlm.nih.gov/pubmed/19519325
That's a really great idea. While I side with @Aniko's answer because it addresses my question directly, your suggestion is definitely more useful in the short term. I think it will take me some time to wrap my mind around sequential strategies.
I suspect you think your data are *correlated*, not [*autocorrelated*](http://en.wikipedia.org/wiki/Autocorrelation).
I'm not completely surem, I mean random forest is completely non parametric so don't worry about assumptions. What I'm not sure is if it will serve your purpose. What I CAN say is that there is a version of Random Forest just for "anomaly detection" called isolation forests: http://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf?q=isolation there was an implementation in R but i'm not sure if it's up and running as of now.
Hi, thanks for getting back to me. I've done some checks. The omega != 0 model yield a more satisfactory fit: H0 can't be rejected for the Standardized Residuals at various lags (1,3,7). The same is true for the standardized squared residuals. The null hypothesis for the ARCH LM test can't be rejected for lags 2,5,10. Sign bias test also yields high p-values. The model with omega = 0 fares worse: Testing the standardized squared residuals, H0 gets rejected for lags = 3,7. H0 for the ARCH LM test also gets rejected for lags 2,5,10. This suggests I should stick with the omega != 0 model, right?
Thank you for those references.  Would you mind summarizing what they say?
I would agree that Cross Validation solves the problem of measuring model complexity. Maybe I'm asking the wrong question, because a practical question is the sample complexity of the fitting procedure. Cross-validated learner would try different models and pick the one with lowest cross validation error. Now the question is -- is this learner more likely to overfit than one that fits a single model by maximum likelihood?
@DavidP Look at it the other way around: Frank Harrell wrote a (great) book (that's his job, he is a professor) and instead of being content with a high-level exposé, he also went to the trouble of developing software to implement the methods in his book. Would you rather have this software be a standalone package or only available to the book's buyers? It's not that it's not legitimate to wonder about the differences but it's in fact quite common for R packages to be paired with a book. Personnally, I prefer it that way than to read books with no implementations or only SAS macros.
Nice answer.  What is unique about time is that it passes in one direction, and once we have withstood a high-risk period we are mainly interested in risks now in effect.  That is what the hazard function tells us.
If the odds ratios are approx. equal, you will state that you found no evidence of statistically significant differences. You can't prove that something doesn't exist. Look at an extreme - how do you prove there is NO diety? You can't, you can only offer evidence that he/she/it exists.