tttjjj/elt_bench huggingface dataset:
Two things: (1) consider an AR(1) with mean 0 is y_t=$\\phi_1 y_{t-1}+e_t$ - you can't fit an AR(1) to one point, you'd need 2 even with no mean and no error df - so you need more that $p$ to fit an ARIMA(p,0,0). So it *can't* be $p+q$ for an ARMA(p,q). (2) With an integrated series, you'll lose $d$ in the differencing as well, so if you're fitting a mean, and estimating $\\sigma^2$ to be anything but 0, you must need - at the very least - $p+d+q+3$ observations, and even then you'd have only a single df for error. You might be able to get a fit for $p+d+q+2$ but it would be pretty meaningless.
Sure. If it's okay with you, I'm going to edit my answer so there's more room (and I can LaTex things).
@MånsT: Thanks for the edits. I was just about to make them myself and was happy to see you beat me to it. :)
Correct me if i am wrong, but a PCA assumes a linear model right? In fact my goal is to analyse the sensitivities of the data that are left once i would have subtracted a multi variate linear regression model from the data. I would like to quantify the variance contributions due to nonlinear, and interactions effects, by using a variance based sensitivity analysis method. I think the fact that i want to investigate especially the nonlinear part, would be distorted by doing the PCA. The physical inputs are uncorrelated, but the sampling points are not, can could i clean my samples by filtering?
@rasole, glad it helped.  One request: rather than re-posting your question on the Statistics site, please click the "flag" button to ask the moderators to migrate your question to Statistics.  Cross-posting the same question on multiple StackExchange sites is frowned upon.
Just to be sure I understand even more, since I am no expert, you are using a truncated density.  The truncated density starts at our value $k$ but the actual density would only integrate to $1 - F(k)$ if we integrated it from $k$ to $\\infty$.  Thus, we divide the truncated density by this value so it integrates to 1.  Similarly, if we truncated it above at $k$, we would use the original density, but only defined from $0$ to $k$, and divide that by $F(k)$.  Is that all right?
Are you trying to choose the best subset of predictors? If so, this falls under the general category called "variable-selection", which has been discussed on this site a lot - http://stats.stackexchange.com/questions/tagged/feature-selection. But, the sentence *What I would like to measure is the impact/significance of each of the other variables on the qty sold.* makes me think it may suffice to simply look at the marginal relationships between each variable and the response, as not mention of the "joint effects" is made.
You said -- "*the OLS coefficients are still BLUE. So this method is still the "Best"*" -- but 'BLUE' is telling you it's 'best among unbiased linear estimators'. The problem is that if your response's conditional distribution is sufficiently non-normal, *all* linear estimators may be arbitrarily bad; in that case, being the best among them is a *very* low hurdle.
@Bogdan thanks for your comment. I knew that qqline went through the first and third quartiles, and was asking why this particular line should be used, rather than a different line (such as the ols regression line).
Not your question, but your regression model is multiple (not multivariate) if you have just one response (outcome, regressand, dependent) variable.
You might have a look at partial correlations http://en.m.wikipedia.org/wiki/Partial_correlation e.g. implemented in R package 'ppcor'.
This question leads me to believe that you should hire someone to help you. Survey data is generally garbage in garbage out. DIY surveys are usually only directional at best.
Thanks for your answer - it's very helpful.  One point of clarification: the sampling took into account only x-ray status, because cultures take weeks to produce results.  Would I then build a model each for groups 1-6 and 7-9?
Not bad at all. The format is static, but the information is very useful.
Thanks! that was the problem!!!
I didn't criticize the use of a prior. I mentioned as a sidenote that there might be more appropriate priors than the uniform one (e.g. Jeffrey's), but this is only marginally relevant to the question. Your solution was perfectly fine, just not as useful to me since it doesn't generalize easily.
Because its embedded in our hypothesis.
What is the purpose of clustering in this case (i.e. what are you trying to find from this data)? Are you looking for sites with similar attributes over the entire 2yr period? What if some attributes match very well and others do not, would these patterns be of interest as well?
(hit the character limit - sorry for the double post) I will take your advice and quote the interquartile range, as I cannot see how that can be misinterpreted. Thanks again for your help
You posted a question relating to your use of `poly` without typing `?poly` in R first? That says '**Compute Orthogonal Polynomials**' at the top in large friendly letters.
Comparison of algorithms is not something you can do on a single set, especially cherry-picked for a particular outcome.
That request is a bit irrelevant to this response. The parametric and non-parametric approaches to variance-component estimation are on polar opposites of the spectrum. 

Another caveat: unlike the t-test, inference on variance components isn't robust to distributional assumptions, so data which are non-normal will have standard errors that are very miscalibrated using the exact normal sample model
Do you have any reason to believe your values should be normally distributed? Dixon's Q-test is based on normality, while real data is often non-normal.
Not directly an answer to the question, but you might be interested in the pgfplots LaTeX package.  It allows you to embed plot code directly in a LaTeX document.  Thus, it can cope with arbitrarily complex LaTeX math expressions, and if you're going to wind up putting your gnuplot plots into a LaTeX document pgfplots would be one less step.
What is the sample size for the smaller of the two outcome categories?
Just make sure to clarify the proof of Theorem 2 in the arXiv document: the displayed inequality is written the wrong way. i.e. one should accept $H_0$ iff $P(\\theta\\in\\Theta_i|x)>\\alpha/2$, not the opposite!
Frankly, I think this is a *disastrous* idea, just about guaranteed to lead to many false conclusions.
I would definitely recommend first reading about supervised and unsupervised learning and the definition of labeled and unlabeled data should come naturally. Then move to semi supervised learning. You are skipping some important steps here!
Are you sure you mean: _standardize_ and not _normalize_?
Please expand your question to include some context, otherwise it is impossible to tell what problem you are considering, and, thus makes it impossible to provide advice. Please see the FAQ for advice on how to ask questions on this site.
+1  This paper exploits information about how the number of apparent modes depends on the kernel width (for Gaussian kernels) to select an appropriate width and then continues as described in my reply.
Consider any $t < \\infty$. Is $\\text{Var}(y_{t+1}) = \\text{Var}(y_t)$?
(The article is "Description of the Entropy(tm) algorithm as applied in the Datex-Ohmeda S/5(tm) Entropy Module", H. Viertio-Oja, et al, Acta Anaesthesiol Scandinavica, 2004, 48:154-161.)
Welcome to the site. This looks like homework or an exam question. If it is, please add the `self-study` tag.
Also as a note to the poster you can not estimate the shared frailty in SPSS as in Thylacoleo's example. You can only include covariates in the Cox regression (either time varying or static).
(+1) You've beaten me on this!
By baseline do you mean a classifier that always says 'positive'?
I am predicting where people will go at intersections, so there are an average of 3 branches, hence .33 chance accuracy. However, each prediction is correct or incorrect, hence the binary.
I made this way harder than I needed to, didn't I?  haha.  Thanks for the input.
Fitting a Gaussian to data like these is described at http://stats.stackexchange.com/questions/70153/linear-regression-best-polynomial-or-better-approach-to-use (including `R` code). Least squares fits *are* maximum likelihood estimates for Normally-distributed residuals. As Nick Cox points out, your residuals do not have such distributions.
Precision is different from variance but related.  The prior is put on precision.  The data with the prior leads to a posterior on precision and from that we derive the inference on the varinace.
Welcome to the site, @EngrStudent. Would you mind saying a little more about the GINI coefficient, & why it's the right answer here? Since you're new here & starting to contribute, you might want to read our [FAQ](http://stats.stackexchange.com/faq), which contains a lot of info about the site.
Great answer. A quick simulation seems to support the proportion in your example.
Ok, but "looking for information on a topic" isn't really a question. Have you read a good textbook? (I like Singer, J. D. and Willett, J. B. (2003). *Applied longitudinal data analysis: Modeling change and event occurrence*. Oxford University Press, New York, NY.)
Equation 7.2.4 is the conditional log likelihood function.  The terms maximum and estimator have nothing directly to do with the equation.  What is the difficulty taking partial derivatives of the function with respect to a$_t$?
can you perhaps treat it like a "regression with measurement error on X" problem?
shouldn't be : chisq.test(c(p1*n,p2*n,p3*n), c(n/3,n/3,n/3)) ?
... In the first 5 hours I see 15 birds. In the first hour, nothing. In the second and third only 1, in the fourth 9 birds and in the fifth 4 birds. Then the frequency of the birds is (0,1,1,9,4) and the expected is (3,3,3,3,3). In the other question suggested the others samples: obs=c(2,3,4,4,4,4,4,4,4,4,4,5,5,5,5): number 1 there is not because in the first hour no birds. Number 2 and 3 only one time because in the second and third hour there is only one bird....
@Altons: You fit an 11th model, using the full dataset.  Under 10-fold cross-validation, each model is fit to 9/10 of the data, and error is estimated using the remaining 1/10.  You then fit a final model on 10/10 of the data (i.e. all of the data).
The solution given by Glen_b is incorrect if you want equal representation of each number between 1 and 9 since in the combinations, 5 appears 8 times, whereas the other numbers appear 5 times. If you require equal representation of the numbers, you need a different weighting scheme.
To take an example, Google ngrams viewer suggests a recent frequency for "the rain" of about 0.0006% and a similar recent frequency of "in Spain".  Meanwhile "the rain in Spain" (a *My Fair Lady* quote) is about 0.0000002%, about 50 times what you might expect from the two parts, but nothing for "in Spain the rain". (It gets more complicated with capital letters, but ignore that here.)  What do you mean by a link between the two bigrams?
+1. You've answered your own question. A difference between logarithms is perfectly meaningful. A logarithm of differences is not. Depending on what kind of lab you frequent, think of pH. It is a logarithm, and people do work with differences on the pH scale.
What results were you expecting? What do you mean by "cancelling each other out"? Why are these results not good?
For instance: $P\\left(\\sum_{i=1}^n X_i= t\\right) = {t-1\\choose n-1} \\theta^n (1-\\theta)^{t-n}\\;I_{\\{n,n+1,\\dots\\}}(t)$.
Can you recommend any reading regarding when it is necessary to have correlated random slopes and intercepts? I can understand in the case of item wise variability, but since I'm from a Visual Neuroscience background, I find relevant literature difficult to come by.
I don't have a pristine corpus to cross-validate against so it would have to be the training set.
See the meta thread: http://meta.stats.stackexchange.com/questions/213/list-of-candidate-questions-to-be-closed where this question is proposed to be closed.
The logit / logistic function is *not* the only function that can be used as a *link function* for regression models when the response is distributed as a binomial. Regarding this point, it may help you to read my answer here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909).
Additive approach, seasonal factors remain constant, a constant is kept within the ARIMA model when calculating it.
_"the p-value is the probability of the resulting test statistic if h0 is true"_. That's not correct. _"[...] the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true"._ (from Wikipedia)
It would help if you could provide more info on how the dependent variables are related to each task.
Also, if I understand correctly, taking the logged variable isn't a free choice - it has to be argumented by skweness-charts (statistically)? And substantively is just more a logical reasoning for taking logged that actually provide proff for doing so? In other words, are there rules of thumb here, defining threshholds above which you should take logged version or is it a matter of interpretation?
Your model only requires the *residuals* to be *approximately* normal.  It does not require the response variable itself to be normal.  An, in fact, that is only for a GLM for a normal distribution. For other GLMs, such as Poisson or Binomial regression, you definitely do not want the residuals to be Normal!
(cont.) (**2**) Viewed this way, it is easy to see that this "least rectangles regression" is equivalent to a form of [orthogonal (or total) least squares](http://en.wikipedia.org/wiki/Total_least_squares#Scale_invariant_methods) and, thus, (**3**) A special case of [Deming regression](http://en.wikipedia.org/wiki/Deming_regression) on the centered, rescaled vectors taking $\\delta = 1$. Orthogonal least squares can be considered as "least-circles regression".
Nope, Jensen's inequality describes systematic biases. If you simulate areas of squares with a random length that has a mean of ten, the mean area will be greater than ten, and the exact bias will depend on the distribution you are simulating from. You can't say "It's 100 because the mean is 10".
For ideas on graphics, I always check the R graph gallery. Here's something from there that is somewhat like what you ask for: [R Graph Gallery parallel](http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=37). I found it by clicking parallel in the tag cloud, but there may be better options.
Just a hint. Think, in what way the results of PCA will change if add more and more copies (or near copies) of one of the variables.
Why not set h = 365 + 292 + 7?
Super. I'm interested in knowing what the value is so maybe leave a comment when you've made the calculation!
I think this is a perfect question to "flesh out" the differences between Bayesian and Frequentist methodology
There is no doubt such a procedure has the best fit *to the data*. That's as fine as it goes, but as soon as you attempt to apply the results to anything else--to predict, to interpret, to estimate, to interpolate, to assess variability, to reason about *anything* other than the raw data--then it's highly likely your model does *not* have "best" coefficients (or even good ones) and it is almost certain its standard errors are too small.
I am concerned about your objective function.  It will be unstable when exploring values where 'Mean Fit' and 'Sk Fit' are near zero, even if these are close to (or at) the optimum.  One wonders about the point of optimizing the sum of relative errors, too.  After all, the uncertainties in the moments get larger as the order goes up: the mean return is a *far* better estimate of the true mean than the kurtosis of the returns is an estimate of the true kurtosis. A natural objective would be something like the sum of the squared differences of the $1/k$ powers of the sizes of the $k$th moments.
"The p-value is just the probability of rejecting the null hypothesis by "accident"" what does this mean? It reads more like the definition of alpha/the Type I error rate. I don't think it's congruent with the meaning of a p value (which is the probability of the result conditional on H0).
Also you seem to miss that the question is not about whether p < alpha (significant), but why the already-low p decreases.
gotcha! Thanks!
Maybe it could be reworded to highlight the statistical aspect of the question (if I understand it correctly): "SPSS include an option for automatic linear modeling. There are several options for data transformation: Which one would be best given *my data* and *my question*, and why". Of course, this assumes that the OP could share more details on his data set. I should note that there is a near duplicate lurking around here: [Is automatic linear modelling in SPSS a good or bad thing?](http://stats.stackexchange.com/q/7432/930)
This is an excellent response; thank you very much for the elaboration.  I just found another reference for that book today, and I will certainly track it down now that I know I'll need NLME.
I've always just called them variance components, but I've calculated them from the mean squares (MS=SS/df) in the past. In this case, under a balanced least squares design, I would have MS(DS3)=var(e)+7*var(DS3), MS(DS4)=var(e)+4*var(DS4), and MSE=var(e), and would use those expected mean squares to solve for var(DS3) and var(DS4).  Thanks for your quick reply, Stephane!
@ttnphs: a minimal spanning tree is just the thing I don't want, since pairwise correlations imply a complete graph. Nevertheless, you're right that this question might fit the mathematics-site better. Thanks!
Note that $n\\hat\\sigma^2/\\sigma^2$ is a sum of standard normal random variables, while $n\\hat\\sigma^2$ is a sum of $N(0,\\sigma^2)$-random variables. And $\\hat\\beta_1\\sqrt{S_{xx}}/\\sigma$ is distributed as $N(0,1)$.
@Misha, why do bootstrap estimates should have normal distribution? Usually bootstrap is used when the distribution is unknown. Or you know that the distribution is normal and you use bootstrap to estimate the covariance matrix?
I don't think I can follow your argument/logic. Are you able to state this in a different way?
Would you please explain the consequences of this sin?
:Adam Thanks for your nice words. You will note that I only opine on time series problems/questions where I have some expertise as I have specialized in that area for the last 40 years.
Thanks for clarifying this, I wasn't sure if there was some nifty technique for clustering without aggregation.
Are these measurements binary i.e., 1 (for yes), 0 (for no)?
+1. Another good way to make packages semi-public is to put the package source up on GitHub - it makes the code easier to find and encourages others to contribute without the implicit polish of a package on CRAN.
@sashkello the sample size for 20% sensitivity was calculated using a significance level of 95% and a power of 80%. So if a 20% effect is observed at 490 samples, can the test be ended and considered _credible_?
This book indeed provides a nice overview of the field. It focuses on a few algorithms/methods (e.g. the well-known silhouette, which happens to have been designed by one of the book's authors) and covers them extensively. It also comes with some code, but 1990 style. FYI: [full table of contents](https://www.quora.com/Data-Analysis/What-are-the-best-books-on-cluster-analysis/answer/Franck-Dernoncourt).
Fantastic. Would it also apply to robust regression (although, I can imagine that calculating IC with those std errors would have to assume a distribution not necessarily normal when using robust regression)?
As an alternative to the above, if you like geometry, think about this geometrically and use spherical symmetry of iid normals.
I have to say that Matlab (thus Octave) is a terrible programming language. It's very popular in the engineering and machine learning fields, but that's due to its power, and being used in schools, not because it's a modern programming language. That said, you'll be more likely to encounter machine learning texts that use it than you will that use R or Python.
thank you Peter! yes, found it on wikipedia.
Sorry, I don't know of a good reference. I think that as the Wikipedia article says, you just use center and circumference instead of intercept and slope.
Could you elaborate on your design a little more. How many times did you measure your covariate and dependent variable? Is covariate and dependent variable aligned in terms of time points? What experimental manipulation did you use? Is the experimental manipulation assumed to influence the covariate also?
Generally, with multivariate distributions, there's generally not a 'the multivariate' case, but simply various possibilities.
Correlations are not "being changed". Why would you expect that the partial correlations would be similar to the ordinary correlations?
2. My deliberately hesitant '_slightly_ more generally' was intended to cover both the complications that arise from the relevant approximation and scope limitations, and from the extensions in the second half of the wikipedia article.  Finally, I deliberately didn't actually use the phrase 'degrees of freedom' and put scare quotes around 'free parameters' because I think both are pretty unhelpful terminology.
Important to look at the comments to that article too.
My view is that good OS libraries should have sane, override-able, defaults. Scikit-Learn is a good example of this (Weka is poorer example). That way the programs are explicit only when required. To program your own just to have more control can be overkill in 90% of cases -- better is to fork to OS library and added in the pieces you want.
Perfect! I had the NR book on my desk all the time, but never thought to look there. Thanks a lot.
The following threads (most of which have good answers) all seem to contain relevant information: http://stats.stackexchange.com/questions/89512, http://stats.stackexchange.com/questions/67422, http://stats.stackexchange.com/questions/69350, and http://stats.stackexchange.com/questions/29860.
With a millon points and an 8 parameter model, a goodness of fit test like chi-squared tells me that there is essentially no chance that the model is correct. (Which is not surprising, as there are endless factors influencing reality that are not in the model)

RMSE gives me a sense as to how good the model fits the data, but does not give me a sense of whether there is a better model
I talked about k-means just to give an example, but I'll try to clarify my question without using it. Consider k distributions with k different means. If I used the real means of the distributions (the means used to generate the data points) and do ONE classification step (associating each data point to the nearest mean) I would have a classification of the data. This can possibly not be the best classification, but in average (classifying many data sets generated with the same distribution and mean, and specifically comparing to the average of k-means results), would it be?
UCLA's statistics help site also has an excellent overview of pseudo-$R^2$s [here](http://www.ats.ucla.edu/stat/mult_pkg/faq/general/Psuedo_RSquareds.htm).
+1, always nice to see when answers to theoretical questions are related to their usefulness in practice
+1 for this leading to a moment of clarity.
It depends on what your criterion for importance is as to whether it's directly available or you need to do further calculation. Knowing there's a nonlinear effect would impact how you model it. In the case of GLMs that might either cause you to consider a different link function, or to look at how to model the curve (which might depend on what you know or expect about it) - GAMs might be used if you don't know its form.
His this estimator been verified to be optimal in a traditional statistical sense?  Binning is usually a suboptimal approach.