tttjjj/elt_bench huggingface dataset:
Your results don't particularly surprise me since MA models need to be estimated with maximum likelihood. Since your model uses the lag of a residual, that original data point would not be included in the maximum likelihood optimization, though I can't say for certain about the second point. My shot in the dark guess is that using the residuals in period 3 and on, they back out reasonable residuals for the first two periods that would be consistent with the maximum likelihood estimates. But I would have to look more carefully at the code to know for sure.
(1) Could you please explain what you mean by "input"? What is the form of your model and what aspect(s) of it constitute "input"? (2) In ordinary least squares regression there is no penalty on (dependent) values, *per se,* but only on the *residuals.* In what sense can "values" (presumably *data*) be "constrained"?  They are, after all, your data! Are you proposing to change them or are you really talking about constraining the fit? Or maybe constraining any predicted values? All in all, it's hard to guess what you're asking.
I agree w/ @Glen_b. Although the context differs slightly, it may help you to read this thread: [What if residuals are normally distributed but y is not?](http://stats.stackexchange.com/questions/12262/)
It turns out we were mis-interpreting a statement on pg 529.
Good start (+1).  The formula needs some fixing.  The variance of the difference of the proportions equals p0(1-p0)/n0 + p1(1-p1)/n1.  With n0 = n1 = n and p0 = .5, p1 = .2, that equals 0.41/n, implying n = 41 z^2.  Note, too, that this is a one-sided test, so z = 1.65 works fine.  (Some precision is needed here because the result is sensitive to the squaring of z.)  Regardless, these calculations establish that approximately 10^2 *independent* subjects will need to be tested *if* this model is correct.  (I do *not* expect the bleach or the new formula to have "random" effects.)
Welcome to the site, @AhmedH.Radhwan. Would you mind adding some peripheral information to situate / motivate / explain your code? Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info about CV.
hi Maarten, thanks for this. You sent me in the right track. I use R normally, so I think the "polr" command from the MASS package will allow me to do an ordered logit as you suggest.
How about "once bitten, twice shy"? No funnier, but commonplace enough to remember. And no ageism required!
you should consider removing the data set and just being clear what the issue is and where you're running into trouble.  Adding the data doesn't seem to help your question.  You may want to migrate this over to stackoverflow.com, as it looks potentially more like a computing/programming question.
@Cam.Davidson.Pilon: $x_i \\geq 0$ and $\\sum_i x_i = 1$ implies $\\|x\\|_1 = 1$. :)
+1, thanks for the plos paper link.
A little more information please. How many subjects do you have (IRT models can require a large number of subjects)? Is this truly a Likert scale or a 7-point partial credit rating scale? Rating scale efficacy is part of the reliability question, but reliability is well defined and can be estimated other ways. More clarity on what you wish to measure or understand would help.
(+1) I am surprised not to find good explanations available from the first several pages of Google searches.  This question deserves a clear straightforward answer here!
It's is; but $\\Delta x = 1-0 = 1$ and $\\Delta y = (\\mu_1+\\delta\\times 1) - (\\mu_1+\\delta\\times 0) = \\delta = \\mu_2-\\mu_1$ and so $\\Delta y/\\Delta x = (\\mu_2-\\mu_1)/1 = \\mu_2-\\mu_1$.
The small-sample sampling distribution depends on the distribution of $y$, and on whether the other assumptions of regression are satisfied. Under the usual assumptions, [this](http://stats.stackexchange.com/a/60097/805) answer covers it (see the later part). If the distribution of $y$ is non-normal, you (often) still have asymptotic normality. If the other regression assumptions are violated, things get more complicated.
What "standard deviation bars" are you  talking about? What are you doing? What sort of graph? What are you trying to show?
Actually generalized Pareto (GP) is a limiting distribution for extreme data (extreme are characterize by all data which exceed a high threshold). A suitable threshold (model gives the good fit to data above this threshold) indicated by the stability of parameter estimates which is not so easy to determined by look at a graphical plot. I want to know whether there is any other way/test to find the best threshold (i.e the best data set for GP)
This question appears to be off-topic because it is about software recommendations.
I know EM. What likelihood do you suggest to maximize? There ain't no data.
@Bash: Because any randomness present will lead to *some* correlation, and the null hypothesis is no effect, or zero correlation.
I think I'm missing the point of this, could you clarify?
The question does not make sense. And sounds similar to one asked a few days ago.
I agree, but I was indeed worried about the k parameter -- if the unbalance will create some class-wise differences in the observation densities in the feature space, the same k will tend to a smaller sphere in feature space for an observation from denser class. Won't it influence the k parameter optimization then?
A method called "thin SVD" is used for dimensionality reduction. See Wikipedia on SVD.
.../... despite the fact that the residuals are normal enough, which you can see from the qq-plot of the residuals at the bottom (& hence $y_i$ would be normally distributed conditional on $x_i$). I think what's important here is to get people clear on the fact that the *marginal* distribution of $Y$ is irrelevant. That's why I use the term "residuals", instead of 'the *data*', which I think is ambiguous, & have found to lead to misinterpretations (an example of which is above).
McCullagh and Nelder is pretty good, although somewhat mathematically dense. I figured out how to use GLM's from the first 100 pages or so. Also, Gelman and Hill discuss GLM's in their Applied Regression Models and this book is pretty good for taking you from the very basics right through to Multilevel models using lme4 and Bugs.
@DrWho So accept it by clicking tick mark.
Note that we have $\\frac{var(\\hat{u})}{var(y)}=\\frac{SSE}{TSS}=1-R^2$ (at least roughly anyway).  This gives $corr(y,\\hat{u})=\\sqrt{1-R^2}$ which is some further intuition about what you mention in later paragraphs.
In epidemiology when monitoring disease outbreaks space-time scan statistics are popular - see the [SatScan software](http://www.satscan.org/) and its library for examples. Basically what it does is calculates the rate for various window sizes and then identifies anomalous windows based on simulated test statistics.
How does it come that you have the same p for one-tailed and two-tailed tests?
Xian received a mail praising Jaynes: http://www.ceremade.dauphine.fr/~xian/critic.html It's a pity if you don't read French, this mail is both frightening and funny. The writer seems to have gone crazy by thinking too much about Bayesian statistics ;)
I want to suggest something using an edgeworth expansion (i.e calculating higher order moments), but I'm also fairly sure this isn't a good idea.
Question 1 is related to [Look and you shall find (a correlation)](http://stats.stackexchange.com/q/5750/930) and Question 3 to [Is adjusting p-values in a multiple regression for multiple comparisons a good idea?](http://stats.stackexchange.com/q/3200/930). More generally results from [this query](http://stats.stackexchange.com/search?q=%2Bcorrelation+%2B%5Bmultiple-comparisons%5D) may be of interest.
@John: If you feel that way discuss the issue at http://meta.stats.stackexchange.com/questions/248/how-to-speed-up-questions, no need to be snarky.
1) If the priors really are conjugate, there's no need to use WinBUGS, as the posterior will have a nice closed form.  2) If you take the log of your data, it will be Normally distributed, and you can use the conjugate priors for the Normal.
@ttnphns: 1) $X$ and $Y$ are two random vectors. They are two vectors of random variables, two sets of columns of data, not two cases (rows). 2) $X=Y$.
Wish I had more ability to upvote. One feature not discussed is "blinding" and the need to assess the success of randomization and stability of results under resampling.
If you want to feed only scores into the LDA training that actually have discriminativ power, consider using e.g. PLS instead of PCA for the dimensionality reduction step.
Would it be correct to assume your combined data could be accurately and completely represented by a collection of ordered triples of the form $(i, x, y)$ where $i$ identifies the experiment, $x$ indicates whether treatment was applied or not, and $y$ is the protein level? Note that this implies there exist no other physical relationships among the data: no two triples refer to the same animal, animals were not housed together, they aren't groups of siblings, and so on.
@David You're right, thank you.  I was (confusedly) equating "length" with "radius."  But this does not render the explanation any less valid.  The question is whether the discrepancies you are seeing are as predicted (by the corrected formulas).
Also note that window(qs,2010, c(2010, 4)) <- 3 will change the qs accordingly.
Thank you for the answer. I am not familiar with the use conditional-sum-of-squares to find starting values - but it sounds like that might be it. Since FitARMA used full ML as far as I could understand and gave the wanted results (same as in OXMETRICS which uses full ML).
@Carlos -- but convergence does not mean equality; this is guaranteed only for unreachable limit of infinity. So there is no contradiction ;-)
I have checked the Greenberg and the Giannone et al. reference but found no mention of a model where the standard deviation of the groups are modelled, like in the last model sketch in my question. The Koop book I don't have access to at the moment.
Yeah, that is revenue on the Y axis. They end at different times because the days are relative the to the market's first day of sale. So some markets are older than others.
I agree, the terminology train/test/validation is not very intuitive
Absolutely relevant - adding some details that suggests why I didn't think it was a duplicate.
Please spell out your abbreviations. Are "Fe" & "Re" *fixed effects* & *random effects*?
Thanks, @DJE. You might also note that [dispersion](http://en.wikipedia.org/wiki/Statistical_dispersion) here is essentially a synonym for *variance* & that this issue is important because the binomial distribution necessitates a specific value for the variance as a function of the mean. Thus a value of sample dispersion that is sufficiently far from the theoretically appropriate value indicates that the distributional assumptions made by your inferential tests may not be met & hence their validity is in question.
Well I don't really want to get into R right now, but I will try to find something similar in SPSS and try it. Thanks for the tip!
Thanks i see what you mean... But my goal is to have statistics rules. The complexity you talk about can be seen in econometrics problems which use classical approach.
@Macro, I've edited my question again...
@Jeromy: Let's see whether I have understood. X and Y could be different measures of "intelligence." In study A, 100 people randomly drawn from a special school are tested. They all have comparable (low) levels of intelligence and r = 0.50. In study B, 50 gifted students are tested; they all have comparable (high) levels of intelligence and r = 0.45. If the raw data were to be combined, r would be almost 1.00. In study C, 50 more low-level people are studied and r = 0.45. The r from the combination of studies A and C would lie close to 0.45 or 0.50. How would a meta-analysis handle this?
Econtalk is one of the most intelligent podcasts out there.
yes please! I will happily wait until tomorrow
It's *nonsense*. By the way, please add the `self-study` tag.
Honestly, perhaps the community wikiness of this question should be discussed on meta. I personally feel that there is a singular right answer to this question - the answer that helps me. However, that singular right answer won't apply to everyone (some people might find an alternative answer to be better). Personally, I want to give reputation to the person or people who help me with my problem, but if the community wants this to be community wiki, I can make it happen (but not without a discussion on meta first).
The caret package for R has a good implementation of time series cross validation that could be used to select the parameters for glmnet.  Thanks for the answer.
Can you post your data ? perhaps ten accounts along with the indicators ?
Depends entirely on what you want to do.  Basic analyses will run fine on anything you can find, on any supported platform.  Whether you need a fast CPU or lots of RAM depends on where the bottleneck is for what you want to do.  In my analyses, I'm usually the bottleneck. :)
What do *you* mean by the "smallest" of two covariance matrices?
I don't have the privileges to migrate posts... but I'd suggest searching the S.O. site. Maybe look into R which is very good for graphing, and can interact with Python, too. Good luck!
I think that you should explore some different data visualisation approaches before worrying about hypothesis testing. Perhaps a dithered dot plot would make the data easier to see.
What, precisely, is "complexity"?  (This is not a flippant question!)  In the absence of a formal definition, we cannot hope to make valid comparisons of something.
Yeah, I used jitter for modifying the rug.
A Markov chain might be an appropriate model, but--crucially--the operational meaning of the "NA" values must be understood. It seems they are not the same as missing values. Perhaps you could explain what the underlying data really are and how your series actually is derived from them? Do you have access to the underlying data?
"Famous" and "strong" are too vague and subjective to lead to a useful question for this site.  Please see our [faq] for guidance about what is appropriate and how to formulate a good question.
A better way to characterize this data is "irregular", rather than non-uniform. My apologies for any confusion :)
@probabilityislogic: Actually I do need this precision. Later I take log of p-values, so I get many Inf values, if p-value is 0.
@IrishStat Actually I think we are close to agreeing on that.  I am a big believer in identifying outliers on the basis of there influence on estimates and predictions. I think your comment about using dummy variables to reduce their "bad" influence on future predictions is legitimate.  It is also in line with the philosophy of Peter Huber and Frank Hampel who advocate robust statistical methods that downweight outliers rather than ignore and eliminate them.  As a practtical matter if you can't find the reason for the outlier I think I agree with what they do and what you do.
This question appears to be off-topic because it is about a request for SPSS syntax only. Please see advice in the help center about software-related questions.
... as long as the Bernoulli assumptions hold -- independence, constant probability of success.
It looks like the title and the first paragraph may have (confusingly) switched the percentages: a one-tailed 95% prediction interval can be derived from a *symmetric* two-tailed 90% prediction interval, but not the other way around.  *Any* reference will document this because the equivalence follows immediately from the definitions.
You are correct with regard to what is happening with the "parts" (i.e logit regression for PR(y>0) and gamma regression for E(y|y>0) but it is the combined estimate (and standard errors, CI) that are of main interest - i.e E(y). Predictions of this quantity are made in the SAS code by (1 - p_yEQ0)*mu. This formulation allows you to conduct contrasts on the coefficients on this combined value.
Do you really intend to include `diff` in the denominators? The values would algebraically reduce to `(x[n]-x[1])/(n-1)` which is a (crude) measure of trend and ought, in many cases, to be extremely close to zero, resulting in an unstable and not terribly meaningful statistic.  I'm puzzled by that, but maybe I'm overlooking something obvious...
You gave the formula for population variance. Sample variance is scaled by a factor of $\\frac{n}{n-1}$, which adjusts for the fact that the sample is closer (in squared-deviation sense) to the sample mean than it is to the population mean; this fact imposes a downward bias on $n$-denominator sample variance estimates. As such, the unbiased version of sample variance is used very widely. see [wikipedia](http://en.wikipedia.org/wiki/Variance#Population_variance_and_sample_variance). So R's answer will be 3/2 times as big as if you just apply the population-variance definition to a sample.
I'm not familiar with how `glmer()` is different from `lmer()`, but in `lmer` you would do: `lmer( bull~p1+p2+p3+p4+p5+(i1+i2+i3+i4+i5+i6-1|school),data=bull, family=binomial(link=logit))` where you've deleted the random intercept since it isn't identified from the first dummy variable.
Related to that, I do routinely compute inter-quartile-range odds ratios (an automatic feature of the R `rms` package).
Most answers and comments here focus on the more specific question towards the end. This is just a flag that time series analysis is about much, much more than Box-Jenkins or ARIMA. Entire fields of time series analysis have quite different (or at least more general) focus. Unobserved components models are just one of several examples.
I don't think there is enough here to allow any really useful comment, except possibly that no regression is designed to maximise "percent correct", whatever that is. You have puzzling results, but you need to tell us what you are trying to do, what you did and what results you got in much more detail.
Huge CIs are relative, but even on a SE scale, there can be large CIs associated with particularly unstable parameter estimates. Likewise, prediction errors are relative. You need to impose some additional assumptions if you want to use prediction errors to determine the probability that a particular new observation will fall in some interval. In particular, you need to actually assume that the data are normal, or else use Chebyshev's rule or some other weaker, non-parametric relationship between interval probabilities and approximations of the SDs of the sampling dist'n of a pred.
I don't understand what you mean by "using the wrong value of n". As I understand it, he/she just took the sample standard deviation of the sample proportions.
(+1) Cute. Welcome to the site. Please consider registering your account so you can participate more fully. I'll have to look at this expression a little more closely later on. :)
Mysterious down-voter - any reason why? Suggestions for improvement?
I doubt there is any state or country in the world whose population is exactly equally distributed among three languages (or any other three categories, for that matter)! This highlights the importance of looking at the metric *in context* and of using it *to make comparisons* rather than on some absolute scale. Of greater concern is the potential for some abstract theoretical answer (no matter how rigorously supported) to be misinterpreted or misused for such a socially important application as assessment of segregation.
@Glen_b I figured it out.  Thanks!
@Tal, @Rob: In that thread, he does say "Be sure to use the hierarchy principle".  Perhaps of interest, this discussion from medstats (scroll down for Harrell's response): http://groups.google.com/group/medstats/browse_thread/thread/86c44163b849572
@Aniko I've supplied an answer that accounts for correlated data. Thanks for your suggestion it was a detail that I overlooked.
I have no idea what that means in this context. The only interpretations I can give it doesn't answer my question.
@COOLSerdash Ok, you may want to post this as an answer.
Not sure what do you mean. I've checked the reviews trying to spot the link to the alternative to no avail.
What's the purpose of the estimation?  To retrospectively characterize past conditions?  To predict future power generation at one location?  To predict power generation within a grid of turbines?  To calibrate a meteorological model?  Etc.  For this question, determining an appropriate solution depends critically on how it will be used.
Simulated annealing is an adaption of the Metropolis-Hastings algorithm (as mentioned in the Wikipedia link).  You could always ask whoever set the assignment whether SA was what he/she had in mind.
@Gerenuk:  I suggest you take into account the comments by Daniel and Michelle about what you want to accomplish using *inferential* statistics.  Based on these 2 time samples, to what are you trying to generalize?
(+1) Note that the reciprocal distribution is *bimodal* when $k\\gt 1$.  All the moments diverge (for any $k$).
I love the idea.  I'll look in to it surely.  As for Markov chains, I don't quite understand them.  Perhaps my other group members will understand it.  Thanks.
It's not quite clear what your problem is here.  What are these "other weights z"?  Also, it sounds like your issue with the constraints on beta is actually the problem rather than anything to do with the weighting.  So perhaps you should be a bit more specific about what those constraints are.
In what way are text files becoming too big? Are you afraid for their integrity? Not able to open them in some other software? Are they too slow to load? Too big for your hard drive (laptop?)?
Thanks for adding R code.  This is extremely useful.
One thing I noticed is also that the version of lme4 I was using was not the most recent one and therefore the simple plot(myModel.lm) did not work, perhaps this is helpful for other readers to know..
Thats Right Analyst. In fact I've been doing that.
But then everytime I have to decide how many lags of each variable to use in the model.

I have to make model to predict demand for 100+ products, I was looking for some kind of automation.
Generalized linear models would be the usual approach
Please don't cross post to multiple Stack Exchange sites. If you (or someone else) feels the question you posted on SO is a better fit for CV, you (or that person) can flag the posting for migration. The entire Q&A can be moved across sites on the SE network.
I'm not R user, I'm SPSS user. As far as I remember SPSS has no random number functions for generalized normal distributions. So, if to follow your interesting suggestion, I'll have to code such generative function myself. If you know the algorithm please forward me to it. I don't insist to start with multivariate normal, I just feel it might be a convenient way: please remember that I need strictly _spherical_ cloud (multivariate uniform - an utmost platykurtic generalized normal - isn't spherical!)
@Azula R. and @Benjamin, I tested with the example data from `clogit()` and found there is no difference whether outcome coded as 0-1 or 1-2. Is your code for unconditional logistic regression like `glm(PHENO==2 ~ x + as.factor(COVAR[,1]), family = binomial)`? It is possible that the results of `clogit` become insignificant, since unconditional logistic regression does not account for the correlation between matched subjects thus fails to make correct and efficient inferences on the mean model.
AFAIK, all of the RF libraries have some sort of `score` function for evaluating the performance. Since the output is based on majority vote of the trees in the forest, in case of classification it will give you a probability of this outcome to be true, based on vote distribution. I'm not sure about the regression though.... Which library do you use?
(+1) I do not understand your situation but it seems to be a situation of a pseudo-Bayesian approach