tttjjj/elt_bench huggingface dataset:
It's one and the same thing. The fitting of the model gives you both the coefficients and the parameters that describe the distribution of the error term. Since the error is assumed to be normally distributed with mean 0 this just means estimating the variance. Any method that fits the model (from memory is usually Yule Walker) will give you variance of the error. Forecasting with an MA model is quite interesting. Basically as you roll the model forward no more error terms are introduced and the MA forecast quickly settles down to a straight line (if the order of the MA model is relatively low
I think that you're right, Rich.  I can't think of any other reasons.  Thanks!
Are you uncertain of the method or just how to do the calculation in your software?
@cbeleites thanks so much for the response!!! I think you have now answered my questions exactly. However, i still don’t understand the exact procedure to follow. Would you mind just elaborating on the last paragraph.
It would be good to write down your reasoning for choosing 3.
Although the O.P. asks for a non-mathematical explanation, it would still be nice to see mathematical (or some equally rigorous and clear) justification for this answer. Upon reading the question my intuition was that "Because the Poisson is a (limiting) special case of the NB and the NB has more parameters, there is more flexibility in fitting, so of course any *reasonable* measure of residuals ought not to increase when replacing a Poisson GLM by a NB GLM." I am wondering whether such intuition was really correct.
I updated my answer to include the line of thought.
Though I'm not familiar with the specifics of your approach, I would avoid [parsimony without shrinkage](http://stats.stackexchange.com/questions/17111/classic-linear-model-model-selection/17113#17113). If LASSO is appropriate, then that might be the easiest solution to your problem.
You could conceivably leave B and C out but that should only affect things in specific instances.  What kinds of sums of squares are are you using for your ANOVA?  Do you know if there is any correlation among your predictors?  What, specifically do you really want to find out.  (It's better if you edit your question, not just add a comment.)
If this is homework please tag it as such.  Please also update your title to something that reflects the nature of your problem.  Tell us about your 3 items.  Do they measure the same construct?  Would you like to average them and get one correlation between your items and the YES/NO response?  Would you like to look at the correlation between your items and calculate a coefficient of reliability?  Please read the link here: http://meta.stackexchange.com/questions/10811/how-to-ask-and-answer-homework-questions/10812#10812 What have you already done to try to solve your problem?
-1, since the proof of Markovian property is not given. Also the hand waving argument is not consistent with formula given. current state = $t$, past means $t-1, t-2,...$, next means $t+1$, but the formula does not involve $t+1$.
have a look here http://stats.stackexchange.com/questions/21152/obtaining-knowledge-from-a-random-forest
It would probably help to know a bit more about how your implementation differs from k-means! Does it just have to do with the constraints on cluster sizes?
Even in the one-dimensional case the t-test is not correct: please see http://stats.stackexchange.com/questions/62634/does-this-single-value-match-that-distribution. You are asking for a *multivariate prediction region.* For certain kinds of multivariate distribution, the [Mahalanobis distance](http://stats.stackexchange.com/questions/62092/) satisfies your requirements for a distance metric. A related problem (but not quite the same) is to [find multivariate outliers](http://stats.stackexchange.com/questions/213).
To be honest my eyes start to glaze over when I see that much greek notation, especially if I haven't eased into getting used to whatever specific notation conventions the author is using.  The title of the test itself certainly sounds omnibus.
You are missing a ' at the end of the options string in your training command.
I see. Thank you so much, Frank!!! One more thing, if I want to compare different models, which statistic(s) would you recommend to use in order to get a "best" model?
I tried to award you the bounty, but it says: "You may award your bounty in 18 hours". Remind me, if I forget it :-)
(1) This appears to be a [routine textbook-style](http://stats.stackexchange.com/tags/self-study/info) question. Such questions are treated differently here. Please add the tag at that link, and read the guidelines there. $\\,\\,$ (2) "84" is not a proportion, which lie between 0 & 1. "84 batteries" is a count. Note that batteries don't last "exactly 2 hours" but *on average two hours*. That means that 84 batteries would only have (about) a 50-50 chance of lasting 168 hours; you'll need more batteries than that.
This is close to what I thought so I accept it.
http://en.wikipedia.org/wiki/Confidence_interval#Practical_example
can anybody provide insight to this test?
Something like [this](http://www.tandfonline.com/doi/abs/10.1198/01621450338861947#.UjuXW8asjjg) might fit the bill.
Next time ask more specified questions.
Great, that's pretty clear. Thanks for helping me out!
Perhaps I'm not following your question but I think the way that you'e worded it the answers practically tautological.  When the parameters are different, then they're different.  When the characteristics of X are different, X is different.  Feel free to imagine whatever world you can.
If it would help to see this exact same algorithm in a less obtuse language like Python, I'm more than willing to add to my answer...
Also, there are a few errors in the formulas in the above answer, and I notice that it's not relevant anyway, since you are using $\\mu=0$ and $\\sigma=1$.
@DmitrijCelov: No, I don't think so. Look carefully. It appears robbrit has used the $z$-transform and then multiplied by an additional $z^p$ factor, which will not change the location of the roots, except for adding one (of multiplicity $p$) at zero. If you factor out a $z^p$ and then substitute $B = z^{-1}$, you'll arrive at something that may look more familiar to you. The roots of the polynomial in $B$ must lie outside the unit circle. But, there is a simple correspondence between the roots of the polynomial in $B$ and the associated one in $z$. Cheers. :)
@Grrr, in addition to Peter's answer you might want to read my opinion [here](http://stats.stackexchange.com/questions/19591/factor-analysis-on-binary-data-and-summated-scales/19595#19595) (with further link there)
@gung Ctrl-C Ctrl-V -- the rest is an Unicode magic. You can also use appropriate keyboard layout or just some app showing special character palettes.
Thanks!  This works perfectly for me!
@Jaradniemi - How do you validate that the model fits the data is the question. For regression analysis one may look at diagnostics of residuals. I  have validated ONLY that the MC has converged to the targeted distribution but is this 'enough', what else I can do as far as validation if the model fits the data well?
many of *which* articles?
If you have lost your login credentials, please follow instructions in our Help Center to merge your different accounts. Sidenote: It is a bad idea to rollback changes that are made by benevolent users who are just willing to help and improve your post.
This looks like homework. Please see the advice on "self-study" at http://stats.stackexchange.com/help/on-topic Even if it is not, there are many resources on the internet discussing sampling. I removed the tag "systematic".
This seems like a much better way to go than using visual histograms of the predictors conditional on the outcome. If you want to make it official, do this and then use the p-value of the predictor from each individual logistic model as your metric.
What are your n's?  Have you checked the ANOVA assumptions?  Unequal n's by itself is not sufficient to completely invalidate an ANOVA.
You should include your specific code to make it easier for people to give an answer.
Patient Reported Outcomes Measurement System (PROMIS) from NIH uses all polytomous items and it would be worthwhile to check them out to see how they are stopping the items.
@Abe Well, this depends on what you want to do...
Also, thanks for the book reference, I'll definitely check it out.
Actually that is another worry I have had. The mean is supported by the central limit theorem.
@Zen-- Thanx man for your help.
I can, but I won't: there are already many questions and answers on the problematic of data-driven model optimization/selection available (http://stats.stackexchange.com/search?q=data-driven+model+optimization). Here are starting points for further reading: http://stats.stackexchange.com/questions/79905/cross-validation-including-training-validation-and-testing-why-do-we-need-thr http://stats.stackexchange.com/questions/5918/cross-validation-error-generalization-after-model-selection
You can try the [`BiasedUrn` package in R](http://cran.r-project.org/web/packages/BiasedUrn/) for help with calculations
+1 to your answer jona. As a somewhat general comment: I would question the realistic difference of a $t$-distribution with 600+ d.f. and a Gaussian... Those $t$-values should look awful lot like $z$-values.
What do these values represent and why do you think you need a test? How did you generate the random profiles? Also, with sufficient sample size, data sampled from two very similar populations will be “significantly different”. Those are two different issues.
To this very helpful answer I would just add that it is very useful to plot your data.  If you have the software to do it, put z on the vertical axis and x on the horizontal axis, and draw the three different lines for the three different levels of your variable.  If you have an interaction effect they will have different slopes (you need to test for statistical significance of course, but visual examination is always a great start).
(1) The problem with logit(x/45) is that it is undefined when either x=0 or x=45. (2) Logistic regression *by definition* uses only 0-1 responses. It is based on a likelihood for outcomes of independent *Bernoulli* variates. Although the mathematical calculations to optimize the "likelihood" function will go through when other numerical values are offered as input, they make no sense.
Thank you very much for your complete answer. do you know how could I write the code for drawing a plot to show conditional distribution of Y|X in R? I would be very thankful if you help.
Unless you are willing to implement a general time series model (which brings in its cons in terms of latency etc) I am pessimistic that you will find a general implementation which at the same time is simple enough to work for all sorts of time series.
@Shane If my memory doesn't fail me, I have seen @AndyW post *fancy* SPSS code. It (code) impresses me all the time!
What do you mean by expected frame? I can understand that by "observed frame" you are tracking the occurrences of the token, but how did you calculate the expected frame? I hope if you can answer because I need this for my application as well?
How do you know one can't just look at the 'size' of the coefficient?
Answer got too long; moved to an edit of the above answer
The problem is not that the network overfits the data. I'm not able to train it because I run out of memory. I tried using Scipy's [`fmin_bfgs`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_bfgs.html) to find the weights for which the output error is minimal. The problem is that the large number of weights (656.128) causes the algorithm to run out of memory. It tries to create an identity matrix `eye(656128)`, with a total of 430.503.952.384 elements.
See the first answer. This is a special case. But, I do doubt that the parameterization for this function differs from the normal definition.
You have identified two data points that appear anomalous. There is no gain to adding the further word "significant", as you haven't carried out a significance test. Significance tests are for tests of hypotheses about parameters. It's much more fruitful to think about the anomalous points: can you be sure that they don't represent measurement or calculation errors? can something be identified about the times or circumstances in which they occurred that makes them less surprising?
+1 to a fellow Canadian, fantastic answer.
I've made some edits to your question which still looks unfinished at the moment (please, fill the `(...)`!).
@gaearon There may be a potential for confusion here due to terminology: "maximum" can refer to a maximum value attained by the distribution function (which, for the normal distribution, occurs at the *average* value of the distribution!) or it can refer to the largest possible number that has any chance of occurring (which for any normal distribution is infinitely large).  @Aniko's response appears to interpret your "worst/best" cases as specifying extreme values in the *second* sense, whereas your comment appears to interpret the maximum in the *first* sense.
Have you checked out Gaussian process regression? You can learn partial derivatives directly (e.g. see http://www.gaussianprocess.org/gpml/chapters/RW9.pdf).
You can follow through by accepting your answer formally; you will gain reputation points thereby. I've not studied the Python code you cite to see what it does to calculate a Kuiper statistic; I do understand that your discussion of calculations with bins refers only to an $r^2$ calculation. It appears that no discussant could accept all your premises here, let alone what was off-limits in discussion, but you get to be judge and jury on what answers your question. When I say "which is closer" I meant graphically; as said I don't favour single measures of fit much.
For the case of Gaussian vectors see [this](http://math.stackexchange.com/questions/469650/probability-of-the-dot-product-between-gaussian-unit-vectors)
had a look at stackexchange: Tal Galili did ask about PSE a while ago, so I didn't ask. But I just pushed a first bit of code to r-forge: [pse.R](https://r-forge.r-project.org/scm/viewvc.php/*checkout*/dev/pse.R?revision=HEAD&root=hyperspec) *please put stars around the checkout* - I don't know how to escape them so they vanish...
Thinking of $f(t)dt$ as an "infinitesimal" probability is a good way to go I reckon.  I think of a PDF as a "infinite" histogram with bins of "tiny" width $dt$ and height $f(t)$.  The integral is how you "add them up",  Which is why its $\\int f(t)dt$  "sum up the area"
Which Tukey post-hoc btw?
I hope this answer is a little more clear. thanks for the input.
And the underlying code is different too (I checked) so they will produce different numbers.
Can you please clarify what you mean by comparing for differences?  In what regard exactly do you wish to look for differences?
Thank you ! I don't know that much about bootstrapping, but the result of running this code on my data seems very encouraging !
@gung is hierarchical clustering not grouping users into same group based on some similarity score ? In my case jaccard coefficient. I'm using <=.2 as the cut off point to check if users are similar.
A word of caution here: it's not clear if correlational analyses are appropriate for the OP's data.
The problem is that the row with the maximum of the first column might also have the maximum of one of the other columns. This might be a starting point for rejection sampling, but I'm wondering if a more direct way exists.
@LadislavNado if the inference is on the mean then yes. But odds ratios, proportions, rates, etc. all have associated standard errors. As with mean estimation, test statistics for these tests will reach "highly improbably domains" according to the null hypothesis as sample size increases.
Yep, I was wrong: QR is not restricted to square matrices. +1, by the way. This question was one of the highest voted unanswered questions with the [tag:pca] tag, so it is nice to see it finally answered.
When specifically dealing with success/fail its more common to label fail '0', but it makes no difference to the analysis if you choose it to be any value other than '1'. If you choose '-1' for example, you halve the coefficient and halve the standard error, leaving their ratio unchanged, and the fit is in no way altered. Personally I find 0/1 substantially easier to interpret. I don't see how the choice of zero conflates with missing data unless you've already made the very unwise choice of conflating missing with 0. The problem would lie with that earlier choice.
Thanks Thomas...Bullock et al were exactly the scholars I was thinking of when I asked this question. Somehow I completely forgot about Imai though!
I can see this now. Thank you. I did not think about the fact I'd need one more subset to find unbiased performance in terms of the model structure. Cheers :) It's very good explanation.
(-1) I smell advertising. Behind the link is just a thin cooperate site, while the OP asked for opensource software.
How does this relate to k-means? You could remove all the clustering/k-means parts from the question, and it's still the same, isn't it?
Just to be sure, are you talking about ST as found in clinical trials, e.g. http://en.wikipedia.org/wiki/Sequential_analysis?
@steffen Sorry, in every phone he sells at most one product and so he might get either 0, 100, 200, 300 or 400 euros. And yes, there is a positive correlation between $X$ and $Y$.
@Glen_b thank you for your reply. Would you say it is a function of bias and variance? From an intuitive point of view, MSE is similar to MADE in terms of what it is measuring.
FairMiles, i want doing second description,problem is that if in correlation matrix where all (p-1) variables have highest correlation than 0.3 , which my variables has . then you cannot perform multiple linear regression because then you find multicollinearity problem (http://en.wikipedia.org/wiki/Multicollinearity). PCA is best to reduce it because of orthogonality . That's why i used it . I got result of unsorted correlated variables  but now i don't know , how to make it in descending order .  But thanks for multiple linear regression idea. Do you have idea , how to sort variable in PCA?
Looks promising so far. So what needs to go in your $1-\\phi()$ expression to make it come out .001?
Besides... Shapiro-Wilk's test is often used when estimating departures from normality in small samples. Great answer, John! Thanks.
Hi, I haven't found any solid literature yet (but I haven't looked very hard either). So far potential candidates would seem to be: regularized linear regression (LASSO or ridge, with large basis expansions), kernel regression, neural networks (with some flavors potentially better than other, eg. I suspect extreme learning machines, dropout nets and deep learning should get better derivatives than standard backprop) and Gaussian processes.
Did you mean $n=1$ instead of the (undefined) $n=0$?
As a general thing, it might be rather tricky; how might one distinguish (as an example) a random variable which has a Pareto distribution with $\\alpha=1$ from one with $\\alpha=1+\\varepsilon$?
1) No. You are right that binning loses information. It should be avoided if possible. 2) Generally, the curve function that is consistent with theory behind the data is preferred.
My concern is about the amount of time and effort that would be required to do such work. I would like to know a more practical alternative.
Instead of _adding_ I would say _avoid removing_.  And it's not just prediction, but using statistical association assessments to guide variable selection causes biases and invalid standard errors and confidence limits.
Sorry, I don't understand at all. When I run your exact code, I get the same result I originally got, not the answer you posted:
Thanks Michael! You guyz answer make my understanding clear on this problem.
Thank you!  I only vaguely remembered the t-test for comparing whether two distributions were different way back in stats class.  Wikipedia has a good write up, and indeed it seems exactly what I need.  Unfortunately no more data available, as this is 'experimental' technology; though the company claims to have data showing cleaner emissions, they will not release it due to an odd argument about it being proprietary.  Anyway, I think the burden should be on them to provide more data, and this will help.
@Decency To look for correlations, begin by drawing a scatterplot of the *raw* (per-person) (MMR, PSR) pairs.  Comparing their histograms (and means and SDs) tells you nothing about correlation.
+1 anyway. Hope you get a chance to return.
Is your response variable a count? That is, are you looking at changes in the number of CD4 cells at different time points?
Now I doubt such code exists, or if it does it is probably someone's property. In general extraction of something concrete, like in your case finding where the desired digit is, is even a harder problem than classifying it. What I would suggest is to have some spacial information of which part of the photo the digit is, and just cut the image around. If you want exact representation as the MINST data set you gona have to convert it to the corresponding pixel x pixel size.
"Finding out" new records for rare classes might be impossible. I suppose the data is structured this way because it is expensive (bioinformatics) or risky (bank loan) to create more rare-class-events.
Could you please supply a definition of the "extended binomial density"?  Your function `pebinom` doesn't fit the bill because it does not define a valid probability distribution unless `sum` is identically 1.
I guess one approach is to do some sort of interpolation
@whuber Added more details,hope that helps
@Bitwise, The reason is one of ease of explaining the simple plots above to non-technical managers.
This is very helpful, thanks!
1) I found AIC and BIC... I will use them as well.
2) You're probably right; I think the likelihood ratio Chi Square is comparing the model against the null model.
How could I calculate the likelihood-ratio test statistics of one model versus the other? I'm working with nested models (the first one only includes control variables; the second adds the independent variables as well, and the last one includes an interaction term). Thanks again!
+1, the point about confirmatory FA being more appropriate is so right...
You need two columns. One column contains the follow-up times (5, 10, 50, 104, ...). The second columns contains an indicator that distinguishes between an observed data and a censored data (1, 0, 1, 0, ...). Software typically requires to specify the name of these two columns. Sometimes the value that indicates a censored observation is also required. In R: ?Surv
@Zia, no problems, usually the questions that should be migrated are migrated sooner or later, since somebody else flags the moderator. However it is always polite to wait for the answers in the first site you posted the question, before asking anywhere else, and if you do it is always helpful to indicate why the answers in the first site were not satisfactory.
I totally agree; if you know that some portion of your sample is corrupted and unrealistic just excluding it from further analysis is the *cleanest* thing to do.
Please provide a small replicable example to demonstrate the problem.
@mpiktas, Actually I just want to clarify, why does $\\sqrt{T}\\overline{x} \\rightarrow N(0, \\sigma_x^2)$? I thought by the CLT, we have $\\frac{\\sqrt{T}(\\overline{x} - \\mu_x)}{\\sigma_x} \\rightarrow N(0,1)$, so $\\sqrt{T}\\overline{x} \\rightarrow N(0, \\sigma_x^2)$ only if we assume $\\mu_x = 0$? Why can we make that assumption? Thanks.
@FrankHarrell makes arguments in this thread that bear directly on this question, namely whether such cutoffs are desirable. http://stats.stackexchange.com/questions/65382/adding-weights-for-highly-skewed-data-sets-in-logistic-regression#comment165335_65382
:) Oh yeah, sorry I mean Matlab. I've changed the original post now.
Shouldn't the -1 be a 1/2?  You want the square root of that quantity.