tttjjj/elt_bench huggingface dataset:
I'm interested in this too - I've done a few techniques on this problem, but I'm guessing there are "standard" techniques for anomaly detection on time series.
Is this a pure R question? Then this forum is the wrong place. However, if the GLM-object is called "fit", then run something like "1*(predict(fit, newdata, type = "response") > 0.5)", where newdata is a data.frame or list with the values of the predictors for your observations.
@user1873 I'm not sure I really understand your study - if you're only running 20 simulations, where do the number of games come into play?
The error generated by Stata means some of your independent variables are perfectly collinear. The likely culprit is within the dummy variables. Either you forgot to exclude at least one of the dummies, or some combination of the other independent variables is perfectly collinear (or one of the dummy variables has no variation). @chl , Stata will automatically drop a variable when perfect collinearity occurs in a regression model (I'm fairly confident this is the error message the OP is talking about).
@whuber I don't understand your comment and/or the question. Perhaps we should chat offline on this.
Okay.  I have a table of data 4300 (yeah, it's actually 4300) lines long.  It has 3 independent variables and one dependent variable.  I want to do multivariable linear regression to find how those three IVs affect the DV.  I also want to automatically generate 3-dimensional graphs of each slice of the three IVs (as in leaving one IV constant while using the other two as x and y, and the DV as z).
Problems here on several levels. There are many threads here on transformations; it's not clear that you are asking anything new and not covered already. Contrary to your statement, it can be as or more important to transform predictor variables as responses. Further, when the focus is on relating response to predictors in a model, working at the marginal distribution is typically the least important goal of transformation: although distributions that are symmetric and not heavy-tailed are, other things being equal, usually easier to handle, additivity and linearity are more important goals.
The t value from your formula (in the first argument to PfromR) matches the t value for the parameter (not intercept) in my bivariate regression, which is confidence inspiring :)
From a mixture perspective, a model like $Y(x,s) = Y_1(x)\\cdot\\pi(s) + Y_2\\cdot\\left(1-\\pi(s)\\right)$ could easily be written down. But here the choice probabililty $\\pi(s)$ does not marginally depend upon $x$. However if we would set $\\pi(x,s)$ the model cannot be identfied. Hence some further restrictions need to be applied. A possibility would be to only look at the case where the impact of $x$ is the same for both groups, i.e., the same for $Y_1$ and $Y_2$.
Hey, welcome to SO. Your question is being downvoted for a few reasons:
1/ this is not a programming question (ask it in crossvalidated)
2/ you have asked too many questions, it appears you want an expert to devote a lot of their time to all of them, rather than a small amount of their time to one of them.
3/ we cannot reproduce your results. We have no idea what you are asking if we can't follow steps to reproduce what you've done. You should always post something which is reproducible (search google for "minimum reproducible examples")
Keep these in mind for next time.
@robertomorales I think they are all well chosen for their purposes. Anyone interested in statistical graphics should study Cleveland carefully.
[This](http://www.tandfonline.com/doi/abs/10.1080/03610920008832632#.U7nyWnVdVNB) might be useful to you if you can access it.
@COOLSerdash: COOL links! Thank you very much. All I need now is to think about the *choice* of Deviance as parameter to stop or continue the iterations and gently ask you to edit your comment above into a question to upvote it :)
A lot's going to depend on the true relation between the predictor & response, as well as on the details of what you did. Suppose it's linear: you improve the accuracy of estimates by taking $n$ observations from the tails of the predictor distribution compared to $n$ from all over it; then worsen the accuracy by discarding information when you dichotomize.
What is "Table XXX"?
The question is: how to specify this covariance structure ?
If you look for "Hausman test" via the search function there are many previous questions that discuss this test in their own context but give the reasoning for the interpretation.
+1, great answer. I'm curious: Could you add a paragraph about small/adaptive sample sizes? In SPC, samples sizes of 3-5 seem to be common. And if the SPC software could tell the technician after 2 samples whether it really needed 3 more samples or not, that would be a *great* feature. With a bayesian model, that's almost a no-brainer: Define a cost for measurements, false positives and -negatives, then estimate the expected cost of taking another measurement vs. stopping. In frequentist statistics, you'd have to deal with weird stopping rule effects (Can you teach those to an ME in 15 mins?)
The data is in https://gist.github.com/anonymous/62dc0c08f16470614c94
Can you clarify the predicted variable, "mean endorsement"?. Is this a 0-100 scale that participants used for response, or is this a measure of the proportion of trials on which participants said "yes, I endorse" (vs. "no, I do not endorse"). If the latter, then it is inappropriate to analyse this data as proportions. Instead, you should be analyzing the raw, trial-by-trial data using a mixed effects model with a binomial link function.
Thanks! very helpful!
I see the point, thank you very much!
Great answer and even better questions.
@phihag - Bummer! It's essentially a bar chart then, but you could align the text labels to the right to give a bit different feel if you want. Good luck.
The AIC values are abnormally high, so I would suspect bad data.
Thank you very much Leonbloy
+1, great answer. You mention some caveats re the asymptotic approach; what do you think of bootstrapping the CI in cases like this? I once did that to show that the peak of a quadratic curve fit for one group was greater than that of the other group.
Zay, this is the third post I've seen from you that seems to be throwing random functions at your problem (and seemingly, missing every time).  Since you don't know what you're doing - I intend no slight by that, we were all in that position once - you are *very* unlikely to arrive at a good solution to whatever your problem is by chance.  I suggest (again) that instead you simply tell us in clear, non-jargon (and non-Matlab) terms what it is you're trying to achieve and we may be able to offer better help that trying to guess what you're doing from what you imagine you should be calling.
This is the chaine rule. We have $\\frac{\\mathrm{d}\\, \\log(x)}{\\mathrm{d}x} = \\frac{1}{x}$ so that
$$
\\cfrac{\\mathrm{d}\\, \\log(f(x))}{\\mathrm{d}x} = \\cfrac{\\frac{\\mathrm{d}\\,f(x)}{\\mathrm{d}x}}{x}
$$
I would guess these were standard deviations from the mean
@NickCox is correct. I'd just add that even a weekly forecast based on only one year's data may have problems, since it will be hard to estimate seasonality (likely to be important for food sales).
This seems to be just what I was looking for. I had a look at the Dirichlet Process material as well, which leads me to believe I could generate an algorithm for $k$ not fixed. Thanks a lot for your detailed answer.
A priori is best, but you need to answer this question: how small could the actual correlation be and still be important to you? Not asking you to guess the result, but rather what's important scientifically.
Apologies I meant 'even if the same kink remains'
Also, I just noticed this [Wikipedia article](http://en.wikipedia.org/wiki/Clustering_illusion)--you might like it. Oh, and I just came across this [pdf](http://www.mpia-hd.mpg.de/~calj/statistical_methods_ss2013/lectures/10_use_abuse_statistics.pdf), too--it specifically mentions the "clustering" of airplane crashes (and describes the issue much better than I have so far...).
You lost me at `"I would like to reparameterise the $(x_i) as...".` Is it not possible to chose a proposal distribution from which to sample the entire vector?
Awesome thank you. You were correct in guessing my confusion, I never realized that Gamma was being used in two different ways - which in retrospect should have been obvious since the distribution can't define itself.
Well, you're a condo association which is presumably a member of a larger statewide condo organization.  :)  Think in terms of scale, and then you're talking clout.  Just ping the largest representative group you can, and see if they can get the data for you.  It never hurts to see who can pull some levers for data.  A few breadcrumbs such as a little report that they can send their members could go a long way in convincing them to check on this.
Thanks for the answer. Currently,I am just exploring means of improving my answer. Additionally, the real data is big data and in real data set only 1% of values are positive, but when i test it on sample set(whose true identity is unknown) its around 6%. which is no good in case of big data.
That provides some good guidance thank you. Jay
I've never heard of this before, but perhaps [this presentation](http://dimacs.rutgers.edu/~graham/pubs/slides/counters.pdf) will be of some help.  "Frequent item analysis" seems to be the formal name.
Any method in spatial statistics used to analyze point processes specializes (usually very easily) to this one-dimensional situation.  See, for instance, http://www-personal.umich.edu/~jiankang/papers/paper/Statistical%20Inference%20and%20Simulation%20for%20Spatial%20Point%20Processes.pdf. Although these are not the only possible methods--one dimension typically has an intrinsic ordering not available in higher dimensions and that ordering can be exploited--they should at least give you a good toolkit to start with.
http://stats.stackexchange.com/questions/89135/trend-analysis-to-determine-threshold/89142#89142 discusses a similar topic. My answer should be of interest to you as time series is my area of expertise.  You might want to read some of my other posts on the subject.
Sorry, this was confusing and I edited it. I realize that similar questions (like the one linked) have been asked before but I have been unable to really answer the questions stated above.
Any ideas on something faster and more precise ?
You are right that you can't use binary logistic regression because your dependent variable takes on 4 possible values.
Can you clarify what you mean by a _single instance_? From the way you put it, my understanding is that each instance is a single observation sequence, for instance a day's worth of observations. Is this what you mean? Also, I believe in the last sentence you mean `simultaneously` not `sequentially`. Is this correct?
Hi BasileL, Your question was a bit confusing. I have edited it to (hopefully) clarify it, but since it was confusing, it's possible I misunderstood your intent and inadvertently changed your intended meaning. Please check it still says what you intend. I also changed your data example to put the response at the end of the columns rather than between subject number and the treatment level. When there are potential 'predictor' variables in some sense, the response isn't usually set between them; given the question was already a little unclear, I did this to hopefully clarify it was the response
The reason is that the sample variance of `y` is $0$ since all its elements are `4.424847`. When `var.equal = FALSE` you get division by 0 (or some similar problem) at some point.
@cbeleites you are right, I had not considered the small sample size when writing my answer (which is why I upvoted yours). Outside of small-sample settings, cross-validation is the weapon of choice for tuning all classification approaches I use often.
*"More efficient implies that the variances are lower"* - I think you are going in circles, because variance is based on the sum of squares. Had you been using some other measure based on absolute values instead, it would possibly favor the absolute values.
(1) An F-test, by definition, compares sums of squares of residuals.  Are you saying you want to test a null hypothesis of equal effects among all groups?  (2) "media absolute deviance" = "median absolute deviation" I presume.  (3) "Parametric" *and* "non-parametric"?  What do you really aim for here?  Typically, only one of these will be appropriate for the data and the assumptions you make of them.  The intention to use a MAD or related statistic suggests you are making nonparametric assumptions, but perhaps that's not really the case?
+1, for humour, even though it doesn't answer the OP
General remark -- try to give questions reasonable, informative titles and make them clear and understandable for people not working in your field. Also good question has a bigger chance for a satisfying answer.
@DrWHO: In answer to your question "Is it better to leave the Level shift or treat it as an outlier for the calculations of Case Fatality Rates of State 1 while dealing with the question". If you don't treat it then one can simply say STATE1 had a Level Shift Change at 2004 while STATE2 did not thus they are different but one can not place a probability on that statement. After treating STATE1 for the Level Shift one has normalized the data for a status change at 2004. The normalized data ( cleansed data ) can then be compared with STATE2's normalized data without loss of generality.
+1, nicely done! I hate to nitpick, but I want to note that prediction doesn't have to be about the future.  For example, an archeologist may want to determine (i.e., predict) the level of rainfall in an area at a point in the past by knowledge of traces (i.e., effects of rainfall) that are left.
@Dmitrij Celov: Thank you.  No, my model is simple linear regression without logs...
There's nothing *stopping* you from reporting the training error, but it's not terribly useful by itself. Test error, since it predicts future results, is the more interesting quantity.
$P(U=u)=0$ for any point $u$ (i.e. the Lebesgue measure of the half open interval is the same as that of the open interval) so I don't think it matters.
You will find both questions are answered in the Wikipedia article on the [Chi distribution](http://en.wikipedia.org/wiki/Chi_distribution).
Thanks for the edit @gung
I definitely read your code but there is no such term as "right cutoff" but the plot Accuracy vs cutoff can give you the correct insight. And using this plot you can figure out how to find cutoff for maximum accuracy.
Thanks, fixed it. Also added more information about the article in case the link is broken again.
For example Kline (2011), chapter 10; Hair et al (2009, Multivariate data analysis 7th edition). I dont see a clear preference to any of the two methods.
@RossXV right, I corrected it. Thanks for spotting it.
For the case of the difference in differences estimator, how do you account for auto-correlation?
@RussLenth: I do have the intermediate observations. In terms of $\\lambda$ they are 0.318388897, 0.0893040538, 0.209697071, 0.0782522406, and 0.265018191.
Please don't cross-post, http://gis.stackexchange.com/q/28262/751
Can you give alittle more context as to what you mean by "what works" or the particular goals of your project at hand. I've used them for visualizing spatial point processes but I doubt that is what you had in mind when asking this question.
Please edit your question to include information about what you mean by the "t-stat of a plot."
@J.M, I don't know, that's why I would hope that the community would help to flesh out the concepts a little bit more.
This cross-validation was very helpful. I will check Agresti out at the library. Thank you for your help.
@SKM 
	
In the second table, the 4 in the first row, second column represents the number of times the value 2 followed the value 1 in the original data series. The corresponding probability in the third table is calculated as the elements value (4) divided by the sum across the whole row (4+4+1=11). Thus, 44% of the time a 1 appears in the series, a 2 follows it.
$SSE=\\sum (Y-\\hat Y)^2$. $SSE$ measures the sum of error squares. $MSE=SSE/n-p=S^2_{\\epsilon}=\\hat \\sigma^2_{\\epsilon}$. It is the variance of the error term of your regression model. Since by assumption, the observation has the same constant variance as the error term $\\epsilon$, we can say $SSE$ measures the variance of the observation from the estimation (expectation of the observation) under current regression model.
@Dirk I was commenting Mike's comment about ggplot. As a general comment I also used to like these seas of dots for the sake that nothing will better represent the distribution -- until we've made figures that were crushing Acrobat Reader ;-) .
@Henrik I have never heard of something like this; nevertheless I've made the change, it seems lowercase z is used more often.
@GregSnow The standard error can be used to construct a confidence interval around the estimated coefficient, but does that imply some kind of distribution with the estimated coefficient as its mean?  It wouldn't be the sampling distribution of the coefficient, since only by chance will the mean of that distribution equal the estimated coefficient from particular sample data.
I really like the question, but using likelihood ratio tests to determine whether or not fixed effects are needed is **not** the recommended strategy, see the [faq](http://glmm.wikidot.com/faq). So the above is **not** fine for comparing models.
Isn't the F to compare variances traditionally computed with the larger variance in the numerator, so F is always greater than 1.0? It won't change the answer or conclusion, but makes it easier to follow.
How was that interval of 15 ms found?  Was it designated before you looked at the data--that is, completely independent of the data and how the experiment was run--or is it perhaps an interval centered around a time of greatest event intensity?
Thanks for posting the solution. I appreciate your help!
Chris, I have edited your post to incorporate MathJax notation. If you click the edit button (lower left), you can see the pretty straightforward TeX implementation of MathJax for future reference.
Thx! I will keep the intercept as I have no specific theory of why it should go through zero (except that literature sometimes does it without explanation). But I still have troubles interpreting the effect on the second model then, where variablity in residuals should be compared before and after an event. Some liteature computes the variability of residuals seperately for the period before the event and after and some even compute the variabilty of the resiuals over the whole period and then compare pre-post - which violates OLS assumptions right?
result=cv.glmnet(x,y,nfolds=55,type.measure="auc",family="binomial") Is really wrong. You have 55 rows you can make 5 folds of the 55 rows each having 11 rows. `nfolds` does not mean your sample size, it means in how many groups should your data be divided. When your data is divided into 5 folds you will have 11 rows in each block. `glmnet` fits a model with 4x11 rows and validate it on the remaining 1 block with 11 rows(cross validation as in `cv.glmnet`). So right code should be: result=cv.glmnet(x,y,nfolds=5,type.measure="auc",family="binomial")
(Btw, a proof that the bootstrap works for the case where we centered can be found in Bickel, P.J. and D.A. Freedman (1981), [Some asymptotic theory for the bootstrap](http://www.jstor.org/stable/2240410).)
Nothing. Same idea, different fields.
How about the tensor-product basis of your one-dimensional polynomials? This sounds like what you were alluding to and they will be orthogonal.
@Cyan is that the same as saying that the missingness mechanism is either "missing at random" or "missing completely at random" but not "missing not at random" (the usual assumptions I learned about for performing MI)? Do you know any reference where this "throwing together" is justified formally ?
On another note, I don't think it's true that the negative binomial is the best alternative in overdispersed Poisson; there are other possibilities & they could be preferable. But if your lambdas are distributed as Gamma(a, B), then the resulting distribution is a NB, & so in that case NB must be best. (Ie, if lambdas have some other, idiosyncratic distribution, it might not be best.)
I've left your "not rejected" in respect of P-values < 0.05. What makes you think Wilcoxon compares means?
@CarrKnight yes!
Since it's a random variable with two possible outcomes, it follows a Bernoulli distribution with some particular value for $p$. Calculating $p$ is not possible based only on the information given in the question. In particular we are missing what $X_i$ (probably some RV) and $h$ are (probably some constant).
Agreed (I love jitter, simply not for this :) )
I would say fairly rarely is an F-test used for this purpose. It's very sensitive to the assumption of normality. Less sensitive alternatives are more often used.
Maybe helps: if Y is distributed exponentially, then X=Y^2 is distributed with f_X. You can have a look [here](http://en.wikipedia.org/wiki/Exponential_distribution) for the MLE of Y...
The derivative is **zero,** because the right hand side does not include any $x$, which is the variable with respect to which you are differentiating. If the $x_i$ are assumed to be functions of $x$, then this derivative is incorrect because it does not account for the $d(x_i)/dx.$
(+1) At first when I started reading the article, I thought the author was maybe conflating the law of large numbers with *regression to the mean*. Then, I got to the paragraph that starts "Also known as the golden theorem...". This reads like someone who skimmed L. Mlodinow's *The Drunkard's Walk: How Randomness Rules our Lives* (an otherwise interesting read) and then thought they knew something.
what do you mean exactly by dependent and independent?
Have you looked at quantile regression and the possibility of comparing the fit of monotonic linear or non-linear vs. broken stick regression for instance ? Just my two cents...
As a guess, it looks like zeros are not inflated. Try a regular Poisson.
Retagged "random-variate" to "random-variable" for consistency with similar questions.
@Kenny - you make many good points but I take exception with 2.  I would never recommend stepwise regression to someone new to regression.  The pitfalls are very numerous and extremely well documented.  Also, "the explanatory variables of a multiple regression [...] need to be uncorrelated" - if they were uncorrelated, then the relationships with Y would be identical to those obtained through bivariate correlation.  In other words, there would be little point in running the regression, which is largely to show how each predictor relates to Y while controlling for the others.
My suggestion is correct.. I asked a mathematician
Hi @jbowman. Sorry, I should have been more clear. What I mean by $score^∗_i=(score_i−α_1AGE_i−α_2Gender_i)$ is that the data I have on score is made of some normalization where the test creator has "somehow" transformed the RAW score using age and gender. I don't know the form of this transformation, I just put that as a "possible" example where the raw score is scaled by e.g. subtracting age and gender. I am assuming that $\\alpha_1$ and $\\alpha_2$ have somehow been chosen by the test maker.
The full split can be computed thus. $\\Delta t = (a+\\Delta a)(b+\\Delta b)(c+\\Delta c)-abc = ab\\Delta c+bc \\Delta a+ac\\Delta b\\color{blue}{+a\\Delta b \\Delta c+b \\Delta a\\Delta c+c\\Delta a\\Delta b+\\Delta a\\Delta b\\Delta c}$