tttjjj/elt_bench huggingface dataset:
Thank you so much Sir IrishStat, I’m totally okay with u, that there a big limitations in distance based methods and i guess the other methods too, this is why I’m testing density base method, i saw a lot of articles speaking about times series anomaly detection, as nasa research, universities ..etc but small progress, for particular data problems 
And recently I found, a good free software for outliers detection : MOA of Weka ! Did you test it before?  it’s an open source software, I’m trying to using it for developing and integrating my small detection anomaly algorithm,
If you have a factor w/ 3 levels, the intercept is the mean of the ref level, & the other 2 will be represented in the output. Their coefficients will both be the difference b/t them & the ref level & their ps will be the significance of those difs. If you have 2 factors, both will have ref levels, & the intercept will be the mean of those people who are in both ref groups (eg, `young F`) & the other levels will be dif b/t the given level of factor 1 w/ the ref level of the other factor & the both ref levels group. Eg `old` is `old F` - ` young F`, & `M` is `young M` - `young F`.
In addition to the linked question above, you might find these threads helpful in thinking about these issues: [when-will-a-less-true-model-predict-better-than-a-truer-model](http://stats.stackexchange.com/questions/22566/), & [what-problem-do-shrinkage-methods-solve](http://stats.stackexchange.com/questions/20295/).
(1) Your `R` code has syntax errors: `group` should not be quoted. (2) The plot is beautiful: the red dots exhibit a linear relationship while the black ones could be fit in several ways, including a piecewise linear regression (obtained with a changepoint model) and possibly even as an exponential. I am *not* recommending these, however, because modeling choices ought to be informed by an understanding of what produced the data and motivated by theories in relevant disciplines. They might be a better start for your research.
+1: very interesting.
Ah oke, that sounds plausible! Actually never thought about it this way. So it is correct to say that when one find an opiste effect in the sub periods, the effect cancels out..?
yes, by inconsistent I wasn't referring to the statements only my personal opinion without knowing how to interpret them. THank you for the clear answer.
what are the first two?
If the baseline probability is .57, then the baseline odds is $\\frac{.57}{1-.57} \\approx 1.33$. So a one degree increase is associated with an odds of $1.33 \\times 5.8 \\approx 7.7$, which corresponds with a probability of $\\frac{7.7}{1+7.7} \\approx .89$
Now that I've seen what your data looks like, I have to say that my suggestion is wrong. The problem isn't leverage. Greg & Aniko are right that it's about separation. @Aniko's answer really gives you what you need here.
@Gavi Simpson-Thanks a lot for the input. I thought we could consider mod2 as a better model since it accounts for the factor Site (knowing that there is a significant Site effect) whereas the other does not. I will look into all your suggestions carefully.
:whuber I don't think so . I think in general there is a relationship between the two acf's although I could be wrong !
Do you want the distribution for *fixed* $n$ or the asymptotic distribution? Are you interested in the *marginal* distributions of $\\overline W_n$ and $S_n^2$ or their joint distribution?
Cross-validation is hardly a solution for n = 20.
Welcome to stackoverflow! You are much more likely to receive an answer if you also provide attempted solutions, why they didn't work, and the expected results. The link you refer to seems to be removed. Cheers.
@Cadu: the ratios shouldn't be "pretty much similar"; they should be *identical* (to within floating point error).
I'm taking a course in theoretical stats, looks like I'll be fully covered between the first two books. THanks!
A very straight-forward approach is via the EM algorithm, as Cauchy-likelihood is a scale mixture of normal likelihoods.  So you get a weighted least squares regression were the weights are adaptively chosen based on the "current" residual.
@chl - Not exactly. The release is purely from business standpoint. This is from an internal process POV. The underlying 'problem structure' could be the same, but are two completely different projects in different domains/organizations.
My last question: How do I calculate p-value in this case?
Thank you, that's exactly what I was trying to avoid (having a lot of samples as baseline), because I would like a really reactive approach (e.g. online detection, maybe "dirty", after 1-2 weeks of baseline)
That could give you 0 for the chi square statistic, and no room for degrees of freedom for error.  These things happen when you have too many parameters.  For multiple linear regression this happens when the number of data points is exactly equal to the number of parameters being estimated.  So for a simple straight line if you have two data points you can choose the slope and intercept so that they go through the two points leaving no error.
See `?heatmap`. For instance, `heatmap(cor(x))`.
I know that R is better in many aspects than SPSS, but for the moment I can only work with SPSS.
Note you can get the odds to base ten by multiply the natural log odds by $\\frac{306}{700}\\approx\\frac{3}{7}$.  The answer you get is roughly $-457$.  and for small odds, the probability is equal to the odds.  So we have roughly a probability of $10^{-457}$ when all covariates are zero.
There is no objective "best". It's **very subjective**. Either it works for you, or it does not find something new.
@Brett : I am aware of the fact that my observations aren't independent. But I don't agree with your statement that N=6. That would mean that a clinical study with 4 hospitals involved would have N=4, which obviously doesn't make sense. It's impossible to give fish in the same tank different treatments, as the treatment is in the water. The 16 fish are independent, but the measurements at t=0 and t=1 aren't. I correct for this by calculating the df using Satterthwaite and then divide them by 2, which gives me a df of appx. 14 for the t-tests.
Unless one of them has written or spoken about this (which I doubt), answers about why this decision was taken are likely to be largely speculation - highly plausible reasons might be given, but they'll almost certainly remain speculation.
Excellent, thanks! I have edited my post for point 2, not sure what I should edit the title of the post to though?
Fantastic, thank you for the advice!
@user3713988 It is normal that the outcomes of cross-validation are somewhat stochastic (though they should be within reasonable bounds). If your models exhibit high variance, symptomized by very large performance differences in cross-validation, you should consider (1) changing modeling strategy, as it seems unreliable (2) performing iterated cross-validation to get more stable performance assessments. I suggest option 1. If you go for option 2 without changing strategy, be aware that your model may perform well below the cross-validated performance estimate if you are unlucky.
I believe the OP means an *exact* solution.  That would be equivalent to $y$ being in the space spanned by the columns of $X$ and $Z$.  Any correct answer would have to be some kind of restatement of this (obvious) fact.
We're using C# but Java is a comfortable language for example code! Awesome. Will study it now.
...cont The number of observations that is required to identify structure is based upon the ratio of signal to noise. The higher the ratio the fewer the number of observations. In this case (aside from seasonal structure) there is no evidence of a level shift or a time trend. A simple mean and an ar(1) is quite adequate while isolating 4 of the 19 values as inconsistent with that model. In a perfect world ( perhaps academia ) one can struggle with wanting to know the data and it's potential causes. In reality having tens of thousands of such series one wants to totally automate the process.
That makes sense even thought it is not so obvious. Thanks, Luca
or are you asking what would be a good statistics to compute accuracy without being too sensible to outliers?
Thanks a lot for your clear answer.
@BrianBorchers: Thanks. Edited. The function I am looking for is something like $\\sum_{i=1}^{n} log(a_i^tx) -a_i^tx$, which is concave as it is a composition of a concave function with a linear function. There seem to be many sufficient conditions for when $f(x)=||y-Ax||_2^2$, but I've not found anything on a general function.
Are you assuming distribution of $\\epsilon_t$ identical? Also what about $m$? $m<t$?
Ah, if I knew... I'd quit my job...
Have you thought about using reclink in Stata?
Your answer to `p(2H|T1=T)` is wrong (I didn't check any after that).
thanks for your answer... I agree with you... modeling the endogeneity will do the job. I was rather interested in an example or code that setups this kind of model... I don't want to spent lots of time in learning GLLAMMs syntax
let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/14246/discussion-between-jsk-and-user134724)
Is this such a bad thing? I understand the usual argument of "the mean is either in the interval or it isn't", or at least I think I do. But the endpoints of the confidence interval are random variables, so why is it wrong to talk about the probability that they take values above and below the true mean?
If you were interested in American football and interested in total number of points scored in a game, the range might be big enough to make an ANOVA reasonable, even if it's assumptions are not strictly met.
Hint 2: What is the constant of integration?  How would that change if you put an extra $x$ in front, as when integrating to find the mean?
Ama>: imho that's much too little. Even if the observations were drawn from a single process (i.e. a single run of 4*16=64 *differentiated* observations), the granger test for causality would probably require more data points. Now in your case you would also have to account for the loss of degrees of freedom to control for the fact that your observations are coming from different process.
Care to elaborate on what ICC(1), ICC(2) and Rwg are? I know ICC stands for intra-class correlation, but what is the distinction between 1 & 2? Also I do not know what the acronym Rwg refers too.
Negative Binomial takes count data, just like Poisson. Binomial (logistic) takes {0, 1} response.
you could simplify the model by excluding seasonal dummies, auto arima automatically fits seasonal models, unless you want to determine the effect of each month. Are you open to other models ? if so you could try unobserved components model.
re neuromorphic computing, somewhat similar question [is there any programming model for self-learning neuromorphic systems](http://cs.stackexchange.com/questions/21491/is-there-any-defined-programming-model-for-self-learning-npus) cs.se
Admittedly I'm not sure what you mean by your question, as I've had fairly little contact with statistics until recently.  I ran a logistic regression model (generalized linear model with logit link) with accuracy as my dependent variable and SOA as my independent variable.
Loess is a nonparametric smoothing function.  I don't know what AdamO is talking about when he says that loess throws out influential observations.  It gives weight to all the data in a neighborhood of a given y.
Here is a minor simplification: define vectors $x$ and $y$ with $x_i=a_ib_i$ and $y_i=1-b_i$. You seek an ordering of their components for which $x\\cdot z$ is extremal where $z_i=y_1+y_2+\\cdots+y_{i-1}$ (and $z_1=0$). Incidentally, there are $n!\\gg 2^n$ such orderings.
possible duplicate of [How often do you have to roll a 6-sided dice to obtain every number at least once?](http://stats.stackexchange.com/questions/48396/how-often-do-you-have-to-roll-a-6-sided-dice-to-obtain-every-number-at-least-onc)
Did you add the same amount of Elodea to each of the 3 'treatment' tanks?
I thought economists believed in "In the long run, we are all dead". Where does asymptotics come in then ?
(1) What *are* you a fan of?  (2) How many dimensions do your "points/lines/..." have?
Interestingly, the link to the popular press article in my comment to the OP gives an example where the the sequence 1,2,3,4,5,6 was the *second most popular* choice in data from a [Jan 2010 lottery in Quebec](http://www.loto-quebec.com/loteries/nav/en/useful-information/popular-selections/accueil).
1)Do the store's generic brands sell better above or beside their name-brand counterparts? ... This is descriptive/arithemtic and lacks rigor                                                    2)Did last month's decision to move product X help or hurt sales of the goods across the aisle?.... This is inferential requiring a model                                                                                                                       3)Are there any unusual sales patterns that jump out?  . This is inferential and requires a model
"is also linear in terms of the number of weights" in terms of space dimension, not the number of weights. You can have linear classifier parametrized with as many weights as you want, but its VC dimension is still d+1 (where d is space dimensionality).

"the VC dimension for SVM models is related to the number of Support Vectors " How exactly is VC dimension realted to the number of SV? I'm aware of the hard margin bound, but afaik in the soft margin case there is no such relation. Even in Radamacher's complexity bounds you won't find the number of SVs as a variable.
The prior should represent your prior information, so it is up to you. If you instead want distributions that make you easy the implementation of the gibbs sampler, i will suggest $M \\sim N()$, $V \\sim IG()$ and maybe $b \\sim G()$, but for the last I am not sure...
Not really. If we restrict ourselves to the continuous symmetric unimodal case with finite 4th moment (since we're discussing kurtosis), in many cases I don't think it makes much sense to call anything *outside* $\\mu \\pm \\sigma$ 'the peak' nor anything *inside* $\\mu \\pm \\sigma$ 'the tail', but sometimes it's hard to say. e.g., consider $f(x) = (3+2a)/6 - ax^2;$ $-1<x<1,  0<a<\\frac{3}{4}$; when $a$'s near $0$, there's no obvious place to start calling any of it the tail. On the other had, with the Laplace distribution, you could arguably call anything either side of the exact center the tail.
Even on Statalist, where I am active, we expect a bit more effort (e.g. > 0) than is evident here. Otherwise "please tell me the code I need" is always a fair question, and in practice it isn't. However, there is a statistical question hidden here, which is what is the equivalent of difference-in-difference design for three periods. My own wild guess is that is not difference-in-difference. So, I think there is some scope for rewriting this as an interesting statistical question, but it can't be acceptably phrased as what Stata code do I need.
(+1) Just to add a bit to the remark "Standard deviations aren't 'good' or 'bad'" - having a predictor with large standard deviation can be "good" because, in regression, it's inversely related to the standard error of a regression coefficient estimate. On the other hand, if you're concerned with precision of a measurement, then a large standard deviation is "bad". I'm guessing the original poster's interest is closer to the former but it's not clear.
You could also try one of the hierarchical methods with an appropriate measure of similarity/dissimilarity. For tha latter, Sokal-Michener, Sokal-Sneath, Rogers-Tinimoto, Russell-Raova, Jaccard, Czekanowski, Sokal-Sneath #2 measures of similarity comes to mind.
are they equal size
This answer doesn't address the time series aspect, including seasonality, at all.
Thanks. This solution didn't seem intuitive for me when I first thought about the problem since the confidence interval for the overall p for me hitting 1 out of 100 and my friend hitting 1 of 1 and me hitting 50 out of 100 and him hitting 1 out of 50 are the same. Subjectively I would give less certainty to the 1% success probability in the first scenario.
Which paper are you referring to? Splitting the node using an SVM sounds like a very deterministic thing to do (not in the spirit of random forests). This way, the only randomization will be achieved by bagging of the training data, whereas modern implementations randomize the split conditions.
It's worth a new post because it's now a somewhat different question (about scale -- your input here really helped though!). Take a look here, it's now much better posed: http://stackoverflow.com/questions/24902226/how-to-combine-multiple-ggplot2-objects-on-the-same-scale
This may be over-relying on a particular choice of phrasing; you're assuming rather a lot from what might be simply a poor choice of words - not everyone here has English as a first language. It's definitely worth raising as a potential problem, but to simply state things so baldly ("absolutely not") implies you know more than we can tell from what's here. (Further, the reference to a 'lab notebook' implies the OP is doing work in a lab. I doubt this is the case. Again, you imply you know more than we have here.)
It's a good question (and may have some discipline-specific answers in addition to certain [legal interpretations](http://legal-dictionary.thefreedictionary.com/Preponderance+of+Evidence)), but my personal opinion is that it's usually a mistake to shoehorn inherently qualitative assessments into a pseudo-quantitative language because it will only deceive others concerning the fuzzy nature of what is being asserted.
I imagine what it's doing is "reconstructing" the matrix from the first few singular vectors. The link to the blog post is broken btw. But that would make absolutely no difference if you go on and do PCA to the reconstructed matrix! Perhaps it's centering/scaling for you, then you would get different results. I'd look into it further; it seems like you want to be working with the scores from LSA to do clustering/categorization/etc
Why do you think they have to be negatively correlated? They don't necessarily have to be, as long as the [covariance matrix](http://en.wikipedia.org/wiki/Covariance_matrix) of the triplet is [positive definite](http://en.wikipedia.org/wiki/Positive-semidefinite_matrix#Negative-definite.2C_semidefinite_and_indefinite_matrices). For example, if ${\\rm cor}(X,Y_1) = .5, {\\rm cor}(X,Y_2) = -.5$ and then ${\\rm cor}(Y_1, Y_2)$ could be anywhere in the open interval $(-1,.5)$ to still produce a valid covariance matrix. Some information about the magnitude of these correlations would be helpful.
(1) The post you refer to concerns a spreadsheet mistake, not a problem with linear regression, and it's about prediction limits rather than confidence limits. (2) I didn't notice this the first time through, but did you see that the question concerns a *quantile* regression?  It's not conventional linear regression.  Even so, @Alexis is correct: given an estimated covariance matrix for the coefficient estimates, an approximate CI can be constructed for any linear combination of the coefficients.
This question appears to be off-topic because it is about programming without a reproducible example.
To me it seems you don't calculate the sum of squares correctly; you have to subtract the total mean of the group from the group mean in the first step.
Yes, it is. But the way is the same. The logical chain is: Put $H_0$->express it through linear models->decide to compare variances of residuals->use F-test. You have said the first and the last stages, missing intermediate.
Thanks, i'll work on that modification.
+1. A check, using `R`: `dpois(2, 0.7 * 110 / 54)` returns $0.2442824$.
I corrected the first point. But I don't think the linked 'duplicate' addresses my question. I'm asking about two contraints at the same time
@Brenton Home, did you run this part of the code:

`for I = 1:length(CI); plot(I,[CI(1,I):CI(2,I)]); hold on; end;`

That should draw the lines. I'll make a screenshot in a second.
Andreea, this answer clearly states you *cannot* validly aggregate these data (without some further analysis, anyway;  taking the maximum is practically nonsensical.) Note, too, that both variables actually are *ordinal*: they carry more information (via their natural ordering) than merely categorical variables. How you deal with them depends on what role they play in your analysis--as independent or dependent variables--as well as their relationships with other variables, which you will need to discover through exploratory analysis.
+1, nice answer
Some sample reproducible code of what you are trying to do would help a lot.
(+1) What you say is true, and a very important point. The triangular array allows for the variables in each "row" to live on different probability spaces than previous rows. On the other hand, if we say a priori that we are considering a sequence of iid random variables, then, implicitly they must exist on a common underlying space for the notion of independence to make much sense.
Looks like a bug. Try contacting the package author.
Thanks, I haven't thought about this problem for a while. I will give your suggestion some thought.
I'm interested in the expected time until the first event $X_i=1$ so it would indeed stop at $X_0=1$. A variant, which is actually more interesting, would be to have a prefix of length $w$ in which no 1 may occur, but I thought I'd start by asking the generic case and try to work out the prefixed case myself. All the better if the probability does not approach 1 for arbitrary large $t$ since I want to find an upper bound on something bad happening :-)
Cool, @Kumara. Be aware of the "abstraction trap": the best way to build your intuition about these things is to compute every little thing for some concrete simple model. Take $X_1,\\dots,X_n$ as an iid sample from a $N(\\mu,\\sigma^2)$ and compute everything: the Kullback-Leibler divergence, the Jeffreys metric, the Christoffel symbols, the Riemannian tensor and its contractions, Ricci tensor and scalar curvature. Solve the geodesic equation. Compute Amari's $\\alpha$-connection and associated curvature. Is it easier to compute things if you use the natural parameter of the exponential family?
There are other situations in which one would prefer probit as well. Econometric selection models (i.e. Heckman) are only proven using the probit model. I'm less sure of this, but I also believe some SEM models where binary variables are endogenous also utilize the probit model because of the assumption of multivariate normality needed for maximum likelihood estimation.
What model is appropriate depends on exactly what 1,2,3,4 mean here, whether they are counts or codes.  Incidentally, and irrelevantly to your question, the term "polychotomous" is common but "polytomous" is better Greek. "polychotomous" is based on misparsing "dichotomous".
Thank you for your comment. My input variables are financial ratios and variables and my target variable is bankrupt (1), otherwise (0). After training neural network (or `SVM` model) and inserting out sample data (new year data), now I can have probabilities of bankruptcy besides binary output ( 0 and 1). Now i want a hybrid method to sort data. I don't want only sorting based on size of firm or probability. Suppose that you want help bankrupt companies. You want know relatively biggest companies with higher probability. Someone recommended me to use `PCA` for this but i don't know how.
It does not sound like computing a covariance matrix would address your problem. Instead, it seems that you may have one or more measurements of the distance of something (along with its orientation), made with a known amount of radial error, and you wish to estimate the location and quantify the uncertainty in that location. Would this be an accurate interpretation?