tttjjj/elt_bench huggingface dataset:
His proof works with his rephrasing of the question in which $v_1, v_2$ are correlated for events having the same index, but bruce has made it clear that $v_1, v_2$ are correlated for events having the same **time**.
Thank you for your nice example. Actually we are well aware of the differences between RCB and CR, see [here](http://stats.stackexchange.com/questions/20806/what-is-a-block-in-experimental-design/20815#20815) for example. We chosed to go with RCB, assigned the levels to the experimental units, started the experiment but, on a successive rethinking, we reckoned that a CR analysis would have been better. Since we cannot stop the experiment, reassign levels to plots in a randomized way and start again, how much bias would we introduce analysing the current configuration pretending it is a CR one?
What's the reason for the vote to close?  How does this problem not relate to statistical analysis?
Are there different dependent variables in the two models?
I have analyzed your data but I can't post the jpg files because "framing is not allowed" . I don't understand. If tou wish you can contact me directly and I will send you the very interesting results.
I don't see where `O` is actually used anywhere to inform your model.
@Sycren You could do a ratio like "energy of top sinuosoid complement" / "sum of energies of all other terms"
+1 thanks for the book
As noted elsewhere: the problem was that your step sizes were _not_ in the hundreds...
I have to admit that, even being a convinced Bayesian, having read several treatments of the subject (popular science ones, in particular Mlodinow’s, and text books) as well as understanding the underlying statistics, this result surprised me. Now, it’s easy to see that it’s in fact true – both by systematically enumerating all possible scenarios or by simulating (I did both). But surprising nonetheless.
This seems like [routine bookwork](http://stats.stackexchange.com/tags/self-study/info), perhaps assigned work for some subject, or perhaps just for your own personal study. Either way, could you please add the `self-study` tag, and read the guidelines at the link I gave above, then edit your question to accord with those guidelines.
What exactly do you mean by "error for linear regression"? Do you mean an estimate of the variance of the errors, standard errors for the coefficients, standard error of prediction, or something else?
Yes it is the case. If you write down regression equations based on the contrast matrix returned by R and try to identify each regression coefficient, you will find that $\\beta_0$ is the mean of all levels (like in deviation or sum contrast) while $\\beta_i,\\, i=1\\dots j$, represent sequential comparisons between adjacent categories (2 vs. 1, 3 vs. {1,2}, etc.).
I am not sure exactly what question you are asking but an obvious error metric for a regression model with a continuous output is mean square error (MSE) between the model output and the outcome variable.
You don't seem to realise that estimates can be produced by any method you like. I could have an estimator that said always guess 42; it would be a lousy estimator in all but very special circumstances, but it would be an estimator. In other words, the class of estimators is not restricted to good or sometimes good named methods such as moments or MLE. The method used for calculating CDF is often just empirical fractions; it doesn't have to be thought of as formal estimation.
I will do a search to inquire as to simulating data in SPSS.  I do you Stata as well so can look at that program as well.
If "a principal component is not a parameter", then *a fortiori* no coefficient of $\\Sigma$ can be a parameter, either (because $\\Sigma$ is completely determined by the principal component decomposition). This is a puzzling assertion.
Do you have any idea, how can i calcualte Java equavalent of  R code `anova(fit)` ?
Feel free to ask it [here](http://chat.stackexchange.com/rooms/18/ten-fold) in [chat] or create your own chat room for this specific discussion.  Interesting question!
BTW something seems to be off with the documentation for the R 2.15.2 / rms_3.6-3 build for Windows.  The help file shows `\\dots` rather than the expected $\\dots$ for the first listing of Usage, then followed by a more typical usage section for ?pol.
Impressionistically, yes, based on the magnitude of the difference of their coefficients or odds ratios (and probably considering those differences in light of the standard errors for each variable).  But if you want to formally test the stat. sig. of their difference, you do need to use one as the reference category.
I've voted to close this question (along with 2 others) because the question solicits opinion. Whilst some questions seek opinion-based answers, those that rely solely on those opinions are considered OT for [stats.se]. If the question can be backed up with experience, supported by other facts etc, then it *may* be on topic. At the moment your question purely seeks advice and such questions are not a good fit for any of the [se] sites. If you can reformulate the question to be more specific, it may not be closed (or could be reopened).
I understand the first part of your answer where using the _Law of total expectation_ you show that $E(E(Y|X) - E(Y|X)) = 0$. This way the expression $2 (Y - E[Y|X])(f(X) - E[Y|X]) = 0$, so could you please elaborate the second part of your answer, following _To finish the proof..._
Thanks for the direction I will look into both options.
Maximum entropy is a principle much wider than multinomial logistic regression. It can be applied to continuous problems too. Terminology varies, but I wouldn't call maximum entropy a model at all.
I tested the data from a social network but I do not know the strength of relationships that links users and their kinds
This question need more information because the answer depends on just how non-independent the data are (as well as a number of assumptions).  What are the values in the table?  BTW, is it a 2 by 2 table (as in the question) or a 2 by 3 table (as in the title)?
Thanks for a prompt response. HMM came to my mind as well, but as far as I understand, there the unknown Y would be a Markov process, and much of the work is to estimate its transition probabilities. My set up is much simpler - it's just one variable..
Your link is very nice! I liked it! Thanks
Thank you very much for this nice post which answers all my questions.
Collinearity is *not* the same as correlation; you can have high correlation without problematic collinearity and you can have low correlation and high collinearity.

To test for collinearity, use condition indexes, not correlation. The links provided byu @gung have details.
Does is make sense to use frequent items even though i wish to cluster the users rather than the tags (columns)?
maybe I didn't choose the headline carefully enough. i agree that a a plain poisson model with some covariates and an intercept is equivalent to an exponential model as you said. but a poisson model with a log time covariate is equivalent to a weibul model. this is at least what is shown in the cited paper...
Annnnd one last question: It would be more intuitive to look at the effects on the original data than the mean-centered data, but can I do that? Maybe the best way to do it is to standardize everything and visualize the effects that way. Its just not as intuitive to look at the +1 SD and -1 SD effects on the mean-centered data, ya know what I mean?
@BSeven In principle no, the larger action space doesn't change the answer, although you might have to do more pruning and use a lot more computing power ;-) Note however that this is a perfect candidate for parallel processing. The question of the stars is tricky, as this implies that there is no simple mapping from kills to stars, although I thought that you got more stars simply by crossing points thresholds (usually this is done by using less birds). If not, you'd have to artificially increase the amount of exploration to avoid settling on suboptimal paths too early.
@Aksakal, so divide the data into buckets (say monthly level). And calculate the median/mean of each bucket. Then look at the depreciation rate between buckets?
"Though for the purpose of predicting sale/no sale it wouldn't hurt to use logits or probabilities in place of the multiple variables used to create them."  That's really the essence of what I'm going for @rolando2.  But how should I approach that?  Using the probability? (But wouldn't that cause problems?  I mean, can you use probabilities as variables in a logistic regression equation?)  Or should I use the logit?
If you're primarily interested in how the SE for proportions works, it can stay here. It might be sufficiently on-topic on either site, so it could be your choice.
I am working my way to there, will reply soon once I get there.  thanks!
Did you try to check your random variable against the definitions for sub-super-martingale?
The _likelihood function_ is a function of the unknown parameter $\\theta$ (conditioned on the data). As such, it does typically not have area 1 (i.e. the integral over all possible values of $\\theta$ is not 1) and is therefore by definition not a pdf.
@pom I hope you can clarify this query.  In addition to the issues raised by @onestop, your modification of both "samples" and "models" by the word "independent" makes one wonder about your precise meaning and your synonymous use of "ratio" and "difference" in the same sentence raises questions about what you mean by those words.  One possible interpretation is that each sample consists of a set of ratios; another interpretation is that you are estimating some kind of statistic in each sample, taking the ratio of those two numbers, and want to compare it to some standard value (such as 1.0).
It might simply indicate that the only systematic dependence between the series is due to the common factor "time".
@whuber yes,it is the density function. The domain is $t>=\\theta$, otherwise, $f(t)=0$.
...and you won't need the CLT for the asymptotic distribution. Think $e$, when the time comes.
Do you know whether this is (similar to) Quinlan's M5?
@manuel.kuehner, a repeated measures ANOVA is a mixed effects regression model w/ all categorical covariates & a random intercept for each subject, but no other random effects. This necessitates a compound symmetry covariance structure (all r's =). Since your data appear to be inconsistent w/ that, rmANOVA probably isn't appropriate for your data. You would ultimately want to fit a mixed effects model w/ an *unstructured* covariance pattern. However, simply for the sake of determining a reasonable N, you can just plug in each of the r's & see how the N's change; that's likely to be sufficient.
Calm down, and go collect some quarterly and monthly data. Higher level of temporal scale cannot be decomposed into lower level, unless you are willing to make some assumptions which are often unrealistic.
Short answer is kernels. Long answer is keeerneeels (-;
@gd047, some tests assume normality of the distribution of the mean, not the data.  t-test tends to be pretty robust w.r.t to underlying data.  You're right though -- with post-transformation tests, results are reported after inverse-transforming, and interpretation can be very problematic.  It comes down to how "un-normal" your data is, can you get away without transforming or applying, say, a log transform which is easier to interpret.  Otherwise, it's contextual on the actual transformation and domain and I don't really have a good answer.  Might be worth asking to see what others say?
All of these are behind some kind of pay wall. Can you provide alternative links or resources?
Yes, nested models are hierarchical in nature. In the case of a variable representing degree (i.e. degree of risk), you do not need a hierarchical model. Depending on how the risk index you're using is divided, you may even want to consider grouping the levels on the index (i.e. if it's 1-9 consider grouping as high [7-9], medium [4-6], and low [1-3]) as this may allow you to better represent the dominance effect where those in the reference group represent a substantially different outcome than those in the comparisons.
Thank you for the answer Mr. Harrell. I understand what you mean.
Dear Jeromy,
thanks a lot for your feedback. Anova seems correct to me. Anyways I am just having a look to a "conjoint measurement analysis" and apparently it seems to fit my problem.

All the best
Thanks a lot, I will study your suggestions and I will get back to you as soon as possible. Any other suggestion? What about PCA? Does it sounds a good idea? I am not an expert in PCA... ;-)
One suggestion is to consider something like ACE or perhaps AVAS to try to get an additive model (see package `ace`). Then once you have a transformation for Y to approximate additivity, if the fit from ACE doesn't grab you (it's not always sufficiently smooth for me) you can do loess in each variable separately, or use splines, or whatever other univariate smoother takes your fancy
I don't understand the purpose of your questions, Vivek: the code already answers them. Different sample sizes are obtained by varying `data`; the curves of the log likelihood are apparent as the contours in this plot. I also do not understand what you mean by "log-likelihood functions for Cauchy(0,1)," because this is not consistent with what likelihood means: the likelihood is a *number* for any particular distribution, such as Cauchy(0,1), not a function: it is the probability of the data for that distribution.
@Dougal It computes a weight per feature indirectly. Linear SVM *still* computes instance weights just like its kernelized siblings. Since a linear combination of linear inner products can be summarized as a single inner product, the solution of linear SVM *happens to* translate to feature weights. That said, you are correct that they are not immune to overfitting, so I modified the answer slightly to reflect this.
Surely, you are aware that robust statistics and outlier detection has improved a lot in the last *150 years*. Is there any reason you are using this test?
http://peekaboo-vision.blogspot.de/2013/01/machine-learning-cheat-sheet-for-scikit.html
I changed my answer to reflect your questions.
He could use the VonMises for the angle and the Beta for the radius.  Either independent of each other, or the parameters of the beta could be dependent on the angle.
@Hempo: That is correct: the likelihood increases with $\\theta$, so you want $\\theta$ to be as large as possible, subject to the constraints that it is less than or equal to each $X_i$. So you want $\\theta$ to be equal to the minimum of the $X_i$ for the maximum likelihood.
(1) My analysis finds that the expectation expressed in the first sentence is not borne out: with a sufficiently large number of variables, if one of the distributions has short tails and the other has just a slight chance of exhibiting more extreme values, then the correlation tends to be unduly high. (2) When you "model ... against different distributions" how do you control for the multiple dependent tests that are implied by that prescription?
Thanks. Great support from a fellow adventurer in this world of statistics! ;) I'm currently reading Discovering Statistics using SPSS by Andy Field, which I'm surprised to enjoy (since it's a statistics textbook). I altered my COX analysis to measure survival on days instead of months, which luckily pushed my the significance of my 'risk' covariate below 0,05... :)
Please, visit our Help Center to [merge your accounts](http://stats.stackexchange.com/help/merging-accounts) and register once and for all.
For $W$ and $Z$ to be uncorrelated (or nearly uncorrelated), we don't need $\\operatorname{cov}(W,Z)$ to be $0$ or nearly $0$: we need the Pearson correlation coefficient $\\rho_{W,Z}$ to be $0$ or nearly $0$.
+1. It's also worth noting that all covariance matrices are positive definite and _all_ positive definite matrices are the covariance matrix of some multivariate distribution. Therefore, saying "non-positive definite covariance matrix" is a bit of an oxymoron. It appears the OP was really just saying that the _sample_ covariance matrix was singular which can happen from exactly collinearity (as you've said) or when the **number of observations is less than the number of variables**.
Because the exact values depend on realisation. Try making 10 RFs on a single set and compare the importance; you'll see what I mean.
They don't have to be independent. CLT's exist for dependent sequences, like martingales and mixing processes. (cc @EconStats)
A rounded x<-c(42,125,81,13,0)
    
    Chi-squared test for given probabilities with simulated p-value (based on 1000 replicates)

results in
X-squared = 95.1776, df = NA, p-value = 0.000999
@Jen, I think that's much better now.
could you provide a link to what you feel is the best tutorial that you've seen using R and survival?
A decision tree looks exactly like a tree with branches and nodes.  You start from an initial node and branch to other nodes which in turn branch down until you reach all the terminal nodes.  Binary trees are special case that have two branches at each node.
@veol A linear SVM is a good place to start, but don't just use the default settings, the key to getting good performance from an SVM lies in choosing a good kernel and in carefully optimising the hyper-parameters (the regularisation parameter and any kernel parameters).  This can be done by minimising the cross-validation error or a bound on generalisation performance (e.g. the "radius-margin" or "span" bounds.  I don't use R, but any decent SVM package should automate this procedure via gradient descent or grid search.
I believe it is absolutely continuous on $(0, \\infty)$. It is realized as a limit of compoud Poisson random variables; as $n \\to \\infty$ a compound Poisson with rate $\\int_{1/n}^1 \\frac 1 x \\ dx$ and jumping distribution density $f_n(x) \\propto \\frac 1 x I(1/n < x < 1)$ converges weakly to this distribution.
I have added the requested info, when using boosted methods generally what i get is the lower portions of the single trees remains very similar.
@PeterFlom, MichaelChernick: Peter, The paper to which Michael is referring is now posted under Early Views on the Computational Statistics: WIREs journal webpage at http://onlinelibrary.wiley.com/doi/10.1002/wics.1266/abstract A preprint can be downloaded at http://www.datamineit.com/DMI_publications.htm All feedback is appreciated. Best,
J.D. Opdyke
Displaying the model and fitting it are two different things. Your examples are precisely what you require, given the data equation is fitted. Blindly fitting the model without domain knowledge is usually not advised, as the resulting model might be spurious.
I disagree with AdamO and Greg Snow on the general statement that nonnormality does not hurt prediction. For heavytailed distributions at least, least squares is overly sensitive to outliers and leverage points. This affects the accuracy of the regression coefficients and consequently your prediction accuracy. Assume the form of the model is correct and the residuals have zero mean and constant variance but are not normal then the Gauss-Markov theorem says that they will still be minimum variance among unbiased estimators. In that sense the estimates are still good.
(i) Please read the [self-study tag wiki info](http://stats.stackexchange.com/tags/self-study/info) - what have you tried? (ii) note that, as given, the above statement is false. For example, if $Y_i$ is Cauchy, no scaled and shifted version of $T_n$ converges to $N(0,1)$. Did you leave out part of the question?
Except, the type I scenario could also end up in significant patient harm if the patient doesn't have a terminal condition and the (falsely positive) treatment has significant side effects. Both type I and type II are bad, depending on the context.
@PeterHopkins: Did you get any warning messages during installation? I tried `install.packages("~/myfolder/bivpois_0.50-3.1.tar.gz",repos=NULL)` and it succesfully installed the package in R 2.15.1 on Linux. I guess that it's possible that this is an R Studio issue - would you be willing to try to install it in "regular" R?
Is `xxoooo` equally concentrated for you as `xxxxoooooooo`? One might argue that the 2nd is more concentrated because despite that there appeared more positional options to break (`xxxoxooooooo`) the 4 x's cling together still.
I actually know this paper. They are converting their 1D temporal signal to a 2D frequency X time matrix with a spectrogram and the used 1D "filters" for the connections to the hidden units. It doesn't seem like they are looking at the frequency-time structure. Am I wrong?
+1 Excellent point. This indicates the preliminary running mean suggested in my answer may be superfluous and that loess alone is sufficient.
Wow. :O Thanks for your time! Will look into it in detail tomorrow.
Also, you cannot just achieve the transformation $$r^2 = x^2 + y^2$$ by making that substitution into the density function ... it is more complicated than that ... you need to calculate the Jacobian of the transformation too.
+1 For creativity.  Actually, your idea is on the path toward practical utility.  It would be quite a bit more interesting to use one of the biking tracking sites (I forget which one now, but do add, if you know) to track segment times.  If the OP were to come back to CV or StackOverflow with a question about plotting segment time and get a density associated with it, it would be a fabulous statistical exercise - GIS, statistical visualization, and density functions, oh my!  :)
Thanks, your answer has completely cleared up my confusion. I have accepted it but cannot upvote because I have less than 15 reputation on this forum.
Presumably the proportion given in the null hypothesis that you're testing against. What does the help say?
Extremely helpful - thank you. I've removed the link to the paper
Don't know of anything on fitting one-inflated Poisson models specifically. As one-inflation is only a slight modification of zero-inflation, reading up on the latter should help. I'd think a few alterations to the `zeroinfl` code should do it - changing the zero-inflated Poisson likelihood & score to match the model above (or try just changing the likelihood & not passing the score to `optim`). You can also of course ask here or on SO as appropriate for references or help with things you're stuck on.
Thank you for your answer :)
A concise comparison: http://baoqiang.org/?p=579
Every time that you do pairwise $t$-tests you have a chance to make a Type I error with probability $\\alpha$. Imagine that you read 20 papers and each of them performs one $t$-tests. Statistically, 1 of those tests will yield a false positive, a Type I error. It doesn't matter if you have 20 papers each with one test or one paper with 20 tests. Further, ANOVA doesn't compare each mean with all other means. It tests the hypothesis that all means are equal vs. that at least one mean is different. There are methods to pairwise compare each mean after ANOVA (e.g. Tukey's HSD).
I encourage you to search and peruse some of the questions on this site. There are several that are very similar and/or related. You might start with the "Related" suggestions on the right-hand bar.
To expand on my comment: I'm not objecting to the basic concept, which I think is sound enough (I've done something similar to obtain prediction intervals for a GLM, and I've seen Ripley suggest the same procedure as I have used for GLMs), but I think it needs more explicit detail about the simulation, and a clearer justification of why it should be reasonable.
and final comment. R has a powerful formula interface, use it.  Look at the code of lm, and functions model.matrix, model.frame, model.response. Using strsplit and parse on formulas and call objects is a bit of trying to fit a square peg into the round hole.
@Erogol I will use labelled data to train the NN, with 1 being a positive case of the class and -1 being negative.
Sorry but I don't see why for each $j$ we must have that $\\sum_i \\theta_{ij} = 1$?