tttjjj/elt_bench huggingface dataset:
@Xi'an Here is a [preprint](http://hal.inria.fr/hal-00666733/) by Victor Kleptsyn and Aline Kurtzmann with a counterexample to the Cantelli conjecture. The construction uses a new tool, which the authors call the Brownian mass transport, and yields a discontinuous function $f$. The authors state that they believe that Cantelli conjecture holds if one asks that $f$ is continuous (theirs is a mixture of two continuous functions).
+1 This is a good answer, in my view.
I would add a resounding "agree" for Harrell's (note spelling) text. It is excellent as is the two-package combination of the R code that accompanies it. I also think "Modern Applied Statistic with S" by Venables and Ripley would be a good acquisition. I had a masters level background (with an undergrad degree in physics)  before using MASS to learn R. There is a wealth of application-wisdom in that text.
+1 The reason why (2) is impossible is trivial: the variance is defined in terms of the mean. Slightly deeper is the fact that when the second moment of $X$ is finite, then the mean must be finite. For if the mean is infinite, then *a fortiori* the second moment must be infinite because the second moment is weighting the values of $X$ not only by the probability but also by $X$ itself ($X^2 = X\\times X$).  Those weights grow without bound, causing the second moment eventually to exceed the absolute value of the first moment.
(+1) Just like to add that data reduction guided by subject-matter knowledge & common sense alone can also be effective - which might be appealing to a non-statistician.
Have a look at the answers to [this question](http://stackoverflow.com/questions/2018178/finding-the-best-trade-off-point-on-a-curve) on Stackoverflow to find the 'elbow' in the ROC.
Corsario, that is the correct answer (although it won't be pretty to implement). I suggest posting it as an answer. Note you've also assumed that each $X_i$ is independent of each each $Z_j$ for $i \\neq j$ (for the convolution part), but I'd imagine that was intended as part of Remy's description.
After a more careful read, the answer might be in the first reference (Neyman, 1934). See equation (28) in Note II on page 597.  But I'm still not sure the FPC originates elsewhere since the concept is fairly basic.
see [R FAQ 7.41](http://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-does-summary_0028_0029-report-strange-results-for-the-R_005e2-estimate-when-I-fit-a-linear-model-with-no-intercept_003f), also FAQ from [UCLA stats page](http://www.ats.ucla.edu/stat/mult_pkg/faq/general/noconstant.htm)
Ahh i see, so it's kind of like utilising fuzzy set theory. This is the approach I eventually went with, or a very similar approach anyway. I call this question closed, it's pretty useless to most people out there!
This statistic is the inverse of Geary's 1935 $a$ statistic for testing normality against symmetric alternatives (with mean taken at zero). He considered its distribution, primarily in the normal case, e.g. [see his 1947 paper](http://webspace.ship.edu/pgmarr/Geo441/Readings/Geary%201947%20-%20Testing%20for%20Normality.pdf) where he considers; it may be worth considering using his statistic and approach.
could you at a minimum provide the input data that you are using?
Plotting the data is one answer. Presumably you know that, so please be more precise about the kind of method you seek.
Of possible interest: [Differences between cross validation and bootstrapping to estimate the prediction error](http://stats.stackexchange.com/q/18348/930).
@JavaCake: I'd look at `cut` and `table` instead of leaning on `hist`, but you need to tell `hist` what your actual range is. Something like: `foo <- hist (x, breaks=seq (1, 10000, length.out=10))` Which says your data ranges from 1 to 10000 and it should be broken up into 10 bins. You should look at the documentation for `seq` (type `?seq`) for more details.
I understand what you are trying to say, but I don't think "varianced" is a word, even in statistics. ;-)
As stated this is simply a test of lag1 autocorrelation. DW statistics and derived statistical tests should be sufficient.
Don't you have 8 parameters as is? Or are you only fitting main effects?
I remember the interpretation of the p-value as (in this case): when assuming that the null-hypotheses (i.e., homogeneity of variances) is correct, then the probability of obtaining this or a more extreme result (i.e., contrary the null) is 57% or 95%. But whatsoever, the conclusion is the same and correct.
What does the 3rd factor look like in the 7 factor solution? What about the 1st factor in each? If the 1st factor is taking a lot of variance, the rest may be noise. How many items in the survey?
Consider this -- if that interval had been $(-.07,\\;+.12)$, would you be getting a different message?
Correct, the horizontal axis is the hour, 0 to 23, and the vertical is the traffic flow, so the dots you see are the traffic flow in that hour over the 30 days, so to say each hour should have 30 days of hourly flow data point.
@Jens, if you consider hierarchical structure of a mixed effects models, than static covariates are more than in spirit of multilevel models. You may look in John Fox http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf for some additional details.
+1 I guess you beat me to it.
The question and its title are not related. There is just one Statistics and ML did not reinvent it, or modify it. I don't know of any standard method that would have differing definitions in the fields. But you are not really asking this question anyway. You are asking whether the implementation of Logistic regression differs in Mahout and R to explain different results. I have no clue, but from looking at the Mahout regression code, they use stochastic gradient descent to minimize the (negative) log likelihood, while R uses IRLS. So both of these algorithms need convergence.
@David I had misunderstood your question. I think I now see your point, and have edited my answer to address it.
@wok Of note, the scikit.learn package also offers efficient implementation in Python for this kind of stuff.
I enjoyed reading this clear, helpful response and am going to share it with a colleague.
Ooops, I wanted to edit that comment for clarity but did not know I only could edit for 5 minutes, can I delete a comment?

I did not realize that I could get mean estimates directly by omitting the intercept, thanks for that tip.

If I understand you correctly, the standard errors of the estimated means are not the same as the standard errors calculated directly from the data. Is it a different set of equations used in each case? And what are these equations? I would like to have some more details to u nderstand the difference better
How much time it takes to build model one time? For forecasting purposes it is always best to build the model with the most recent data. However since you have lots of data, I suggest doing various forecasting scenarios and then compare them. Then you will have a feel what is the cost of each scenario.
Thanks all for your suggestion. Definitely I will try those packages in R.@hrbrmstr,yes I want to run it locally in my windows system
If process depends on previous values of itself then it is an AR process. If it depends on previous errors than it is an MA process. To identify whether it is an AR or MA process it is always better to look at both ACF and PACF (partial autocorrelation function). Although ACF of AR goes to 0 slowly by lags sometimes the decay rate is high so that it will appear as if you have 3 lags in ACF and incorrectly conclude that it is an MA. PACF helps you distinguish some ambiguities.
Some other resources I have found on this subject:
[Levin 2011](http://onlinelibrary.wiley.com/doi/10.1002/pmic.201100033/abstract)
[Dicker et al., 2010](http://www.mcponline.org/content/9/12/2704.long)
Some key `R` functionality to look up includes `ecdf` (*empirical cumulative distribution function*) and `density` (*kernel density estimator*).  Concerning the histogram approach, please consult http://stats.stackexchange.com/a/51753 before you go any further.
By the way, any reason why you are not using R package _caret_ that supports this feature? (and lots of other goodies) - see bottom of the page 19 of [caretTrain vignette](http://cran.r-project.org/web/packages/caret/vignettes/caretTrain.pdf).
what about the second model and residual analysis (and what analysis made you sure that no anomalies are present)? what estimator do you use for AR(1) with deterministic trend (the first model)?
"I am not sure if this method is valid in my case for the following reason: each of these two data series, Data1 and Data2, is itself a weighted-average of a bunch of standardized individual series.": can you explain why you think that is a problem for using order statistics?  Transforming using the cdf just means saying that the 99% highest value of Series 1 is comparable to the 99% highest value of Series 2, the 75% highest value of Series 1 is comparable to the 75% highest value of Series 2, etc.
Hi StasK, thanks for your response. With regard to Neutral categories - they are forced choice responses. E.g. strongly agree / agree/ disagree / strongly disagree (4 options), very easy / easy / difficult / very difficult / cannot do (5 options), and often / sometimes / rarely / never (4 options). I would think that they are equidistant... If I were to be testing discriminant validity between constructs, wondering which processes would be best for these differently 'weighted' response options?
(1) I am curious why MaxEnt might apply here, because it seems to me that the more long-tailed the underlying distribution becomes, the more uncertain any sample-based estimate of its maximum will be.  That suggests this principle might not even be relevant to the question. (2) Of what value is an inequality relating the maximum of a sample to its expectation when the concern is about the maximum of the *population*? This inequality seems to ignore the potentially huge negative bias in using the maximum of a sample to estimate the population max.
The p-value shouldn't systematically increase or decrease if the null is true.
Nick, mediators != covariates. The OP has a multivariate system, not just a single equation. Mediation is a term from causal effect modeling literature, and that the stuff that social scientists had to come up with on their own because statisticians did not bother helping them out.
Since there was some specific recommendation made in terms of using orthogonal polynomials elsewhere I'm holding off on accepting an answer until I see a response from the opposing camp. Please vote up Frank's answer if you agree with his position.
Thanks user777 and dilil. Appreciate your pointers.
Thanks that helped a lot!!! Is it necessary to use the single-step method?
@jefflovejapan, I had a chance to glance through Canova's book. I am not economist, so I cannot judge on economics content of the book. However mathematical content is awful, inconsistent notation, strange definitions of well known concepts, and plainly wrong mathematical statements. So I definitely do not recommend this book if you want to get acquainted with new methodology.
If you remember your last course, then maybe useful indicators for characterizing the empirical distribution, such as *central tendency*, *variance* or *scale*, and *shape*, might be a good start. The second question about sampling distribution of the mean has already been addressed elsewhere on this site.
+1 Great point about setting the seed values.
Thanks... that does clarify the things.
There's only one game structure, however the number of distinct categorical answers is unlimited... This seems like empiric distribution applied to categorical data.
With parameters this large, for many purposes you can closely approximate the distributions of the $\\sqrt{p_i}$ with Normal distributions. For instance, $\\sqrt{p_0} \\sim$ Normal$(0.241186, 0.0186454)$ approximately. The CDFs differ by at most $0.004$. This enables you to capitalize on the extensive knowledge of ratios of normal variates. You can go further and approximate the root odds $\\sqrt{p_i/(1-p_i)}$ with normal distributions, too (although the approximations are not quite as good).
Regularization methods can not generally be seen as having one degree of freedom. The relevant concept is that of *effective degrees of freedom*, which sometimes can be defined, e.g. for ridge regression, and which is generally an monotonely decreasing function of the penalization parameter.
@Max Thank you. To be fair, mathematical models in Epidemiology are what I do for a living, and there are some *very* interesting collisions of applied math and statistics to be found.
It sounds merely like a comment. However, I understand that you may be looking for an update to the OP and that you will update this reply accordingly, so I'm leaving it as is.
Additionally it means that $\\frac{r\\cdot\\sum_{i=1}^{t}\\tau_{i}^{2}}{t-1}\\leq{0}$, which would mean that $r$ would have to be negative or 0, since squaring $t_{i}$s gives you all positive numbers.
You can basically choose between small but wrong standard errors (due to the correlatedness of the error terms) or larger but correct ones. I prefer the honest option ;).
You can test your model through cross validation.
Thank you ars! :)
At first I thought this was fairly straight-forward, and that you could use Poisson regression with time to 5 hotdogs as the independent variable to get the finish time 10 hotdogs given time to 5. But the more I think about it, I suspect there may be too much correlation in those two times, since the outcome is some time plus the independent variable. You might have better luck modeling the outcome as time to finish the second 5 hotdogs conditional on time to first 5 hotdogs.
These histograms are inconsistent, because one (blue) represents approximately five times as much data as the other.  They also don't show much, because they cannot portray the *paired* comparisons of actual and predicted values: you need a scatterplot (or something similar) for this.
+1: This agrees with my recursive solution for smaller values of $N$, but breaks down as $N$ gets large.
@whuber: Thanks for your comments. I have added what $Z$ is. And for $c$, I would write $c$ indeed since I have stated that $c$ is positive
+1 thanks for the link to the Feldman/Cousins paper.
(+1) If you have the time and inclination you might strengthen this answer by expanding on your claim that including X1*X2 makes the effect of X1 on Y vary with X2.  Specifically, a model Y = b0 + b1*X1 + b2*X2 + b3*(X1*X2) + error can also be viewed as having the form Y = b0 + (b1 + b3*X2)*X1 + b2*X2 + error, showing precisely how the coefficient of X1--which equals b1 + b3*X2--varies with X2 (and, symmetrically, the coefficient of X2 varies with X1).  That's a simple, natural form of "interaction."
@mpiktas Good suggestion, but bear in mind that the stated interest is *technique* rather than *theory*.  Recommendations at math.SE will likely favor the latter.  Moreover, you don't need to know measure theory (beyond absolute basics) to learn about NP Bayes methods, so the main focus here should be on intros to probability that focus on statistical applications.
I don't agree with your premise: assume you have a bunch of data points which you just interpolated with a gaussian process. Then the posterior mean and covariance functions fit your data and can be observed at an infinite number of points.
What exactly do you mean by "the correlation amongst a set of individual correlations", & "the overall correlation"? Those don't sound like the same thing. The former sounds like you could correlate two corresponding sets of correlations, whereas the latter could be just the mean.
@Gleb_b: I am attempting to write a question.
@AdamO Thank you for your advice. I have included a few more info. TF-IDF and hierarchical clustering have been used.
Excellent :-) with the standard error/p-value/confidence interval you can now also say whether this difference in differences is statistically significant.
You are correct, @Dilip: a difference of half-normals does not have a normal distribution.  The problem is not with the variance of the difference: the very shape of the distribution is not normal (its kurtosis is too great).
Oh ok thanks so much for the great answer.
I simply want to know which groups have different means. What specific multiple comparison method would you recommend? Can I use Tukey's?
Use Bayes's Theorem.
Given that `princomp` documentations states: "COEFF (your `pc` variable) is a p-by-p matrix, each column containing coefficients for one principal component." and you are using the rows... nope, your projection is wrong. And besides given your using `princomp` you might as well ask to return two arguments and get your projections immediately.
@JohnFInn - Yes in both cases.  Bias is normally thought of in terms of "if XX were the true model, and we had YY as our model (which may or may not be the same as XX) and used estimation method ZZ, what would the difference be on average between the estimate and the true parameter?"  So either model misspecification or estimation method may lead to bias.
"Expected" implies you have a probability model for the graph in mind, at least under some null hypothesis concerning its construction as well as for the selection of edges: what is that model?
Oh, so I can make many thread to ask my question separately??
Secondly, have you looked at Gibbs sampling? This is likely much easier for steps 3-5.
0 down vote
	

This is sort of an interesting issue to me - isn't the sort of regression you choose motivated by either the conditional distribution of y, or the loss involved in predicting the wrong value? So if you do a beta regression are you estimating both the conditional distribution and the regression parameters simultaneously?
@Macro my data comes from face/fingerprint recognition systems and by looking at the data having excessively large values, I am able to see that they typically correspond to poor registration/noisy image etc. And I completely agree with the reasoning given by whuber. But the question is, how to do robust estimation?
Do you know the reorder points of the goods at the warehouse?  What sort of leadtimes are you dealing with?
The joint density determines the marginal densities and the conditional densities uniquely, and so the answer is No, you cannot construct another group of samples as you desire. If the joint density is being _estimated_ from the samples instead of being known a priori, then different samples from the population will typically give slightly different estimates, and so the _estimated_ $p(x,y)$ will be different as will marginals etc.
The mean survival time is most easily found by expressing it in terms of the moment generating function of a normal random variable evaluated at $t = 1$.
I've seen this (parameter free) formula: x <- log(x + sqrt(x^2 + 1))
Nothing is said in the question about which countries and/or periods are being included, but if the range of population or GDP is at all large, it is difficult to see that including GDP and population as they come makes sense, as any regression will just be dominated by its outliers. Hence on substantive grounds alone log transformation is indicated. That makes the point in this answer especially cogent.
Sorry, I do not see your point concerning model 3. May you explain?
(+1) Much more straightforward and clear than what I did in my answer.
Another problem is that we can have may words which are frequent but do not relate to the topic of the document such as "the", "of", "to" etc ... and I hopefully prefer the method to be independent from the language.
Are $e_i$ the empirical residuals? I thought I understood why they sum to zero but I am not sure now. Can you show me explicitly? As for the remaining term, the only thing I can think of is to substitute $\\hat{y}_i=\\mathbf{x}_i^T \\beta$, and I don't think it leads anywhere.
I agree that there are big differences between those who have health insurance and those who do not.  Why do you say there is no self selection?  Also, I would be careful not to oversell the Heckman correction... unbiased parameter estimates still require some fairly strong assumptions http://smr.sagepub.com/content/18/4/395.short and http://onlinelibrary.wiley.com/doi/10.1111/1467-6419.00104/abstract
Maybe a mixed model with a random effect for household? I've never tried to do this with this with a Poisson model, but perhaps you can patch it in somehow.
If I do believe my data, can you please explain why the mean will not be biased? Thanks!
Thanks a lot for answering the question! I'm sorry that I was not clear enough. $\\theta$ is the same for all the pairwise potential functions, so I don't want to solve the inverse covariance estimation problem. I think there is a simple way. What's more, the unary potential $\\psi_u$ may not be Gaussian.
Asterisk symbols are commonly used, but they shouldn't be. Much better to give exact p values.
+1 for the figure. Illustrations are always nice to see.
I guess I don't understand the dilemma. $E(c) = c$ for any constant, $c$. Multiply through by $-1$ and you have $-E(c) = -c$. Were you asking anything else?
However, it's your point about analyzing the distribution of monthly counts that's at the heart of my question.  There is a definite decline in TB, both nationally in the US and in my district.  For example, when I compare 2009 to the previous years, there are decidedly fewer cases.  2010 is on track to have fewer still.  What I'm trying to identify (which I did a poor job of explaining in the question) is whether or not these declines are part of an ongoing downward trend, or just a downward wobble.  Thanks - you've gotten me to think much more carefully about the problem.
(1) On what basis do you say that a t-test "shows that the data are significantly different"? I get a p-value > 0.8  (2) why are you using a t-test for these data?
I've edited your title; the question about nominal data is important in that it's why this question is not a duplicate.
You also need to give careful thought to *why* those 600 observations are missing. If their missingness is at all related to the response variable, then no technical solution that assumes "missing at random" is going to give meaningful answers.
I believe the question has more to do with the advantages or popularity of some measures of variable importance than how to print the ones available in R for a particular method.
You can observe data, not a "distribution". Each model takes a shot at guessing the unknown distribution of observations around an estimated expected value; residual analysis is a way to assess how good that guess is.
Note that you will need the [development version](https://github.com/mike-lawrence/ez) of `ez` to perform type 3 ANOVAs. And you need to set the appropriate contrasts: `options(contrasts=c('contr.sum','contr.poly'))`
@Tim: See page 64 on http://www.amazon.com/Testing-Statistical-Hypotheses-Springer-Statistics/dp/0387988645, or grab any other statistical theory book.

Also note that my reply only justifies the example you saw. I does not mean it is a good way to do it, as it is a type of "post hoc power" analysis, which is NEVER recommended.
I voted to close as no statistical content.  But to troubleshoot your problem I would start by checking that your forecast object "train" is forecasting as many time units ahead as you think it is; or (to put it another way), is there some reason to think that "test" is longer than the period your model is forecasting for?  I might hazard a guess that since you first wrote the command the data source you are using has grown and you might find that test is now 31 or more values?
@venkasub, the center of normal distribution is its mean. If you know that bimodality comes from two normal distributions you can fit mixture model to estimate the means, which will be two modes.
**Very closely related**: http://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia.
-1, imposible to debug code in r?