tttjjj/elt_bench huggingface dataset:
The gentle nudge by @ttnphns was meant to encourage you not only to use a less complicated means of illustrating a (simple) idea, but also (I suspect) as a hint that a more directly relevant illustration might be beneficial here. You could do both by finding a more straightforward way to graph the transformation when it is applied to the min and max actually supplied by the O.P.
Already asked (but not answered) at http://stats.stackexchange.com/questions/76948/what-is-the-minimum-number-of-data-points-required-for-kernel-density-estimation The absence of answers perhaps reflects a feeling equivalent to your last question; and also that complicated (e.g. skewed, long-tailed, granular, multimodal) underlying distributions will need even more data than simple ones. One thing you can always do is try estimation with a smaller subsample to judge the extent of deterioration in estimation quality.
I agree with the answer, but I think the example is a bad one. We know from calculus that are under $e^{-x^2}$ is $\\sqrt{\\pi}$. That means we normalize it by dividing it by $\\sqrt{\\pi}$. And to keep the area constant, if we need to divide it by a factor of $\\sigma\\sqrt{2}$, we have to divide $x$ by the the same factor as well... which gives us the formula after a simple shift by $\\mu$. It's so trivial that you should *definitely* understand where it comes from.
Was the treatment randomly assigned?
+1, I think this is a good question. Note however, that some similar question have been addressed before on CV. It may be worth your time to [search the site for regression assmptions](http://stats.stackexchange.com/search?q=regression+assumptions) & see what already exists. I can recommend [what-is-a-complete-list-of-the-usual-assumptions-for-linear-regression](http://stats.stackexchange.com/questions/16381/) & [what-does-having-constant-variance-in-a-linear-regression-model-mean](http://stats.stackexchange.com/questions/52089/).
Since all multivariate MANOVA test statistics are equivalent in the case of two groups (and equivalent to Hotelling's $T^{2}$), you can also use any of the built in ones, like the Hotelling-Lawley trace with `summary(manova(), test="Hotelling-Lawley")$stats`.
1) Latent factor _is_ its factor scores, so how can you write `I would like my factor scores to have the same...as the latent factor` - I don't grasp it. 2) You can rescale the scores to a wished range _or_ to wished mean and sd, but not to this and that at the same time, - it is impossible.
This is kinda what I was looking for. Thanks.
@whuber The reason for the -1 is mostly that computing distances to **centroids** is exactly what is not possible here. He is clustering executable files (see his previous question), which he compares by compression distance. What is the centroid of a bunch of executables?
Could you please try to ask several shorter and more defined questions?
@Mat you might also want to try quant.SE since they probably deal with this sort of problem all the time and are familiar with algorithms. It might be possible to migrate your question if you desire (but check with a moderator like Suresh)
(+1) Shouldn't it be `cut2(x, ...)`? Also, I don't think you need to assign a value to `m`: The only constraint in the question was about the No. groups.
I think there are some details of the question you may have missed. Unless I've badly misunderstood the question, it's not simply geometric.
I would personally not use either as the default approach but instead would recommend the basic bootstrap confidence interval.  See p. 8 of www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf‎ .  I've been doing a lot of simulations for the binary logistic model and have seen better confidence interval coverage using the basic bootstrap than using the percentile or BCa bootstrap.
I find the question too vague. You need to define "anomalous", e.g. as you did in the example. In statistics terms, you need to define a model for "normality" (i.e. lack of abnormality)...
Assuming age is measured in years, then an odds ratio of 1.059 implies a difference in odds between a 20 year old and a 50 year old of $(1.059^{30} -1)\\times 100\\% = 458\\%$. I would not call that a small effect. However, I implicitly assumed you were talking about humans. If instead these are mice then a 30 year span is not very helpful and you will need to change the evaluation of the size of the effect accordingly.
@Glen_b Yea, the approach 1 is extended from Wikipedia Lilliefors test from my understanding into different distribution. Just wonder whether my understanding or extension is correct. Or other approach is more correct? Thank you.
FYI: http://www.cs.ubc.ca/~mpf/2011-hybrid-for-data-fitting.html
This sounds like a mixture estimation problem (despite atrocious notations!) EM for mixtures is fully processed [there](http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#Example:_Gaussian_mixture).
@Patrick, It's self study. I've been studying ANOVA and the relationships between the $F$ statistic, $r^2$ and $r$ by extension. I'll add the tag, Thanks for the tip!
Hi @Dualinity, thanks! But this is a little different from what I'm looking for. Please see comments I made in response to Nick Cox above.
+1 Thanks for your insight.  Welcome to our community!
@whuber: I should have specified more clearly what I meant by "appropriate truncation". I was thinking of truncating (shrinking) the value of $f_n$ at the specified point (so that $F_n$ becomes unity). I think that notion is still valid in the case you mention, just that the truncation would not result in a change of value of $f_n$. I'll try to clarify this in an edit shortly. Thank you!
@clancy: Looking at iterations essentiall measures model (prediction) instability. This is quite different from estimating the binomial distribution behind testing a single (surrogate) model. OTOH, you are probably right with the practical conclusions: if you have enough test samples to prove that the model is better than some other approach (look at the uncertainty of the estimate as derived from binomial distribution), quite often you don't have problems with instability any longer...
Again afraid you might need someone more familiar with R than myself to help you on the use of this sample function. However, you could always generate a much larger population and then randomly pick 10000 samples from that.
It it correct to assume that the complication will not happen anymore if you are able to predict it based on imaging?
I don't follow how your $net$ frequencies are related to the numbers listed & your dummies d0-7. If you have counts, why not use them directly as your covariate?
Thanks for the detailed anwser. I believe my intent would be to classify data more like the Affinity propagation.
As an aside, there are better penalties to use than L1 or L2, such as the generalised double paretro, the bridge penalty, horseshoe.  These penalties have much better "tail" behaviour, as they shrink the larger coefficients less (unlike L1 and L2). Further than can all be fit via the EM algorithm, using a least squares algorithm similar to IRLS (so no loss of computer time).
To get predictions for factors, you use the same formula (at least for linear models), or, more likely a multidimensional version of it in matrix form. I don't remember the exact formula off the top of my head, but these are standard in textbooks. Moreover you would need a Poisson or logistic (etc) specific version, b/c the variance scales w/ the predicted value (note my answer to Q3 above). You might just ask for the specific formula you need on CV.
@steffen By observation; I mean it is obvious that no one significant from NN community will come out and say "Hey guys, let's drop our life work and play with something better!", but we have tools that achieve same or better accuracy without all this ambivalence and never-ending training. And people do drop NN in favor of them.
@bigTree, No, PCA is generally not an adequate means to cluster features into groups. But factor analysis (as well as cluster analysis, of course) is a means for that (see pt8 in the "list" [here](http://stats.stackexchange.com/a/95106/3277)). If V1,V2 "represent" PC1 and V3,V4 represent PC2, it _doesn't_ follow that these 4 variables don't intercorrelate _outside_ of these 2 PCs. But with factors, it is the other way round.
The number of successes in zero trials is always zero, so the probability function is a spike at zero, and the cdf is a step function that jumps from 0 to 1 at x=0. So ... nothing like a uniform on (0,1).
Can you say a bit more about that $2^m$ figure? Given $k$ groups, one has $k(k-1)/2$ maximum possible pairwise multiple comparisons, and that number + 1 for the omnibus test... Are you including all possible (e.g. pairs < triples < $k$-sized tests)?
Please name all your predictor variables.  I can't make sense of "z is two factors which is present in x."
Did you mean to say you have two *independent* variables?  BTW, this looks like an *exact* duplicate. In order to keep the thread open, could you please be more specific about where you think the differences are?
Nice answer, it is a good thing to answer old questions!
@whuber That may be up for discussion but it was not part of the question.  The OP just wanted to known why s$_n$ and √n appear in the formula for the CLT. Of course S$_n$ is there because it is consistent for σ and in that form of the CLT σ is removed.
A very important but often neglected situation is covered by #4 here, but I will spell it out. Often -- indeed usually -- you should want to compare your results with those of previous workers with similar data. If others found particular covariates worth including in their models, you should want to compare your results with theirs, regardless of whether your covariates achieve (conventional) significance levels. Note that cases here can vary from reporting model(s) you decide are not (especially) good to reporting model(s) you decide are good.
I guess simulation is always a possibility but if you have the values for all the observed independent and dependent variables and you assume a Gaussian noise term you can directly compute the variance of the estimated Y.  Are you looking for a situation where only group averages are available? Then I could see a problem.
I don't understand how this answers the question.  It's possible that e.g standardize=TRUE will give you the loss sequence back on the original scale, right?
@Skippy Fixed, thank you :)
(+1): I should have waited for clarification from the OP.
Thanks for the answer. However, the problem is that I do not want to test the probability of the event given a one-day rise in the stock market, but rather the correlation of the probability of the event with a bull or bear market, for which a one-day rise or drop might not be enough.
I am dealing with roughly similar issues in my daily work live. Even pushing simple ab-testing beyond those easy-cheesy binary metrices and make it usable in a commercial environment was only possible by first studying statistical methods independent of the application area and THEN, now armed with the correct mindset, apply this methods to the web. To the best of my knowledge, there is no such book, but I guess it would be a bestseller. Good luck !
Gelman and Park have an [article](http://www.stat.columbia.edu/~gelman/research/published/thirds5.pdf) which compares the practice of creating three categories from a continues variable, as opposed to two.  Usually it is best to leave the variable continuous for the reasons explained by others below.
It's a transformation of the difference in log-likelihoods from nested generalized linear models to a [0,1] range in an effort to mimic the R^2 that people understand in the context of comparing ordinary linear models. See: http://www.palgrave.com/psychology/baguley/students/supplements/9780230_577183_04_sup04.pdf
Although I can find plenty of annotated outputs of SPSS binary logistic regression on the Web, *none* of them directly provide confidence intervals, but they all report standard errors (from which CIs are easily computed), including for the constant term. Do you think you could document the behavior you are asking about by means of a reproducible example?
Both the answer from Peter Flom and Andy Clifton made it more clear to me why one cannot get D from clustering in the original post. However, I think this answer is the most thorough, which can more easily make somebody else understand it. Thanks for the help!
If a parent has one child, then half of the parent’s genes have made it into the next generation.

However, if a parent has two children, it does not follow that all off the parent’s genes have made it into the next generation, because some of the genes in the first child are shared by the second child. The correct number is somewhere between 50% and 100%, and it grows in some fashion proportional to the number of children, n.
If I haven't erred, here's a function to do that ratio-of-Bernoullis interval estimation: `binrat.confint <- function(x, y, n=Inf, m=n, p=0.95) {
  s2 <- 1/x - 1/n + 1/y - 1/m;
  x/y * exp(c(-1:1)*pnorm((1+p)/2)*sqrt(s2))
}
`
+1 You identified the essential idea and stated it well.
It's certainly possible, even for common or garden regression, so long as the response (dependent) variable is be treated purely numerically. Depending on your software, you may need to push or force that to happen. With a suitably wide definition of regression, to include logistic or ordinal regression, it's not only possible, it's commonplace.
Hi Peter, can you elaborate on your idea of bootstrapping to estimate the variances of $\\mu$ and $c$?  Also that still leaves open the question of what distribution our estimates would be following... what were you assuming for that?
+1 ... but: If they're the same shape apart from location shift, it's not just a test for difference in one location-estimator, but all of them (or at least all that exist).
@Macro The downvote seems a little harsh, because there is a useful and valid observation in this reply: the tests for overall significance and for individual variable significance "answer different questions."  Admittedly that's qualitative, but no more so then the first answer with many upvotes; and to that answer it adds some valid intuition, arguably making it an improvement over that answer.
Let me know if you find it useful, I would like to know!
table B is not a table, and probably you have the letters mixed up.
@D.W. There's no cause for embarrassment.  In fact, your question was worded in an unusually clear manner.
This answer is really helpful and brightens one's day. Thank you!
Another option, also mentioned in the paper is pLSA (su-2010-projekt.googlecode.com/svn.../hoffman2003collaborative.pdf‎). Is easier to implement, and factors are easier to interpret
@chl, I think this is one of those rare occasions where I wouldn't  want the x-axis limits to cover the entire range of the data.  He has outliers which are so far from the rest of his data (he said mean=500, max=15,000) that showing the entire range will make it hard to see the bulk of the distribution.  You are right that he may need to vary bin size or number as well.
A lot of good thinking comes through here.  However:  1. You'll want to change "stepwise" to "in sequence" or "in separate steps."  "Stepwise" has another meaning.  2. The fact that Time-2 seems to eclipse Time-1 may be misleading. A tricky topic that involves statistical control, partial correlation, collinearity, and variable importance.  3. There's an interesting strand in the literature debating whether and when to use ANCOVA as opposed to gain score analysis.  e.g., http://stats.stackexchange.com/questions/26529/help-with-change-scores-and-or-ancova-design
Does the Y column give the means?
Are you using the fact that the posterior density is proportional to that quantity? i.e. $$\\pi (\\Theta |x) \\propto \\exp\\left \\{ -\\frac{1}{2}\\left ( \\Theta ^{2}\\left ( n +t^{2} \\right ) - 2n\\mu \\Theta \\right ) \\right \\}$$
Thanks Momo. This is really helpful. My software provides me an option of choosing Family as "possion" and Link as "log" while running GLM alogorithm. So i think that's exactly what is required here.
But I don't think the geometric will give you the answer. For 8 tosses, the probability is 1/256. For 9 tosses, there are 3 combinations that work: all heads, a tail then 8 heads, or 8 heads and then a tail, for a probability of 3/512. How do you get the answer for 9 tosses using a geometric?
The properties of your data are a bit unclear. For each project there is a population of software modules, but inside each project, how could you compute the correlation between the modules if you only have one value per module?
Could you use the (out-of-sample) predicted likelihood or some function thereof (e.g., -2 loglik or proper scoring rules)? The predictive likelihood is determined in full by the parameters you estimate and you could simply "plug in" the real values. A scoring rules provides a summary measures for the evaluation of probabilistic forecasts, by assigning a numerical score based on the predictive distribution P and on the event or value that materializes x. In your case you have the Dirichlet for P and might use the logarithmic score S(P,x)=log(px) (which is related to entropy btw.)
@David - cheers for the link
@rolando2 In some applications 100 is not large at all.  For instance, the literature on accessibility to high-speed broadband points to approximately 150 variables (out of many hundreds analyzed) and all of them are reasonable: they relate to ease of engineering (terrain, rock type, climate, etc), to economics (SES, etc.), to demographics, to the geometry of the local transportation infrastructure, to politics (tax and business climate), etc.  I believe that many economic models similarly can include many hundreds of theoretically important variables.
Good answer. LDS is also called "quasi-random sequences", because it exhibits some properties similar to uniform random sequences (see [wikipedia](https://en.wikipedia.org/wiki/Quasi-random_sequences))
I am traveling now and so will leave it to others to answer. However, this is in pretty basic statistics territory and I suggest finding a good intro textbook - or reading further if you already have one.
@ttnphns Well, r-squared would always improve when adding a new variable.
There are potentially many "sandwich" estimators -you should include in the question which one you are referring to.
@whuber Shouldn't the windows' p-values be uniformly distributed, so that I can adjust for the number of windows I have? I am not even sure how to tell if a given test would work for me, or what I should check about it. What if I look at something like the length of the longest run of windows with p-values $<0.5$ (assuming the window length is fixed at something reasonable).
Thanks, that was a silly oversight.
Before I look into other models, I would like to have a more precise idea of how good a fit my logistic regression is. And I have a good graphing calculator, with which I can do some basic programming operations, so all I need is an explicit formula.
Hi Chl, I put this question as I've very little idea on non-linear regression technique. I want to learn it from scratch. The reference I'm looking for is more theoretical, with some application example from any field.
This question appears to be off-topic because it is about how to use R without a reproducible example.
You might want to read parts of [this answer](http://dsp.stackexchange.com/a/488/235) and [this one](http://dsp.stackexchange.com/a/1169/235) on dsp.SE.  In the simplest formulation, all the random variables are defined on the same sample space and the probability of the outcome Boy does not change with time.
Thanks Jeromy. The question was based on a conceptual idea, and I was questioning there exists any formalised practices in data integrity on how to deal with these problems with a mechanical approach. In my example, the points refer just to ages with a -99 as a data error, along with the 'F1'.
Thanks for the answer. I made up the example to try to simplify it. What I am doing is more like collecting a dose-response curve for each subject, fit a cumulative normal distribution and estimate the mean and confidence intervals. So I collect many many data for each subject. The effect of the drug for each subject is very reliable (no overlap of confidence intervals). But if I use the estimated means to perform an ANOVA, the effect of the drug is not significant.
Agree with @whuber and I think it is better to think of estimators/algorithms as being paradigm free with the *derivation* of the estimator being motivated by such-and-such paradigm and the estimator having such-and-such properties under such-and-such paradigm. I'd also like to add that from a frequentist perspective the minimax estimator of $\\theta$ in your example is $\\frac 3 4$ which agrees with the Jefferys prior.
@Mog: The bars seem a waste of ink/space; they force you to include 0 on the y-axis, which is often not necessary; and they prevent you from showing the lower halves of the intervals. The bars could be made worse (eg with gratuitous 3d).  With small datasets, I prefer to show the actual data points.
I advise against stepwise model selection (you can see my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856)). In general, I think finding the "optimal model" is overrated. If you want to build a predictive model, you should look into cross validation.
@statquant And I should be summing over the joint probability distribution?
This, I guess, drills to the heart of the perceptions. Early versions of `glm` in Stata were based on IRLS. In  later versions the default algorithm became Newton-Raphson, just as with other maximum likelihood-based commands. You need to look carefully at the output and documentation for `glm` for each version of Stata.
While variance and skewness are unrelated in the normal distribution by definition, that does not mean this is always the case...
You may have to correct for multiple testing in this context. You can find more information about that here: https://en.wikipedia.org/wiki/Multiple_comparisons
In response to the R portion of your question, the VARS package has a vignette that describes how to create a VECM. Type in vignette(package = 'vars').
Refer to [S. A. Murphy and A. W. van der Vaart. On Profile Likelihood. *Journal of the American Statistical Association*. Vol. 95, No. 450 (Jun., 2000), pp. 449-465](http://www.jstor.org/stable/2669386) for details.
Thanks Glen_b, I realize that, but what I want to know in addition is whether those differences are statistically significant. If there is say a five point differences and this differences is not statistically significant and the power of the test is high, then the fact that the null hypothesis is not rejected helps a bit to argue that there is no difference.
I tried training on only the first 100 data points and testing on data points 100-400 but get the error shown in my above edit
I upvoted both your answers, so you can edit this answer into your first one which I think would be better for the Wiki feeling of the site.
I got your second point. For the problem above I just used the same PP setting that I had. So both blue and red generations are from the same PP. There are however something makes me not satisfied with the resulting realization. 1) It could be from a region that locally is dense so not matching well with superimposed one. 2) original PP was a PPP, however after merging only one region not sure the realization stays Poisson. Hope you got the point, if I am not talking precisely, please keep kindness and explain more.
Do you have the actual weight measurements, or is it just those categories? (I hate this medical approach of taking an arbitrary threshold and classify everybody either above or below it; economists, at least, would've broken this down to five quintiles and have the same number of people in each group.) I would think that the s.d. needs to be computed within the diagnosis, rather than across everybody, to make the standardized effect sizes comparisons meaningful.