tttjjj/elt_bench huggingface dataset:
+1 nice answer. If you now could explain me how you do some post-hoc tests or planned comparisons (e.g., is there a dfifference between morning and evening in the groups 1&2, pooled) you would answer me a lot of questions. See also my question on R-help: http://article.gmane.org/gmane.comp.lang.r.general/237681
It depends on your design, of which there aren't enough details. ... e.g. was the order of the two surveys randomized? Or were the particpants assigned by design to either a survey 1 first or a survey 2 first treatment sequence? ... Or did everyone do one first and then the other?
I might be mistaken, but can't you use DBSCAN or OPTICS? See also http://stats.stackexchange.com/questions/67686/partitional-vs-hierarchical-clustering-with-distance-matrix/ of which I think this question would count as a possible duplicate.
Thank you so much for your reply. I will definitely consider estimating Cronbach´s alpha, especially since I consider looking at different "clusters" of components of the aforementioned questionnaire.
Ok, I see. It's using 1/2 bh formula along with Pythagorean's theorem to get the formula for each of the white triangles. I originally was thinking the 1/2 was because there are two people meeting you had to split them up evenly... Completely missed the graphical representation of the triangles. Thanks for clearing that up!
Regression can't indicate whether data were sampled well. That is a matter of considering the sample design and (as far as possible) comparing population values. Plotting data is the most obvious and by far the most useful preliminary to regression.
Do you mean the SHARE frailty model? I think you will need to provide more clarification to allow people to help with answering your question.
Perhaps the answers at http://stats.stackexchange.com/questions/8019/averaging-correlation-values will be helpful.
@Massimo, please provide link, internet is a large place.
But what is that value. I know it must provide some information about that chi-square distribution. What I want to know is what is that value.
Ok I see! The problem with (1) still holds, though, even if you create "synthetic" data, because the synthetic data is usually very similar to the original data it is mimicking.
Thanks for the great answer. Definitely gives me some specific terms to start Googling to hopefully start making heads and tails of all of this.
+1 Thank you for your support! :)
I believe I understand you're comment, but I'm not sure the interpretation is the issue. I thought the comparison would be those that delayed enrollment to those that did not delay, all else being equal (including college characteristics). The college characteristics I have include things like public/private, enrollment size, and funding. Does that help to clarify? I've added more detail to the original post.
@Glen_b: It boggled me and they wouldn't sit around long enough to discuss it.  So, I figured I'd poke here to see if I was off base.  I concur that the degree to which broader inferences can be made would depend on the characteristics of the sample and ones willingness to make such inferences.
So your contact person couldn't make the argument that R is still relatively newcompared to SAS, and he wanted to wait to make sure R was established before using it?
Ok, I had a look, and I cannot use relogit(), because as I said, I have a value of tau for each ABC category, instead of a global value, and this function does not allow me to enter a vector of the same lenght as my dataset as tau.
From what I have understood about how the function is written, I think what is did is right (except I did not made the more advanced bias correction part...).
Likelihoods for contingency tables are critical to the [G test](https://en.wikipedia.org/wiki/G-test), so the calculation of the likelihoods of such tables is done. Exactly which likelihood you get depends on the assumptions you make. If you condition on both margins as with the Fisher test, clearly the likelihood is hypergeometric for the null, but the G-test already deals with the log of the likelihood ratio. You might find [this section](https://en.wikipedia.org/wiki/G-test#Relation_to_Kullback-Leibler_divergence) a useful starting point.
This is exactly like another question I just answered on Math.SE: http://math.stackexchange.com/questions/576048/calculating-the-p-value-for-a-difference-between-bootstrapped-parameters-using-9/576189#576189
"Amazon obviously can't do it". Yes they can and they are famous for that. Amazon is known for spending billions in R&D.  They have been around for 20 years, are worth 6B$ and employ 50,000 + people. I wouldn't be surprised if 1000 of them are quantitative analyst, risk analyst, economist...
One does expect the code to behave sensibly, if the `predict` method has been set up in a standard way. I've edited my answer to to address the issues raised in your first comment.
I'd say that a path analysis based in SEM is not appropriate - you could try, but I don't think you're going to have the power to give you the advantages that SEM has, and you could well run into convergence problems.
It sounds like you're looking for an [interaction effect](http://en.wikipedia.org/wiki/Interaction_%28statistics%29).
Thanks! Would upvote but not enough reputation points.
Any three-parameter distribution is guaranteed to fit three quantiles *perfectly.*  Thus it makes sense to use this approach to fit only one or two parameters.  It also does not make any sense to compare a one-parameter fit to a two-parameter fit (with a different family) based on likelihood alone.
Again, I don't know why this was done. But someone sometime thought that a 0-100 scale was more layman than a 0+/- scale. I also now see an additional problem as using this formula a high standard deviation in the population against a low reference can yield higher than 1, but a low standard deviation in the population against a high reference can never be lower than -1, so in reality the normalized scale goes towards infinity as the reference goes towards 0... This is now officially above my paygrade ;)
There are lots of examples of convergence in distribution without convergence in relative entropy -- any time the $X_i$ have a discrete distribution and the CLT applies.
Some very good discussion here.
Yes, you can consider plotting a 3-D surface (x=age, z=gray, y=outcome), or you can consider binning gray by, say, quintiles and then proceed to make five smaller outcome vs. age scatter plots to show the changes in slope as you move up in the %gray quintile. Of course, you can overlay the five regression lines onto the same plot as long as they don't look messy.
Note that the two treatments *cannot* be "different from each other but not from the control", although you *could* find a pattern 'significant' contrasts that logically necessitates there are some type I or type II errors (more [here](http://stats.stackexchange.com/questions/32840/how-to-interpret-a-post-hoc-tukeys-test/32846#32846), if you're interested).
this may not give an interpretable or easy usable expression but can't you definitionally write $E(X|X<c)$ in terms of $f$ using integration by parts?
@Glen_b, Nestor,  thanks for your input. I remain confused though:  How does $\\theta$ impact the expected value?  How do I include $\\theta$'s distribution in the calculation of $E_{y|\\theta}$.  Or does it not matter?
Why would you randomly divide the controls? If the controls vary on the same phenotype as the cases, you could divide them on that, but if they don't then you can leave them as one group. 

If you are using controls as matches for cases, then they will be automatically split. 

E.g. A comparison of "HIV+ with AIDS", "HIV+ people without AIDS" and "People without HIV" is fine. But if it was "Men with HIV" and "Women with HIV" I would want male and female control groups.
The classic "eight schools" problem seems like a great candidate to introducing HBMs. http://andrewgelman.com/2014/01/21/everything-need-know-bayesian-statistics-learned-eight-schools/
There are several entire books on seasonality directed at economists and econometricians (I am neither), so I suspect this question will seem (a) too broad (b) lacking in research effort to those in the field. Are there any other methods? is a way to cut your readership in this forum, as users with high reputation much prefer specific questions to invitations to write a review. I'll give you "unobserved components models" as one positive.
Kwanti, could you please summarize what this article says?
Once I have done what you describe, what statistic could I use to determine which initialization point lead to a better partition? Thank you for all.
Actually, this answer is quite appropriate  (+1), given that $X$ and $Y$ are resistance values and are therefore, in reality, $>0$.
The update still makes the calculation look harder than it is.  The problem has a natural symmetry: because it's just as likely that $i$ calls will be made as $14-i$ calls, the expectation clearly is $14/2=7$.
Do you need to know about FA, or only how to get one from `R`? If the latter, this question would be better posted on Stack Overflow. You may also find [this page](http://www.statmethods.net/advstats/factor.html) helpful. If you want to know more about factor analysis, you might find [this page](http://www.psych.cornell.edu/darlington/factor.htm) helpful. If you need more, or different information, could you give a little more detail about what you need to know?
+! Yup; @Max What do you thing this "obvious" clustering would be?
thanks for your reply. Could you plase describe a bit more precisely how to apply this method to my kind of problem?
If you want to compare them over different eigenvalues, that might make sense yes.
+1 Thanks for helping me with this, I really appreciate it!
The _independence_ part is very important because we can have a sample in which all the variables are identically distributed (have the same marginal distribution) but are _not_ independent. Such a sample can still be considered as a _random sample_ but not of the experiment that you think it is a random sample of.  See [this question](http://stats.stackexchange.com/q/17868/6633).
The answer for the case where all the $g_j, j\\ne i,$ are equal to each other appears at http://stats.stackexchange.com/questions/5850. The general case is much, much messier. Writing down a formula would be unilluminating and not provide an effective computational method. Thus, you might benefit from refocusing this question on whatever the underlying problem is so that your readers could have some guidance concerning how to approximate the answer.
Anyway, it should be clear from such categorization, which techniques are linear and nonlinear and which parametric and nonparametric.
Also, in any decent vector graphics package (I recommend inkscape) you can align line segments to a grid, which would make the graph just as accurate as anything painstakingly created in any maths/stats package or spreadsheet editor (at least for the linear segments)
"*a list of all the families or classes of statistics that are asymptotically normal*" - such a list might be infinitely long.
@amoeba This is a good point. A distance being from $x$ to $y$ - i.e. not just of one variable $x$. I think the expression was lifted from the wikipedia article which is a little unclear.
Hi David - it would, but I don't see any great need for orthogonality in this case, and with just 4 predictors, not much need for the data reduction that PCA provides.  There's also the risk that Y's relationship with the derived components would be less interpretable than relationships with the 4 original predictors would be.
I'd suggest, if at all possible, to avoid including code that deletes all variables (`rm(list=ls())`), especially not without any warning. Someone may copy-paste your code into an open session of R where they have some variables already (but none called `x`,`y`,`df` or `spline1`) and miss that your code wipes out their work. Is it kinda dumb for them to do that? Yes. But it's still polite to let them decide when to delete their own variables.
@Alexander, are you interested in the *change* in the level of corruption or are you interested in detecting a change in the *rate* of increase or decrease of corruption? They're quite different questions.
So I think of the F statistic as a ratio of variation explained by predicted values to variation explained by residuals. Is using a scale parameter equivalent to estimating variation explained by residuals?
Aghh.. yep; you are right on, whuber.  Thanks for correcting my error!
The wiki page appears to be correct. The Markov Blanket is a shield from the rest of the network, such that if we know the values in that 'shield', then no other variables in the network provide any additional information about A. The key here is that we are talking about what happens when we observe those values, this doesn't change the factorization of a joint given the structure of the BN.
Awesome - that works out great!
$\\cdot$ My data is not exactly a signal subject to noise like in signal processing. I don't understand you idea of applying a filter. There is no "noise", the strong variations of my data is due to the strong variations of my predictors i.e my external variables which influence completely my data.
$\\cdot$ What do you mean by $\\chi^2$?
Better how? What does that mean? Eg, the prototypical statistical test would be a t-test on the response of a patient to a candidate medication vs a placebo to see if the mean of the treatment group is higher than that of the control group. It is not at all clear how to create an analogy b/t that & Euclidean vs Manhattan distance. It is not clear that this is even a meaningful statistical issue; it seems more like a *theoretical* issue that might better fit on [cognitive sciences.SE](http://cogsci.stackexchange.com/).
Why has no one mentioned theoretical properties of information theory and entropy yet? That seems to be pretty apparently what the OP is asking about given the statement "*Let's pretend that I have the fastest computer in the world and all the time to use it on my hands.*"
I am not sure whether you are using likelihood in its common meaning.
(ctd) links to pdfs are handy, of course.
Equality constraints can be accomodated with reparameterization. Equality to a constant can be done with an `offset` in straight GLM.
That depends on chance, of course. Usually you can find out which one is it by profession, location, etc. though I saw cases where there were 3 persons with the same full name, in the same profession and living roughly in the same area. Then of course it becomes harder :)
if you have (1|independent5) and 1+(1|independent5) then you put a 1 into the equation. why? I do not know. what does happen if you do that? will the 1 be another parameter? I do not thnk so. does the intercept change? 

if you want to include multiple nested levels  you can do (1|independent5/factor6)  then I think 5 will be nested in 6 or the other way around.
excellent example, thanks.
How do I calculate `qq`'s arguments for a given vector? Also, could you advise on using your `qq` function with `ggplot2` package? I was thinking about using `ggplot2`'s `stat_function` for this.
For some weird reasons the references ended up helping me out in another question of mine: http://stats.stackexchange.com/q/105201/6793.  
For this question I am honestly still keeping a moving variance of the last observations and stop regressing whenever that variance drops to 0 (which is equilibrium). It's ugly, but it gets the job done.
+1 Did you notice that the value of $0.8416$ is irrelevant? You can obtain $11.18 = \\sqrt{(10^2+5^2)}$.
With only slightly more work, you can select an additive constant that makes the resulting distribution as symmetric as possible (measured in any convenient, robust way; a handy method is to match outer percentiles, such as by achieving $z_{0.90}-z_{0.50}=z_{0.50}-z_{0.10}$ after transformation).  This has a much better chance of accomplishing the objective of the transformation, which frequently is to achieve symmetry.
I think that in my original question I mixed up two completely different things: confidence intervals (interval that x% of the time contains the true value) and prediction intervals (interval that should contain a new observation x% of the time). Maybe the Bayesian framework is more adapted to what I'm trying to do?
@BobJones: You can see http://www.mathworks.com/matlabcentral/fileexchange/20355 for variograms or http://mgstat.sourceforge.net/ for a full-out matlab geostatistics toolkit
You said "change over time." I wanted to see if you meant you are interested in clustering one variable measure at multiple time points. In that case, like @alto said, growth mixture modeling could be useful if you have enough data.
+1 I prefer credible intervals myself.  Regarding point 1 I would argue that Bayesian alternatives are less likely to result in rote data analysis, as the definition of a probability is not so counter-intuitive, which makes it much easier to formulate the question you actually want to ask in a statistical manner.  The real problem lies in that performing the test requires intergrals, which are too difficult for such methods to be widely adopted.  Hopefully software will develop to the point where the user can concentrate on formulating the question and leave the rest to the computer.
You need to take the means of the two samples into account. Consider the simpler case of one variable. If different samples give different (conditional) means and variances, then the combined variance is the (weighted) average of the conditional variances plus the variance of the conditional mean. A similar effect will arise with covariance: the two covariances are likely computed using different means, $(\\mu_{X,2},\\mu_{Y,2})$ instead of $(\\mu_{X,1},\\mu_{Y,1})$, and so just a weighted sum of the covariances will not do either.
Cheers, I actually have that book open now :)
Related question on another site: http://cogsci.stackexchange.com/questions/3267/analyzing-control-questions-data-for-a-survey
I asked the students to pose questions. Question-posing by students has come to be recognized as an important skill in science education. As you wrote, I am interested to see if the distribution of question types has changed from pre-test to post-test. As the students ask a larger proportion of advanced questions, we believe their comprehension of the material taught has grown fuller and deeper. (A full discussion is beyond the scope of this comment.)
How do you mean "represent data" and "good estimates"? Do you simulate data according to some copula and check if this copula can be reconstructed by estimating the parameters of another family of copulas?
CP: Thanks for your help.  I think I understand things (about as well as I'm ever going to) now, and I have a software algorithm worked out.  I'll give you the "answer".
By default, the argument 'replace' of 'sample' is set to FALSE.
Basic question like this are almost always duplicates - [How can adding a 2nd IV make the 1st IV significant?](http://stats.stackexchange.com/questions/28474/how-can-adding-a-2nd-iv-make-the-1st-iv-significant) Also essentially duplicated [here](http://stats.stackexchange.com/questions/30363/factor-significant-within-model-but-non-significant-after-drop) and [here](http://stats.stackexchange.com/questions/26014/univariate-non-significant-results-becoming-significant-on-multivariate-analysis). That took approx 40 seconds.
I'm sorry I didn't quite follow your explanation there. Could you clarify either your explanation of what you did or rephrase your question more precisely into a form that doesn't require an explanation of what you did at all?
@PeterFlom, well said! I think the OP got hung up on precision/recall definition of F measure, which gives you an undefined (0/0) answer.
Boruta works well for regression.
I'm not sure I understand correctly: how would you calculate a correlation between only one tupel (two consecutive values)?
Insert `assume(sigma>0)` before computing the integral. Tried it on www.sagenb.com, and it works. Edit: also, it'll be hard to find a more complete _free_ CAS than Maxima (which is what Sage uses).
Please tell us *why* you are comparing variances.  What do you hope to learn from this comparison?
Does the "day of study" mean "total number of days in the study," or "Monday, Tuesday, Wednesday...?"
We need more information regarding the nature of the function you want to extrapolate in addition to how far you want to extrapolate away from known values.
You should explain that you show Stata output. What's the question precisely? On the logic of the Breusch-Pagan test? On what Stata commands you should be using?
Say it this way: $\\epsilon$ is the half-width of the confidence interval. Since your empirical cdf will be within $\\epsilon$ with probability $1 - \\alpha,$ the confidence interval has width $2 \\epsilon.$ Here is another way to look at it: If $G(c)$ is your empirical cdf at $c,$ the confidence interval is $[G(c) - \\epsilon, G(c) + \\epsilon].$
$\\epsilon$ is a zero-mean error term.
I'm using an integral because the arrivals are uniformly (i.e. continous) distributed over the interval noon to 2:00 PM. Your answer doesn't specifically answer my question.
Because of there are only ratios (between counts and mass; between concentration $A_1$ and $A_2$ ) , the final cumulative uncertainty is quantified with this formula


$ ( \\sigma_f / f )^2 = ( \\sigma_h / h )^2 +( \\sigma_g / g )^2  $ 
with covariance terms equal to zero.

the formula is used to quantify the uncertainties $\\sigma_{Ai}$ and $\\sigma_R$
Thanks, this explains! As for the validity of the method, I share your doubts - just wanted to make sure that survreg() is doing what I think it is before going into this.
+1 for `ggfluctuation`, hadn't seen that before! This post has other useful code to visualize this type of dater: http://stackoverflow.com/questions/5453336/r-plot-correlation-matrix-into-a-graph/5453398#5453398
Could you please provide context, details, and an explanation of what you mean by "diverging" and "significantly different" regression models?
Could you please explain (or give a reference) why the variables need to be continuous in order for Pearson's correlation to make sense? Ditto for why the arithmetic mean makes sense only for continuous random variables?
Does that mean in this situation, Pearson or Spearman correlation  coefficient is not a good similarity metric for this two binary vectors?
The left hand side should be $p_j(t+h)$ then expand the RHS and take $p_j(t)$ to the other side as a negative. Then divide both sides by h, take the limit as h -> infinity so that the LHS becomes $\\frac{d}{dt}{p_j(t)}$. After that multiply both sides by $z^j$ and sum both sides over j=0 to infinity.
Technical details on how to actually split the dataframe into groups for the `survdiff` function you can find here: http://stackoverflow.com/questions/17338774/r-formula-how-to-constrain-calculations-to-two-groups-using-formula/17339707
The text of the question and its title ask two different things.  Are there two questions here or just one (and if so, which one is it)?
Cross-validation *is* validating the model.  Don't use a single test set.  If you need to tweak the model in a way that requires cross-validation to determine the tweak (not usually recommended) then you will need nest cross-validation or double bootstrap.
thanks. Very close. I have to teach it though. Can you think of a way to get this while avoiding ggplot/lattice's machinery? (both of which I love, but I want to keep is as simple as possible). I want to avoid having to discuss 'reshape', aes() and facet_wrap().
For the purpose of data visualization, the high-dimensional case might be limited to 3D (or 4D using faceted plots or small multiples), right?
Excellent, thank you Dimitriy. I use SAS but the Stata manual looks like a good resource to have handy.
Some intuition might be afforded by the analysis at http://stats.stackexchange.com/a/66295/919.
After the normalization I have the same pecentatge of success turn to 68%.