tttjjj/elt_bench huggingface dataset:
This question appears to be off-topic because it is about the use of software instead of about statistics per se.
Hi Caracal - where might I find this proof? (thanks)
"Never" is too strong. There are reasons to be cautious about interpreting correlation coefficients involving proportions, especially those based on small counts. But the same analysis supporting those reasons also shows that when proportions are based on large counts and the proportions are "sufficiently far" from $0$ or $1$, then the correlation coefficients are not problematic. Furthermore, one can *always* report a correlation coefficient for any set of paired data (where both components exhibit variation) as a *summary* (descriptive) statistic.
1) There is no Anderson-Rubin "PCA" scores; they are for factor analysis only. Or, to put another way, Anderson-Rubin method in the context of PCA will give usual linear PCA component scores. 2) If you applied standard (linear) PCA to your Likert items this automatically means that you treat your items as interval, not ordinal. And so are the principal components - they are always interval variables. 3) So go ahead boldly to use any clustering method you like: all methods work with interval level data. 4) I don't see reasons to prefer crude hand-made variables to PC scores in your case.
This question may confuse people because it does not actually ask for a CDF. It could be understood as asking how to estimate the *conditional* CDF of success. A standard procedure for constructing such an estimate is logistic regression. It requires additional assumptions concerning how the conditional mean might be related to the values in column 1. You can find out a great deal about logistic regression on this site by [searching it](http://stats.stackexchange.com/questions/tagged/logistic+regression?sort=votes&pageSize=30).
If you simulate data where the predictors truly have no effect, this will give you a reference. If you permute the response variables, you will also have a situation where the predictors have no effect, so that would be another way of getting a reference.
Yes, that seems to do it. Thanks for your help!
Are you sure that you need a statistical test? If the variance is negligible then you can just look at the difference between the means and declare a winner. If there is no difference between the means then there is no difference...
It looks like you're overthinking this one.

You have only four distinct categories, corresponding to $(x_1,x_2) \\in \\{(0,0),(0,1),(1,0),(1,1)\\}$.  Allowing four parameters is tantamount to fitting the $y$'s separately in each category.  Therefore you really have four independent problems and each of them reduces to summarizing a univariate dataset.

Where to go from here depends on the nature of the $y$'s and the purpose of the model, neither of which is apparent in the question.
"Dummy out promo periods (plus post-promotion dip if present)" can you please explain this line..?
@whuber: I want to do a deeper analysis with this data. My hypothesis is the proportions will change with the gradient. The counts are also available.
`R; function??` - R has inbuilt help. you can also usually search for "cran" to find R stuff, and I find that most major search engines can handle the single letter well enough.
To what "paper" do you refer?
The question in your Add2 is actually somewhat unrelated. The question has to do with the interpretation of the $y$ axis and whether it has a direct probability interpretation. That obviously is incorrect. However, integrating a KDE over distinct bounds *will* give you interpretable probability estimates.
Thank you, perfect, exactly what I was looking for.
This question appears to be off-topic because it is about asking for statistical packages.
@mbq +1 for snowfall- abstracts snow even further and makes parallel computing with R pretty simple.
@DilipSarwate, no, because the question assumes that you'll be making a statement about the state of your knowledge on the matter
this depends heavily how do you get estimated probability distribution. At the moment the questions in my opinion is too vague, although it is an interesting one.
After giving this more thought I think that my recommendation would not work in high dimensions because (1) although a judicious choice of bins is practical in 1, 2 and possibly 3 dimensions, it does not seem to me that identifying such bins in 30 dimensions could be done (2) because of the curse of dimensionality even if such a selection could be achieved points in 30 dimensions spread out in such a way that it would be difficult to detect differences between the distributions without a very large number of points. So cardinal makes some good points.
I think leave one out is the best you can do.  I have a strong aversion to any bootstrapping techniques statisticians employ to get around small sample sizes - it's basically cheating, it has zero logical justification, it's only used because it has nice properties.
There's a (very) little visual indication of an increase in variance with mean, but it's driven almost entirely by three points. Since the lone leftmost point would be expected to have a small raw residual, we're left with an even weaker impression generated by two points, and the lower one doesn't stick out much. I'd attach no weight to the visual impression; there's little to indicate non-constant variance there. Such an impression wouldn't be all that uncommon with the same pattern of X's and random normal data.
Hello @Behacad,  I had a quick look at the abstract of the article in question, and they didn't just sample from individuals with social anxiety disorder.  It appears that they had 3-cohorts: 1) socially phobic patients enrolled in 5-treatment trials (n=456), 2) an independent clinical sample (n=?), and 3) an independent student sample (n=?). So it appears that they might have had a heterogeneous enough sample to get item statistics that would be invariant.  I am going to run out of room here, but will try to put my thoughts in order and place them in the longer answer format.
This question should be split into 6 distinct questions.
Good answers elsewhere e.g. http://stats.stackexchange.com/questions/27724/do-all-interactions-terms-need-their-individual-terms-in-regression-model A keyword to use in searching is "interaction". Note that the issues are totally independent of which software you are using.
With all the respect: Do not bad-mouth your colleagues in public. More to the point of your question: from the looks of it, a mixed-effect ANOVA seems like an obvious first choice actually. Your random effect would be your participant ID and your fixed effects your different behaviour types interacting with your professional types at first glance. Try it, see how it goes, post some results and we can take it from there.
Both are worth a periodic re-reading, maybe once a decade, just to refresh the ideas.  Concerning Tukey: it's great to sit down just with pencil and paper once in a while and do a deep analysis of an interesting dataset.
@Glen_b Since $\\sum{x_i}=T\\sim GAM(12,\\theta)$, $\\theta T\\sim GAM(12, 1)$. So, $2\\theta T \\sim \\chi^{2}(24)$. From there one can find upper and lower bounds ($c_1$ and $c_2$ respectively) on a $\\chi^{2}$ table, and solve $P(c_1<T<c_2)=0.95$. I think that is it... just leaving this comment as to not leave others hanging. Thanks for the help everyone.
+1. To add to the last thing Michael said, probit and complimentary log-log are two other functions that map $(0,1)$ to $(-\\infty, \\infty)$ that are implemented in many software packages.
I think it's important to keep in mind that you can get a bachelor's, a master's, or even a doctorate in biology without ever having taken a statistics class, even at many tier one universities. My degree in biochemistry required two semesters of introductory calculus and one semester of differential equations. The substance of these classes is quickly forgotten because many students will never use these skills again! So I really do think it's necessary to dumb down the explanation for typical non-statisticians.
This seems not related to statistics. It also looks like homework.
Depends on what will you do with ID column. Without more context, this question does not have a single "correct" answer.
this is a better fit for cross validated.
@user3280 - the procedure I outlined does exactly that - the summation over the Gibbs samples (the b's) is an approximation to integrating over that parameter space.  This is using the probability distribution of the betas to get predictions.  But remember that the ultimate goal is prediction here, so you shouldn't really care about the betas per se, only about the predictions.  the betas are a secondary concern.
What does it mean that a vector is negative? If $\\mathbf{x}$ is an eigenvector of the matrix $\\mathbf{A}$ then $\\mathbf{Ax}=\\lambda\\mathbf{x}$ for some $\\lambda$. But then $A(-\\mathbf{x})=-A\\mathbf{x}=\\lambda(-\\mathbf{x})$ so $-\\mathbf{x}$ is also an eigenvector.
@ATJ: Neither the point estimate of nor the ($1-\\alpha$) confidence intervals for a parameter become "useless" when not significantly different from zero (at level $\\alpha$) or containing zero respectively.
Thanks for your reply. I don't quite understand why if there is an interaction between the factors this means that the difference x1-x2 is significantly different to the difference y1-y2. I would really appreciate it if you could be more specific.
If the $p_i$ are very small, you can use Poisson approximation. Let $X_i\\sim \\mbox{Be}(p_i)$ be independent and let $Y\\sim\\mbox{Po}(\\lambda)$ with $\\lambda=\\sum_{i=1}^np_i$. In a [classic paper by Hodges and Le Cam](http://projecteuclid.org/euclid.aoms/1177705799) it is shown that
$|\\mbox{P}(\\sum_{i=1}^n X_i\\leq x)-\\mbox{P}(Y\\leq x)|=3\\cdot (\\max_{1\\leq i\\leq n}p_i)^{1/3}.$ If the $p_i$ are all close to 0, this difference is small.
@user25786 you're welcome! As I said in the post: If you have the data at the individual level (information for each of the 2000 people), you should do a logistic regression because the variants might not be independent. Exponentiate the regression coefficients and the confidence intervals to get the odds ratios + CI for each variant.
@YanRaf 1) What convergence criterion do you use? You shouldn’t wait until non-diagonal coefficients are null, but stop for example as soon as they are < 0.001 in absolute value... this should be precise enough. 2), the SVD. This is much more complicated, there is a part to construct a tridiagonal matrix that I don’t remember at all. The brute force version is to compute the product $A \\times A'$ or $A' \\times A$ to have the smallest possible square symmetric matrix, and to compute its PC that are the singular vectors of A.
Thanks for pointing out. Have corrected it. :)
You are correct - thanks for your input. Although my DV is not strictly accuracy, those papers were still useful.
Thanks for the link. And FWIW, HTML5 Canvas drawing code should also work on the server: http://stackoverflow.com/questions/9409106/generating-html-canvas-image-data-server-side
There is an equivalence between the Poisson model and the Cox's PH model when the baseline hazard is (piecewise) constant. In such setting, you assume an underlying **exponential** distribution. A little bit more details here: http://stats.stackexchange.com/questions/8117/does-cox-regression-have-an-underlying-poisson-distribution/8118#811, or on the Internet...
@HorstGrünbusch I am running a Monte Carlo Simulation and am sampling randomly from about 65 Normal Distributions, each with a different μ  and σ and I put this data into an equation which upon obtaining 1000 samples from 1000 iterations gives me a Normal Distribution of the equations solutions. Secondly, nothing is known about the overall data. Thanks a lot.
thanks for your insightful comments, if $N$ is 109, then normalized split/join as you mentioned above would be 8/218 = 0.037, and 68/218= 0.31, resp. Not too much intuitive to be gained, I guess. I think the beauty of split/join shines more when *unscaled*.
For reference, the slide above comes from Thomas Minka's slides here: http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-tut-slides.pdf
Also, with a Poisson GLM you can fit log-linear models. See Agresti: Categorical Data Analysis Section 8.6.7.
@NickCox That would indeed be interesting! If the models are linear and of equal complexity, and the errors have a Gaussian distribution, $R^2, AIC$ and $BIC$ should give the same answers. The truth is that I don't know how to produce an example ad-hoc where scatterplots, $R^2$ and $AIC_{c}$ lead to different answers (maybe someone could help me?). Sorry.
I am trying to see whether the results changes if I have more participants, more like a prediction. In my paper, I will only share the current results and, if possible, mention how the results might or might not change with a bigger sample. As a Statistician, @MichaelMayer, what do you suggest I should do? Would re-sampling or bootstrapping decrease type 1 error rate and have a correct prediction?
I just found great notes on the subject here:
http://www.bwgriffin.com/gsu/courses/edur8132/video/delta_r_squared/
Hi fredd and welcome to the site. Your question seems off-topic here because it's only about some code. It would be good, though, if you provided some more information about your problem: What statistical problem do you want to solve? In what way exactly isn't your code not working?
Doesn't your software produce the eigenvectors corresponding to the  first two components?
@shabbychef, it was late. I have given a more detailed explanation.  Note that for the asymptotics to work in theory as explained the model needs to be correct and the estimator must be the MLE. More general results can be obtained in the framework of general estimating functions and estimating equations, see, for instance, the book *Quasi-likelihood and ...* by Heyde.
I am not sure, but I think `PROC KDE` may do what you want.
I have asked the Stats Department, but they couldn't support me before the deadline of the thesis..
thanks for your answer. In regards to using $Y_{ijk}$, do I need not to subindex two of the indices – e.g.,  $Y_{i j_i k_{ij}}$ --  (because persons, raters, and items are only partially crossed) or is that understood in such notation? I think I understand your first point that the between person variance depends on $k$ (items were randomly assigned to item position). The linearity is hypothetical.
@whuber - updated the questions. Ideally I want opinion on #1 question. Otherwise, for #2 I don't know how to keep green jobs ordered top (by using different states which can have different weights may be). Bottom line, yes - I don't know how/what algorithm can solve this issue of using "hotness" to order the jobs.
A mention of a potentially useful search term: if it can be the case that there are alternatives that can produce a lower-than-$\\alpha$ rejection rate, then the test is said to be *biased*. And indeed, using that or related terms (e.g. *bias log-rank test*) in a search does seem to turn up some potentially relevant references.
@Bill I added a plot of sample size vs. error that illustrates what I'm seeing. It's more like 20 times out of 20. As for the random sample, it's as random as `ORDER BY RAND()`, which isn't perfect but should be suitable for this task.
Thank you , do you mean there is no relation between variability and smoothness(irregularity)?
@Scortchi I have included an example at the other question which covers your first comment (i.e. where the tests are done using the common error variance and df), though all the tests (F and multiple comparisons) are done at a pretty low significance level (0.0025, not 0.05). When compared to the individual ordinary two-sample t-tests as is being asked by Nick S. here, it shows that quite a substantial difference in significance is possible (in this case, $p_t>.05$ for all the ordinary t-tests, yet $p_F<0.002$). I believe with many groups, it's possible to go much further.
@Glen_b I have called it "when is it so" because I intend to first judge if the IDD assumption is too strict. If it is then I will think about how to deal with it. I am hoping that it is not too strict.
@Black, take the simple case: $u_t=\\rho u_{t-1}+v_t$, where $v_t$ is white noise. Unit root test that $rho=1$. Suppose $y_t=\\beta x_t+u_t$ and you want to test whether $u_t$ has unit root or not. However residual $\\hat{u}_t=u_t-x_t(\\hat\\beta-\\beta)$. So if for example $u_t=u_{t-1}+v_t$, then $\\hat{u}_{t}=\\hat{u}_{t-1}+v_t-(x_t-x_{t-1})(\\hat\\beta-\\beta)$. So although $u_t$ is a unit-root with iid white noise, $\\hat{u}_t$ is not. Since the ADF expects the former, the usual p-values might be incorrect.
Is this homework? And the plus sign in the first equation is unnecessary or something is missing?
Dear Eldan: This looks to be a simple solution, but could you please let me know of any supporting documents for the same? i have tried wikipedia and few other docs, but there is no indication of gamma-gamma conjugate prior model. May be I am wrong in understanding the text, please do guide me to some good reading material on the same. Thanks a ton for your help.
Please make more concrete the expression "increases with dispersion": increases as variance increases?  Some other measure of dispersion? Increases as the support gets larger?
"*Why wouldn't expected utility exist?*" -- It seems like you're engaged in the fallacy of argument from ignorance. If you can't produce a reason to think it should, there's nothing to respond to, particularly since it's already not to be finite, nor even to exist at all in particular instances. I don't understand the difficulty - if you posit conditions where it isn't finite, it isn't finite, end of story. Why would there be anything to do about that?
@CrisStringfellow: Best approach will probably be guided by purpose. i.e. What is the reason behind trying to find if there is a relationship? Is it just to test mutual independence? Or is the aim to construct a predictive model? Is there some knowledge of the underlying phenomenon that might allow a better guess about possible model forms?
I might suggest updating the title of your question to include the terms "convex hull" or "border of irregular points" since it has now been clarified. This will help others in the future.
@Macro I know I can have school-level predictors in the model. My understanding of such a predictor is that it is one that does not change at the pupil level, only at the school level (so it's constant across all pupils in each school). My question is whether having the _count of observations_ at each school in the data (which **is** a school level predictor by the definition I just mentioned), makes sense statistically. For some reason it just seems a bit weird to me.
Your question could be improved by providing a citation or two pertinent to "meta-regression" and, perhaps to the use of moderator analysis within the meta-regression context. While effect moderation (aka effect modification aka interaction effects) is common enough in applied data analysis, I wonder if you are interested in a very specific kind of application? The "edit" link in the lower left of your question can be used for you to clarify your question along these lines.
If you think that linear approximation is not accurate you could use cubic spline interpolation method such as quadratic match sum method as in http://forums.eviews.com/viewtopic.php?f=7&t=3799 or in SAS http://support.sas.com/rnd/app/ets/proc/ets_expand.html. Hope this helps, I don't know of a function in R that could do this.
So, if I understand, you could define $\\alpha_{ci} = \\mu + \\gamma_c + \\delta_i + \\epsilon_{ci}$ where $c$ indexes conditions, $i$ indexes participants, and $\\mu$, $\\gamma$, and $\\delta$ are the effects of the grand intercept, condition, and participant, respectively? If that's the case, you can use the BayesFactor package. You can email me for further help if you like. You'll use `generalTestBF()`.
@Srikant: Yes it is. It is known by both names.
I should have commented earlier -- I love that this answer is succinct and complete. :D Thank you!
Thanks for the link to the post about fitdistr. I will use that to determine what distribution best fits my data.
What have you looked at?  I searched the obvious on Google and the first link seemed pretty straightforward (it had errors but at least it would put you on the right track). Two things could improve your question. The first is to provide clarification on what all your p-values come from specifically (i.e. what test is 0.03?); and the second is to explain what you have looked at and ask for clarification where you were stuck.
Excellent!  That really helps settle the issue for me.  Given that I'm looking for a correlation to economic indicators, I'll remove it from the regression, as the r-squareds are not impacted much when I factor in just HousingStarts.  Thanks for the help!
The question of whether you should do it yourself or whether you should hire someone comes down to your statistical knowledge and the importance of the project.
"$4!$" means, by definition, $4\\times 3\\times 2\\times 1 = 24.$  It is not a $4$ with a surprise after it (!).  You also need a better scanner: it did not pick up the negative sign before the "$1.8$" in the exponential.
I'd have awarded you the full bounty if I'd been available at the right moment. :)  You seem to have grasped the nature of my problem - and you're not alone in recommending GGobi to me.
If you blindly apply normal CV methods when working with highly correlated data, like things with temporal/spatial characteristics, you're going to have a bad time (and likely be highly overoptimistic about true model performance).
If you want to add more info, please edit your question (note you can always do it regardless of your reputation).
Yes I got 8 ≈0.98, so that sounds reasonable.
Are you sure this is a constrained regression problem?  As I read the question, you seek a relationship of the form $y_4$ (one Forex series) = $\\beta_1 y_1 + \\beta_2 y_2 + \\beta_3 y_3$ (plus, I presume, a fourth term representing a prevailing safe rate of return).  That's independent of the investment decision.  If a customer wants to invest $c$ capital in $y_4$ using $y_1$, $y_2$, and $y_3$ as proxies, then they would just invest $c\\beta_1$ in $y_1$, $c\\beta_2$ in $y_2$, and $c\\beta_3$ in $y_3$.  That adds no special complication to the regression, does it?
@Zhubarb, no the problem described there is different. I have a balanced training set and enough samples to balance the classes. My problem is about modelling the imbalance in the real-time scenario without biasing my classifier.
What's that huge spike in counts just below 10000 replies?
What is the question you want to answer? Do you want to compare males to females, or wing length to tail length, or something else? And why are you restricted to a univariate test?
Something is missing: "$E(u|X)$" is a *number* (or perhaps a vector), not a logical criterion, so it is nonsensical to write that it "holds."  What did you intend?  That $E(u|X)=0$?  By the way, some explanation of your notation (at least to distinguish vectors from numbers) would make the question more readable, and therefore more likely to get good answers :-).
@conjectures Why?
It seems wikipedia can help you out with this one: [probability plot](http://en.wikipedia.org/wiki/Probability_plot). [QQ plot](http://en.wikipedia.org/wiki/Q%E2%80%93Q_plot), [PP plot](http://en.wikipedia.org/wiki/P%E2%80%93P_plot). If you have a more specific question please clarify!
One solution is to run Kaplan Meier on men and women separately.
This attempt at an explanation fails because it invalidly conflates ignorance (we don't know which of three answers would be correct) with probability (assigning 1/3 to each answer).
Hi @Luna - OK but there is no mention of weights in this post. What kind of weights? Can you edit the post to clarify?
I generally use boxplots with superimposed points, I find it very "visual". Violin plots, instead, are a bit hard to understand in my opinion.
I have edited the question in order to make sure that this question  is of interest for the community.