tttjjj/elt_bench huggingface dataset:
Dave and Frank - thanks for your comments. How should then one interpret the second part of that statement, that is, "we did find that high and low-risk scores were associated with a clinically meaningful gradient of readmission rates"? What's the intuition there?
(+1) Have to try this!
ANSWER: Read sections 2.2-2.4 of the link you provided.
It is difficult to know where to begin.  I wouldn't recommend Excel for any data analysis.  If you seriously want help, you need to provide more information:  how many universities, what ranking do you have (research, satisfaction...), what variables do you have, is the variable of interest really a proxy for something else that might explain the association (e.g., is socioeconomic status a confounder).  I am sure others can think of information they need to begin to help you.
The variability of the mean (=what you are trying to estimate) in the population is zero.
Truncating a Poisson at zero is not the same as shifting a Poisson to the right. I find your question confusing for that reason. Actual shifting (by $\\delta$, say) is easy to write down, you just replace the variable ($x$, say) by $(x-\\delta)$. Truncation is also fairly easy but *completely different*.
If you don't get good answers here, you may find them in [Paul Allison's book on survival analysis with SAS](http://www.amazon.com/Survival-Analysis-Using-SAS-Practical/dp/1599946408)
@doug - Thank you, I've done just that.
Shouldn't $i_c$ be $1 - \\sum_i (w_i n_i / t_c)^2$? if you increase the weights too much the impurity goes to 1.
Thanks.  I appreciate the answer.  I tried both ways, and ended up getting a better result for the one when I *didn't* force the correlation.  Not what I was expecting...
For what it is worth, in the UCLA stat consulting group, we tend to use genmod.  We talked about demonstrating the same analysis in various procs (or other functions for other packages) but the thing is there is often many means to an end.  Other examples in SAS are proc reg and proc glm which overlap substantially, proc glimmix, proc mixed, and proc nlmixed also come to mind.  They all started differently so they each have their own quirks but they have all been extended so they overlap.
Right, so change (3) to calculate the probability of the sequence occurring given the chain. The two chains are similar to the extent that the probabilities they give to each sequence are correlated.
+1 It's good to know (and it seems I forgot about that package).
That was really informative; thanks!
Whether $E(X_1)=\\pm E(Y_1)$ or not would yield quite different results. And a hypothesis more precise than "$X_i$ and $Y_i$ are dependent" might be needed. (Duplicate from math.stackexchange... You might want to choose one of the two sites and stick to it.)
+1, though in respect of your last sentence, it's possible for the slope coefficient of a linear model to be 0 while there is nevertheless a relationship (for example if we fit a linear relationship, but the relationship is nonlinear in a way that's orthogonal to the linear term). I'd have said "no *linear* relationship"
Sorry, rules. You can try your luck on [chat](http://chat.stackexchange.com/rooms/18/ten-fold).
It is robust for 3 to 5 but could also be larger sample sizes. The nromal assumption is made to get the exact distribution under the null hypothesis to set critical values.  So the test does depend on normality.  However, for non-normal data it preserves its significance level approximately.  This hurts power compared to Grubbs' test. But it will still detect large outliers.  What you do regarding outliers will depend on how far from normal you think the data containing outliers really is.  What is an outlier and what to do about them is always difficult and controversial.
This is correct but i think it is more common to say that the posterior is proportional to the likelihood times the prior which is the same as saying that it is proportional to the joint denstiy of y and θ.
Thanks for the thoughtful response! You raise a number of great points, especially about how decisions are made on a "what feels right" basis vs. a cost-benefit analysis. If the most important attributes of the item aren't observable (to us), associational rules are the way to go. It would be interesting to think of other domains and applications that lay somewhere between ecommerce recommender systems and real estate agents.
"on a computer computer time is discrete" - well strictly, yes, all measurements are only finitely precise. But Poisson is for *counts* of events, not their duration. The time between events in a Poisson process *does not have a Poisson distribution*. It has an exponential distribution. If your process is being interrupted by many events, though, its the distribution of the time taken by their process, not the inter-event time, that you'd need to sum up to figure out what additional time your process takes.
+1 Actually, the double integral simplifies into a single integral since $$\\int_y^\\infty p(x_0)\\,\\mathrm dx_0 = 1 - \\Phi(y-\\mu)$$ giving $$P(E) = 1 - (N-1)\\int_{-\\infty}^\\infty \\Phi^{N-2}(y)p(y)\\Phi(y-\\mu)\\,\\mathrm dy$$ which is the same as in my answer.
I was stating an impression, and do not have an expert opinion, as I said, and I am happy that you want to qualify or indeed to correct a throw-away remark, which is how this started. So, thanks for your remarks. Incidentally, least squares was introduced in 1805 or earlier, depending on whether you believe Gauss's priority claim, does predate MAD, which was 1816 if I recall correctly. I'd regard Boscovich as relevant here too.
thanks for your help. A few questions: In your excercise: what kind of distribution are you referring to, Binomial? What do you mean make up a discrete probability? You mean calculate all the probabilities of 2 and 6 over different sample sizes?
@JonathanBone I tend to use AIC rather than significance to make that kind of decision, but it can definitely be reasonable to prune the model if some of the variables you included aren't worth estimating. It depends on what your goals are.
@ttnphns What is this mystical other you speak of?  How can I not have clusters in my data?  How do I know?
Thanks! @whuber
Wrong: 200 C and 0 C average to about 100 C, which is the boiling point of water. Ovens only go down to about 150 C, and 75 C is still too hot. Now, if you have one foot in scalding water (about 55 C) and another in cold icy water... then you are probably a strange person.
what was the sample size of positive cases?
I recalled it getting a brief mention here: [Colton, "Avoiding mean-square error bias in Designed experiments"](http://www.minitab.com/uploadedFiles/Shared_Resources/Documents/Articles/avoid_mean_square_error_bias_in_designed_experiment.pdf). On p.3 "Another inappropriate model-reduction technique can result in a positive bias in the LOF-error component" et seq.
First, resolve whether you mean *all* possible interactions (which is how @PeterEllis interpreted your question) or just all possible *two-way* interactions (which is how @macro) interpreted it. 
But I'd be cautious about a model with 100 variables, much less interactions among them. It's hard to conceive how such a model could be interpreted. What's the context?
This is just the standard "no omitted variables" assumption that is implicit in all regression analysis.
"However, in case 4, there is zero predictive power for that trait, (every position has to have one or the other value, so expected value should be 1), but most tests report a high value."
I'm not sure what you want the result to be in this case -- knowing what kind of specimen you have (black or grey) is very informative about the gene position (if black, then A or B, if grey, then C, D, E, or F).
Do you mean you took the test?  Were you in a group of 259? Or were there 259 questions?
Correlation is something one computes between variables, not between users.  Are you looking to see how variables relate to one another, e.g., whether smaller size tends to mean lower price?  Or are you looking to devise a system to assess how similar different users are to one another (a less straightforward problem requiring more subjective criteria)?
@PeterEllis has provided a good answer below. Let me add one small detail regarding your concluding question (which I suspect might have been an oversight): It depends on how much data you have. Eg, if you use an 8th degree polynomial, your model would be saturated if you had only 10 data.
Thank you so much for your opinion...
Thank you Macro! BTW, this means that if I compute the model $y \\sim x_1 + x_2$, I can simply transform its coefficients to those from the model $y \\sim \\eta + x_2$ without even computing this model (but I need coefficients from $x_1 \\sim x_2$). Interesting!
@user603 Ah, no, that's not the case: take a look at the histogram of the quantified values yourself.  They will appear to slope almost linearly downward, exactly as in the question.
Thanks very much Sven. Your reply enlighten my knowledge towards interaction output. ^^
Well, it's not just you. I feel like there are 5 guys writing all the Bayesian nonparametrics papers. They all write the same way, and lose me in exactly the same spot in every paper. Unfortunately, that spot is before page 5.
Actually, user1983395, that document appears to mention, just in passing, the term 'binomial deviance' (without 'capped')... and it doesn't seem to define it. What is the capped binomial deviance?
An exponential distribution will plot as a straight line against $-\\ln(1 - $plotting position) where plotting position is (rank $- a) / (n - 2a + 1)$, rank is $1$ for lowest value, $n$ is sample size, and popular choices for $a$ include $1/2$. That gives an informal test which can be as or more useful than any formal test.
I am trying to understand the regularization parameter for linear regression. And not sure how to take the derivative $\\frac{\\partial }{\\partial \\beta} \\lambda \\beta^T \\beta$. Is it correct that answer should be $\\beta = [X^T X + \\lambda I]^{-1} X^T Y$
@propofol: I think you would weight them if you concluded that the two samples came from the same distribution and wanted to derive a new estimate of the standard deviation for the combined sample; more importantly, you would also need to take account of the difference in the estimates.  But I was assuming that at this stage David was interested in the question of whether the estimates from the two samples were significantly different - on these numbers there is insufficient evidence to conclude there has been a change.
@Frank H: In the presence of static covariates you are suggesting that we condition their coefficients on a deterministic impact of "memory" or "time".There are two diametrically different ways of conditioning or adjusting for "memory" or "time". The second approach would be to use a suitable lag structure for the Y variate as causative/support variables. I have found that trying both ways i.e. conditioning upon dummy time trends/level shifts vs lagged memory,one can determine an optimal strategy to evaluate the significance of the static covariates.
@mbq But the problem actually isn't the sampling rate.  In my $sin(x)$ example above, DFT will always tell me that I have 5 repetitions, regardless of how dense the sampling is.  Simply because it will always give an integer number of repetitions.
I agree with @timbp. But if you require summaries for variables not in the regression then stata can mark your sample by creating a dummy variable. Just type"mark sample" and you will get the dummy. Thus you can use the IF sample==1 along with your specific summary command.
Thank you, but I am not testing for agreement. I am trying to find out whether Adam and Claire agree more often with one another than Bertha and Adam on whether a painting is worth buying and I have a sample of their answers for 170 paintings.

Let's say that if Adam and Bertha agree they buy a reproduction of the painting and that if Adam and Claire agree they buy a reproduction of the painting. Now if I am selling reproductions of paintings, would I rather have that Adam and Bertha come into my store or that Adam and Claire come into my store or am I indifferent? That's what I want to know.
Most likely you will have to simulate to determine the required N. I have an overview of the ideas [here](http://stats.stackexchange.com/a/35994/7290) (albeit framed in a different context). You may need to work with a statistical consultant.
I can understand why some people didn't like the question as it is not very well posed. Yet it is not irrelevant. Sometimes you want to keep a time-invariant variable in a FE regression and I answered this question with two methods. I put some effort into this answer, so if I should have made a mistake along the lines it would be nice to know about it and not just receive this -1
@user32145 was this any useful?
Thank you for the idea - I guess I sought an "official" explanation ;)
Do you mean in your last sentence "or are these methods only suitable when the dataset is not that big?"
Is it that universities are intelligence factories, so Scotland makes more per capita; or that Scotland needs plenty of good universities to use up all the smart people it has? The most obvious retort to the assertion is not that the quality of a country's further eduction provision is a somewhat rough & ready measure of its population's intelligence, but that it's not a measure at all.
What literature on kernel-density estimation have you looked at? There is a fairly rich set of results for mean-square like metrics with pretty strong results regarding the order of accuracy you can (and cannot) expect to get.
Think about this as a picture.  2 distributions have the same mean, which is the same as saying they are both centered at the same point on the x-axis.  The SD is a measure of how far from the mean, or how spread out the data tends to be.  So what would a SD 2X as big as another SD (with same means) say about its distribution?
I'm not sure if I can write a simpler version of my question. To remove your doubts, I might add that soil samples are collected uniformly (say 50-100 samples) everywhere. Therefore, there's no bias in comparing one region to another. A straight-forward solution can be: high proportion bad = high risk. But adding the weight of mean would be interesting in the scenario and that's what I'm looking for ... @whuber
This isn't quite clear. You state that your independent variable is a proportion, if so, you don't have to transform anything linear models (ie regression ANOVA) don't make any assumptions about the distribution of IVs. Why are you asking about MANCOVA? Do you have multiple Y variables? What is the "(p<.05)", where did that come from?
You've asked an extraordinarily broad question. In its current form, it is unlikely that anyone can give you a meaningful answer. Please consider narrowing the focus. In particular, if there is some specific problem you are trying to solve, it would be best to detail that instead.
@jona: Yes, I've thought about bootstrapping. To emphasize the issues, I've simplified the original data structure in the following sense: 1) actually there are two factors involved, one between-subjects (patients vs. controls), one within-subject (pre- and post-treatment); 2) the amount of data is so large, making bootstrapping a last resort if not prohibitive.
OK, the answer for the guys who will read this later: if you just want to calculate a robust scale estimator for a piece of data 1-install latest version of R 2-install the robustbase package 3-ready to go! but if you are developing a code outside this environment you need to use weighted high medians to minimize required calculations for Sn or Qn.
@Nupul Observe Pearson's multiple references to "gravity."  Clearly he is reasoning with a physical analogy.  This pushes the question back to why *physics* uses the term "moment" for such things.  I believe it simply is a natural generalization of the idea of *moment of inertia* (a second moment), which you find referenced in the etymology links for "moment."  *That* is why the etymology is relevant.
Ok.  We can focus on the Fisher's exact test. Thanks.
One thing to add - in my data set the score may not be linearly correlated with the features. For example, my data set could have very low average score for say people with Location = x, whereas it may not vary significantly for other Location values.
He, thanks for the answer. Ok thats good to know that I can not do this with only one value per group. That makes sense that there is no variance. You say it compares the medians of the values? I thought it compares the rank sums? So it schould work then if I use the column "domain" as grouping factor. This represents the dataset the variables belong to. Then I could check for differences in between the datasets, asking if there is a difference in  variable impact factor ("lifecycle") between the datasets.
@leonbloy I've expanded the question another bit, I'm still not sure how should I calculate those $m_i$ values.
I believe the scaling device you choose reflect practical matters and an underlying measurement model. [Likert scales](http://en.wikipedia.org/wiki/Likert_scale) were proposed as a way to scale persons (not stimuli, in contrast with Thurstone's approach, or both, as in Guttman's approach), and adding individual response scores altogether would make sense in this context, see, e.g., [Unidimensional Scaling](http://www.sagepub.com/books/Book445) by McIver and Carmines.
I should say that the 18s time was for the autocorrelation of a single ~1e7 element array.
The on-line help for `ranksum` in Stata is pretty straightforward. What do you mean precisely by "dummyvalue 3 and 6": are countries coded in separate columns, or as one column?
While I don't doubt the veracity of your claims, it may be more useful to the OP and to the community in general if you describe how the lme4 package handles missing values.
I'll use that train of thought to get a starting point for @EpiGrad's power plot.
We don't yet have the density data, but I'm trying to justify the cost of acquiring it. I think I'll have to make up some values just to get a better understanding of your answers! PS: I've had quick look at that book and it seems very well written. I've been trying to find a clear explanation of how to measure regression to the mean and this is the first place that I've found it. (p.14-15)
Cool, make sense to me.
Thanks, that's the answer I was looking for, so SPSS adjusts the p value so I don't have to adjust my interpretation of it.
Indeed, justifying normallity or any other model is going to be difficult with only 3 observations. Nonparametric methods are not safe in this case either, given that their convergence is typically slower. I would check several methods and see if the resulting conclusions agree.
I think a lot of people skip path analysis per se and jump to the more comprehensive (and demanding) structural equation modeling, e.g., using AMOS.  Can't help with your question about nominal variables, etc., but at 1st glance some interesting sources on path analysis in SPSS Base are http://psych.unl.edu/psycrs/942/q2/pathex1.pdf and  http://cw.psypress.com/multivariate-analysis/medical-examples/chapter07/med_path_regression.pdf .
+1; This answer seems more appropriate and general than the accepted answer however in the particular use case wouldn't it fall apart for r values of 1?  Is something like an emperical logit reasonable here where one would just "add" a datapoint that lacks the correlation?  If so, where would one add it?  Would one have to conduct a monte carlo sim grabing two random variables from the source distributions? Alternatively would one just adjust r to some value slightly less than 1?  By how far should one adjust?
*Good* answers to this question would exploit information about the error structure of the $z$ data.  Different kinds of data will benefit from different statistical treatments of this problem.  The fact that some of the $z$s can be zero already indicates that taking logarithms will be inappropriate, but it doesn't really help decide among the many available alternatives. Could you perhaps edit your question to indicate what $z$ represents and how it is measured?
Nice explanation. Thanks!
Just to clarify on my last comment (misunderstandings are too easy on the interwebs): I do *not* want "foreigners without credentials out of my wonderful country". I am simply afraid that Austrian culture is not very welcoming to someone in this situation.
Hi @Zen, I haven't leant anything about absorption identity yet. Are there other ways to prove this?
Can you clarify what is actually observed? If you really observe both x and the CDF, it seems that the parameters become exactly known after a couple of observations. So I suspect you don't really mean that the observations are of the form $(x, F(x))$ where $F$ is the CDF.
I'd post the data as well, but I'm no good at formatting and can't get it to look right. There are 5 varieties of orange trees, each with sample size of 7 (For the one-mean ANOVA test, N = 35, I = 5, ni = 7)
@chl: see update to the question
@nico: My feeling is that Peter has responded to the question exactly as expected and intended given this is a homework question.
As with @gung I am uncomfortable with the way you have conceptualized this.  As others have said you will need to take several statistics courses to be able to do a competent job with this type of analysis and its interpretation.  And the fact that you or your boss automatically assume that LINEAR regression is the way to go is quite telling.
I believe you will find the answer at http://stats.stackexchange.com/questions/91402 to be revealing and helpful. With minor changes, it applies directly to your situation.
Reread the assumption! It tells you precisely when one would consider this approach. :-)  (Of course there's more to say about it, for otherwise this procedure would be used far more often than it is. The key is to consider whether accounting for the errors in $x$ might make a difference in the analysis.)
Very Python indeed.
Thanks Yang! If I understand your answer properly, you want to say that in (1) Income is including all the direct & indirect effect. but in (2), due to the presence of Inv & c.p. condition Income is taking only the direct effect on GDP & not the indirect effect. So, that's why there's difference in coefficients.
Oh dear god, please never use the right assignment operator `->`
Thank you for the example. That's exactly what I needed.
(+1) Thank you very much, @whuber, this is a great answer! Special thanks for mentioning the word "frustum". It so happens that I have accepted another answer just minutes before you posted yours, and I wouldn't like to de-accept it now; hope you understand. Pity that it's not possible to accept both! By the way, note a very simple proof of $1/D$ expression for variance from that answer: one can see it directly without messing around with beta functions! Variance of the dot product is equal to variance of any sphere coordinate (as you wrote), and a sum of all $D$ of them should be $1$, Q.E.D.
Not quite... perhaps I was a bit loose on my description. The diagonal elements of your transition matrix for the threshold model receives the original self-transition probabilities from your other models. The transitions from a state to other states are initialized with uniform probabilities. I know it may seems lazy to point it out to the code, but sometimes [code is easier to understand than formulas](https://code.google.com/p/accord/source/browse/trunk/Sources/Accord.Statistics/Models/Markov/Learning/HiddenMarkovClassifierLearning.cs#162).
It is strange to see this done with a plot of predicted vs. fit: it makes more sense to see the intervals in a plot of predicted vs. *explanatory* variables. The reason is that (except in the simplest case of a straight line fit to one explanatory variable) the SE does not depend on the predicted value: it depends on the values of the explanatory variables. Thus the plot isn't even well-defined as a curve; it should be thought of as (at best) a collection of points, each with its own little vertical error band.