tttjjj/elt_bench huggingface dataset:
I'm trying to more fully grasp the theory behind the calculated values (to match with the sim), but I'll answer you as best I can. If you're asking about the y -- that is y=x1+x2 where x1,x2 are independently realized random variables.  This is often known as the joint sum of independent probabilities.  If you're asking about the last case, where P(y=0.25)=0.8,P(y=1.5)=0.2 -- that basically says "if the random sum is <= 0.25 then it is 0.8. if the random sum is >= 1.5 then it is 1.00".
Do you have the full paper pdf that you might share?
@Brett There are a few book recommendations on my [related question](http://stats.stackexchange.com/questions/1815/recommended-books-on-experiment-design) and some others may accrue in time. Not sure how well they fit your criteria, but might be worth a glance.
Carl, you can't derive expected scores ($E(Y|X=x)$) *from* the correlation in that way. To make those kinds of statements requires a a few [additional pieces of information](http://stats.stackexchange.com/questions/32464/how-does-the-correlation-coefficient-differ-from-regression-slope) -- and then corresponds to interpreting a regression relationship, rather than a correlation. You can make such statements in terms of *standardized scores* from a correlation, but that requires substantially more explanation.
(1) looks suspect because its conclusion is incorrect: Bayesian results with uninformative priors are *not* necessarily the same as frequentist conclusions. In fact, in [many cases](http://en.wikipedia.org/wiki/Prior_probability#Uninformative_priors) there appears to be no agreement about what an "uninformative prior" even is!  (2) also is debatable because it implicitly assumes that every assumption made in a frequentist analysis is not subjective--but that's not the case. Finally, what exactly do you mean by "better in interpretation"? Your question is ambiguous without a definition of that.
@kjetilbhalvorsen Thanks for the add on contribution in comments.
Thanks kjetil for welcoming me the acronyms: UCLF mean Unplanned Capability Loss Factor and OCLF mean Other capability Loss Factor they are the types of energy loss.
Bad idea - [see here](http://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable/68839). And using the response variable to decide where to break a continuous variable is especially bad, biasing the coefficient estimates. Did you check for curvature in the relation of the logit of the response to the predictor?
Unfortunately, this question is too old to be migrated to Stack Overflow. I believe it belongs on Cross Validated but if you want to ask on Stack Overflow (putting emphasis on the *programming aspect* and providing a minimal *reproducible example*), let me know and I will close it here.
I like the Cholesky version better - you have square root of chi-square on the diagonal and standard normal on the lower triangle.
Thanks M- Adams. In response to 1: I've noticed that the structures are nearly consistent. For 2) Thanks! I am validating against a sample (split my big set in half) 3) Are you suggesting a model where they may be a factor driving other factors? Thanks!
please be more specific what you're trying to ask.  Are you trying to distinguish between the standard error of the mean versus the true standard deviation of the sample mean?
This is much simpler than a binomial distribution. If there are $n$ equally likely outcomes, the probability of each is $1/n$, and the probability of a set $S$ of them is $|S|/n$. This tells you the probability the winning ticket will be among those in the set $S$.
Nice answer, and the association with Condorcet's jury theorem makes sense. Formally though, the reason it works well is because of jensen's inequality!
Thank you Jeromy Anglim for your response and the links to "related questions." They were very helpful.

I worked on computing the Wilcoxon signed-rank stats option in SPSS and have prepared a table with the T, z, p, and r (effect size). I will now try out the paired-samples t-test to see what the results look like.
Can I perform a Kruskal-Wallis with 4 groups? (male+patient, male+control, female+patient, female+control)
I suspect that this is not what you mean when you say "something wrong with your regression equation", but you should be aware that the NLSY is a complex survey and fitting a regression in standard software ignoring the design will result in invalid coefficient estimates and hypothesis tests. See [the NLSY documentation](https://www.nlsinfo.org/content/cohorts/nlsy79/using-and-understanding-the-data/sample-weights-clustering-adjustments) for details. I've found that Stata is the most user-friendly tool that supports regression on data from complex surveys.
So it is related mainly to the sense of variables, and is not important for the regression accuracy and future prediction of values?
You can construct the weight vectors from the support vectors and their coefficients. See http://stackoverflow.com/questions/10131385/matlab-libsvm-how-to-find-the-w-coefficients
Can you provide more info regarding the data? Which scale-levels? Are both binary variables like: treatment (replacement-therapy/no-replacement-therapy) and baby(does-better/does-not-better)? Then you need to use Chi-Square test.
I just toyed arround with this nnls package a bit. Is there any way to get an adjusted R-squared value (or something equivalent), or would I have to try and calculate it myself somehow?
$E[X\\mid Y]$ is a _random variable_ taking on values $\\m_0$ and $\\mu_1$ with probabilities $P(Y=0)$ and $P(Y=1)$ respectively and so the expected value of its square is ....
Would you like to explain your reasons?
I can see how such a model might provide a more precise model of observed responses. However, in the typical practical research situations that I have seen, researchers are interested in whether the two groups differ in terms of the mean (e.g., did the training group report greater performance than the control; was student satisfaction higher one year to the next). The proportional odds ratio model does not test this question exactly as far as I am aware.
If you want to make comparisons of the means over time (which I'm assuming you do) you need to know that the construct hasn't changed, otherwise the test of the mean difference doesn't make sense. E.g. if the reliability is much lower at time 2, you'll want to know that, because that might create an false difference.
Good point. It requires that there not be different variances for the errors by group and that the errors not be correlated in the different groups.
See [What's the difference between correlation and simple linear regression?](http://stats.stackexchange.com/q/2125/32036)
"*I also know that a Mann Whitney U test is more powerful than a t-test when these assumptions are not satisfied*". That's too strong a statement. Let's say the data were uniformly distributed (for example). You say you know that in those circumstances a *U* test is more powerful than the *t*, but it's not the case.
+1 This seems like a really good idea. Thanks !
I did ask my professor and he is not giving me the answer. He is just telling me that I am not right
@DonutPhil If you have just one point, what is there to discriminate?
Assuming that you refer to samples rather than populations, it seems that it is necessary to have the original data in order to fill the ANOVA table. A workaround probably is to create random data from the assumed underlying distributions of each samples and apply F test in these data.
Sounds like you want to test coefficients in a multiple regression/ancova
You have three stimulus and these will be an effect on your conditions. You should compare your condition 1 vs. 2 based on Stimulus A, B or C. Do the same thing for Condition 1 vs. 3 and 2 vs. 3.
KW does not assume the same distribution in each group. It is simply a test for stochastic dominance.
1. How due to (not necessarily strict) monotonicity of $G$, we have $G^{-1}[F(X)]\\leq z \\Rightarrow F(X) \\leq G(z)$? I know that for strict monotonicity that argument is correct. But, how does it follow for nonstrict case?

2. Why due to monotonicity of $G$, $F(X)<G(z)\\implies G^{-1}[F(X)]\\leq z$?

3. Why $G^{-1}[F(X)]\\leq z$ implies that $[F(X)\\leq G(z)] \\cap [G^{-1}[F(X)] > z]$ is a subset of $F(X)=G(z)$?

4. How $F(X) \\sim U(0,1)$?
@PeterFlom Thanks for your attention. The number of columns is as less as 4 to as much as 10 in some cases. There are no missing datas fortunately
By inspecting `model.fit(lm(log(z) ~ x*y))` you will quickly see what `x:y` represents and then you can directly apply the solutions you reference.  It's a good idea, though, *not* to transcribe coefficients from the output, because they are rounded. To retain their full precision--which becomes more important with interaction terms like `x:y`--extract the coefficients from the result of `lm` as doubles and use those for subsequent formulas.
Uwe Ligges on r-help has already given the ['answer'](https://stat.ethz.ch/pipermail/r-help/2011-July/283220.html): You cannot have standard deviations of 0. Are these missing values?
There is another discussion relevant to these topics here: [what-problem-do-shrinkage-methods-solve](http://stats.stackexchange.com/questions/20295/).
I don't completely understand your question - can you flesh it out a bit more? Incidence rates are just that, rates, and thus have person-time "exposed" in the denominator. You are correct about the 24/2, which reflects the  assumption that the people who develop lung cancer did so at the mid-point of the interval, and thus censors them at 6 months. In contrast, you could estimate a prevalence ratio (24/2300), but a prevalence is a function of incidence and disease duration, thus less useful if you are interested in identifying causes of disease.
You might want to see my related [question](http://stats.stackexchange.com/questions/56681/gauss-markov-theorem-blue-and-ols), and clearly the answer seems to be "yes", but only among linear estimators.
I like the nuance about "different levels of discussion."
I do use R, but I have to use jmp in this case. Maybe I have not explained myself properly but I have already modelled the data - fitting a maximal model then reducing to minimal model which is Y ~ A + B + C + A:B. This tells me A has a significant effect but not which level of A, it also tells me A effect depends on B or vice versa - but not which level of A. I can just plot it and compare visually but as I understand it you can do this statistically with contrasts
1) Yes, they are just different versions of the same test.  2) I think, insofar as validating the significance of the test is concerned, the only issue is the missing data.  If you look at the scores on test 1 of the nonrespondents on test 2, that may help you decide whether to consider the missingness to be "at random" with respect to the responses.  (Maybe it won't, also.)  Otherwise, you can do the analysis, but put a caveat in about the missing data - "this analysis assumes the nonrespondents can be treated as missing at random" or some such.
I wouldn't want to advise on this, I do not consider myself as being especially proficient in the theory behind simulation methods. Since it appears that your interest lies more on applied derivations of distributions, I suggest that you update your post to include these questions, and also tag it appropriately so that it attracts users that can help you with them.
I dont want to calculate similarity matrix. I need only feature matrix with numerical values, since i need to run Non-negative matrix factorization later which requires feature matrix.
Note that (a) you have three total parameters, (b) you are using only two pieces of data (i.e., statistics) and (c) you state you have per-month totals as well (but aren't using them). So, you're throwing some potentially valuable information away since two statistics won't be sufficient for three parameters. Imagine, for example, that there are no failures in a given month for one population, but there are some in the other. This immediately gives you additional information about the rates that the running totals won't.
Estimate it from the data.
Don't regard me as an expert on survival analysis (I'm more nearly a beginner) - but ... (oops, my comments grew too long -- I'll edit some more and add it to my answer).
In terms of regression and ANOVA, controlling for a variable usually means that variable was included in the model.
@Inq This sounds closely related to [changepoint](http://stats.stackexchange.com/questions/tagged/change-point) detection.  The statistical technology is the same, and the same issues come up, but the purpose seems a little different.
I think you answered your own question. You should standardize the data if your inputs have different scales.
@Damien: The issue is, while the event is one day, you are asking about its effect. IrishStat's suggestion of 0's through time 255 and 1's afterwards addresses the case where the event has a permanent effect after it takes place. As he then says, it's possible that the effect started before the event (say rumors were swirling), or to not kick in until after the event (say the resignation was Friday after the close of business), and that the effect could trail off over time.
It is unusual for boxplots to show confidence intervals. Don't these show quartiles, fences, and any outliers?  It also seems worthwhile asking why you would want a p-value when the difference is so blatantly obvious: clearly the simulation does not reproduce the observed values.
This sounds like a description of two-way ANOVA (days by cycles) followed by planned tests of 11 pairs of days.  Plain old stats software is likely going to be more flexible and powerful to use than specialized time series software; it will certainly be easier.  BTW, the indexes are days (into the exam period), not week of the semester.
An answer to [your previous question](http://stats.stackexchange.com/questions/29882/when-does-the-law-of-large-numbers-fail) demonstrates that it cannot possibly fail in this application (for which it is implicitly assumed the repeated observations of the event are *independent*), so this question is devoid of content.
(1) Did part of your answer get cut off?  (2) I believe the thread Sarah references disputes your "of course," Eric.  (We should also wonder what it means to be "statistically invalid" in the first place.) Exceptions to your "also" have also been discussed.
For example, there are assumptions you don't have in the question [here](http://stats.stackexchange.com/questions/55113/where-do-the-assumptions-for-linear-regression-come-from).
Thanks.  I should have made it clear that I know how to calculate a p-value given a test statistic but I find it strange that they don't output the p-value for you as in almost all the other SAS procedures.
@Ryan I just had one final read through and noticed an error - where I had $σ_O^2 = \\sqrt{}...$ that should have been $σ_O  = \\sqrt{}...$ (this was caused by initially giving the formulas in terms of the variances and then editing to give in terms of s.d.s). I hope that change doesn't cause any problems.
I am not using an kriging package. I am writing it on my own. So it's inefficient, that's why I was looking for alternatives
These are good points.  Outliers and other data problems plague *any* dataset, no matter how large or small.  In my experience they actually are easier to identify and deal with in large datasets, because you have the power to discriminate them from the mass of data and, especially if you use robust methods, they are less likely to influence the results.  BTW, you're *always* doing "data cleaning" throughout any analysis. This isn't something that can be segregated and referred to a specialist to be handled once and for all. *An outlier is only an outlier in the context of a particular model.*
I don't think SEM or similar approaches would work as the correlation structure of DVs is really unknown and is not really of interest. For example in my gait example, there are tens of outcome measures so things can get really complex.
As pointed out by @RobertF (in a comment to one of the answers) the reason why options #2 and #3 were not acting as expected is because I was scaling (and centering as well in #2) BOTH the predictors AND response vars.  Only the predictors should be centered & scaled... when I do that, then option #2 gives me exactly what I needed to compare the relative magnitude of effects on the response (which also correspond quite well with the `lm.beta(nb)` values I discovered later.  Now I have more confidence in my results.  Thanks for the help!
"Analyzing trend" would be the topic of more than one very large book: your question is too general to be answered. If you can tell us something about the context, the purpose, and the nature of these data we might be able to help.
If a large majority of observations is around 20 and the mean is, say, 25, then the linear regression coefficients will be optimized for this center value. This is where the prediction interval will have the smallest range. By "discretizing" the observations at the 60 score threshold the logistic regression would not suffer from the same problem?
Because the "reflection" is straightforward to interpret, this question effectively is a [duplicate of questions](http://stats.stackexchange.com/search?q=interpretation+log+add) about interpreting the "started log" $\\log(1 + Z)$.
Equal sample sizes are not required in Kruskal Wallis. This looks as if it might be more of a matlab issue (either an issue with the implementation or with your usage of it) than a statistical one. Should this post be flagged to be moved to SO?
@whuber I think the future population of *potential* sales is known, just not which will be won & which lost.
You can try [Frobenius norm of a matrix](http://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm), which is basically $trace(\\Sigma^T * \\Sigma)$.  For choosing the threshold, you can add a [Regularization](http://en.wikipedia.org/wiki/Regularization_(mathematics)) term in the information gain expression.
Note that ridge regression doesn't *select* predictors; it shrinks their coefficients *towards* zero. LASSO also shrinks coefficients towards zero, but some may reach zero & thus be removed altogether.
@Michael: I don't believe Didier unintentionally dropped your proposed nonnegativity constraint. Indeed, this situation is completely analogous to a comment I made on one of your answers a day or two ago (namely, that artificially imposing a similar constraint did not yield the desired answer). Because this is a Gaussian model, the unbiased estimator of $\\beta$ will be linear functions of the $Y_k$.
@user2432701 No I would like to use all 100.
It all depends on what exactly you mean by a "normal" and a "lognormal" distribution. The conventional mathematical definition of "normal" makes nonsense of this question, because the only normal distributions supported on finite intervals are the degenerate ones: one might go so far as to say there is no such thing as a "normal distribution on the interval [0,1]." Do you want to *truncate* your distributions, perhaps? If so, then why do you propose truncating *twice* in your procedure, when once will do?
Thank you Glen. I'll make some tests with real data and will post feedback.
@CarlosAccioly I actually have commented about "death by Beamer" - it's a welcome change of pace in talks dominated by PowerPoint, but after the 50th generic Beamer presentation, it's hard not to tune out.
@Michael Chernick Yes. But i could also use the t test as the sample is large enough.
Prediction error refers to the discrepancy or difference between a predicted value (based on a model) and the actual value.
Yes, the presence of a bit error is what I want to find.  I arrived at a similar solution as you, but I didn't trust my decade-old one-semester grasp of statistics.  I thought there might be more to it that this.
i don't think i understood you fully, but regarding the heterogeneity of the correlations you are trying to combine, you can always do a Q-Test to check whether or not the null hypothesis that they were all sampled from the same population makes sense (which is the usual purpose of meta analysis as i know it ...). [Q-Test](http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2006-07641-005). Additionally i would recommend that you get old of Michael Borenstein's Introduction-Meta-Analysis-Statistics-Practice which is a very complete introductory book on Meta Analysis.
... and is your only research question "which groups have higher satisfaction" or is also "which dimensions is there satisfaction with" or "what is the relationship between the different dimensions of satisfaction".  I note for example that supervision seems to get noticeably lower scores...
Fisher [works on general tables](http://mathworld.wolfram.com/FishersExactTest.html); it's not limited to 2 x 2.
Because this might help future readers, I offer a geometric translation of the general part of this question: "How can a shape whose area does not exceed $1$ possibly extend more than $1$ in any direction?" Specifically, the shape is that part of the upper half plane bounded above by the graph of the PDF and the direction in question is vertical. In the geometric setting (shorn of the probability interpretation) it's easy to think of examples, such as a rectangle of base no greater than $1/2$ and height $2$.
@ocram It's definitely *not* binomial. The phones are sampled *without replacement*, so the trials are not independent.
Essentially no regression assumes X and Y are independent- otherwise there would be no point using a function of X to predict Y
What do you intend to *do* with this matrix? How is this question related to statistical and data analysis?
I'm using R. I just learned about the sins of stepwise regression....my original analysis and two stats classes were pro-stepwise
(+1) Let me just add the amazingly detailed an accurate [A History of Probability and Statistics and Their Applications before 1750](http://www.amazon.com/History-Probability-Statistics-Applications-before/dp/0471471291/ref=ntt_at_ep_dpi_2) by Anders Hald as a possible reference.
Is this really a meaningful or important question to ask? What you really want to know is *whether the procedure works.*  That can be answered only in a context of what images you intend to apply it to. For instance, it's a terrible procedure if your images are rotated, as you could easily tell just by looking at the results.  So, in order for this question to be answerable--if you still need it answered--please characterize the images you intend to process.
"Frequency" as a physical measurement actually has little intrinsic meaning in western music, most of which is based on the *relative* frequencies within a particular paradigm of composition, performance, and listening. Coupled with the impossibly vague referent of "western sheet music," this would seem to make the question unanswerable. Perhaps you would like to make some clarifying edits to remove these ambiguities?
@chl, thanks. I updated the question. I already understand things like [this](http://stats.stackexchange.com/questions/2356/are-there-any-examples-where-bayesian-credible-intervals-are-obviously-inferior-t/6431#6431), but not how it actually works. Thanks, Tomas
Thanks, it helps a bit. I think I should give me some guidance
Do you mean "For example a correlation that has a p-value of 0.000 is highly significant?".  In which case, it is also worth looking at the magnitude of the correlation as well.  Are you able to show us a small piece of the SPSS output?
1) I'd have used the `ecdf` function. 2) The function doesn't have an actual inverse. What did you need it for? Random sampling?
@John Salvatier the goal is redacted :D ... actually I want to do some analysis on what kind of information generally gets redacted in free form text documents. In other words what sorts of information do people consider sensitive - and if its possible to use some machine learning algorithms to aid automatic redaction. I read sometime back Xerox had something like this - but there's no more info on how that can be done. I have some idea I want to try out, but no data ...