tttjjj/elt_bench huggingface dataset:
I never heard "multiple discriminant analysis". Perhaps it is a synonym to LDA or DA (LDA+QDA). Multidimensional Scaling (MDS) is very different from DA because there is no groups; MDS is more related to PCA. LDA is more related to MANOVA and Canonical Correlations. I recommend you to search this site for `MDS`, `Canonical correlations`, `LDA`.
It seems like a simple correlation would be what you're after I think?
thank you for the help
i also found gllamm and denom() option is also useful
(+1) `psych::alpha` does a better job than `score.items`, thanks for adding that. I never remember the one I should use!
Are the three experimental conditions independent or is it repeated measures?
@wolfies Yes, I obtained that expression too: it integrates the tail of $K_0$. Because the exact distribution does depart from it in the extreme tails, it did not seem worthwhile carrying the analysis of that integral any further. The logical next step is a more discerning analysis of the tails, which means going beyond the Normal approximation.
Thanks fabians!
Not your question, but 8/30 values in your data are equal to the maximum 300. You are evidently fitting a three-parameter version of the Weibull, but even so, these data don't look like a good candidate for such a distribution. Some kind of censoring issue?
The `predict` function often has a variable `type` and for a logistic regression you could use `type="response"`.
I agree with you Peter, but I am slightly more optimistic. An important aspect of using R/SAS/Stata/SPSS/... is the statistical way of thinking. Learning how to use such a package will require the student to learn both the specific package but also the general statistical way of thinking. I'd say that learning to ask the right statistical question is more useful (and harder) than learning the language to actually pose the question in. In that regard, learning *any* statistical package will be helpful even when the student will end up using another one eventually.
@MichaelChernick I took the liberty of moving your post + comment where I believe they should have been posted (and clean up nice comments you got on them from Macro and cardinal). As they kindly pointed out, this site shall be viewed as a *wiki* (with full support for edits, history, $\\LaTeX$, etc.) where we intend to collect long-lasting responses to specific questions about statistical analysis, with minimal clutter in the response thread. Should you want to ask for clarifications or comment on the present question, you can always make use of the comment functionality. Welcome to this site!
This sounds like a condemnation of *mindless* power analysis, not of power analysis itself.  The bigger question is whether this is an attack on a straw man or if indeed there are many people who conduct their power analyses (or any other analyses) without regard for their sensitivity to assumptions.  If the latter is true it is good to enlighten them, but I hope they do not become so discouraged as to abandon all efforts to plan their experiments!
Welcome to the site, @Don.S. This appears to be a comment on B_Miner's answer, rather than an answer to the OP's question. Please only use the "Your Answer" field to provide answers to the question at the top of the thread. If this is an answer, please edit to clarify, if not, this comment should be deleted. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq), which covers info like this about CV.
Why do you believe the same dataset will be separated with $K_1(x,y)+K_2(x,y)$?  Is this homework?
Neat and precise!
Lamentably, in that document Crawley perpetuates the myth that ANCOVA can be applied to scenarios when the covariate is correlated with an IV, even when this explicitly violates an assumption of ANCOVA.
@Macro He did his undergraduate work at OSU so naturally he would not consider his arch rival school. Gschneider was a grad student that was a TA for one of his courses.  Maybe you would feel better to know my Cousin's daughter went U of Michigan.
This should have one caveat though - there should be some measure of how representative their previous record is.  For example a change in income could easily make previous defaults or successful repayments irrelevant.
(1) Variance-based stopping rule: It affects the variance estimate, & therefore can affect error rates when the experiment's analysed as if the sample size had been fixed beforehand.  There's a tension between the caveat of "beyond a reasonable minimum N" given in your comment & the "small sample sizes" referred to in your answer; doubtless you have the statistical nous to know what approximations are good enough when, but not everyone does. More generally, an unimpeachable approach is to clearly define the stopping rule before the experiment.
I posted an answer below, but also think about how you would be able to answer this question if the functions `dbeta` and `rbeta` did not exist. Those functions are for convenience but you do not actually need them to solve the problem.
+1, pressed post the same second your answer appeared. I would add that for a full tree rsq.val[nrow(rsq.val),] is probably what OP wants.
@soakley: what should it matter since you almost surely don't "look at" both variables at the same time.
This would be a better answer with a short description of Gigerenzer's main claims.
Well, Alexis, GASP indeed, but this is why when reading a paper, never skip its preparatory sections -it is where the author defines the symbolic language he will use -and if he doesn't, he is sloppy.
Welcome to the site, @willy. Note that stepwise variable selection routines are not considered valid anymore. If this doesn't make sense to you / you want to know why, it may help to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856).
(+1) Since it is easy to characterize exactly what the (theoretical) correlation will be, adding that small bit might improve this answer even further.
@bruce: +1, I also think that $b_1, b_2$ can be replaced with a single Bernoulli since the only time that the dependence between them matters is when $N_1$ and $N_2$ have simultaneous events, but this happens almost never.
As bayerj suggests (+1), the tanh is just a rescaling of the logistic sigmoid (which is the normal choice for binary regression).  Just replace a_output with 0.5*(a_output + 1) and this will undo the scaling.  However, there is no good reason to use tanh for binary classification, just use the logistic function instead.  See Bishop's book on Neural Networks for Pattern Recognition for a good discussion of the relationship between cost functions and output layer activation function (and neural nets in general).
You are not computing variances correctly.  The variance of $(1,3,5,7)$ is $5$ and the variance of the means of all its two-element samples (with replacement) is $5/2$, as expected.  (This is the multiset $\\{1^1,2^2,3^3,4^4,5^3,6^2,7^1\\}$.)
"*In situations where unequal variance t-test is appropriate, if we use equal variance t-test, type I error would be inflated*" --- not quite. If the smaller sample has larger variance, type I error is inflated, but if the *larger* sample has larger variance, type I error is decreased from the nominal. If the two samples are equal in size, type I error is pretty much right.
@AlecosPapadopoulos Yes you are right. But let's say that in this question OLS means OLS with normal disturbances.
Ultimately, the question of relevance to you concerns comparing two empirical distributions.  This is a popular topic here, with many answers available: please [search our site](http://stats.stackexchange.com/search?q=compare+distribution) and read over any threads that might appear relevant. You might also consider viewing your data in a different way: as a [tag:time-series], in which you retain the information about when each tip was received. Time series analysis provides models and methods to track your data and (to an extent) forecast future values.
@Frank thanks I caught your attention. I have modeled the $AUC$ and  with same model, given the optimum decision. I will like to go ahead to make predictions. I do not get your point of $ML$ estimation. Dodd and Peppe(2003) parameter estimates were $ML$ estimates.
@whuber absolutely. My bad. By the way, I like your geometric answer (to the duplicate of that question) better :D
Excellent! Exactly what I needed; thanks!
You can use model comparisons with likelihood ratio tests for a full and reduced model using `nnet`'s `anova()` function.
@Glen_b In the Excel sheet I believe the expecteds are worked out by doing Total Population * the 'P' value I worked out above. Is this going to be a problem? Also the total population does vary - most of the time it is 2000 but it could be any number really. The Excel sheet I'm trying to recreate here doesn't actually take into account the p-value, so if the statistic won't be affected by this then maybe it's not a problem...
Why don't you drop the categorization and do a regression analysis? You would get rid of the unbalancedness and you would gain power, because categorization wastes information you have. Also MANOVA won't give you a result to interpret the way you want ("variable 1 differed...").
This wikipedia article on endogeneity should get you started http://en.wikipedia.org/wiki/Endogeneity_(economics)
How does this question differ from your preceding one, http://stats.stackexchange.com/questions/13433/repeated-measures-anova-in-r-and-bonferroni-adjusted-intervals? Moreover, some users already suggested you to provide *more details about your data*, which is important to get a useful response (for everybody).
@mbq: Can you provide a link?  It's not in the FAQ.
This is a standard topic in any linear regression textbook. The statistic for testing the hypothesis is called Wald statistic.
There are cases when the effect is huge but the p-value is small, and the opposite can happen too. Trying to combine the two in some sense amounts to going from frequentism to Bayesian, i.e., you believe there is a continuum of truth with some probability distribution. In that framework, you can simply take the expected value of the effect size.
(+1) I came across this interesting blog post: [Handling Non-Normal Data in SEM](http://rudyanto62.blogspot.com/2008/01/handling-non-normal-data-in-sem.html).
By "matched pair of binary data" do you mean you have two vectors of binary numbers, with the rows matched on the basis of some criteria?  That doesn't seem to correspond to your next-to-last sentence very well, since obviously the sampling distribution isn't approximately normal (if that's the case.)
One could write down multinomial formulas but this way is more fun :-).  These algebraic methods come into their own in more complicated situations, such as [probability calculations in the game of Scrabble](http://stats.stackexchange.com/search?q=user%3Ame+scrabble).
I would recommend Proc UCM over ARIMA because, it is much more flexible and more interpretable especially when you want your time series data to be decomposed into trend and seasonal component. you can go to SAS website for more help. I can provide you with an eexample below. If you can post your data, I can see if I can model the data in proc ucm. http://support.sas.com/documentation/cdl/en/etsug/60372/HTML/default/viewer.htm#etsug_ucm_sect040.htm
If you *have information* (which is about the only reason I'd use an informative prior in this situation), then use it to construct the prior. Alternatively, if you don't have the information to construct a finite variance prior, why would you insist on an informative one?
+1 extrapolation and prediction intervals is an excellent point!  For regression of one scalar onto another, plotting the prediction intervals ought to be standard operating procedure.
@AleksandarDimitrov: It would be good to show the box-whisker plots here so others can see how transforming solved the problem. Were there any "outliers" in the transformed data? Or (my guess) were the outliers before simply the tail of a nongaussian distribution? Did the transformed data pass normality tests?
To be precise (pedantic?), we should clarify that the bootstrap is *not* per se a Monte Carlo technique. However, it is usually *approximated* by using Monte Carlo simulation since it is not often tractable to calculate the particular quantities of interest. I find the wording of the problem very prone to misinterpretation as well! One must be careful to distinguish *sampling error* from *Monte Carlo approximation* error. The latter is only sensibly estimated by repeating the MC bootstrap approximation many times on the *same* sample, as you point out. :)
As a wrote, this was a "toy example". Thus, there are no meaningful p-values to report, I'm afraid. The point is that the test may stick with the null hypothesis although it is false (type 2 error) for two decimal places when omitting the last one. Thus, omitting decimal places may give the test a lower power.
The data points can be larger. 30,000 here is just used for illustration purposes. My interest is that how to setup the sample size, which can be 1/4 or 1/8 of 30000, without losing any statistical significance. For example, I would like to be able to say something, such as we can just draw a sample for every 10 points. How to design this kind of plan is my question.  I think there should have a formal way to design this kind of plan statistically.
..i guess i'm confused.  if data set A has a thousand households and data set B also has a thousand households, why wouldn't you know anything about within-household variability?
I think the robust analogue of CV would be the Winsorized standard deviation divided by the trimmed mean.
+1 for the rigor and for addressing the second question about the EDFs.
On the question: "*Is it possible to calculate the probability that two samples are drawn from the same distribution?*" -- the short answer is 'no'. How would you distinguish between $F=G$ and something *very* close to equality but still unequal? The longer answer is "maybe you kind of can do that, if you're a Bayesian".
I am interested in imputing - the first point you made. Double checking if this works for categorical variables - what would be y and x variables ?
@suncoolsu : it does say 'nearly'...
@Scratch I'm sorry but I don't understand this "you can only move on the triangle plane by definition of the support" :(
That's 1.6 s.d.'s either side of the mean. If the given mean and sd were population figures rather than sample values, you could use inequalities like Chebyshev. The problem is Chebyshev doesn't take into account the impact of the sampling error in the parameter estimates. You may be able to bound the percentage from the sample values, but I can't see a neat way to do that.
See also [Problems with pie charts](http://stats.stackexchange.com/q/8974/930).
I find that discrete wavelet transforms might help me conduct clustering, but I'm not an expert on the theory behind it. Any lay-man interpretations of how to carry it out?
It would be great if you could comment on your code, for future readers or people not used to R.
Also it sounds like you're really asking about *multiple regression* (a scalar-valued response with several predictors) and not *multivariate regression* (a vector-valued response value). Is that accurate?
Why not forget about the histograms, because the problems you describe are well established, and consider alternative tools such as qq plots and goodness of fit tests?
I don't understand how the first paragraph describes pooled data, can you explain a bit more?  Perhaps the key is what you mean by ignoring the bank effects, does this mean you are actually taking the average across banks for each ratio?
The book, Discrete Multivariate Analysis, by Bishop, Holland, and others, has some techniques for finding patterns in sequences.
Re: $\\alpha$, I didn't know it was notoriously difficult, but it sounds right, thanks.  @probability, you are right, and the only reason I picked 3 versus 1 was because it meant less fractions.  And BTW, I liked probability's answer better than mine (+1).
TSP? Is http://stats.stackexchange.com/questions/22293/explanation-of-statistical-simulation/22295#22295 of help?
This is a very widely discussed topic on this site. This is probably due to collinearity. Do a search for "collinearity" and you will find dozens of relevant threads. I suggest reading some of the answers to http://stats.stackexchange.com/questions/14500/how-can-a-regression-be-significant-yet-all-predictors-be-non-significant
@user As I pointed out, the second derivative of the logarithm *must* have non-positive eigenvalues.  Yes, there are links between variances and negative second partial derivatives, as the theory of maximum likelihood estimation, Fisher information, etc., reveals--Macro has referred to that earlier in these comments.
@user2806363 yes it does.
@Peter Flom:
Is a queuing system with one 1 server and 1 queue where services times are $exp(\\lambda)$ and the interarrival times are $exp(\\mu)$.
It would help to make your question more clearly on-topic here (rather than SO, say) if you framed it as 'how do I fit a constrained natural spline?' and then added that you work in R and any suggestions for how to achieve it in R would be especially helpful (that is, present a statistics problem which you're implementing in R, rather than an inherently 'what R code do I use' question, which is a programming problem)
You'd fit something like $E(Y) = \\frac{\\theta}{1+\\exp(-\\{a+bx\\})}$.
I believe you need to use a kernel density estimator and calculate P(x) from there. meaning first you estimate the probability density function of your vector x and then calculating P(X) from this PDF. This P(X), is not the frequency of your word in the document, but is the probability of occurring P(X) in all documents.
`untable` does exactly what I mentioned in my comment :-) Thanks for reminding me of that function!
@whuber Do you have R code perhaps to share on how you did this?
I don't think the standard deviation of {2,5.33,4.33,3.66} is 1.21; I make it nearer 1.4 (or over 1.6 if you use a bias adjustment).
(+1) Of note, the `Hmisc::latex` function makes it easy to print the above as $\\LaTeX$, e.g. `latex(fit.1)` may yield [this output](http://i.imgur.com/7FlMr.png) (using [sample code here](http://stats.stackexchange.com/a/9808/930)).
Well, you could assess the reasonableness of the assumption visually - "does it look like one sample's distribution is a shifted version of the other or doesn't it?" - But if you have no particular reason to think it should be a location shift, you might also question whether a test of location (whether via means or anything else) is particularly meaningful anyway. Which really comes down to asking yourself "what is it I actually want to find out?"
@gung - thanks, I should have provided the link.
A search here for [dummy coding](http://stats.stackexchange.com/search?q=dummy+coding) suggests you can find a few hundred answers and examples :-).  Here's one that's particularly close to your set of questions: http://stats.stackexchange.com/questions/24242/how-to-apply-coefficient-term-for-factors-and-interactive-terms-in-a-linear-equa
From the paper you link to it follows not `Ward algorithm is directly correctly implemented in just Ward2`, but rather that: (1) to get correct results with both implementations, use squared Euclidean distances with Ward1 and nonsquared Euclidean distances with Ward2; (2) to further make their output dendrograms comparable (identical), apply sq. root to fusion levels after Ward1 or square fusion levels after Ward2, before constructing dendrogram.
Right, so the idea being that state interactions can interfere and then the **intersection** of two states can "be negative".
Welcome to this site! Could you provide a brief description of the resources available on that page?
FYI, besides purely personal aesthetic preferences, there is some current work being done in information visualization that would suggest the XKCD style theme has other benifits. See for example *Sketchy
rendering for information visualization* ([Wood et al. 2012](http://openaccess.city.ac.uk/1274/1/wood_sketchy_2012.pdf)). And some other related research [via Robert Kosara](http://eagereyes.org/blog/2012/visweek-2012-digest-part-2).
Wait, you're computing a 600x600 covariance matrix from only a 750x600 data matrix? Are there any missing values?
After plotting data, you might simply calculate the correlation between the actual data and the predicted values.  Does that make sense to you?
Please tell us what $Z,$ $\\varepsilon,$ $\\hat{\\varepsilon},$ and $\\sigma$ represent.  In particular, what formulas do you know that express $\\hat{\\varepsilon}$  in terms of $\\varepsilon$ and $Z$?  (In so doing, I suspect you will find the answer to this question yourself.)
Out of curiosity do you have a theory where the line should be beforehand? It is pretty simple to draw the hull on any plot, so finding the boundary [ex-post facto](http://dx.doi.org/10.2307/3213451) beyond those points is difficult.
$X, Y, M, C$ all have probability distributions, so they would be @Stochastic class. All you said about $\\hat g$ is that it is a 'model', can you write out what that means exactly? Does $\\hat g$ have random components? It seems that you have defined the prior of $C, M$ already?
My bounty expired with no solutions or comments (and few views... people must really not like this question for some reason?). I officially am in protest of the rule that says I don't get my 50 points back. I got nothing in exchange for them and they were hard earned. :(
Here http://stats.stackexchange.com/a/105577/49130 you can find some additional information on how to draw a ROC curve
Thanks. Looks good. I was not aware of plyr. Migrating from SPSS, it seems - at first sight - to be an advanced version of Data/Aggregate.
You can't read this one without the alt text.  it said something like "But because of that we're totally breaking up"
I think this might need to be a wiki question :) ...
Also, if you want to get some quick idea, you can (1) run 20 t-tests, one for each measured variable, (2) standardize all 20 variables, do PCA in this 20-dimensinal space, take the two first PCs, and look at the scatter-plot of your 12 points. Neither procedure is a statistical test you are looking for, but it can give you some idea about whether there is some difference or not.
The solutions to $-q^\\prime(\\theta)=\\theta$ are $q(\\theta)=-\\theta^2/2 + C,$ not $C\\exp(-\\theta)$.  BTW, most people will not understand what you mean by "definitizing."  I still don't, [even after seeing it used in this context.](http://definitionsinsemantics.blogspot.com/2012/03/humpty-dumpty-principle-in-definitions.html)
@cardinal Hum, yes, I wrote that last comment too fast! Many thanks for all the comments.
As with what Mike said, turn on ODS, check your documentations, and look for a plots=ALL option somewhere in whatever procedure you're running.
Welcome to the site.

I don't understand your question.  Perhaps you are assuming some substantive field knowledge that the rest of us lack.  Can you explain it in terms we all would get? (pretend you are explaining it to someone you meet who asks "What do you do?")
The answer given here holds for *each* member of the original population, no matter who they were.  Summing over all members gives an *upper bound* for the probability that we are descended today from *some* person in the year 1300 of $360,000,000 / (2.66 \\times 10^{249}) \\ll 1$, which is obviously way wrong (unless alien clones were introduced along the way...).
+1, this is a good answer. Another couple of points to note is that the total N is growing in this scheme (which is generally associated w/ more power intuitively), and you have more group means, so you should get an increasingly clear picture of the b/t group variance as well.
If you say so. It's odd that the word "company" doesn't show up once in your answer. Sorry, I don't see the connection.
I'm trying to datathief the plot, so I need to understand exactly how that y axis is constructed.