tttjjj/elt_bench huggingface dataset:
Thanks chl. I think the problem is that many of the answers were too good - e.g. a search for "patient cluster analysis" in Google scholar returns 650,000 results. So rather than saying "here are some good directions to go in," I wanted a more discrete bibliography.
Thanks, Srikant! Sorry, I somehow missed your comment earlier. The upper cluster is really just a spike right at the ceiling - there's no variability there except for the long stretch of uniformity that links it to the lower distribution, which is basically as you describe.  It'll take me some time to parse out your answer (especially since I'm stuck in IE and can't see the LaTeX properly right now), but I really appreciate your dedication to this odd little question.
please see the edit in the question.. =)
...always remembering that you are reinventing the wheel with that: use it as a "proof-of-concept" that you can program, if they want to align short DNA fragments to a genome they would use BLAST.
+1.  This reminds me that I've been using your scripts to learn asymptote by jumping in and changing stuff around, trying different things.  Thanks again for that, *very* helpful to get started.
Extrapolating the model would require sampling the error term which would induce the desired randomness.
mh, to be honest: I am not good enough to get this thing working. What I was interested in is: Can I get the same result with my algorithm as the implemented version of the mixtools package. But from my point of view this seems to be asking for the moon. But I think you put effort into your answer, so I will accept it! Thanks!
$a,b$ are constants and known. $z_1,z_2$ are Gaussian noise with known mean and variance. And LMMSE is very straightforward method and I have all the information i.e., variances and co variances of $X$, $z_1$, $z_2$ to do LMMSE. There is no question about feasibility. Only question is about the two errors. Thanks.
@ssdecontrol Thanks for your interest in this question. The book is "Econometrics Methods with Applications in Business and Economics", Oxford Press.
Could you please edit the spacings in the result portion that things become a little more clear? Besides, you mentioned about your factor loadings, but I can't find them above!
Gee, it worked for me on my machine.  I created two vectors with your data, that formula, then `nls(form)` (without starting values.)  At this point I don't know what to suggest.
@Ret I copied wrong URL of paper. I linked to authors we page to get around pay wall issues but linked to the wrong paper by mistake.
Welcome to the site, @LX_Gould. We appreciate your help with answering questions here. You may want to read our [FAQ](http://stats.stackexchange.com/faq), which contains information on site policies & how to best help with answering questions.
Can you please suggest based on the edited question?
Thanks to @whuber for the correction. If you have the three principal radii  a, b, and c, (which you'd have, presumably, or at least could figure out, if you can plot it, since it's the distance from the center to where the ellipsoid crosses the principal axes), the volume of the ellipsoid will be $\\frac{4}{3}πabc$.
Seems to me one of the main differences between machine learning focused on prediction and statistics focused on inference is exactly what you say, B_Miner. Machine learning is more concerned with what works while traditional statistics pays special attention to assumptions. In both cases you need to be aware of assumptions/properties of your approaches then make an informed decision whether they matter or not. You might be fooling yourself in predictive modeling about whether your model works if you don't understand the assumptions/properties of the approach.
Use entering in blocks. Block1= A C (method enter); Block2= AC (method stepwise or forward). Hereby, you force main effects first. Then, if the prediction allows, the interaction will be added.
Thanks a lot for your answer
@whuber, I was also torn as to which to accept as "the" answer because both were very helpful.  I went with this one because it included an R package reference which I can use; the book reference in the other answer has been good reading and should not be discounted.  Thank you for offering the bounty which prompted these two good answers.
Wow, I never really appreciated how "smart" the `[.data.table` function can get ... I guess I didn't realize you didn't create an `.SD` object if you didn't really need it. Nice one!
@Ajay I edited the solution to give a simple worked example.
I left out the "not" in that sentence. Sorry about that!
What does it mean to 'make a shot' and 'defend it' in a way that allows these probabilities to be consistent?
Hm, I don't know. Have you looked into the reference provided in the help page for `spm()`? http://www.stat.tamu.edu/~carroll/semiregbook/ Unfortunately, the help page does not clarify whether the penalized splines it uses are restricted to be linear outside the training set (which I certainly could imagine to be the case).
A randomization (/permutation) test would be another possibility.
Allow me to quote directly word for word from pg 5 of Theory and Method of Survey Sampling. "We shall denote by $y_i$ the value of $y$ on the unit selected at the $i$th draw, $(i=1,2,...n)$.  Thus for example, if the sample is $S=\\{4,5,1\\}$, $y_1 = Y_4$, $y_2 = Y_5$, $y_3 = Y_1$.  Clearly, $y_i$ is a random variable whose possible values lie in the set $(Y_1, Y_2, ... Y_N)$."
+1 Well put, helpful, and to the point.
@NickSabbe When posting code, please state what software is being used. Thanks.
@Xi'an Do you have a reference on the problems of using  with BIC (in particular) or the others?  I'm unaware of any problems with how to define the no. of parameters for a Gaussian mixture model.
I think the answers to this [question](http://stats.stackexchange.com/questions/11009/including-the-interaction-but-not-the-main-effects-in-a-model) might be helpful.
Please [merge](http://stats.stackexchange.com/help/merging-accounts) this account w/ your [other account](http://stats.stackexchange.com/users/9587/savedbyjesus), & maintain only 1 account in the future. This has advantages for you: you will be able to follow what happens to your Qs & As (eg, you will be notified of comments), & will accumulate reputation & privileges for the good works you do.
To extend the answer slightly - the point about the limit is that it shows that the expectation is infinite for $\\beta = 1$, as $\\lim_{a \\to \\infty} \\int_0^a x^{1-\\beta}\\text{d}x = \\infty$.  (Note since $x^- = 0$, technically $x$ has an infinite mean rather than not having a mean at all.)
+1 This thorough discussion touches on many important aspects of the question.  Welcome to our site!
According to [Wikipedia](http://en.wikipedia.org/wiki/Margin_of_error), the margin of error for a particular statistic of interest is usually defined as the radius (or half the width) of the confidence interval for that statistic. Thus, the above calculation is correct. Probably, the mean is also reported for completeness.
Really?  Are you sure you want to do this?  I'd be concerned about overfitting and correlation between linear combinations of the predictors.  Not only that, with 100 predictors but only 10 observations, you have $p>n$ and linear regression isn't going to work at all.
Each treatment control difference is an RV, the covariances are zero, I am allowing for unequal variances in each of the groups.
I thought about this. That is the reason I put the weighted sum in the bracket. I guess MAPE ( mean absolute percentage error also a candidate for error metric.
I don't think that that is the problem here, as I don't see any mention of additional control variables. I suspect that Tano ran one multinomial logit model, than computed marginal effects, and than saw that a raw coefficient was positive, but the corresponding marginal effect was negative.
Coding is a must, but also knowing how to work with the OS not against it is important.  You must understand that sometimes splitting the work has extra costs associated with it, as accessing disks and networks carries additional costs.  You gotta understand different ways of blocking and waiting and doing interprocess communication.  I've seen great scientific code that would spent most of it's time waiting for some system calls to finish.  Befriend the sysadmin of your system, you can get a lot of help with optimization of yours systems by bringing them coffee ;)
Welcome to the site, @Jason. Please don't remove the HW tag, even if this was an in-class question, & not technically homework. The tag doesn't exist just to label questions that come from someone's actual HW, but to identify any "routine question from a textbook, course, or test used for a class or self-study". Your Q does come from a course, & it seems you are using this for self-study, in a sense, now. You can read more about this here: [should-we-tag-questions-that-smell-like-homework](http://meta.stats.stackexchange.com/questions/1172/) & on the [FAQ](http://stats.stackexchange.com/faq).
You could have also mentioned one notable inconvenient of the HPD interval : its lack of invariance under reparametrization: if $[a(x), b(x)]$ is the HPD credibility interval for $\\theta$ then, given a monotone function $f$, the interval $[f(a(x)), f(b(x))]$ is $\\textit{not}$ the HPD credibility interval for $f(\\theta)$.
This classical problem is covered by standard Bayesian textbooks, eg our own [Bayesian Core](http://www.amazon.com/gp/product/1441922865/ref=as_li_ss_tl?ie=UTF8&tag=chrprobboo-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=1441922865).
Out of curiosity, did you get the code from [this answer](http://stackoverflow.com/a/4909480/333599)? If so, then just forget about it: it is plain wrong!
@Zhubarb Are you referring to 10x k-folds (5 for instance)? If so, I believe the best practice would be to use the model with best model selection parameters that comes out of each of the 10x k-folds. For each k-fold you get an average RMSE or R2 in the case of regression so you get already a kind of aggregated model. But care should be paid in the presence of unstable models. Perhaps the "best" model is not the one that comes from CV but the second or third, depending on the train/test data sets. Difficult to know without trying I think.
@Vikas, have you looked at [Wikipedia](http://en.wikipedia.org/wiki/Hidden_markov_model), the references there and the tutorial by Rabiner (also mentioned in the R package)?
I don't know about that package, but if you use the TreeBagger class in Matlab (in the stats toolbox) then it uses the ClassRegTree class to build the trees which has options about what function it bases the split on but the Gini Index is the default option.
Thanks. This is a great answer and makes perfect sense. In my application, the assumption that $P(Y_i|X_i,D_i=1)=P(Y_i|X_i,D_i=0)$ is not realistic. But, it would just as good to add $D_i$ as a predictor and consider the distribution $P(Y_i|X_i,D_i)$. Using a similar derivation, I think you can show that 
if $P(Y_i=1|X_i,D_i,S_i=1)=P(Y_i=1|X_i,D_i,S_i=0)$, then you're fine. This is a reasonable assumption in my case. What do you think? BTW, would you happen to have any references that mention this problem? I'm not familiar with econometrics literature.
Thanks so the comment. Yes there is an outlier in my data. However the KS test doesn't make any difference after excluding the outlier.
@Luca, there was a similar question that i answered here: http://stats.stackexchange.com/questions/79547/metropolis-algorithm-what-is-the-target-distrbution-and-how-to-compose-it/79559#79559
+1 I have had good success using a slightly simplified version of this. Instead of Theil regression, just fit a line using the most extreme points to the right and left (3 at each end does well).  After subtracting that line and *closely* smoothing, the range of the smooth estimates $|A|$, the location of the maximum estimates $B$ (assuming $A\\gt 0$), and $C$ can be estimated from the proportion of smoothed values exceeding some sizable fraction of $A$, such as $1/4$.  You need to do this *twice*: once assuming $A\\gt 0$ and again (with suitable changes) assuming $A\\lt 0$.
N = 7468 and N = 7451, so not very unequal. I guess, in the worst case, I could remove 17 randomly chosen rows from the larger sample to make them equal. If there is a difference, I would like to be able to infer that it reflects a population difference. With the large sample size I have, p value is likely to be low, so I need to look at the effect size.
Do you have a 'ground truth'/answer key/etc? For example, if 900/1000 people vote the same way, is it possible that they're all wrong (e.g., you're showing them an optical illusion or perhaps they're all just dolts)?
Could you elaborate on how the distinction between mediators and moderators is relevant to whether a partial correlation can be greater than a zero-order correlation?
Thanks for your answer. Hypergeometric distributions tend to be used more frequently for gene list overlaps from what I have seen in literature. The question is what is the probability of getting 100 or more white balls in a sample of size 400 from an urn with 3000 white balls and 12000 black balls? I think I'm still perplexed by how to explain this to a bunch of biologists? The way they see it is 3000:12000 is a 1:5 chance of white:black. So in a sampling of 400, 80 should be white. So how come the probability of getting 100 or more is so much lower than 20% (1 in 5)?
While I've fixed pages before, here the task of making sure it's right would require a fair level of effort and constitute such a major change that I wouldn't want to do it without some other knowledgeable people agreeing to check it. However, the typical standard of conversation on that particular wikipedia talk page is so low that I wouldn't want to rely on the denizens of that talk page at all - any error I made could linger indefinitely. I find myself in a dilemma, then. I plan to post something to the talk page about this when I work out what to say, but I doubt it will do any good.
Hi Andy,  it could also well be that y has an effect on x.  So you think the VAR model is the best option?
"Accurate" is for you to define, since it's your question. But the usual ones at the links I referred to in my answer are common (the links mention comparing variance and MSE, I believe).
There are many approaches; one simple one would be accept-reject (generate an enclosing rectangle, and then generate uniform points inside it, keeping any points inside the polygon). You could overlay it with a sequence of strips, and do accept reject in each strip, with each strip being selected in proportion to its area (in effect generalizing the ziggurat method).  You could split it into triangles, find the area of each, and then at each iteration select between the  triangles in proportion to their area and then place a point uniformly in the selected triangle.
Centring in this case makes no substantive difference, as you can confirm algebraically & check using `predict`.
Amazing explanation!. Thank you, Your examples helped me in time
Just some comments, in case they're of any help: it looks like your responses must be non-negative and converge to $0$ (or close to it) at $0$, whereas these bands evidently are erected using a model of independent additive error.  That makes them unrealistic, especially at the left.  Moreover, the patterns of blue dots suggest the error has strong serial correlation, which also needs to be accounted for in constructing these bands.  Although you might not want to cope with this additional complexity in your data to do the fitting, it indicates the bands you have drawn aren't worth much.
@Micheal - the place where they will differ is that a fudicial interval must be based on a sufficient statistic, and condition on all ancillary quantities.  Neymans confidence interval does not require this property, and so are subject to the "95% confidence interval" having varying coverage for particular sub-classes of samples.
Dear Andy, 

I would first like to thank you for your response, it has been very helpful and very detailed.

since the fixed effects estimation is considered doing a longitudinal study and I've never done that before, I would like some advice on how to do this. Do you have a helpful tutorial / book to get me up and running?

Thanks in advance!
Did you find [this](https://www.kaggle.com/wiki/MultiClassLogLoss)?
This is great!  Thanks a ton. However, I think that your logic at the end there is assuming that this person will always get the joker. It's just another card in the deck that he may or may not get. Your explanation was good enough though, that I think I can figure it out from here!
This is immensely helpful. Thank you.
One possibily to obtain @Henry's table is the 'dnorm()' function in R ;-)
Did you have a look at http://stats.stackexchange.com/questions/7351/bayesian-statistics-tutorial?
Thanks for the answer. Somehow, I've overlooked the fact that I have two cores (my computer isn't very new). I should've looked at multicore a long time ago.
I did not downvote but you should see the other thread as to why this is not ok.
Although this is the only Paul Allison text I have read, I would like to say various other texts of his have come highly recommended for me to read. Even if you don't use SAS (I have in the past so I was familiar with the code, although when I read this book I had completed migrated to other software) it is a really excellent book on survival analysis.
I have feature sets such as (hue,shape,size) of 2 objects and I want to use FCM for classification. FCM needs membership function. What I do not follow is how to represent and construct a FCM model. The learning of the weights can be done using Genetic algoritm or Neural network. Will it be possible to illustrate with an example to show what will the inputs to the nodes be, membership function and how to classify using such graphical model?Thank you for continued replies.
Wow thanks for the detailed answer. The interpretations were particularly very helpful and got me to thinking as to whether the TI is really appropriate for this study. It might seem like a simpler approach would be sufficient. Again, thanks for the time you put into this! I really appreciate it.
Support of a distribution is the set on which the pdf or pmf is non-zero.
+1 For making some sense of the question!
I agree with Zach.  David Cassell and I wrote a paper about this, concentrating on SAS but not entirely. It is [Stopping Stepwise](http://www.nesug.org/proceedings/nesug07/sa/sa07.pdf).
I question your claim that standardizing predictors is "generally not recommended, especially when interactions are involved."  Neither Gelman and Hill, nor Raudenbush & Bryk mention this concern in their texts.  But when I have a chance I will look at the references you mention with interest.
@Bosley 's suggestion would be very constructive except for the fact that cell sample sizes are too small even for chi-square testing.  Sometimes Ns are so small that no hypothesis test is going to be appropriate in the sense of adequately powered. And you may want to look up 'statistical power.'
By definition, we are fixing the type I error rate in setting the p-value threshold. However, it seems like the difference between "statistical" and "practical" significance is the issue here. When the sample size is able to detect differences much finer than the expected effect size, a difference that is correctly statistically distinct is not practically meaningful (and from the perspective of the "end-user" this is effectively a "false positive" even if it's not a statistical one). However, as you say this starts to get outside the realm of statistics.
I'm not clear on why a lack of correlation is needed for ANCOVA to work.  Great point about dichotomization; this will likely ruin the analysis.
The answers in this thread may be relevant: [Learning to map vectors to vectors](http://stats.stackexchange.com/questions/111515/learning-to-map-vectors-to-vectors/112031#112031)
Thanks a lot. I should also mention that I want use validation function as fitness/goal function for particle swarm clustering.
(+1) I located and installed this package less than two months ago.  For more sophisticated work, consider using censored regression models.
+1 as a general maxim, I would suggest that the first task is to have a clear idea of the problem to be solved, and the second is to approach the problem with the method that gives the most direct answer to the question that is actually being asked.  This is a slight generalisation of Vapnik's maxim, but only very slight!
Tell us this: (1) Is it possible that $f_{X,Y}(1,0) \\ne 0$?  (Obviously yes.)  (2) Is it possible that $f_{X,Y|X}(1,0|X=0) \\ne 0$?
Testing what hypothesis? Equality of $p$ parameter? Something else?
This doesn't answer my question. I asked about the scenario of $\\Pi$ being full rank while at the same time both variables being $I(1)$ (hence apparent contradiction).
The 'compression scheme' point is from the fotnote in Lawrence's paper.
We're happy to take R questions here. See http://meta.stats.stackexchange.com/questions/252/should-we-allow-more-computing-questions
Ok, got it, thanks! So my data consist of a sample of around 7000 observations. With such a large sample size and when considering the above relationship between sample size and Hausman statistic, is it obvious that the Hausman statistic is significant?

Because my Hausman statistic is significant, but the differences between the RE and FE estimators are not substantially (only after the 3rd digit after the comma).


Thanks
I have added some details to the question above...
Seems you want [tag:multiple-regression].
Could you clarify what the "this" is that you want to read up on?  CART?  Re-expressing variables?  Logarithms?  Skewed datasets?  Things not working out?  Analyses not making sense?
A key word to look for within the xkcd site is Fermi.
That is a one clear and beautiful answer!
Simulation (10,000 trials with normally distributed main effects and errors) indicates [21, 58] is a symmetric two-sided 95% CI for the mean.
thanks for your answer but the functions you mentioned seems to take only one criteria like either sales or profit. what I need is forecasting/prediction based on relationship between two criteria like sales and profit. How to do that? Correct me if I am going wrong.
@ Ben Bolker: Thank you so much for your fast answer. I do not know if it makes sense or not (still trying to think this through), the only thing I can say so far is that `mod.glmer4= glmer(cbind(marked,unmarked) ~ area + (1 | site) + offset(sampl_effort), family=binomial(link="cloglog"), data=data)` gives me the Error `Error in (function (fr, X, reTrms, family, nAGQ = 1L, verbose = 0L, control = glmerControl(),  : 
  Downdated VtV is not positive definite`
@Stat It's an $x$, sorry about that.
I agree with @Dimitriy, especially about the necessity of domain-specific knowledge as opposed to some statistical criterion for clustering.  The clustering is probably going to be somewhat arbitrary.  There are few rules to go by.  The very choice of hierarchical cluster analysis vs. some other type, and the options one uses within that method, are going to be somewhat arbitrary.  And to tell you the truth, I wasn't entirely convinced by your rationale for avoiding regression.  If you explore you'll find a number of ways to make regression results visually interesting.
@Glen_b - this is true only if you are interested in "A" vs "not A".  If you're interested in all three, then surely you should split the "not A" region into "B and not A" vs "C and not A".  The "CI" is a 2-D region here I would have thought.  Admittedly, the question is not clear on this.
I remember having a similar problem about a year ago when I played around with the `fda` package.  I was writing up a response that involved getting predictions manually, but a big chunk of it was lost due to not saving it.  If someone else doesn't beat me to it, I should have a solution for you in a couple days.
Thank you so much. I do appreciate your time and help. I haven't used R before. Could you please let me know if I can estimate the missing values using Minitab?