tttjjj/elt_bench huggingface dataset:
@Prasenjit: Always. It's just that sometimes we are made to take shortcuts. :)  By the way you should *at least* have noticed that row 1 cannot possibly have been selected if seat 15 is.
I voted to migrate to CV; I think it stands a better chance of getting answered there than here.
Thank you, that is exactly the sort of thing I was looking for. But is there a mistake in the formula? The dimensions don't seem to match in $a^T(\\mathbf{X}^T\\mathbf{X})^{-1}a$. Should $\\mathbf{X}$ be the $n \\times (p+1)$ matrix having ones in the first column?
Indeed. And one way to see this is to note that probabilities are always between 0 and 1, while odds can be anything.
I agree that it depends entirely on what sort of weights you have. Decent statistical packages distinguish between several (sampling weights, frequency weights, inverse-variance weights...) and treat them differently.
Thank you, the reference to the permutation distribution is helpful!
@Comp_Warrior You're welcome!
@user9171 I am not an expert but I would feel confortable with doing that. To my opinion, the main question is more why building a kind of "full" model for independent variables? but I can't answer this question without understanding the context. Hope it helps.
And don't forget SDeffect = sqrt(2 * VARpooled).  It is NOT 2 * SDpooled.  It is sqrt(2) * SDpooled.
Could you possibly share your syntax (do file) so we can exercise your solution?
Thanks. I will pose another question as I am not a stats expert and am not sure my question was clear.
I made some edits to the problem. Also I have data for the previous year and the lagged variables for the Z's could be used as instruments.
@BlueRaja-DannyPflughoeft: Pay attention to what I wrote: either there is non-determinism or there is non-local information and you can not have complete knowledge. There is no point in bringing the interpretation of quantum mechanics into the discussion because the situation is independent of which interpretation you choose.
If you're using software like WinBUGS, you can incorporate this quite easily. For example, you could specify a uniform prior on the range:
`age ~ dunif(age.lower,age.upper)` (where the ages are known, just set `age.lower = age.upper`).
Dividing by (k-1) is when you compare multiple algorithms against a control method. But this a each versus each, so NxN. The dividing part I can understand, but the relation to the Studentized range distribution is beyond my understanding.
Thanks for your answer, it's really helpful!
Excellent.  Thanks.  Is it in the documentation for resettest that it scales y?  I did not see that when I looked.
I second this answer and would just like to add another great reference on this topic:  Singer's Applied Longitudinal Data Analysis text <http://gseacademic.harvard.edu/alda/>.  Though it is specific to longitudinal analysis, it gives a nice overview of MLM in general.  I also found Snidjers and Bosker's Multilevel Analysis good and readable <http://stat.gamma.rug.nl/multilevel.htm >.  John Fox also provides a nice intro to these models in R here <http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-mixed-models.pdf>.
(+1) (Inside joke: Well that's one way to handle a flag...) :)
After trying this, it doesn't work as well as I imagined. I want the Dirichlet to be centered on my current categorical i.e. $p_i = \\alpha_i / \\sum_{j=1}^k \\alpha_j$ so $\\alpha_i = p_i*\\sum_{j=1}^k \\alpha_j$. But if $p_i = 0$ then $\\alpha_i = 0$, yet there is the restriction that $\\alpha_i > 0$. So I tried $\\alpha_i + h$ where $h > 0$ is a stepsize (or inverse stepsize since bigger $h$ makes a more peaked distribution). The problem is that large $h$ (small steps) never seems to explore the boundary and small $h$ gives steps that are too large. Is there a way around this?
Err, what happened to reading the response from the same Q you asked on SO? http://stackoverflow.com/q/5686940/429846 Use `prcomp()` instead! The help page `?princomp` even tells you this.
Can you please indicate *exactly* which models you fitted and tested in each case.
Do you need to generate according to a distribution whose mean is $100$ or do you need the mean of the generated values to equal $100$?  (The two are different!)  I also ask for similar clarification concerning the proportion over $110$.
Clamping the data as recommended here will bias the slopes in a regression.
I asked a similar question that you might find useful: http://stats.stackexchange.com/questions/3199/clustering-of-a-matrix-homogeneity-measurement
First of all, it's not my recommendation; I have referred the questioner to a peer reviewed paper, and have not given an answer myself. Second, AIC gives a measure of significance, not the accuracy of prediction.
(1) Please give a link so we know what you're talking about and (2) Please make sure you have spelled the name correctly.
Can you say how you are selecting your confounders? You seem to be focused on a causal question (rather than a purely predictive question) because you are asking about a main "effect" and confounders. If so, you should include potential confounders based on theory, not statistical attributes. If "all potential" confounders is a reasonable number, well thought out, and justified by prior knowledge, then leave them in...if "all potential" confounders is just a long list of covariates in your dataset that haven't been given that much thought, then you have a different problem.
"Cluster Analysis" means?
@aL3xa: I've added the plotting code I used. Hope this helps.
Could you point me to some good references?
+1 Thank you very much, nico. Let me see if I got you right: at the end, this is basically like setting some global threshold and defining any point with value < threshold as a part of a valley. The sampling etc. is just used to get some meaningful measure (quantile) to set the threshold. Why can't we use a single threshold for the entire points, I mean, if we did enough simulations we would get straight (read and yellow) lines. Also, correct me if I'm mistaken, but this does not take into account the surrounding environment but examines the absolute value of each point.
charles, you just attempted an edit from a different account than the one you originally posted from. Please see the help on [merging the two accounts](http://stats.stackexchange.com/help/merging-accounts)
This is too broad to be answerable as currently written. If you can write a focused, concrete question, we can help you. You may want to read [our materials](http://stats.stackexchange.com/help/asking) regarding asking questions on CV.
@Rachel I don't think that's right. If they were measuring how outsiders perceive comments on SO, they'd have needed quite a larger sample set than 20 people.
To get a good answer, please provide specific descriptions of what you mean by "window" and by "correlation."  You should also specify the sense in which "takes into account" is some modification of the usual definitions of correlation ([Pearson](http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient) and [Spearman](http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient), among others).
@ Glen_b: Hi thank's for your answer, I 'm sorry but I don't see how to apply mixing CLT in my context, moreover I realize I should have added that all my Bernoulli have the same parameter. For your last point could please provide some reference for the continuity correction ? Best regards
@mpiktas I have never seen this notation for variance.  My first reaction was that $D$ is a differential operator!   The standard notations are mnemonic, such as $Var[X]$.
Thanks Dimitriy.  The data is from an incredibly dumb sentiment analysis tool.  I basically took 50 user comments, subjectively rated them on positive or negative sentiment, then ran them through the sentiment analyzer.
Let me make sure if I understand it correctly. In your suggestion of uniform sampling along the vertical line to $T1$ for each sample we are going to find the weight by multiplying normal probability densities for sample $\\varepsilon_1$ and $\\varepsilon_2$. Thus each sample will have a different weight. I think it is a good idea, but I would prefer to have equally weighted samples, since I look at the envelope of the distribution. It becomes a little complicated to back out the envelope from the weighted samples.
1. The data does not need to be normal, the residuals need to be. 2. What is a "good enough" $R^2$ varies by field. 3. Both these questions have been asked before (many times).
Of possible interest: [Using lmer for prediction](http://stats.stackexchange.com/q/4187/930). And it's probably worth to check what Doug Bates has to say in his draft on [lme4: Mixed-effects Modeling with R](http://lme4.r-forge.r-project.org/) (I didn't check it, though), or maybe [LMERConvenienceFunctions](http://cran.r-project.org/web/packages/LMERConvenienceFunctions/) on CRAN.
Given the extra detail here it seems that overall counts ignore the detail you have on individuals. In that case a transition probability matrix and Markov chain ideas seem relevant. I can't advise on how much of that you can do in SPSS.
@Penguin_Knight; could anyone explain entering the mean of difference in G*Power3...i could say Mean of difference=10 and SD of difference=10...will that mean 10secs or 10mins?
The G2 estimator from `e1071` gives the 'standard' bias correction; see http://cran.r-project.org/web/packages/e1071/e1071.pdf . Using this estimator instead of g2, implemented by the `moments` package, had little effect, as I noted in the Q. The dependence on the eighth moment would indeed imply the sample size is too small here.
At the moment, I don't know of a simple way to do a GNLMM (generalized non-linear mixed model) within R: I would suggest looking into AD Model Builder (http://admb-project.org)
Are you suggesting to fit m separate models, sum the log likelihood from each L1= SUM(LL_i, i from 1 to m) and then proceed with the likelihood? Also, is L0 a model with the categorical predictor in question included (with m-1 dummy variables for example)?
I noticed that the SPSS output linked by @user1357015 notes that Lilliefors Significance Correction is used.  Maybe that is the difference?
@whuber when I think about the SD I think it should be on the same scale as the mean (164) so 4103 isn't intuitive to me. I'm a beginner so I am probably making some basic error, so thank you for your time and I'm sorry if I'm wasting it, but I am very grateful.
Depends on the model, so tell us if you have a specific one in mind rather than wanting to know about general approaches you might take. The user guide for NLINs gives [an example](http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_nlin_sect005.htm) for the Michaelis-Menten kinetic model.
Well, look at distance and density based clustering algorithms; as you have a similarity-matrix-like thing.
Thanks Kevin. I think it is very interesting that such a well-articulated rule of thumb exists in any experimental field. Do you have a citation or two to papers in plant breeding that discuss this rule of thumb? I'd be interested to see where it comes from.
This proof seems to be for diagonal covariance matriz only. With all this work It is really easy to extend it to any covariance matrix $D$. You have to use the cholesky descomposition to find $D = UU'$ where $U$ is an upper trianglular matrix. Then if  $U^{-1}=S$, you have that $D^{-1}=SS'$ and the transformation you need to apply to the model is $SZ=(SZ)\\beta + S\\varepsilon$. Then all proof goes the the same way
Also see [Free statistical textbooks](http://stats.stackexchange.com/q/170/2981) for other online resources for studying statistics.
I think so, yes, but you might want to provide more detail on how you're handling the clustering.
This discusses a lot of the books listed. It is a very nice link for the OP. It doesn't mention anything about books specifically directed to biologists though.
What is "anomalous" to you? This is very domain specific.
@MichaelChernick, I don't understand what the controversy is. There's clearly no statistical question here. It seems the opportunities you're taking to bash the moderators here are becoming less and less justified. The "proper place" for this question is probably an internet search engine or, *maybe*, http://sports.stackexchange.com/.
If you want the classical *theory*, it's hard to go wrong with Brockwell and Davis.
Check whether odd.dummy (or the relevant variable in your data set) perfectly predicts membership or non-membership in any of the outcome categories.  If it does, you have perfect separation and the mlogit command is choking when it tries to estimate the standard errors.
@Peter: yes, you're right. It can be *semi-*definite, not just definite.  Edited accordingly.
For cov(Xt,Xs) t≠s, the 2 variables are independant, so it's 0
. For var(Xt)=E[(X-E[X])²], X-E[X]=$\\epsilon_t$ and $E[\\epsilon_t^2]=1$ from the chi-squared law with 1 degree of freedom?
The first argument describes a property of any location family of distributions whose pdfs have monotonically decreasing right tails.  The second argument is not correct unless you add assumptions (such as the two distributions have the same shape and location).
So f0 is known and fixed.  But is f known or from a parametric family or are doing this in a nonparametric framework with all you know about f coming from your sample?  I think it makes a difference when attempting an answer.
Please edit your post to include the `self-study` tag: self-study questions are treated differently. See http://stats.stackexchange.com/tags/self-study/info
He should have measured the standard deviation.
Also here http://sifaka.cs.uiuc.edu/course/498cxz05f/hmm.pdf (Section 7) says that "Like in all other cases of applying an EM algorithm, we will start with some random guess of the parameter values".
@gung Yes, this post does ask for code. But it also implicitly raises a question of how one could correctly handle mis-coded data in which true zeros are confounded with missing values. That question could only be answered here, not on SO, and a good answer would likely be much more useful than a pedestrian answer that points the OP to some blackbox code (which, if employed, would likely give bad results).
With `expand.grid()` you're testing the same pairs of variables twice as well as each variable with itself; so you will need to filter out the result.
Actually, (assuming that the sample space for $X$ is $\\mathbb{R}$) we have $E(g(X))=\\int_{-\\infty}^\\infty\\int_{-\\infty}^\\infty G(t)exp(itX)p(x)dtdx$. You have the right idea about how to show this, but you need to think about whether or not you can change the order of integration.
Thanks Joel for this post, however when applying this code, it gives me very low accuracy than when using the two classes mehtods. Is this true? I am currently comparing both methods to choose which method to use; especially in my case having no data on absences (the other class) and I am currently comparing the 1 class method with 2 class methods (with pseudo-absences). Any help on this regard would be appreciated, Regards.
Hi Torvon. I have never came by a solution. I guess permutation tests will be the safest route, but I never got the time to sit and make it work.  If you do - please come again to post your answer.  Cheers, T
Ok, thanks for your answer and your help!
To be honest, I'd probably fit a GLM for this, possibly with a gamma or perhaps inverse-gaussian distribution family, depending on the skewness and the way the variance relates to the mean. The null hypothesis you have there would be simple enough in that framework. You could do a permutation test, or there are still other alternatives.
Do you intend that X and Y be independent? [Your variance term looks wrong](http://en.wikipedia.org/wiki/Variance#Product_of_independent_variables). If they're more generally dependent, it looks even more wrong. But yes, Yes, if the coefficients of variation are small, the product of two independent normals is fairly normal.
This is a good question, biofreezer.  I just wanted to remark that there are close analogies to other methods of work.  My favorite is, pen & paper EDA is to modern stats as hand tools are to modern woodworking.  ("Modern" woodworking employs many power tools like tablesaws and routers that enable even beginners to turn out acceptable results in much less time.  However, these tools also account for thousands of missing digits and limbs every year.  People who learn to use hand tools generally learn to work better and more efficiently even when they employ power tools.)
This one reminds me of the recent bailout in the States, where they just made up 700 billion number - they said they just wanted a really large number. :)
Perhaps I don't see why this is such a problem. Hypothesis testing a small sample size using a normal distribution, sure, but using a more conservative/nonparametric test, is this so bad?
Is the example at http://stats.stackexchange.com/q/5206/919 helpful?
Regardless of your context, there is nothing to compare here. Whoever designed the questionnaire built a dead end. The only possibility is that some people said No to Q1 but answered Q2 any way. If you are an optimist, they provide data for a comparison. If you are a pessimist, nothing they said is credible as they evidently didn't understand the instructions.
Chi-square tests generally work well whenever _expected_ frequencies all exceed about 1; the ancient advice about a threshold of 5 still lingers on, despite being rebutted many times. As for your "missing" cell, if there are no relevant items the observed frequency must be entered as 0. Your problem must be converted to a problem about frequencies, not percents. I can't see that Kolmogorov-Smirnov applies to arbitrary categorical variables, as the cumulative distribution is then not uniquely defined.
@guy - could you describe one or two such "catastrophes" and tell how they were caused by dropping insignificant terms?  Dropping such terms is extremely common and I'm having trouble imagining how doing so is likely to be so very harmful.  I understand that low power may prompt someone to simplify to the point of neglecting a potentially important effect, but  if there is already some independent evidence for the lack of such effects,....
Define resilient. If the samples are iid and you missing only small portion of the data (and the missing values are distributed randomly), then linear regression can be thought as resilient to missing data.
Here is a similar related question about modelling proportions on this site, http://stats.stackexchange.com/q/24187/1036. I suspect it would be good to search Sociological journals as well for literature, the same diversity measures are of frequent interest to demographers interested in segregation.
Sounds like a problem that calls simply for trial and error.  I don't know of any named method that would specifically accomplish this task.  Checking for nonlinear relationships ought to give you better chances.
why would a *cost* function be linear? That would imply erring hugely one way would incur a negative cost...
Welcome to the site, @Kat. This is a really nice answer (+1), I hope we'll see more. Since you're new here, you might like to read our [FAQ](http://stats.stackexchange.com/faq) to help learn your way around.
I'm referring to a threshold for the scores in the 3rd and 6th columns of the table. A classifier means picking a number and calling all scores higher than that "positive" and all lower scores "negative". Numbers 7 and 8 are truly negative, so if you lower the threshold to class them as "positive" you'll get two fewer correct decisions. A threshold of .5 gives 5 true positive and 5 true negative for accuracy 10/20=.5, not great.
It's not that the hierarchical model is hopeless; rather it appears that your sampling scheme needs some tweaking. Have you tried different proposal distributions for the metropolis step? What is the acceptance rate for these proposals? It should be around 15-30%. If it's not then it needs tuning. Also, i'd be very surprised if you needed anything like 500k iterations to determine if it's converging.
+1, this is one of the best (& least partisan) descriptions I've seen.
I believe your question is similar to this one: http://stats.stackexchange.com/questions/19103/how-to-statistically-compare-two-time-series
What you are doing, in essence, is discrete time survival analysis. [Here](http://www.ats.ucla.edu/stat/mplus/seminars/DiscreteTimeSurvival/default.htm) is some information from UCLA's stats help site. I haven't seen these slideshows, but the UCLA site is of uniformly high-quality.
No problem, however, if it does answer your question / provide what you need, you can accept it by clicking the check mark to the left under the voting mechanism.
See here : http://scikit-learn.org/stable/modules/decomposition.html#truncated-singular-value-decomposition-and-latent-semantic-analysis
@genneth - you have put additional information in these comments which makes my answer not really applicable - you should update your question (and "unaccept" my answer) to indicate where your uncertainty is, and what you want to use the interval for.
May we assume that you meant ECM, i.e. error corection model, not esm?
I know this doesn't answer your question (sorry), but have you tried a univariate time series? How far into the future do you want to predict?
Uniform priors are not necesarily un-informative and vice versa, it depends on the nature of the problem.  I think the idea of a minimally informative prior is often more realistic, using MAXENT and transformation groups to decide on a prior distribution that conveys the minimum amount of information consisten with known constraints (e.g. that the prior should be invariant to scaling).  For many problems, improper priors work well, so there isn't necessarily a need for there to be a well defined minimum and maximum.
There is a statistical problem in understanding what the specific model needed is.
Ah yes, even reviewers are human beings. The difference you found might suggest the source of the problem, but in general the number of failed convergence runs does not sound like a very appealing criterion to decide upon the imputation method.
The third paragraph is about the distributions discussed in the book. See edit.
And what if you are working at larger scale data (but not big data) of 10000<N<1000000? At that point splitting seems reasonable.  This fits many, but not all, situations I encounter.
@Shane It's not a time series, these are coordinate along the genome (chromosome).
I'm not sure about how to do this in `aov`. (I never use `aov` as I prefer to do all linear models in `lm`.) Is there some reason why it must be done in `aov`? You now know of two other ways to get the answer you desire...
You may be aiming for too high an $R^2$, the data has noise in it.

Also, you say "time and complexity" but your graph has "lines of code" and "complexity"
Thank you Peter, your comment is very helpful
Thank you for your response. Sorry for not being as clear as I could have. What I meant was that, there exists a set of items. These items are all indicative of a latent variable (Having a correct conception). A subset of these items are also indicative of another latent variable (A particular misconception). This subset of items would cross load onto both latent variables mentioned. I hope this makes it a bit more clear.
@MichaelChernick this question is independent of my previous post although that doesn't discount the possibility of overlap in answers.
@downvoters: Could you please leave a comment as to how the question can be improved? I am not sure if downvoting without leaving a comment is helping the OP.

@Elaine I think you should fix the qn to give the correct value for R^2 and any other edits that you feel would help others understand your question.
@pertsevin, if it's a pure Gibbs implementation, shouldn't it scale pretty well?
@GaëlLaurans Ok.
you do not appear to have attempted to answer the question yourself or give any indication as to where you're struggling.  Please add more info