tttjjj/elt_bench huggingface dataset:
I second @Frank's remark and would add that the very first thing one suspects when residuals appear non-normal is that one or both of the first two expectation assumptions is wrong: that is, there may be lack of fit ($E[\\varepsilon_i|x_i] \\ne 0$ for some $x_i$) or heteroscedasticity ($E[\\varepsilon_i^2|x_i]$ varies with $x_i$).
Scortchi that question doesnt mention anything about distribution of a variable before/after being split by group
Thank you for this, it really helped me understand!
(+1) Actually, treating ethnicity as a random effect *would* let you see how each ethnicity deviates from the overall level (by looking at the level-2 residuals, e.g. by using the `ranef()` command in R), but as you point out, you get no significance test.
If anything, your edit made it less clear, not more. You no longer really seem to discuss the distribution at all, and you removed the only part that gave me any real clue what you were doing. Now I'm even less certain I understand what you're asking. Are $Y_i$'s count random variables? Is the distribution of $\\bf{Y}$ essentially multinomial,  specifically a [mixture](http://en.wikipedia.org/wiki/Mixture_distribution) of $X$-distributions? I am no longer confident that this is what you mean. There remain several other questions or requests outstanding that would help.
For level you want one with its rejection rate as close as possible to the specified significance level (or possibly 'closest without going over', depending on what you're trying to achieve), while for power you want the *highest possible* rejection rate, given the same actual (rather than nominal) significance level. You should change your question to incorporate the new information. Before I go into much detail, I ask again -- is this for some subject?
+1. That book can't be recommended enough...
Is there a theory that says what predictors you should include? If you have a lot of variables that you have measured, and no theory, I would recommend holding out a set of observations so you can test your model on data that was not used to create it. It is not correct to test and validate a model on the same data.
When you don't center the samples (subtract the means from the columns), the first eigenvector usually is *not* the vector of means.
[CLT](http://en.wikipedia.org/wiki/Central_limit_theorem): large sample sizes make the means normally distributed.
I looked at rolando's suggestion and went to that link. I downloaded Eureqa and it seems to be the exact algorithm that is binding the best fit to my data. It seems extremely powerful in the sense that it does not bog your computer down and try to do 10000 calculations per second. The results so far are great.. it basically finds the g(E(Y)) that you suggested. Thanks Momo and Rolando.
@Macro, a stock reason for declining a flag (as supplied by the SE team) is "flags should not be used to indicate technical inaccuracies or an altogether wrong answer."  For more info please read the SE meta posts at http://meta.stackexchange.com/questions/80269/to-flag-or-not-to-flag-and-a-request-for-auditing/80273#80273 ("If an answer is simply low quality, please just downvote it instead of flagging") and http://meta.stackexchange.com/questions/10848/what-are-acceptable-reasons-for-flagging-as-requires-moderator-attention.  If you have additional questions, let's start a meta post.
Thanks for the response. Lagged differences is an interesting suggestion. Would using mean(diff(x)) be more telling of a "trend" than sum(diff(x))?
You are right, this is exactly what bothers me. How to select a sample of _non-positive_ samples that leads to a good decision boundary? Is doing a random selection the best I can do?
+1 @mpiktas, You're right, clustering may be a more efficient model for the heteroskedasticity and autocorrelation structure given in panel data than HAC standard errors. This approach is similar to the block bootstrap as well. They are all discussed in the Bertrand et al. reference that I give.
@Alexis All participants take part in all 4 conditions, I updated my description to make it clearer.
@probabilityislogic Thanks! I added some more detail to the question to help clear things up. In this case, the upper and lowerbounds are discarded from the routine.
So, in that sense, it's the same interpretation as for any normal OLS estimation?
I have tried the graph with boxes, as per your supplied syntax, with great results! The only drawback is that the data points are a bit clustered, so that even with the transparency a lot of occlusion is going on. 

If you have the time to show me the syntax for adding error bars instead of boxes, and perhaps even to try ellipses instead, I would greatly appreciate it. It would be great to have a look at what those two graphing options would look like with the data I have. They might make the clustered data points easier to distinguish.
I added a link to the actual problem set from which this problem is taken (I taught that course for more than 35 years, and recognized the problem....)
How is your data collected? Is it errors per trial?
Nice answer, though from an aesthetic standpoint (_not_ a practical one) a probit regression model might be even nicer. The probit model is equivalent to starting with a bivariate Gaussian R.V. and thresholding one of them (to zero or 1). Really it just involves substituting the Gaussian cumulative normal ("probit") function for the logit used in logistic regression. Practically this should give the same performance (and computationally it's slower since normcdf is expensive to evaluate (1+e^x)^-1), but it's nice to think about a Gaussian with one of the variables censored ("rounded").
I've read in other areas that cronbach's alpha may not be the best measure of internal consistency for our kind of questionnaire, but i'm afraid i'm out of my depth when trying to understand exactly how to evaluate the alternatives.  The ones I've seen cited include guttman's lambda 6, which is calculated by the function alpha in r package 'psych'; Revelle and Zinbarg discuss "Coefficients alpha, beta, omega and the glb" in their article of the same name (2008) and seem to hold that cronbach's alpha is a poor measure of reliability.
great!  thank you so much
Following up on that thought, I found: [Bien et al (2012), "A lasso for hierarchical testing of interactions"](https://statistics.stanford.edu/~tibs/ftp/hiertest.pdf), which uses two different $\\lambda$s just like you said.
Generating points uniformly on a sphere is described at http://stats.stackexchange.com/questions/7977/how-to-generate-uniformly-distributed-points-on-the-surface-of-the-3-d-unit-sphe; the method generalizes directly to all other dimensions. But the title of this question appears to have little or nothing to do with the question itself, which asks for a uniform distribution on the *circle* together with additive "Gaussian noise" in the radial direction. I have therefore edited the title to change "hypersphere" to "circle."
There are many ways missing values can be handled in decision trees, but the `randomForest` package in R only has the imputation method you described.  If you want to stay in a similar environment, `gbm` has a somewhat smoother method of handling missing values in new data (it's not perfect, but it is useful).
This is a lot of information and will take me some time for me to digest, but I'm glad to hear the tabulated percentages and logistic regressions will essentially give me the same results.  I ran the aggregate function on two of the predictors and it finished immediately...the results for the logistic fit has been running for a number of minutes and hasn't finished yet.  I'll continue to tweak things, but I may end up just using the tabulated percentages.  Thank you!
Concerning the 2 equations that you have up there. If the regression line (as a linear function) is of the form $y = a +k x$ then the linear model is $E[Y|X] = a+ k X$ and using error terms this is $Y = a + k X + \\epsilon$ where $\\epsilon$ is an error term with zero expectation. This is the sense in which the two equations fit together.
Unfortunately, your question cannot motivate any decent answer as it stands. You might consider adding information about your design (crossed effects--random or not--vs. nested ones), what you qualify as multivariate (is this on the response variable(s) or the number of predictors), etc.
Just a point of clarification: Is $\\mathcal X$ assumed known?
You might find http://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models useful
I think this maybe what I am looking for. I am going to try apply some of this to my data and report back.
I came across some more recent and extensive ressources on testing for exponentiality. 1) An article:
A Henze, N. and Meintanis, S.G. (2005): 
'Recent and classical tests for exponentiality:
a partial review with comparisons'. Metrika, vol. 61, pp. 29–45. 2) A CRAN R package named 'exptest' implementing the tests of the mentioned article.
Thank you! one question regarding the offset. so the outcome variable is indeed rate (crude death rate), the numerator is death count, but the denominator is population. I actually want to model the death COUNT (not the rate), but am only using the crude rate precisely because it has been divided by population (i.e. it's been normalized by population),in other words, I actually want them to be weighted differently, do you mean even if the crude rate is already normalized I still need to add the offset to the model? thanks for your help!
+1 This demonstration (specific to the logistic regression model, without trying to generalize to all the GLMs) is also given in [Maddala (1983) Limited Dependent and Qualitative Variables in Econometrics](http://amzn.to/nCl2kj), pp. 25-26.
This is a really good book. But I don't think there is any solution available. There is another similar book "The Basics of S-PLUS" by Andreas Krause Melvin Olson that can be considered as an "introduction to R". It has many exercises with detailed solutions.
I don't think that is quite correct. Feature selection before   cross-validation leads to biased classification performance: http://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation?rq=1
Bonus point for finding this information on wikipedia. Bizarelly enough only one variable functions are considered.
(+1) Good suggestions. Even the "theoretical version" is an easy and enjoyable read. :)
Did you try links from http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples?
Yes, there's a [prim](http://cran.r-project.org/web/packages/prim/) package, with an R vignette: [Using prim for bump hunting](http://cran.r-project.org/web/packages/prim/vignettes/prim-2d.pdf). It's not obvious to me how it will work in this case, though.
This doesn't really answer the question but you could just use an independence GEE and then this is no longer an issue.
@ttnphns: It is not an R question, I just used R to exemplify the problem as that is my stat-language of choice. The question is regarding how to interpret the V matrix of the SVD. As I'm not an expert and multiplying matrices I don't feel comfortable in just blatantly saying that prof. Leek was wrong in his notes. His interpretation of the SVD is really interesting and could be extremely useful - given that the logic isn't faulty. This could be perhaps a math question but I figured I start here as those answers often go beyond my mathematical skills...
(To clarify, the sign of the coefficients (weights) depends on whether you measured your angles clockwise or anticlockwise from north.)
Ben Lambert - Undergraduate and graduate Econometrics - youtube Burkey Academy - http://www.burkeyacademy.com/home/statistics-econometrics Econometrics by Mark Thomas - YouTube
Are you referring to the clustering algorithm which SPSS calls two-step? http://www.spss.ch/upload/1122644952_The%20SPSS%20TwoStep%20Cluster%20Component.pdf
@Jonathan Because this is not a question about samples or estimation, one can legitimately take the view that $\\mu$ and $\\sigma^2$ are the *true* mean and variance of the empirical distribution of a batch of data, thereby justifying the conventional use of greek letters rather than latin letters to refer to them.
is turing the site into a discussion forum a revolution ;) ?
Strictly speaking, these are not quite fixed effects in a panel data setting. They're really just dummy variables for the states in a linear probability model and logged y regression applied to panel data. Other binary variables would have the same intercept shift interpretation.
Re the wording of your question: under a frequentist interpretation of binomial sampling, you can't say that the population parameter $p$ has that distribution, or indeed any other distribution; it's a fixed (but unknown) value, not a probabilistic outcome. Instead, one thing you can sensibly attempt to do is [construct a quantity](http://en.wikipedia.org/wiki/Pivotal_quantity), $q=Q(p,\\hat p)$ whose distribution doesn't depend on $p$; that allows you to make probability statements about $q$, and hence to obtain an interval for $p$ with the usual characteristics of a confidence interval.
I did not notice it at the time but it seems that this is a duplicate of http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs
I'm not really up on SVM theory, but the 'flatness' in the kernel-machines discussion you link to seems to amount to: 'has small *second* derivative' (think of the typical motivation for spline smoothing models).
Can you make vary A?
Thank you for giving the name of the JIRA diagram class, as well as pointing out the dangers of it. I'm now using two views on this data, both the Cumulative Flow Diagram, and a bar chart of totals.
What makes it multinomial logit is defined by the response, not the predictors ($x$'s). It's possible the relationship of the x with the linear predictor with the isn't linear, but you can deal with that by a transformation of $x$.
For more information, see http://stats.stackexchange.com/search?q=%2Bstandard+%2Bdeviation+%2Bvariance
Answeredj in http://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data - the answer is "yes", if you want to use the standard errors and F tests spat out by a default stats routine (which is implied in the OP and in @Cesare's comments).  Better to use a bootstrap for any inference.
@ Dmitrij I tried ZA test, its result is: Teststatistic: -9.1024  
Critical values: 0.01= -4.93 0.05= -4.42 0.1= -4.11
In cases like these, my advice is to consider your audience and their expectations. Personally if you only present one quantity, I will want to know the p-value, but more generally, I want to know more than that - in two tailed tests, for example, the test statistic conveys something that the p-value does not.
It would be worthwhile studying the foundations of statistical inference.  You will learn that 'prove' is an inappropriate word in this context, for example.
Just for the record, note that any such distribution will not be unique, except possibly within its own family.
*Writing* the likelihood is trivial--it's no different for one continuous distribution than for another (it's just the product of the values of the PDF evaluated at the data, assuming they are iid). Is your problem really one of *computing* the likelihood?
OK, but three grumbles, one major, two very minor. Chi-square on two ordinal variables ignores the ordering. This isn't _the_ SPSS document, but an elementary introduction by someone else, and they over-simplify, as just mentioned. They didn't copy "Asymp." correctly (example on previous page). The bigger issue for the OP is that correlation is the wrong word here: "association" is the key-word, in terms of measuring, testing and (best of all) modelling the association.
You've built a Markov model for spelling in English and fit it to data. But sampling from the fitted model is _not_ MCMC.  (What's the "desired distribution" it samples from? Clearly, not the distribution over "properly spelled words in English", since the model still makes mistakes after training). I don't mean to criticize the exercise; it's a nice demonstration of a Markov chain model for language. But the key idea of MCMC is to _design_ a Markov Chain so that its equilibrium distribution corresponds to some distribution you have in mind, and it's not obvious that this achieves that.
The model mispecification issue is the key.  Bayesian methods (especially the "poor-mans" Bayes of evidence maximisation) can perform very poorly under model misspecification, whereas cross-validation seems to work pretty well almost all the time.  The gain when the assumptions (priors) are "right" is generally much smaller than the penalty when they are "wrong", so cross-validation wins on average (as it makes almost no assumptions).  It isn't nearly as intellectually satisfying though! ;o)
Ok, thx ! 
Now, can one see that these reasons do lead to negative variance from a mathematical point of view?
If $X_1$ and $X_2$ are uncorrelated with $X_3$, the linear combination $aX_1+bX_2$ is uncorrelated with $X_3$ since covariance is bilinear.
What do you mean by straightforward? To convert categorical variables to metric scales is not a simple task. There are some latent variable models (like IRT models) that, given certain assumptions hold, allow to do that. Another way is optimal scaling. But for this I suppose you must be willing to do things with R and delve into some advanced statistics like latent variable models or nonlinear PCA. Does that sound okay?
Just a note for anyone unfamiliar with the term 'C-test' - this is just the usual 'condition on the sum and do a binomial test'.
Annoying, but a very easy fix.  Thank you very much!
+1 for suggestion of learning curves.  State of the art classifiers are likely to reach the same accuracy given enough data, but some are likely to get there sooner than others.  Trying to estimate the variance of the learning curves is also a good idea (which is straightforward to do if there are unlimited training examples as then there is no need for resampling)
@Tomas, There are some potentially useful links about contrasts [at this StackOverflow question](http://stackoverflow.com/questions/2352617/how-and-why-do-you-use-contrasts-in-r).  Also look at the help file for `contr.treatment`.
Thanks a lot. This is what i was looking for.
@user603 in the case of no offset, for support vectors we have $y^{(i)} \\theta \\cdot x^{(i)} = 1$ (I said no slack and then later asked whats going on with slack). If $\\theta = \\sum^n_{i=1} \\alpha_i y^{i} x^{(i)}$ then the constrain becomes: $y^{(i)} \\sum^n_{i=1} \\alpha_i y^{i} x^{(i)} \\cdot x^{(i)} = 1$. Which is what I wrote. For support vectors we can change the inequality to an equality because its a support vector. There is nothing wrong about what I wrote.
Interactions are certainly tricky. You are talking about an interaction between a continuous and a categorical variable. The essence is the same, but the interpretation is now slightly simpler. The main effect of ability is now the effect of ability in whichever group was coded with a 0. To get the effect of ability in the group coded with a 1, you have to add the interaction parameter. (This assumes the groups are coded 0-1, which is usually a good thing)
(+1) Might be worth clarifying the meaning of $\\Sigma_1 \\oplus \\Sigma_2$.
@whuber, It means I have to use mixture models and I can not fit my data with famous discrete distribution. Because Just for generalization, I want to simulate large sample data from a famous discrete distribution.
@ConjugatePrior hmm, I see. I will try out the packages once more as told by you. thanks.
I see no problem applying the definition $f(t)=\\sum_{x\\in\\mathbb{R}} p(x)t^x$ to a discrete distribution with probabilities $p(x)$. Whether this converges (and therefore defines an actual function) or has any useful analytical properties is another matter, but clearly if the support of the distribution is integral it will be quite tractable (it's a [Laurent series](http://en.wikipedia.org/wiki/Laurent_series)). I wonder, therefore, what motivates this question: is there a particular difficulty raised by the possibility of negative support?
So much effort has already gone into using deep learning algorithms for summarizing texts but not in legal domain. So I am trying to find out how will that work. Like you said, these algorithms by itself are not summarization algorithms, they just give out features. I have figured out a way to generate summary. Will update once it works.
Thank you for detailing the full solution, that's very helpful!
... and to understand the reason for this answer in more detail one good search term is 'shrinkage'.
This is also what GibbsLDA++ seems to do. I'm surprised that this is the favored approach since, as you said, there are severe theoretical problems. I would think it would make a lot more sense to at least choose the iteration that maximized the likelihood of the model out of the last $T$ iterations instead of just choosing the last iteration.
Have you taken any sampling classes, or worked through any sampling books, like Lohr (2009, http://www.amazon.com/Sampling-Analysis-Advanced-Cengage-Learning/dp/0495105279), Kish (1995, http://www.amazon.com/Survey-Sampling-Wiley-Classics-Library/dp/0471109495), or Cochran (1977, http://www.amazon.com/Sampling-Techniques-3rd-William-Cochran/dp/047116240X/)? It looks like you are familiar with the main terms, but if you did have training in sampling methods, you'd know the answers already :)
JAGS does not have a burn-in parameter that I know of; it does have an adaptation parameter (called `n.adapt` in rjags).
What do you mean by "a timecourse dataset"? Do you have multiple measures per patient over time? Also, is your response variable pain assessed via a 10-point rating?
Compare http://stats.stackexchange.com/questions/67571/how-can-i-group-numerical-data-into-naturally-forming-brackets-e-g-income
Here's a similar example from John Cook using binary Amazon seller rankings with different number of reviews. The discussion of choosing a prior in the comments is particularly illuminating:
http://www.johndcook.com/blog/2011/09/27/bayesian-amazon/#comments
Nice work the mystery is solved! I need to actually buy the book instead of previewing it in various tid-bits and having my library send me chapters every now and then. IMO this form of visualization reminds me alot of what cartographers call "dot maps", and one could utilize literature from there to justify how dots are a better visualization tool than the lines and cross-hatchings. It is also a good literature in terms of preferential placements of the dots.
I completely understand your point. Thanks for the answer. Do you then mean that the selection should be done on the training set or that there should be three sets one for training one for testing and a third for selection alone?
@hiberb This is becouse you don't bootstrap your variables in class 2. This procedure is described in the page you have cited and is implemented in the function _createClass_ which is located in the source file _rfutils.c_ (line 25) and _createClass_ itself is called from _classRF_ (line 238, file _rf.c_).
Thanks, this looks promising. I will take a look at the Gomez et al. article. Is there an implementation?
oh yes,this helped :)
+1 For the example.
OK, but please note that the answer is *not* "Poisson distribution"--that's merely an ingredient in the approximate calculation.
I have seen this approach (aim for a given CI width, assessing it as you go) recommended by various authors (sorry no reference at hand). It seems related to the [accuracy in parameter estimation](http://stats.stackexchange.com/questions/16985/how-to-report-general-precision-in-estimating-correlations-within-a-context-of-j#30287) approach.
You would probably want to treat k/n>0.25 cases as $\\alpha=0.5$ and not 0, but that does not change the bias issue.
(+1) I can confirm that Pearson correlations are used, by default. (With raw data, pairwise correlations are computed.) The 6 by 3 matrix of beta weights is returned as `fa$weights`, in the present case.