tttjjj/elt_bench huggingface dataset:
If I decide to use a dummy variable for each group but the base group (effictively g-1 dummies) and also include a dummy that takes value 1 if the family income lies below the poverty threshold - will this lead to near-multicollinearity? [Correlation between inc1+inc2+inc3 and poor = 0.79, correlation between inc8+inc9+inc10 = -0.54]
(+1) Nice concrete recommendation! I was about to suggest the use of the `mvoutlier` package (which relies on mahalanobis distance and displays outlying obs. in an interactive display).
Here's a link to one of the references in the above top link http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.3325
Thanks for looking, but these seem to be distribution-fitting applications rather than OLS regression.
I've entered the formulas in your images as LATEX. Please double-check that I have not inadvertently introduced any errors.
This is a great answer! I should have thought of the effect of approaching extremes of possible CIs. Thanks for writing this!
I think you might be looking for a stratified sample for some of the groups, but this is not an experiment. The idea would be to make sure all relevant groups are present in your data set and then use design weighting to the population. Still n=50 is perhaps too small to stratify for all of the socio-demographics you mention. Also sampling error might bust the quality of your predictions, even if the associations / effect sizes would be strong.
(+1) Perhaps you can provide a more general statement, since the OP's seems to be an example. I'm thinking along the lines of $\\mathbb P(A \\cap B) = \\mathbb P(A) + \\mathbb P(B) - \\mathbb P( A \\cup B )$. So, for example, if $\\mathbb P(B) = 0.6$, instead, then the first statement in the answer no longer holds. :)
1. Haven't you forgot to divide your average by the number of trials? 2. How are your Pavg different from each other? Your formula doesn't have i in it. 3. Is this (extended to t dimensions) what you are looking for? http://stats.stackexchange.com/questions/55999/is-it-possible-to-find-the-combined-standard-deviation/56000#56000
This is trivial. Use independence of $\\epsilon$s, then relationship between raw and central second moments. Done.
Can you include a link to where you saw these plots & the context in which the term was used?
Thanks a lot! Just to check if I understood everything correctly: Using the method you provided, I calculate the expected number, put the observed number into the formula, which would be 12 for the first column and then repeat the process?
A test of normality is usually too severe.  Often it suffices to obtain symmetrically distributed residuals.  (In practice, residuals tend to have strongly peaked distributions, partly as an artifact of estimation I suspect, and therefore will test out as "significantly" non-normal no matter how one re-expresses the data.)
thanks, this is exactly what I was looking for.  I don't have a lot of experience with writing functions, so your example is really helpful.
@dsimcha re large datasets: depends on what you mean by "large". Many observations? Use good graphics (boxplot, jittered dotplots, sunflowerplots). Many independent variables? Yes, you have a point there... But if you have so many IVs that you can't plot the DV against each IV, I'd question using an ANOVA at all - it looks like it may be hard to interpret in any case. Some smart machine learning approaches may be better (Brian D. Ripley: "To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'.")
Thanks, this will work nicely I think.
thanks for the suggestions, I think the methods you suggest are too technically advanced for me. I'm surprised that R doesn't have a package that solves this problem. It's quite a standard reparametrization, since $\\phi>1$ suggests overdispersion and $\\phi<1$ is underdispersion.
@Claudio If both the account from which this question was posted and the account from which it was edited belong to the same person, you should probably seek to merge your accounts. The fact that the gravatar looks different suggests they're under different email accounts (which implies it might not be the same person), but if they're both one person (i.e. you), it would be best to stick with one account (not least, to avoid accusations you may be voting for yourself, for example).
That's Sir Harold Jeffreys, so Jeffreys prior, Jeffreys' prior and Jeffreys's prior are all defensible, but Jeffrey's is wrong. He preferred the last form.
Hmm...20 ages times 2 genders times 3+ marital statuses times 3 to 20 or so levels of educational attainment times at least two "licenses" (whatever those are) times ... etc. = a minimum of 20,000 combinations (and likely millions).  What useful purpose will this exponential explosion of numbers serve?
Very closely related: http://stats.stackexchange.com/questions/91561, http://stats.stackexchange.com/questions/85878, and http://stats.stackexchange.com/a/80960.  Looking at the power curves in the answers may be helpful.
It is categorical. Thanx for your response.
"Thinking through the process" is good advice. Although beta is a tempting model based on a qualitative inspection of the histogram, I think that if you look closely at the values strictly between $0$ and $1$ you will find they depart substantially from any beta distribution.
Are you suggesting I reduce the sample size of the treatment group to a lesser amount or equal to the size of the control group.
It sounds like you don't actually know what the model is fitting and how to interpret the parameters in your model.  This is probably more appropriate for the stats stackexchange site.
a couple of recent discussions here have focused on modifications to the KL divergence that result in a proper metric. In case you're interested, see **[here](http://stats.stackexchange.com/questions/7630/clustering-should-i-use-the-jensen-shannon-divergence-or-its-square)** and **[here](http://stats.stackexchange.com/questions/6907/an-adaptation-of-the-kullback-leibler-distance)**.
How many "big outliers" are we talking about? Do you have any other unusual information about them that could justify their exclusion from your sample? If it's less a matter of outliers than of a non-normal distribution, you might consider fitting a robust/nonparametric GLM to reduce bias in your results.
Think about binomial distribution and notice that your answer is exactly the half of the correct answer. Update with another hint: C(2, 1) = 2!/1!1! = 2.
Thanks for your reply, Peter. Not sure if I was clear enough. What I want to show with this analysis is that they items are simple and for that reason most of the people will solve them. Visually it is quite clear. For example, for the variable1   participant obtained the following: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 and for variable 19, they obtained the following: 0 0 1 1 1 1 1 1 1	1 0 1 1 1 1 1 1 0 1. So what I want to know is whether the score in these variable are significantly different from 1
Comment on edit: this should be changed to be consistent with my equation 1 below. The root of your misunderstanding is directly related to the difference between my statement and yours. The interpretation is intricate enough that I don't think stating it in this form is very useful, but it would be something like: ``The difference between the effect of $X_1=B$ minus the effect of $X_1=A$ remains the same regardless of the level of $X_2$'' i.e. the difference is the same whether $X_2$ equals $E$, $F$, or $G$, etc for each line.
I am accepting the 'use a parametric test' advice. it doesn't exactly solve my question, but my question was probably too open-ended.
I am using SPSS. Whether to use the actual value or saved predicted probabilities from running a regression analysis, I am at a complete loss.
Fair enough, particularly on MSE. 

I remain a bit confused by the interest in bootstrapping and population inference but the lack of interest in testing since, perhaps naively, these seem to be equivalent concerns differently addressed. I'm also having difficulty tightly distinguishing out of sample prediction from inference to a population, but that's probably pre-coffee knee-jerk bayesianism (where prediction is just another population inference problem) getting in the way.
Thanks, you are right that FNN falls into the category of neuro-fuzzy systems.  The links you gave are helpful.  I think I will contact the author of the paper to see if there are closer implementations.
This really should be migrated back to stats.stackexchange since it's actually a statistics question
thanks! @cbeleites
@Ernest That's an interesting point.  Using orthogonal polynomials cures the problem.  E.g., `u <- poly(t,2); summary(fit <- lm(y~u[,1]+u[,2])); library(car); vif(fit)` shows the variance inflation factors are as low as possible (both equal to $1$).
Many thanks for your reply. I appreciate you getting back to me about this. Based on the responses to other questions in the questionnaire, I would assume that the 220 schools that responded to the questionnaire are a random sample of the total 1618 schools. Should I use 85/1618, 85/220 or 85/217 to work out the margin of error for this specific question? Thanks again. N
Just looking at the amount of 0s, there is no way that a Poisson is going to fit well to this data
You've learned that, if you know the errors are normally distributed around the regression line then the least squares estimator is "optimal" in some sense, other than arbitrarily decreeing that "least squares" is best. Regarding ridge regression, this solution is equivalent (if you are a bayesian) to the least squares estimator when a Gaussian prior is placed on the $\\beta$'s. In a frequentist world it is equivalent to $L_{2}$ penalized least squares. Logistic regression coefficients are not the solution to a least squares problem, so that would not be analogous.
Is this mainly about what makes sense statistically or do you seek Stata support? If the latter, it is arguably off-topic here (and would not be well received on SO, but you do not show a programming problem). In either case, please explain ADL and please don't assume that abbreviations used in your field make sense to all readers.
Okay...that's not how I understood your question. Now it sounds like you want to compare your two datasets 32 times (once per code). There are many ways to do that, and the choice will depend on whether you're trying to infer anything about broader populations to which your datasets belong, and whether you want to test hypotheses.
For reference I would like to add a number of suggestions that I've found in Makridakis, Wheelwright and Hyndman regarding DVs selection: 1. Principal Component analysis (Draper and Smith, 1981)2. Distributed lead/lag analysis 3. Best subset and stepwise regression (although I believe these techniques require to run the regression first).
@Dib I will extend my answer with some examples later when I have more time.
I changed the title to make it more visible.
There is no finite formula for the CDF of a bivariate normal distribution or for the CDF of a (univariate) normal random variable either. There _are_ rational-function formulas that approximate the CDF of a univariate normal random variable.  See, for example, [Abramowitz and Stegun](http://people.math.sfu.ca/~cbm/aands/page_932.htm)
Thanks. For the record, the **majority preference** hypothesis seems the appropriate choice as suggested by the paper.
Nice, and thanks for the link.
The lambdas (and betas) are chosen via a genetic algorithm. If I let the algo run without any constraints it will pick collinear often highly collinear sets of lambdas. As these provide the best fit in the Least squares sense and there is nothing theoretically that I know of that would prevent the actual parameters from being collinear. So I am using ridge regression to try and get more accurate estimates of the betas for these multi-collinear lambda values.
You shouldn't take my comment about basic pharmacological research as a specific criticism of that field, it is just my own particular discipline and thus the one with which I am most experienced. I am confident that you would find many areas in basic research with exactly the same shortcomings with respect to hybridized P values and error rates.
mmhmm intresting! GEK 2900!
You tagged the question as "cointegration", but do not mention it... are you asking for a "normal" relationship, or one where R and R_m might are potentially unit roots?
@AlexB. My version of Statistica is quite old and doesn't have integration with R. But even if it did, I wouldn't use it because R gives me pretty much everything I need. I have one "social network" site I use but I don't want to share that here. There is a chat facility here but I haven't used it before.
Similar to the zip, hurdle models have been suggested as well. See Porter & White [*SELF-EXCITING HURDLE MODELS FOR TERRORIST ACTIVITY*](http://arxiv.org/pdf/1203.3680.pdf)
Have a look at this question.
Stackoverflow: SVM equations from e1071 R package?  http://stackoverflow.com/questions/7390173/svm-equations-from-e1071-r-package
Those statements sound fine to me.
Couldn't you discretise the data afterwards, and then run any tests that would be available for discrete data?
The index in $\\hat{\\theta}_n$ is just notation to show that the estimator depends on the sample size $n$. You've found the likelihood function $L(\\theta)$ and want to maximize it. How would you go about doing that?
For instrumental variables it is required they do not correlate with regression error, **not** with the response variable. In general practically all econometric textbooks with chapter on simulataneous equations give an example with supply and demand models, where you model the volume and the price. The sales is then simply volumes times the price.
@MichaelChernick haha please forgive me i'm use to using artificial intelligences like svms and clusters that have more defined metrics of correctness. I think I'm going to explain this in terms of true and false positives, negatives and so I want the data to be as optimized as possible. do you think a non linear approach would provide any more optimization?
It is bad in the sense that the contribution of the small-range features to the overall distance will be minimal. If we exaggerate, the distance will be determined by the large-range features only.  But if the range of a feature is proportional to the "information" it carries, then maybe you don't need normalization.
Let's simplify discussion down to just looking at means (though much of it carries over to other things). If the question of interest is "are they reasonably similar", then a hypothesis test *answers entirely the wrong question*, and at sufficiently large sample size can be relied on to always reject the null, *no matter how trivial the differences*. Instead, you need to think more about what the actual question of interest is and work on answering that (I expect it will have something to do with effect sizes - such as an estimate of the difference in mean, or whatever else is of interest).
I agree and I think it applies generally across several stack exchange sites. The moderators sometimes wield too much power to close a question. Even when many people up-vote, it often only takes the negative attention of 3-4 moderators to get a question closed. I think questions should be innocent until proven guilty, so to speak, and a long, drawn-out discussion on the meta sites should be required, with tons of votes to close, *before* the question can be closed. User votes naturally do away with bad questions, so there's no real danger of too many "junk" questions.
@Mr.Cat, you have to transfer from $X$ to $Z$ in order to use a z-score table. Since a z-score table contains a small finite subset of values, you often must settle for an approximation. So you could also settle for $P(Z<3)\\approx 0$ and $P(Z>3)\\approx 1$.
thank you very much for your answer, it is very clarifying.
Just to be sure, you do know about `predict(m,type="response")`, right?
@Alexis I am not sure I understand your downvote. Care to elaborate.
I think this has been the most "authentic" depiction of mathematicians I've seen in a movie that's very good and entertaining.
@probabilityislogic, I intentionaly wrote non-rigorous mathematical text, since strict definition of singular and regular processes involves Hilbert spaces, which I do not think are really necessary to get the point accros. I will update the answer to reflect that. Concerning $c_t$, it does not need to be summable. The equation must hold in $L^2$ sense, hence summability of $c_t^2$ is sufficient.
It is not clear to me what you meant by steps. Is it an algorithm? Do you reduce the number of classes each step? Do you want to classify $N$ classes using $M$ classes?
Could you please explain what you mean that it is only valid asymptotically? I'm specifically unsure what asymptotically means in this context. Thanks!
I am having great difficulty following this. Can you describe the situation you are studying, the data you have, and the algorithms you are applying?
Have you tried to solve it yourself? If so what specific problems are you facing?
What's the matter with removing redundant ink from overlapping points in the plot, @Peter?
@Vic you can have a confidence interval on an effect size measure, but the confidence interval itself is not the effect size.
I will have a look asap. Thanks :)
+1 One *never* has complete information about sediment characteristics: these data are measurements of very small portions of the total sediments in each lake.  Therefore these data make perfect sense when either (a) inference is about this set of lakes and/or (b) inference is about all lakes and these form a representative sample.
I'm also very much interested in this.
Please let me know if you found anything!
And could you expand on what you mean by "moderator relations".
@gung I used a variant of the code at http://mathematica.stackexchange.com/questions/11350/xkcd-style-graphs for the first figure. It has long been known that adding a little random jittering to text or an image can make it look warmer and more accessible. ($\\TeX$ itself was originally built on [Metafont](http://en.wikipedia.org/wiki/Metafont), which Donald Knuth designed after a deep study of typography; he included such built-in variation.) For some graphics, like the pseudo 3D scatterplot in the second figure, the jittering technique adds little or is distracting, so I did not use it there.
+1: "...based on your null-hypothesis data model" is a nice way to provide a general yet useful answer.
My comments are just warnings that double-counting specific sequences can easily happen if you think of just placing the $X$ successes at some point in the $Y$ trials regardless of the remaining successes. Let's say $Y=5$, $N=4$, and $X=2$. Using your suggestion, $11011$ (with $1$ being success and $0$ being failure) would be counted twice. This is because we can begin the $X$ successes in the first position, or we can begin the $X$ successes in the fourth position. Either way, we can place the remaining $N-X=2$ successes together beginning in the fourth or first position, respectively.
I don't know anything about the nature of the data, but perhaps a [nested logit](http://elsa.berkeley.edu/eml/qca_reader/7a.nested.pdf) might be appropriate? I've +1ed your question because I'm interested in a solution for your "in practice" case.
I was thinking that this might be simplest way to come up with a ranking. Thanks!
I assume this is homework. Here are some hints. 1. show that the distribution of $F(X_i)$ (or $1-F(X_i)$...) is uniform on $[0,1]$. 2. What is the distribution of $-2 \\log U$ where $U$ is uniform on $[0,1]$ (the results is well known)? 3. Conclude.
Please explain what you might mean by "$\\lt$" when comparing loss *functions.*
Ok, thanks a lot, I was not aware if this.
PERFECT!  Thanks so much, this makes a lot of sense.
Thanks! I'd just stumbled to the conclusion that it may be more appropriate to use a mixed model (where subjects are treated as a random factor) for this very reason. For example, using `lmer` I'm playing with models such that: `m1 = lmer(choice ~ cost + msg + rank + (cost|id) + (msg|id), data=data)` if using lmer. Does this seem like a better approach?
@lejlot: I'm comfortable with having a difference of opinion: My main concern was to note that your first statement isn't a universally held opinion. Just to clarify my earlier comment with respect to your reply: when I said "Multiple imputation...is the current preferred option in biostatistics" my intended meaning was "the current **academic position in biostatistics, based on theory and formal investigation**, is that [properly applied] MI methods are less prone to bias than complete case analysis" rather than "everyone does it."
+1, I expect many teachers and professors are unfamiliar with IRT, as well.
Chi Sq does not like zero values in cells. Fisher Exact Test can handle cells with zeros.
I think that you would need something like an aggregation or clustering/comparing if you are actually interested in the outcome of an organiziational process but you have only collected data about the knowledge of individuals.
This is a good point to make; it's often not actually the mean that people deal with but some function of it and other things. However it's not only $s^2$ that can be messed up - you also lose independence of numerator and denominator, and that can have some surprising effects in the tails.
(+1) I don’t get well why it is correct to simulate samples under the estimated distribution. I would have think that we needed a prior for the parameters, and sample from all the possible distributions... can you explain a little bit more?
Even better, for any fixed $k$ it's a univariate polynomial (in $p$) which has infinitely many zeroes, and thus is zero. No complex calculus necessary!
Thanks for confirming. I'll edit the question to remove the ambiguity.
Mann-Whitney and its extension to several groups, Kruskal-Wallis, is the test of "location". "Location" (quotes are intentional because different statisticians differently define it) is, vaguely, a nonparametric counterpart of concept "mean" (rather than median): you may look in Wikipedia on Mann-Whitney - the key words there are "stochastically greater" and "Hodges-Lehmann"
@AndyMcKenzie: Good catch.
@Nick, thanks for great response and advice, will investigate this approach
Orthogonal parameterization has a broad range of uses.  Please be more specific.  What are you applying it to?
There are several examples of its use here, though not necessarily explicitly for *naive* Bayes. However, that hardly matters, since the idea of the trick is quite straightforward and readily adaptable.
It's not clear what you mean by "percent error."  For general guidance, take a look at the Wikipedia article on "level of measurement" (http://en.wikipedia.org/wiki/Level_of_measurement ).  As a rough guide, data considered to be on an "interval scale" are compared additively (e.g., by subtracting a mean from an individual value) whereas data on a "ratio scale" can validly be compared multiplicatively (e.g., by dividing a value by the mean or by a percent difference).  In a regression setting where the slope can be positive or negative, this rules out using percent difference.
(+1) This link doesn't refer to Hadley's website.
And Nick Stauner, I am indeed looking at item response theory (-> see: http://www.statalist.org/forums/forum/general-stata-discussion/general/1884-cfa-with-binary-variables). IRT has helped me to specify my measurement model as part of the generalized SEM. In a further step, I wish to correlate my latent variable with another exogenous observed var., while controlling for other factors. Yet, trying to model this in gsem, Stata gives the following failure notice: "var2 does not identify a gaussian error or latent variable".
Does anyone have any advice what I could do about this?
Sorry I forgot to say. the check variable is the sum of the preceding columns and is there only to check if there is missing data across the other columns, as each one should add up to two.