tttjjj/elt_bench huggingface dataset:
You're using hypothesis testing language but yet talking about a very exploratory, know-it-when-you-see-it approach @ the same time - so it's not clear what your goal really is for this part of your analysis. If you have hypotheses you're testing later (e.g. predicting cluster membership or using clust membership as predictor) then you can choose not to do things that will tempt bias there. But "multiple comparison" issues don't really figure into the exploratory problem you're describing. The viz cutoffs are just to help you see what's there - but your trust may still be misplaced.
Are these states latent? I am not sure what to make of the FA and SSM tags here.
Why don't you estimate the standard deviation directly from data?
Thank you, your answer is helpfull.
My communalities are 0.5 or higher ...
That's great insight, thank you. So, in the latter case where we have 10 out of 20 over 3 standard errors apart, what would you do to formally say that null is not supported? I'm guessing in a paper you would want to do the home run in that case and point it out formally. Would you say, look, we have a large effect size? What's the definition of "large" in that case?
$E(log(y))\\sim\\mathcal{N}(\\beta_0 +\\beta_1 x,\\epsilon^2)$ is a valid OLS model: you simply have changed your response to be $log(y)$ instead of $y$. But, e.g., $y\\sim\\text{Poisson}(\\exp(\\beta_0+\\beta_1x))$ requires a GLM, in this case, using the log link function.
Have you had a look at rpois()?
I'm not convinced this is correct. The potential error I made lies in asserting that the conditional likelihood is independent of the permutation: but it *does* depend on it. And that, unfortunately, gets us right back to where you started. This pseudo-reasoning, though, suggests you might construct a successful test anyway by using the sum of squares as your test statistic. It can be justified as a first-order approximation to the correct log-likelihood.
The only reason time was a factor was because I wanted to show how the company was progressing over time. But to answer your question, it doesn't make a difference when they were reviewed. So I think I can just normalize them all without regard to time and average them out that way.
@JolJols: Just doing it as a kind of warm up? And then  using Dunnett's test for the analysis you're really interested in?
+1, very clear exposition; nicely done.
@whuber, I didn't want to take any credit from you. Thanks for the assist!
Have you looked for a multivariate logistic regression solution, say with an autoregressive style correlation structure to account for time proximity? Example: http://www.google.com/url?sa=t&rct=j&q=multivariate%20logistic%20regression&source=web&cd=7&sqi=2&ved=0CF4QFjAG&url=http%3A%2F%2Fwww.jds-online.com%2Ffile_download%2F347%2FJDS-1040.pdf&ei=V6uWUv2jBpHnoASqtIGYAw&usg=AFQjCNHL02Y-87AHMvJ0hpqocN7S0Lra1A&sig2=1tpz_MGv30riRtAKX-fhsA
@Tal I was writing in your language, not that of your students, because I am confident you are fully capable of translating it into whatever you know will reach them.  In other words, I interpreted "intuitive" in your question to mean intuitive to *you*.
Actually my idea was to build it on the current standing, upcoming program and how well the teams are currently performing, based on wins/draws/losses during the last few rounds. 
Maybe with some kind of ranking, so that a win against a better placed team in the standings table last match counted more than a win against a lower placed team.
But I understand that there really is a need for more parameters, like injury status, and available players and so on.
First you need to see if the desire to reduce a nonlinear relationship to a single number is a good idea.  If so, you have to check if the relationship is flat over an interval.  If so, beware of the next step.  Next, choose an interpretable measure that handles varying scales and nonlinearities, such as inter-quartile-range odds ratios for continuous predictors.  This last step is automatic in the R `rms` package.
@Chethan The last example shows explicitly how this sum is computed.  I learned about the FFT method from the geostatistics literature long ago, but it's a stretch to characterize this as "simple."  You can find an explicit description by Denis Marcotte in a [1996 paper](http://w3eos.whoi.edu/12.747/problem_sets/PS05/marcotte_96.pdf) in Computers & Geosciences.  A [2010 PowerPoint presentation by Jim Jennings](http://reproducibility.org/wikilocal/docs/data_flows.ppt) summarizes the approach (starting p. 14).
You've proved that pictures are worth 1000 words. Thanks @whuber =)
+1 Practical advice.
Really, mathematically speaking the difference disappears. You'll likely never draw a 0 using np.random.uniform anyways. Not will you ever draw a $1/\\pi$, nor a $1/2$, nor a ....
VEry helpful - thank syou
I was coming from a standpoint of MLE, sorry if my posting wasn't the greatest, it's my first post on this forum!
The variance of $x$ is directly related to the *fourth* moment of $y$. This becomes obvious when $y$ has zero mean, for then the mean of $x$ is the variance of $y$ and $\\text{Var}(x)$ = $\\mathbb{E}(x^2) - \\mathbb{E}(x)^2$ = $\\mathbb{E}(y^4)- \\text{Var}(y)^2$. Thus knowledge of just the first and second moments of $y$ (and related variables) will not suffice to find variances and covariances involving $x$.  The answers to your questions will therefore depend on precisely what "bunch" of statistics you have about $x$.
I corrected my typo. Thanks for pointing out my error.
+1 for this lengthy one, deleted mine wrong guess with negative default pm = 0 parametrization.
I admit, I don't get it.
$\\ell_2$ penalty does not mean your coefficients cannot be zero. You can see the $\\ell_2$ penalty as a constraint on your coefficient. You can have more details on why this is equivalent [here](http://math.stackexchange.com/questions/335306/why-are-additional-constraint-and-penalty-term-equivalent-in-ridge-regression)
Another example:  If the true relationship between Y and X is non-linear, then alpha and beta, and thus Yest, are only a local linearization of a non-linear function. The error term will pick up effects that can not be captured by a linear fit. In such a case Yest can very well be a biased estimator of Y[i], i.e. the expected error if we made a new observation should not be zero.
@user: I for my part cannot think of a statistical or mathematical reason. But my experience is a bit limited, let's wait for more correspondents.
I wouldn't say that number of pills taken in a day is continuous.  In fact, I would guess that it's Poisson distributed (count data, i.e. discrete), with the interval time between pills taken as continuous (exponential).
You might want to make your question more specific.
(+1) Very handy.
It's the right idea and the sample sizes should be large.  You would get insight into why by actually computing the power curve rather than simulating it.  Be careful in your choice of example: I don't think anybody would take a new failure rate of $1/10$ to be an improvement over $1/20$! :-)
It is also questionable whether assymptotic results are of practical use when looking at datasets with a large number of features and comparatively few patterns.  In that case the variance of the procedure is likely to be of greater practical importance than bias or consistency.  The main value of LOOCV is that for many models it can be implemented at negligible computational expense, so while it has a higher variance than say bootstrapping, it may be the only feasible approach within the computaional budget available.  That is why I use it, but I use something else for performance evaluation!
I think I got it.  Would the final equation (adjusted also for the change in looking now for the min of the group with smallest max) be:    
$$=\\Sigma_{n=1}^{n}[ m\\int_0^x f_{n}(w)[\\int_0^w (F_{n}(y)-F_{n}(w))^{m-1}g_{n}(y)dy]dw]$$
Thanks so much, your answers have really helped.
See also [questions/tagged/regression+effect-size](http://stats.stackexchange.com/questions/tagged/regression+effect-size) -> Abelson, [A Variance Explanation Paradox](http://wolfweb.unr.edu/~ldyer/classes/396/abelson.pdf), 1985, 5p.
I also hope that if this is a HW problem, you will not simply copy paste.
How many statistics courses have you had?  This kind of analysis would require you to have had several.  Why should anyone help on an urgent basis?  You question is actually quite complicated, you did not provide the experimental design, and you dit not say which measurements are made on different subjects or experimental units and which are made on the same unit.
What if you do not know why the sample is unrepresentative, are you still justified in discarding it and drawing a new, random sample?  If not, why not?  Also, let's say you do discard the first sample and draw a second one, are the inferential statistics that you might calculate based on the second sample in any way inappropriate due to the discarded first sample? For example, if you subscribe to discarding unrepresentative samples, are you changing the sampling distribution that your statistical test is based on?  If so, are you making it easier or harder to find statistical significance?
thanks for your explanation! Then do you know is there any indirect method to quantify the relative importance of the predictors? mike1886 mentioned the "ROC curve analysis" in his answer, but is has some issues as mentioned by rolando2 . Thanks a lot!
The definition for absolute risk is the risk of arithmetic difference between two  groups.
Thank you for your feedback, Thomas. I have made quite a few adjustments. If you, or anyone else, can provide more feedback, that would be greatly appreciated.
They usually have sparse representations - you don't need to store $mn$ numbers for a low rank approximation.  For example, a rank 1 approximation requires $n+m$ numbers.
"*I tried to use binomial family - logistic regression*" -- why? You stated above this that it was conditionally Poisson. Wouldn't you use a Poisson model if you think it's Poisson?  (I'm not saying that would be the solution to all your problems there ... it just seems a rather odd choice not to use the distribution you have in your model)
This has a frequentist interpretation too.
Actually I wanted to create the distribution model, THEN perform a monte carlo simulation on the same model to extend it out. I guess the followup of going from a kernel + random number between [-1,1] -> random sample from distribution is a separate question altogether!
@Christian Perhaps you can contact me directly and I will try and help you. See my contact info .
Another note: this optimization problem seems perfectly suited to be solved by a genetic algorithm: the DNA is given by a logical vector specifying class membership of the vars, the initial population consists of individuals with randomly assigned DNA sequences; also mutation and selection operators can be defined in a straightforward way. The result is a population of really well performing individuals.
Indeed, choosing the pareto distribution as hyperprior for the sample size works and has the advantage that it’s easy to implement in Stan. I implemented both approaches and as expected the pareto prior provides slightly more shrinkage.
have you considered appending the example described over the course of these several comments onto your answer, instead?
I was stuck in a long day of meetings. :)
The whole question of multiple comparisons is vexed. It is arguable whether they are need at all; and people forget that correcting for them (esp. with Bonferroni) will lower power.
@Zach, I think the `flare` package for `R` implements the [Dantzig selector.](http://cran.r-project.org/web/packages/flare/vignettes/vignette.pdf) I have no experience with it though.
Except in respect of the emphasis on correlation/multicollinearity, this question is at the least a near-duplicate of http://stats.stackexchange.com/questions/74140 ... whether you want to remove variables at all depends on whether your goal is statistical inference or predictive power.
The last one is slightly more complicated than you present it, but I agree much of the observational associations found between soda/diet soda and obesity should be looked at with a critical eye. Theoretically some have posited that the fake sugar/fat substitutes have other physiological effects beyond the simple calorie intake. See for example [this experiment on rats and synthetic fats](http://www.freakonomics.com/2011/06/30/do-synthetic-fat-substitutes-make-you-fat-ask-the-rats-who-ate-pringles/) (taken from the Freakonomics blog).
(+1) Good reference!
If I'm not mistaken, the first step of Weka is to load the entire dataset into memory.  I believe every value is stored as double, so that's 8 bytes per value.  With 1000 features and 200 million records, that's 200 billion values and thus 1.6 trillion bytes.  In 2012, it would be difficult to get even a small portion of that into memory, so it seems to me that perhaps plain-vanilla Weka is not the best place to start.
Hi Nick,I have updated the data. I realized that only one observation per factor does not allow further test analysis. Thanks and I would really appreciate any help.
The gamma distribution *is* defined by its first two moments in the sense that specifying the first two moments (and the fact that it's gamma) is sufficient to determine the two parameters, since the first two moments depend on the two parameters in such a way as to make it possible to solve for them. That doesn't imply that the higher order moments don't exist, any more than it does for the normal distribution, so why would authors not refer to the higher order moments when they're relevant, just as they do for other distributions which are determined by their lower order moments?
Actually, in 7-card stud, a 4 of a kind *is* more likely than a 7-card straight:
There are 13 possible ranks for the 4 of a kind, and the remaining 3 cards can be chosen from the any of the other 48 cards in the deck (${48 \\choose 3} = 17296$) yielding $13\\times17296 = 224848$ possible hands, or slightly under twice as as likely as a 7-card straight.
The important thing to note is that the probability of 5-card hands in 5-card stud do not match up with the probability of 5-card hands in 7-card stud. (However, the relative probability of the hands is the same, and so the rankings are the same.)
+1, excellent overview.
Gamma integrals can be "controlled analytically," so what distinction are you making?
See [here](http://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection). Reducing a model is fraught with problems & you should only consider it when (1) your primary concern is predictive accuracy, & (2) you can demonstrate poor performance owing to over-fitting: even then stepwise methods are not generally a very good solution.
(+1) for making the distinction.  However, I believe the OP is asking for (1), not (2) (and I have edited the question's title accordingly).  Also note that your formula appears to assume the regression depends only on one variable.
Hi Peter, when you say "and treat one variable as independent" do you mean treat 'male' as an independent variable? With that in mind, should the title become "which independent variable is more likely to influence the dependent variable?" ?
I see two problems with Fisher's test: it assumes i.i.d. and even sampling (which I don't have). The only thing for what I can assume even sampling is for the modelling of the ARMA model for which I really don't need the physical description (where it would be better to use the continuous CAR and CARMA models), just the detection of it.
Thank you for the detailed and very useful advice. If I use only Region 1 and Region 2, and apply Pearson's chi-squared test to the data for each year separately, would it be better to do a Meta-analysis also for these 2 regions? There is heterogeneity in the sample data and a Random model looked more appropriate and gave a significance at a p value of 0.000. Egger’s linear regression method intercept (B0) 10.34631, 95% confidence interval (1.05905, 19.63357), with t=3.54535, df=3. The 1-tailed p-value is 0.01911, and the 2-tailed p-value is 0.03822 (significant funnel asymmetry).
In addition, `n!` = `Gamma(n+1)` for n >= 0. So try to look for a function called `Gamma` if you need to calculate the factorial (or log Gamma if you're calculating the log likelihood)
Thanks for the pointers. The first one explains a technical difficulty related to a kind of degeneracy: H0 implies that the mixture model parameters are undeterminate. The second reference indeed provides further links to solutions such as the Lo-Mendell-Rubin test. Given the level of complexity of these approaches, I think I will consider a simple normality test (single component model vs arbitrary model).
@Glen_b Not "looking for" a confidence interval, but perhaps should consider using a confidence interval to understand the data. The specific question would require a confidence interval on the difference between the two means rather than the usual pair of confidence intervals for the two means themselves.
@Alexander i do not think ANOVA applies to my query of interest. ANOVA generalizes the t-test and compares the average of groups. my query is to compare (for each variable) 1 region vs all other regions and determine if this variable (i.e. its value) differentiates it by strength and/or significance. if i had a bunch of data on region 1, region 2, ..., region N, by variable, then i could see how ANOVA could help. but in this case, i only get an aggregate value (the value is at the region level, not sub-region level). thanks though.
They *think* (i.e., bring knowledge to bear).
Thank you very much, I'll give it a go and come back if I have any extra concerns. Again, thanks very much for your help.
if k is the number of groups, the total possible pairwise comparisons would be k*(k-1)/2, so the α ′ should be 2α/(k∗(k−1)).
@mpiktas If those four values represent a bivariate distribution each with probability 1/4, the `cor` function returning zero will indicate a population correlation of zero.
@genesiss Wayne is right; it's the relationship of the model to the data that makes an outlier an outlier. Data in and of itself is just data; it can only be discrepant with regard to some model making it so.
If you pick only one level of each categorical variable, you won't see the interactions.
@JRideout No, it's not. I just like to show it in a column rather than a chain of numbers on a line. When it's applied to many countries, it would help to arrange the number nicely. Another option is data.frame().
Could you maybe answer in intuitive way the following question.Figure 2.1 can be viewed as representation of 3d plot where the color of points simply represents a third dimension. There is a cloud of point where majority z=0; and there is another cloud where majority of z=1. By building linear model one simply builds the plain separating two clouds of points. Question, how does this plain projects on 2d plot? Or is it not, and decision boundary line in 2d plot has nothing to do with plane in 3d plot?
Use Gram-Schmidt orthogonalization of the monomials (or of any other basis, such as the Hermite polynomials). I posted an example [here](http://mathematica.stackexchange.com/questions/19492/how-do-i-solve-n-simultaneous-equations-for-n-variables/19504#19504). That code--which is symbolic--embodies the mathematical principles. It also happens to be executable: if you change `limits` to range from $-\\infty$ to $\\infty$ and the kernel `k` to be the mixture PDF, then apply `Orthogonalize` to the list of polynomials in `vectors`, you will have an answer.
Well, the pun was international -- but in retrospect, I really should have asked for some **tips** for weighting on tables... Thanks for the note on $\\TeX$ markup
(+1) @hydrologist: As a complement to whuber's answer: One possible criterion that will lead to uniqueness is to insist on the *square-root* itself being positive semidefinite. Uniqueness of $\\mathbb A$ then holds under the weaker condition that $\\mathbb V$ is positive semidefinite. An instructive example to see what can "go wrong" is to look at the possible square roots of $\\mathbb I$, even just the *diagonal* ones! :)
@Sam Glad to help out. Feel free to post another question if you get stuck.
$a$ is the number of measurements for each individual. If there is one measurement at each timepoint then $a=7$.
It's a discrete math question...
Do both approaches and for each calculate the given result. Then use a measure to find the appropriate one. The measure, e.g., can be given by the prediction error found in a cross-validation procedure.
Actually, thinking about this again, the question still applies to linear least squares, in the case that $d$ is non-linear.
Welcome to the site, @user21859. I don't really follow your question. Could you perhaps state it succinctly at the end?
@Torvon This is mainly used in sales modeling where the sales is measured in revenue or sold units etc. You don't have to center the variables to use this measurement. Your beta values will adapt to fit the data you use. The basic idea is that you decompose the total sum of the response variable in it's pieces by multiplying the corresponding beta value with the total sum of the corresponding variable. By definition adding all of the variables sums multiplied by their corresponding betas have to give the same as the sum of the response variable. But as whuber has pointed out, it has flaws.
Not sure how much to trust a presentation from someone who can't spell their own university name. More seriously, beware all advice that implies that things are fine above a certain sample size and dire otherwise. It's a matter of approximation quality increasing smoothly with sample size and also depending on the distribution of the data. Simple advice is to be very cautious, plot everything and cross-check with bootstrapped confidence intervals.
@David Thanks but I would like to use the same framework for all my graphs. Moreover I do not have an Excel copy.
Try here: http://mitpress.mit.edu/txbkreq/195422
Hi @Hibernating, thanks for your well documented answer. I really appreciate your effort. I'll have a look at this.
Are a subset of the results going to be verified using PCR or some other method? Or are the microarray results meant to stand on their own?
*What is the purpose of the clustering?*  As with most statistical questions, there are multiple answers and knowing the purpose is an essential guide to selecting appropriate or good ones.
Do you have a link or more specific citation?
+1 Great question.  I was wondering the same thing.
Method 2 is quite clever. Thanks :) I think that works well enough at this stage of my work.
thanks for your answer. I used only one `scatter` call. I agree that the the order is determined by what is plotted first. Here's the [code](http://pastebin.mozilla.org/1254318)
Have a read at `vignette("longintro", package="rpart")`. `rpart` works by recursively finding the best split for the data. You don't choose the order, it is chosen automatically by the algorithm to optimize the classification, that's the whole point of it.
Good grief... constructing the covariance matrix is never the best way for SVD. I displayed an example of why explicitly forming the covariance matrix is not a good idea at math.SE: http://math.stackexchange.com/questions/3869/3871#3871 .
Thank you for an interesting post. I suspect that a lot of epidemiologists "borrow" terms from statisticians, adapt them to their setting and then... due to the lack of a sound base... it becomes somewhat of a mess
+1. The usual question: have you got the data for the parents' BMI?
...and by keeping them separate, you can actually ask the question: "Is the estimated effect of FF consumption vs snacking similar", whereas if you combine them, you have already assumed it is.
These terms appear to be taken from an answer on this site at http://stats.stackexchange.com/a/16460, where they are clearly defined. Could you please tell us what your reference is and what the stumbling blocks are for you to understand its terminology?