tttjjj/elt_bench huggingface dataset:
(1) The ANOVA analysis is suspect due to the strong heteroscedasticity in the response (the variance of a count is proportional to its square root), so obtaining a significant result with this method means little. (2) Which better describes such data--the Poisson model without interaction or the ANOVA one with an interaction? Many people would say the former, because it's more *parsimonious.* (3) Stevens' taxonomy is not helpful here because "ratio" is a misleading characterization of what counts are and how they can appropriately be analyzed.
Thanks. I didn't even know this field exists! After Googling I also found this related question in the Stack Exchange network: http://cstheory.stackexchange.com/questions/822/a-reading-list-on-experimental-algorithmics
Perhaps if there is no precise hypothesis to test, the research should always be considered exploratory.
You don't have a MLE of *data*. You have a MLE of some quantity, like a parameter, or a distribution function. Please clarify what you're estimating by ML.
Hosmer & Lemeshow recommend to categorize such continuous predictor and they give quidelines how to do it right: See their book "Applied logistic regression"
"probably normal distributed" - plot them?
(+1) great question @infinity!
Is this categorical (i.e. counts for values between say 1 and 2 hours, 2 and 3 hours), or something like that?
@ttnphns: yes you are right. My question is with respect to the right eigenvectors $V^*$. I should have been more precise I guess.
averages of more points will tend to be more precise. If the variance of individual observations is constant, the variance of averages will be proportional to $1/n_i$. You need to weight the averages so the more precise ones get more weight.
Here is the link (note that there are spaces within the link, which causes problems): http://gseweb.harvard.edu/~faculty/singer/Presentations/9) ALDA Chapters 14 and 15.ppt
Can you post a `str` of your data and the call to `lmer`? However, from your description I have a wild guesses: Yes, `lmer` only uses proficiency scores when available.
I'd also love an answer to this. What I did is just used the mean value for the fixed factor variable (I am using time and group FE, not individual).
Did you consult the wiki article [on estimation of parameters for the von Mises distribution](http://en.wikipedia.org/wiki/Von_Mises_distribution#Estimation_of_parameters)?
I am only analyzing the data, and as such, the data don't actually belong to me. But once the data is published (should be soon) you should be able to use it as you like.
Re the edit: The Lebesgue measure of this set is irrelevant. You need to be looking at the probability that the CDF of a uniform distribution on Y does not agree with the CDF of X. This probability is still zero. Take a closer look at the Web page I referenced and think harder about the Bernoulli case: once you understand this (the simplest possible situation) all should be clear. (This comment was originally posted Dec 19 '10.)
Chapman Hall / CRC and Wiley also publish quality statistics texts.  But one shouldn't judge a book by its publisher...
@ttnphns, what would you suggest for smoothing? I have a pretty erratic series (daily download counts of documents), which I feel might benefit from some smoothing.
There are many ways to create a bivariate distribution in which both marginals are Weibull.  Thus the "spearman correlation between the two" distributions is not well-defined.  It would help to quote a source for the statement in your second paragraph.
Another possibility is to do a permutation or randomization test.
Please tell us a little about the objective of your analysis. What would you like to find out with these data?  What actions or decisions will be made as a result of your analysis?
@Thomas the answer is to be found by inspecting the code of dgpd, which is a very programming-specific issue.
Does [this similar thread](http://stats.stackexchange.com/questions/6067/does-an-unbalanced-sample-matter-when-doing-logistic-regression/) answer your question?
I appreciate the suggestions and will investigate.  I was attempting to use the simplest approach in my understanding.
Are you sure about this expression? I think here should be an $\\ell_1$ norm somewhere. Could you put a link to the paper you are referring to?
(part 2/2) The other parameter to set is the scale parameter for Zipf that defines the spread of the distribution (this value in my case is 0.9). I have all the parameters (size of sample, frequency per user, etc.) but i have to verify that my dataset follows the zipf distribution.
I agree in practice, which is why this exercise I found seemed rather strange.
@Dov your first link is broken - I can't tell where it's supposed to point.
May I ask what the purpose of the outlier detection is here? Is determining that $.9$ doesn't fit of scientific interest here, or are you looking for a way of weeding out "bad" observations, such as measurement errors?
While said F-test is relevant for univariate multiple regression (one predicted variable), I could not find information about its use in multivariate multiple regression (many predicted variables analyzed simultaneously).
It is rather striking that $y_t = x_t - x_{t-1}$ = $1/2(x_{t-1}-x_{t-2})+w_t$ = $(1/2)y_{t-1} + w_t$.  Does that help?
As far as I can see it, you just could run a regression including "aid expenditure" as independent variable and look at the coefficient and its confidence interval, $p$-value to assess significance.
After you have paid two dollars and won three dollars, your *net* winnings are one dollar, not three.
I have checked and rechecked and I can't seem to find it. Sorting those in descending order and choosing the corresponding features resulted in a good-ish accuracy. in ascending order the accuracy was horrible. but since you're telling me that MI is non-negative maybe there's something I'm missing
Thanks for the plot.  The Poisson would be visually indistinguishable from a Normal at those high x-axis values.  It almost looks like you have a mixture of two distributions, one with a peak around 190 and the other with a peak around 290.  A problem with very large sample sizes is that you can almost always reject the null hypothesis (because it's almost always false, in reality) unless you increase the level of the test (e.g., from 95% to 99% to 99.9%) as the sample size increases.  Have you tried overlaying plots of the two cumulative density functions?
I have verified these abbreviations. Your searching skills seem better than mine. Thanks.
Cube roots work for negative values too. The core of the function would then need to be something like `sign(x)*abs(x)^(1/3)`. (Square roots also work for negative values, but few people want to bring in complex numbers in this context.)
@ttnphns: Yes, these are two different things. But note that I was talking about two matrices: U and V. If X is the data matrix with variables in columns, then SVD decomposition $X=USV^\\top$ gives principal axes (eigenvectors of the covariance matrix) in V and loadings (up to the scaling) in U. Both matrices have orthonormal columns. In case of V it means that principal axes are orthogonal, and in case of U it means that principal components have correlation zero. Hope it makes sense.
As far as (1) goes, I read your reference (to Dave Gile's blog) differently: he argues that *formal testing* of multicollinearity is misguided. I do not see him claiming that multicollinearity is not a problem.
Johnny, $s$ is the square root of the variance and $\\bar{x}-\\mu_0$ is the mean difference, so both of them *are* in the formula. Indeed, no term in the formula depends on the significance level. What do you suppose that means will happen to $t$ if the significance level is changed?
@whuber Thanks for the comment. I have revised the first sentence as suggested.
You seem to be confounding errors and residuals together there. The assumptions apply to the errors, but the diagnostics are done on the available estimates of them (the residuals). You're interested in checking whether the entire set of errors are consistent with being sample from $N(0,\\sigma^2)$
First, correlation isn't measure by t-tests. Second, your general question is one of multiple comparisons. [Here](http://stats.stackexchange.com/questions/tagged/multiple-comparisons) is a list of threads with that tag. Look around in there and see if you find an answer.
This question might also be of interest: [Is there an alternative to the Kolmogorov-Smirnov test for tied data with correction?](http://stats.stackexchange.com/q/35606/10525).
If you don't know where the missing data are, you'll need to backcalculate them from the imputed data. This will incorrectly classify points as observed if, by happenstance, all imputations for that cell are identical across the m data sets. As a consequence, the diagnostics may incorrectly label imputed points as observed points (in mice terminology: some red points are incorrectly plotted as blue points). However, this does not affect the validity of the statistical inferences. So, with some extra effort, you can.
Those SOAs... they must be just transforms of each other with a little noise.  They really look like they're measuring the same thing.  What are they?  Are they centred?
You might start by checking your definitions: some of them are incorrect (particularly in 1 and 5).  It's hard to understand a concept when you haven't got the definition right!
No I don't. The data is collected as rounded: 0,1,2,3,...,100 even if the actual interval is 0-100.
Do I understand correctly that you run 2 pieces of code and are unsure why they provide different results? In this case I recommend you to post the relevant code fragments as it is a bit unclear what you are doing exactly with the current information.
Great point on the ruby gems, I forgot about those. Perhaps other people can suggest few ruby gems on the matter of using Ruby as a workbench which are well known to work and are robust? Thanks for adding up an answer on what I am looking for again.
@CravingSpirit it would return the number of observations for each variable. Read `?dcast` which would have told you this (see the details for argument `fun.aggregate`).
Other people have had this idea, too jf, so you're in good company: after some improvement, it was called "[forward] stepwise regression." For some indications of the problems with this, please search our site for that phrase.  With 10K variables, the problems will be so overwhelming that there is little hope for this approach.
@jbowman Not sure as to your question, but in an attempt to clarify, I'm using a bootstrapped sample and using a T-Test to compare means with some other sample. If I had two samples (neither of which are bootstrapped), then the sample sizes to use is straightforward in a T-Test. I'm asking about which sample size to use when one of the samples has been bootstrapped.
Thank you gung, you´re explanation clarified it all :)
I already read that and framed the question from that. Maybe it goes something like this. For example 1, "Knowing that 39 out of 549 times I see `something` under C1, and that 100, 100, 180 and 169 of them are from P1, P2, P3 and P4 respectively, What is the probability that these 39 `something` are so unevenly distributed between the 4 plants? And this is what I desire.
I agree the second table is a bit misleading. It might or might not be significant (depending on the parameters that fisher's exact test computes).
A simple general rule is "do whatever you want in the training part and do not ever touch the testing one". Thus, you equally sample the classes in the training portion and can apply feature selection over this training fold and finally, report the results of the model over the untouched and hidden testing portion. Also, you may look at nested cross-validation. More details are at:
http://stackoverflow.com/questions/2595176/when-to-choose-which-machine-learning-classifier
This question is [routine bookwork](http://stats.stackexchange.com/tags/self-study/info) (please read the link carefully). As such it should carry the `self-study` tag (please edit your question to add the tag), and it should follow the guidelines at the link (please edit your question to be in accord with the guidelines).
Mpiktas' hunch is right. There exists multivariate multinomial logit models and I have seen it in Agresti 2002. I will get back with the exact page numbers as I dont have the book on me. But I have seen the theory being developed in the same chapter in which they introduce Mutivariate logit; I think immediately after the horse-shoe crabs or crocodile examples.
...besides: I thought that in all cited cases support means that $X$ and $Y$ both appear.
This is a *three* variable optimization problem; the variables are $a$, $b$, and $c$.  The objective function is quadratic in those variables, so you don't even need calculus: you can use elementary methods to find the minimum if you like.  But it's easiest to characterize the unique global minimum as the point where the gradient $(\\partial/\\partial a, \\partial/\\partial b, \\partial/\\partial c)$ vanishes.  (Consider the one-dimensional analog: if you want to minimize $\\alpha a^2 + \\beta a + \\gamma$ with respect to $a$, which derivatives would you take?)
Conjugate, thank you for your answer! So I need to standardize my variables first and then average them? How about rank order them and weighted by ranking order scales? or use decile? Do you have any book or good website to recommend? I only have a very basic stat background...Thanks again.
Look up references in this post, http://stats.stackexchange.com/questions/68864/general-theorems-for-consistency-and-asymptotic-normality-of-maximum-likelihood/68866#68866
Sorry about the shape/scale mistake, I fixed it. Please edit your original question to make it clear to everyone what you are after.
Excellent answer. The [Morgan & Winship book](http://books.google.com/books?id=lFTP49c5wJoC&lpg=PP1&pg=PP1#v=onepage&q&f=false) is quite a bit easier than Pearl, with a focus on social science problems.
(1) For some estimation methods see the [Wikipedia article](http://en.wikipedia.org/wiki/Log-normal_distribution#Maximum_likelihood_estimation_of_parameters). (2) When either or both $l$ or $k$ are small, all bets are off: there will be no universal analytical expression. Deeper analysis of the cumulant generating function of the empirical distribution function of $(a_{ij})$ might produce something, but that would take a lot of work. For a practical application, why not just simulate the distribution? All it takes is the second line of `R` code in this reply.
I **strongly** suggest against optimizing recall exclusively. Without accounting for anything else, the 'optimal' classifier would simply always yield `classA`, with maximal recall (1).
Is your GLM Generalized Linear Model or General Linear Model?
I think you should reconsider the statement that the fixed effects are similar.  Look at gamma.  They differ by many standard deviations.
You can convert to $data.frame$ using $as(mcX,"data.frame")$
Done, Cheers ! Chap
There is always ambiguity when somebody mentions "orthogonal components", because this can either refer to axes (meaning eigenvectors are orthogonal) or to projections (meaning components have zero correlation). Obviously scikit's LDA returns "non-orthogonal projections", that is equivalent to saying that correlation is non-zero... To your question: I have no idea why it works! I think somebody should convert scikit's code into formulas and then we can try to analyze them. I am quite surprised that scikit developers on github have not provided such an explanation yet.
I look at likelihoods, R-squares (despite what people say), confidence intervals around parameter estimates, plots of observed vs fitted, residual vs fitted, etc. If there were science favouring one model over another, that would weigh too, but in my experience the science is not so well formed. How else could it be done?
Could you elaborate on this idea a bit?
Setting the whiskers at these fixed percentiles seems strange to me.  Do you have a reference for this?  (Tukey, who originated the boxplot, did not use this method: he set the whiskers either at the extremes, if they are sufficiently close to the quartiles, but no further than 1.5 "steps" (equal approximately to 1.5 times the IQR) out from the quartiles.)  This is much more robust for outlier detection than using an extreme percentile, which--by definition--would *always* identify 10% of the data as "outliers," which wouldn't be a very useful procedure.
is that a feature on the program? I havnt no. Im just having trouble getting my head around it. there is a box you can get up that is labeled "repeated measures" after you click analyse and compare linear models. in this there is an option to add a between subjects factor. but i entered the grouping variable (the column in the data view that groups 1, 2 and 3 for the levels of the independent variable) but I'm not sure this was the correct way to run the statistical test that I'm looking for.
@EDi thanks - please cite the included reference if you use it!
That's a pretty nice solution. And the graphics are impressive. There is also a `partitions` package that might have provided a structure for enumerating the possibilities.
That small detail that was a bit important... I thought you were talking about `R`. OK, a bit more rigorously then : $-\\frac{N}{2}\\log(2 \\pi) - N\\log(\\sigma) - (1/(2\\sigma^2))  \\sum( lm_{residuals}^2)$ where $N$ is your number of readings, and $\\sigma$ is the standard deviation of your error. Do you have `lscov` functionality, so I can present an example using that or you want the standard *matrix left division* notation for MATLAB?
What makes the hypothesis you listed interesting, subject-matter-wise?
There is nothing more here than a [perfect] analog to the conundrum "there's a $1/2$ chance that a fair coin will land heads up, but after you observe that it has landed heads up, it is no longer possible to say that the chance of heads [on *this* toss] is $1/2$."
does the following link help ? http://archives.math.utk.edu/ICTCM/VOL13/C013/paper.html
@amoeba Ok thanks so much for the input either way!
Thanks, this is what I was looking for! Ill look at your sources, hopefully I'll get a better understanding.
How would you get negative squared prediction errors?
+1, great answer!
Are you talking about binning a single Likert item or the sum of several Likert items?
Did a full nights worth of reading and realise how wrong I was. Thanks for the heads up. Could I confirm my findings with you quickly?

"y= m1x1 + m2x2+... + -6701*Train_time + -7740*Walk_time * 94*Train_time_x_Walk_time"

With this result, would i be correct to say that for 2 houses at 10 mins train ride of the CBD, a HDB further by 1 min would be expected to be 5761 dollars less. While for 2 houses at 20 mins train ride of the CBD, a HDB further by 1 min would be expected to be 4821 dollars less.

Hence, the further we are from CBD, the less one would value the walking distance from the MRT?
@mbq: thanks for the advice. I have edited the question accordingly, hope it is more clear now !
@mbaitoff, I'm not sure it would quite fit your application, but in related application areas, magnitudes of waves that undergo (many) random reflections between source and receiver are modeled by a Rayleigh distribution or one of its generalizations, e.g., Rice or Nakagami-$m$ distributions.
@HCAI: I have added textual definition which may circumvent notational issues.
Reading a step by step derivation from the seminal papers from Saltelli or Sobol' might help you to get a better grasp on the fundamental concepts.
@Anne and others: Since your data are not time series and you're not using a time series procedure please disregard my answer. Others who have observed negative R Squares when involved with time series might find my post interesting and tangentially informative. Others unfortunately may not.
Yes it is likely because of class imbalance. You can also look into various tweaks to random forest for class imbalanced data including using class weights ("Weighted Random Forest") or sampling the data to balance the classes ("Roughly Balanced Bagging").
Have you learned about [Venn diagrams](http://en.wikipedia.org/wiki/Venn_diagram)? Drawing one might help you make reasonable guesses concerning the intended interpretation of this question.
What prevents you from calculating the correlation between each of the substances and the concentration, just like you did for one? By the way, which kind of coefficient do you use to test for the correlation?
People interested in the IRT vs. sum score issue may have a look at the literature, e.g., [Culpepper (2013)](http://apm.sagepub.com/content/37/3/201)
I added the error message I get. Is that because of the NAs? Removing NAs in MatrixA and 0's in MatrixB (which is equivalent) does not yield matrices of the same dim. The str of both is '46175 obs. of  741 variables: with all integers.
FOLLOWING ON FROM PREVIOUS COMMENT........I ended up with different p-values depending on whether I used standard between-group or repeated-measures ANOVA as the test statistic to reveal effects across columns (permutations used were the same for each test-statistic). Here is the simulated experimental data I used (spaces separate columns and colons separate rows): [4 3 6; 8 3 4; 10 4 3; 9 2 4; 6 3 4; 4 6 2; 5 4 5; 8 3 6;7 4 2;3 4 9;8 7 7;5 3 5;4 5 7]
1) So you did a hypothesis test of multivariate normality? Failure to reject normality doesn't mean your data are normal. It just means the sample size was too small to find the non-normality you have. Hypothesis tests of normality don't answer the question you want to answer here.  2) it's not necessarily helpful to formally test normality if you're then choosing what procedure you use on the basis of the outcome of that test.  3) "How non-normal" == What kind of non-normality is present and how severe is it?
I don't know what you mean by a "chi square qq plot for the entire set of variables."  When I first read the question I thought it meant a QQ plot comparing the sum of squares of the values to a relevant $\\chi^2$ distribution, but it now occurs to me you might have had something else in mind--I just don't know exactly what.
Many thaks for your help
@whuber The y axis is a measure of frequency of two levels of a categorical variable (whether an individual earns more or less than 50k). I would be content knowing how to recreate this type of plot.
A normal distribution is completely determined by two parameters, $(\\mu, \\sigma)$. Since you have 5 percentiles specified above, my guess is that no normal will fit preceisely those percentiles. You may try to fit a normal to the data, which is only an approximation, or use a 4-parameter distribution (Johnson, Generalized Beta, etc.) for a possibly better fit. Otherwise, just interpolate, knowing that there is no basis for choosing any monotonic interpolation scheme if you don't know about the distribution.
The "similarity" indexed by correlation refers to the degree to which high/low values in one distribution (relative to that distribution's mean and variance) are paired with high/low values from a second distribution (relative to that second distribution's mean and variance). Since high/low are relative, the two distributions can be very dissimilar in both mean and variance and still obtain a high correlation, so I don't think you can use the correlation as a means of comparing distributions.
Closely related: http://stats.stackexchange.com/questions/26988 (a similar question involving binary strings) and http://stats.stackexchange.com/questions/12174 (the same question, perhaps, involving quaternary strings).
Are familiar with utility from economics?
@JackTanner, The model that I have (MARS with logit function) gives outputs in the range of 0 to 1, similar to logistic regression. It's basically the same, but includes a few more features. To get precision at different recalls, I simply set the thresholds at different points. I just use the standard way to calculate PR or ROC from a ranked list.
@NeilG Thanks - very helpful!
Why the Poisson? That is used for modelling counts.