tttjjj/elt_bench huggingface dataset:
Tangential remark: I've always thought that for many purposes and many algorithms 'scale-freeness' obfuscates cluster structure, due to the large pull exerted by hub nodes. It depends on the application, but I would usually look for a way to filter out hub nodes. Have you considered algorithms like Louvain (which yields a tree), Affinity Propagation clustering and MCL? The latter two do not require the number of clusters to be set (but have parameters that affect coarseness). They probably won't work well on a scale-free network though.
@PeterEllis: seems to me that prediction falls in the definition suggested by A. H. Welsh in the sense it is indeed a question about the underlying $F_0$.
http://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d.
ARD is probably going to be a bad idea if you have a large number of features as you are likely to just over-fit the marginal likelihood when optimising the parameters of the covariance function.  Feature selection is generally best avoided unless identifying the relevant features is a key aim of the analysis.
+1 thank you. often I get some "weired" results, for example, a normal distribution gets a higher p-value then a poisson one, where lambda is relatively small (so by looks only the normal and poisson are not similiar at all)
When you integrate $f(\\text{data}|\\text{params})f(\\text{params})$, you are integrating over the parameters and so the result has no term depending on the parameters, in the same way that $\\int_{x=0}^{x=2}xy\\;dx = 2y$ does not depend on $x$.
Thanks for your response. That would help better understand if you have a piece of code.
or `x=c(11,15,9);  z=zoo(c(rbind(NA,NA,NA,x,NA,NA,NA)))` etc. But you'll want to divide `x` by 7 first.
@Torvon, I'm not sure that I completely understand 'information [] about the probability of rejecting the null hypothesis...'. It doesn't sound right to me. Perhaps you mean something like 'In order to show the strength of evidence within the data against the null hypothesis, exact P-values are displayed." The probability of rejecting the null hypothesis is something like the power of the experiment combined with a Bayesian prior, I suspect.
Thanks Bernd. I thought unique can be applied only for a specific column. I didn't know that it can be used for the entire data frame as well. thanks again
If the second is a percentage, why not similarly normalize the first?
@Aniko, thanks, I've updated the answer.
To further add to confusion, note that the notation you used for cross entropy is also the same notation used for the joint entropy.  I've used $H^x(P, Q)$ for the cross-entropy to avoid confusing myself, but that's for my benefit and I've never seen that notation elsewhere.
Thanks! More explicitly, I am working with a training set of N = 5000 observations and D = 200 dimensions. It takes 2 mins to train an SVM RBF.

During prediction on a test set of T_s = 50,000 observations,  it takes about 6 minutes, which is too slow for my purposes. I know I could parallelise the prediction but was thinking if it would be possible to store the full kernel matrix before hand. 

Can you please provide more details on how much space would be needed to store a NxD data set. Will be it N^2 x D? Also does the prediction time scale linearly as T_s increases?
@PeterFlom  Harrell's book was published by Springer. Here is a link to it at amazon.com [Regression Modeling Strategies](http://www.amazon.com/Regression-Modeling-Strategies-Applications-Statistics/dp/1441929185/ref=sr_1_1?s=books&ie=UTF8&qid=1346100618&sr=1-1&keywords=frank+harrell)
+1 I would be interested in learning how anyone can learn about equality of variances (in the usual applications of ANOVA, regression, t-tests, and so on) through consideration of the *means*!  @Gung If we interpret "qualitative" as "exploratory" or "potentially revealing of unexpected data characteristics," it becomes clearer how such a method could supplement--and even be superior to--a formal test.
Homework or "help-me-to-solve-my-problem-possibly-in-the-next-five-minutes" questions are accepted on this site provided you add some context and say what you know or what you tried, i.e. what's prevent you from finding the right solution. Please update your question, otherwise we'll have to close it.
@suncoolsu `ncp` is a function of the parameters (kappa and sigma, through c). No I have not used the the `data` snippet.
(ctd)... that's at least a few keywords to get started with. If you'd like a text that covers some of this, *[Elements of Statistical Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/)* (often called ESL) is widely used, and a (legal, legitimate) free version is available from the book's website I linked above. You might find more elementary treatments better to start off with however. There's a lot of hits under one or another of those keywords here. In your case, it looks like there's a lot of important domain-knowledge that may be important in setting up a classifier.
see also http://stats.stackexchange.com/questions/79149/how-much-do-self-organising-maps-suffer-from-local-minima-problems
You can't speak of "eigenvalues" after rotation, even orthogonal rotation. Perhaps you mean sum of squared loadings for a principal component, after rotation. When rotation is oblique, this sum of squares tells nothing about the amount of variance explained, because components aren't orthogonal anymore. So, you shouldn't report any percentage of variance explained.
@Remi.b Thanks. I removed `as.factor` intentionally and stated that I assumed that `var2` was a factor already - to simplify things. `as.factor(var2)` would create a factor and thus it would be expanded to $l - 1$ columns in the model matrix (assuming the default contrasts and inclusion of an intercept). I wouldn't use `as.factor()` in a formula - you should separate the model fitting from the data analysis code, but people do and it is valid as `model.frame` arranges for the function to be called as it is creating the data from which to build the model matrix.
It might help to distinguish "infinite" from *undefined* from *unbounded.* For example, in the case of a Uniform$[0,1]$ distribution relative to Gaussian distributions, (1) the KL divergence from the Gaussian to the Uniform is undefined (due to the zero values of the Uniform PDF at points where all Gaussians have positive density) and (2) although the KL divergence from the Uniform to any Gaussian is finite, there is no upper bound to the divergence among all Gaussian distributions.
Is this a homework problem? 

If not, how do you know that the classifier is right 60% of the time without knowing the true probability of each class?
Oh! Thank you very much for you exhaustive answer! It helped me a lot but still I am a little bit lost - because I am beginner. Are there any tutorials anywhere to use these methods..?
thanks, that is very helpful!
Was it supposed to have been a trick question? How could sex & smoker status be independent or dependent w/i a single individual? What would that mean?
It's not necessarily detrimental, though. I recall an interesting, and quite old, paper in the *American Statistician* in which the author shows several examples of the main effect being hidden in the smallest principal components. Unfortunately, I don't recall either the author or the date. It was probably late 1970s or early 1980s.
[tag:effect-size] is a very broad class of statistical estimation. It's not a bad potential answer to your question though.
To make sure I understand: you are projecting the end-of-fiscal-year balance each month and obviously the projection becomes more accurate as the fiscal year progresses (and you have fewer months to project to the end of the year)? And you want to determine how far into the fiscal year you have to be to have a "firm" estimate of that year's ending balance?
.. and it is not the best place to comment the question ;)
@K Hein 1) I suggest to use Random Forests (RF) instead of DTs.  See e.g. http://stats.stackexchange.com/questions/10001/implementations-of-the-random-forest-algorithm. 2) numeric variables: RF can handle both numeric and discrete labels, you should try both approaches; director_id,actor_id is not a numeric feature, it is either a boolean (actor participated ?) or a nominal (main actor); budget can be discretized or let RF handle them. In this case the algorithm searches for the optimal split point. I suggest to play around and come back later with more specific questions ;).
As per the help page (`?arima`): "If supplied, only ‘NA’ entries in ‘fixed’ will be varied.". So `fixed=c(NA,rep(0,9),NA,NA)` means that the 2nd through 10th AR parameter are fixed to zero, whereas the 1st and the 11th AR parameter as well as $\\sigma^2$ (which is the twelfth model parameter) can vary. You could in principle also fix parameters to any other value. Of course, if you have an ARMA(p,q) model, your `fixed` needs to be of length p+q+1.
For quartz, it's faster to draw a white rectangle over the existing plot and then just redraw the lines.  See `tourr:animate_dist` for an example of this approach.
Is that community special in some sense (other than the high proportion of missing data)?
I just realized, since this is rather a programming than statistic question, maybe the question should be moved to Stack Overflow. Shall I delete it here and repost it in Stack Overflow, or is it possible to move?
@whuber: Thanks for the example. I actually have 14 subjects in each group, leading to 91 unique correlation coefficients within each group. Does it make sense now to compare the "average" correlation between the two groups? Thanks again for your help!
I just proposed a very simple explanation of why a conditional distribution is needed. If you notice a mistake, please let me know.
Averaging $R^2$ isn't something that you'd want to do. I guess that I don't understand what you are trying to achieve.
It's also possible for the correlation over 6 years to have the opposite sign of the correlation in each of the years. See Simpson's paradox. http://en.wikipedia.org/wiki/Simpson's_paradox
OK. From your last plot, it seems you get no fitted values about 1 or 6, 7. If that is a problem, ordinal regression might be better. Or it might just be because those values 1,6,7 are few in the data.  Otherwise, your linear model might be OK.
But it's log can be negative. That's the context
thanks for the update simon !
Hi Matt, would you care to a more self contained description of how the method works?
Interesting ! Does this distinguish multiple trend lines or does it assume 1 trend. How does it deal with the third row third column example from the OP. Do you have to manually ignore the anomalies after the visually obvious floor trend line ?
In terms of the random effects, yes, because only an intercept term is specified in `random=~1|subjectid`. In fact, there are no slopes in your model (fixed or random) because you treat timeslot as a factor.
Instead of passing 1 as the first argument, did you try passing `length(a)`?
How is the third definition rigorous?  And what is the problem with the sample size not being defined?  Since we say $P(x_1, x_2, \\dotsc, x_n \\mid \\theta)$, which naturally brings into existence a corresponding sigma algebra for the sample space $\\Omega^n$, why can't we have the parallel definition for likelihoods?
Per the Wikipedia link, the *exchangeability* of observations is a requisite assumption of permutation tests, and different variances between groups violates this assumption.
Have you ever heard of the cross correlation function - http://en.wikipedia.org/wiki/Cross-correlation#Time_series_analysis?
I did not know about this. Now I can manipulate the `h$count` vector to my nefarious purposes. Marvelous. Thanks
Among other things, i want to simulate the Dickey-Fuller distribution by montecarlo methods in order to compute critical values.
For the curious: the figure can be obtained in R as follows: `x <- dbinom(0:2000, 2000, 1/12); y <- cumsum(x); plot(diff(y**12),type="h",col="blue",xlim=c(160,220)); points(diff(y**12),pch=19,col="blue",cex=0.8);` whuber, I think there’s a tiny mistake: I find p = 2.2% with `sum(diff(y**2)[204:2000])` (you have to start at the 204th element, because the first element if 0; if I start at 203 I find 2.8%). Do you know how to compute this without making the independance assumption?
Nothing in what you wrote provides any indication that the method is safe from overinterpretation (a form of overfitting).  One reason I worry about the method is that it doesn't treat continuous variables continuously.   The problem with Fisher's test is that the P-values are too large.  See for example Crans GG Stat in Med 27:3598; 2008.  The word "exact" has misled practitioners for decades.  The ordinary Pearson $\\chi^2$ is usually more accurate.
What exactly is a "Gamma(0,0)"? Perhaps you should cite your reference--and preferably provide a link to it.
Welcome to the site.  You should know that it's going to be a challenge for anyone to really help you without more information on your research question and especially your methods.
Please provide the example. Now it is unclear what is your actual problem.
just adding to @JMS answer, a great reference to check out about is **Bayesian Inference in Statistical Analysis** by Box and Tiao. It presents conceptual ideas behind it too.
?? This seems to have nothing to do with stats.  (The curve is modeled after energy potentials in physics, not after anything in stats.)
That makes sense. But how do I get a comparison of means (for intended infidelity) based on male and female sensation seeking scores? All the Regression Equation gives me, is the change in slope for females as compared to males. Of course the R-squared change from adding gender to the model later is useful, but I would like to just compare the means for male and female sensation seeking scores. ANOVA doesn't let me because they're 2 separate variables. Btw thank you for answering my question! I am new here, and for some reason Cross-Validated never sent me an e-mail when your response came in.
Ok, the model is little better than originally proposed, but similar to @javlacalle.
@Chi On one level, yes, undeclared and uncorrected interim analyses are inappropriate (but it is done in ignorance, an ignorance that I believe points to inadequacies in the methods of teaching statistics to basic biomedical researchers...). However, if we consider it at a meta level, then it is possible to find some partial justifications. Many experiments involve such small samples that an increased false positive error rate may be a reasonable tradeoff for more power. Convention precludes a declared level of alpha higher than 0.05.
Also I wish to know, whether a paired t-test performed on each of the groups is to be retained? or is not an appropriate tool?
I believe the 'discrete' description refers to the fact that draws from a Dirichlet process are discrete with probability one (it follows from the stick breaking representation of the DP).
FWIW, I have a fuller treatment of essentially the same issue here: [Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?](http://stats.stackexchange.com/questions/78828//78830#78830)
The apparent circular reasoning is why it was nicknamed the bootstrap.  It felt like people were trying to lift themselves by their own bootstraps.  Later Efron showed that it really did work.
I guess I got it. I would use survival analysis if I wanted to know if users A stop contributing earlier than users B, if I knew when they stopped contributing. Is that correct ?
Are you truly interested in the linearity of the *data* or in the linearity of *the underlying curve*?  The distinction is that because the data (may) sample the curve unevenly, a measure based on the data would vary according to the nature of the sampling, whereas an estimate of the linearity of the curve would be more stable under changes of sampling program.  Also, do you conceive of "linearity" as an absolute property (and thereby dependent on the units of measurement) or is it a property of the *shape* of the curve (and thereby invariant under affine transformations of x and y)?
@user603, I've never heard of FA (not PCA) with n<p. I even doubt such algorithm exist or is needed.
Perhaps you could give a reference to where it's called linear and also say why this does not make sense to you?
Would it be better to say "the reciprocal" instead of "1 over" to account for precision matrices?
Logistic regression?
@Alexandru, We are happy to _bear_ with you -- that's what the site is for -- but out of modesty we may choose not to _bare_ with you. (For that, the internet offers, I believe, alternative venues.)
`with all 13 products as break` This is right advice only if the OP meant: combination V1 V2 V3 _is_ "these 3 all=1 _and_ the rest 10 variables all=0". If he meant only "these 3 all=1", your advice won't give the wanted result unless a postprocessing on it is done. (As for me, it is so far unclear what Michael wants.)
Usually the intent is to make the (asymptotic) variance independent of the parameter of interest. This is particularly important in inference where we need to know the reference distribution to calculate related quantities of interest.
I would say the opposite - it has special meaning (the S3 dispatch you mention), but some old naming conventions caused function names that are not S3 generics to have names that include a `.`. That pertains to names of functions. As for names of (non-function) objects, then yes, there is no special meaning.
All good points (+1)!
Some comments by Frank Harrell (http://stats.stackexchange.com/users/4253/frank-harrell) on why stepwise regression is bad: http://www.stata.com/support/faqs/statistics/stepwise-regression-problems/
Thanks, @whuber!  This is exactly what I was looking for, and my atrophied brain was unable to get to. As an aside, the motivation for the question is that for various reasons we essentially do not have the full $X$ available, but have `cov(z)` from previous calculations.
@Joris Good point! I forgot the sample size issue when I initially wrote my answer. The GLM would certainly yields poor estimates... Re-randomization test should be better, you're right. Thanks!
Have you tried http://en.wikipedia.org/wiki/Sample_size_determination
Are your three Facts the underlying truths which were unknown to the researchers?
Good suggestion @Glen_b....now I see your confusion. I've edited the question.
I know MAD (http://en.wikipedia.org/wiki/Median_absolute_deviation) is used a lot for wavelet shrinkage.
@coffee F-tests are almost always one-tailed.  The reason is that the null hypothesis of equal coefficients means that all variation is random, whereas under the alternate hypothesis there is an added component of variation arising from differences among the coefficients.  That would cause the numerator only to *increase.*  The other (left) tail corresponds to unusual consistency among the data.  Fisher famously used it to suggest that Mendel's pea data were a little too good.
Hi Michael,  Your method of bounding the CI for the variance does not give the numbers in the example as near as I can tell.  Did you compute this?
I love complicated alternatives! You can always account for more variables (e.g.: age), construct and fit a complete model (including occupation, gender, age, education etc.) and calculate the effect size or marginal means of gender.  Even fancier, do a random forest regression to calculate the marginal effect of gender. :-) or just look at the difference in means.
I can't see that zeros in the independent variables have any bearing on the merits or use of Poisson regression.
The IMS now says they posted all published articles on arXiv. Of course, this claim is difficult to verify.
@Michael, I do not understand why your statement is true. A p-value less than .05 will occur under the null hypothesis 5% of the time *regardless* of sample size. So how can it be possible that larger sample sizes will solve this problem? It seems to me your recommendation implicitly invites readers to confuse the size and power of hypothesis tests.
(+1) "Should only increase" is key: you are asking about how to perform (robust) *monotonic regression.*  It would help to emphasize that point more in your question: you will get better answers.
@Gabe There is no mention of infanticide in the question, it is a mathematical excercise as opposed to a gritty analysis of a real country where murder is common place. Equally the real ratio of births of boys to girls is closer to 51:49 (ignoring social factors)
thanks for the extra info. From what i understand of the OP's description the answer should be a vector with 33*32/2 positive numbers in it... –
In what sense do you mean "increase with sample size?"  After all, $\\bar{X}$ changes with sample size, too: you can't hold it fixed while increasing $n$.  To put it another way: $t$ is a *random variable.*  Therefore, for any fixed population, you obtain a *sequence* of random variables, one for each $n$.  Could you clarify what you mean for this sequence of random variables to be "increasing"?
As Dilip points out, how you generate the plots is dependent on the kind of plotting software you're using.
The conditions I have are gene expression measurements from disease and normal tissue. Ideally I would like to obtain a measure of "differential" expression.
Note that $-99$ is often an automatic value that is used by some software (SPSS, I think) to mark missing data.
I'm kinda nervous because all of this is based off of an ad-hoc literature review I did.  I am an embodied cognitivist, so while I get the gist of how thought and language work, I only know a skiff of computational linguistics.  I hope someone else can step in and validate this.
@Derek see my updated answer. Sorry, it's getting quite long now. You might find my answers to Fly interesting; especially (i) there doesn't seem like there's likely to be all that much of a power difference between t and Mann Whitney; and (ii) consider that you *might* be better off with a two-sample Kolmogorov Smirnov test, though it's a general test for differences, it is sensitive to location-shift and scale-shift alternatives; the tendency of that test to be conservative in the presence of ties can be dealt with by sampling the permutation distribution. Check out the references there.
At present, I think this question can remain closed. If you can clarify how your question is distinct from what's in the comments here, in the linked thread, & already covered on the site, maybe we car re-open it for you.
It was only possible to add the reference that contains this rather long derivation - two lines are not enough to explain the differences. (And this would also be a little bit beyond my knowledge of such calculations.)
Hi there. I'm going to use SPSS.
Depending on the meaning of the `i` variable, this test might not be applicable.
It's not the best reasoned answer (and I have nothing substantive to back it up), but it's working for now.  If anyone has a better one, please let me know and I'm happy to open it back up.
@user13907, that's not just you. This is indeed a good answer that deserves to be up-voted, but it would benefit from having a nicer tone.
if you could provide example how to write and read the data with R XML package it would be great.
@SamuelBenidt Thanks. For asymptotics to definitely apply, you need $n\\to\\infty$. In general we have $n$ smaller than that (it's not safe to assume n is large here). Sometimes even surprisingly large $n$ can still have the asymptotic results not hold. If we assess that the distribution of residuals is perhaps not too strongly non-normal then it might be reasonable to ignore that it's not actually normal (i.e. to use the asymptotic inference that would result from applying CLT+Slutsky, say, which would for example make our t-statistics go to $z$ for a wide variety of conditional distributions).
Exponentially weighted moving averages are a special case of a kernel smoother (assuming you used a 2-sided MA rather than 1-sided). Better estimates that are generalizations of this are loess or splines -- see my answer.
This sounds like a different approach to my original problem and is in fact very helpful. I will look into it, thank you :)