tttjjj/elt_bench huggingface dataset:
+1. +2 if I could. I just had a chance to glance at the first linked paper and it's very similar to Mark Fox's problem. I'm a novice at statistics, but the paper looks well-reasoned and the environmental dataset it uses may well be helpful to Mark Fox as well. (Naive question: people talk about "only positive data", but I wonder how often wildlife studies actually have negative data. How sure can you be that there are NO deer in those woods?)
Nice! If you calculate the density "by hand" you can get the lines in the right places `d <- density(vBetaSim[abs(vBetaSim)<=5]);
ggplot(data=data.frame(x=d$x, y=d$y), aes(x=x, y=y)) + geom_area(fill="lightblue") +
  geom_vline(xintercept=c(0, d$x[which.max(d$y)]), color="purple")`
The qt() function does this.
@ralu do you have even read the question ?
@Etienne I added details you asked for to the bottom of this answer http://stats.stackexchange.com/a/48859/3277. Thank you for your generosity.
Does the Wilcoxon test better handle the zeroes that will occur in the higher ages, or do you recommend it over, say, a Chi-squared test, for other reasons?
If I use the absolute value between the scores, |PostScore - PreScore|, then I use an independent t-test to evaluate if the absolute difference is different from zero, can I say that I examine the magnitude of the effect without consideration of the direction of the effect?
Bagging can improve the standard errors, but I am not sure it completely deals with multicollinearity (perhaps it can help: [here](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.1557&rep=rep1&type=pdf) they indicate it can help with cases of many variables, but do not explicitly touch upon multicolliearity or highly correlated predictors). Other places I checked, like [here](http://learnitdaily.com/six-ways-to-address-collinearity-in-regression-models/) seem to know bagging, but advocate other approaches for collinearity.
Hi Tim, and thank a lot for your answer :)
This chart is wrong, correct percentage is about fifty-fifty.
One thing to be aware of here, which became apparent when I aligned your code for easier readability, is that for your updated example, the vector you use to populate the initial matrix `test` is not an even multiple of `ncol`, so presumably R is doing some recycling to fill out the matrix completely.
Well, that pretty much answers that :-) Thanks for weighing in!
If you are comfortable solving the same problem with ordinary (least squares) regression, then why not just express the response as log odds (which is just $B_0 + B_1X_1 + \\dots + B_nX_n$) and use the techniques you know?
Hello Charlie Parker, I am a little confused by your question, but in regression/classification contexts as the one in your cancer example, we always have observations $(Z_i, X_i)$ at our disposal, ie. observations of both the outcome and the inputs of observation each $i$.
Thanks for the explanation! I did not know about ClustalG.
Much depends on what you are trying to accomplish with this analysis: are you trying to make predictions about incomes of other families based on educational attainment and family size, or trying to make inferences about factors that are "important" in determining family income? Some cautions in either case: this doesn't seem to be a very large sample, there are many other variables that are missing (like adult ages, geographic locations), and this particular data set might not be representative. And for inferences, I'm sure you know that correlation doesn't imply causation.
$i=1(1)n$ means $i=1,2,\\dots,n$
Can you provide some more information about exactly what your data is and how you plan to use HMMs to classify?
If A, B and C are separate groups (not levels of a continuous variable) then you can't really draw lines between them, or place them equidistant on the x-axis.
There's no change. In GLS the objective function ("SS") becomes a little different, but this approach of factoring the optimization into two parts still obtains values of $\\hat{\\theta_i}$ that minimize the objective.  After all, the values of $\\hat{\\theta_1}$ and $\\hat{\\theta_2}$ are correct because they are obtained by GLS itself, and if the value of $\\hat{\\theta_3}$ were wrong, then it would not minimize $\\lambda$.
That F is not exactly zero, but it is very small, about $2.6 \\times 10^{-6}$. The means are *very* close together compared to the variation in the data.
@Zen thanks.  The OP is have a difficult time explaining his problem.  I feel that if I can understand what the real question is I can solve it easily. So I tried to help him explain the problem.  Based on his responses to my queries my answer is given based on my best guess about the problem.  But I am still not sure I got it righr.
(+1) It's not even clear that plant height *is* the dependent variable. When relating chili-pepper plant height to number of chilies, I'd have thought number of chilies would be the dependent variable.
I give a +1 to mbq for enlightening me about things I never knew before.
The differences between observed and fitted values are available through the [`residuals` command](http://stat.ethz.ch/R-manual/R-patched/library/stats/html/residuals.html) in `R`.  Use `cbind` to join them to the original dataframe.
It occurs to me, you've set specific types here. Those numbers are supposed to be factors. You should probably enter `dput(theDataFrame[1:30,]` in R and paste the result into your question. That way people writing answers will have exactly what you have. After that, the only code you need to include here is what you've tried for correlations.
I think that by "sample frame", you meant "sample". There is nothing random about a sampling frame. See,e.g.  http://en.wikipedia.org/wiki/Sampling_frame
In the 2 d case, suppose all the points lay on the line y=x. Then the first eigen vector will be the line y=x. There will not be any other eigen vector since there isn't any unexplained variance. Or if the values were close to 0, then the loss of variance will also be low
I'm still struggling to understand equations (biologist ahoy), which is why I turned to the community here, hoping it will help me explain the difference in layman's terms.
+1 for this nice, readable and freely available resource!
Since they are count data and non-negative, what about quassi-poisson or a negative binomial regression model, that accounts for dispersion?
The technical name of "one hump" is *unimodal*
Note that you are only rejecting at a specified significance level.  You can still reject both hypothesis by raising the significance level to 10%.  you would also fail to reject both if you lowered the significance level to 1%.
But if I use a sliding window of only 30 seconds for instance, the values fluctuate more, but it seems that the small window nevertheless approximates the values of the large window very well.
@Dikran - Thanks for understanding!  I didn't feel "opponent" was exactly the right word, either, but I stuck it in there anyway partly just for fun, and partly because I couldn't think of a better one that still preserved some sense of opposition.
This estimates the standard error of the "accuracy" as measured from "total trials" draws from a Bernoulli variable.
This question is **[crossposted](http://math.stackexchange.com/questions/25546/algorithm-to-dynamically-monitor-quantiles)** on math.SE.
I somehow missed that! Dr. Williams' answer (http://www.stata.com/statalist/archive/2013-04/msg00950.html) avoids all my nonsense with preserve/replace/predict/restore, but the basic idea is the same.
Could you please explain why you cannot use the "usual approaches"?
This question suggests logistic regression. To be cautious of confounders, you could include not only the treatment but also history of drug abuse as predictor.
My personal view is that we should not mix comments and question. Comment are made to give feedback or to ask questions about the question, most often it is a dialogue between two people. The question should be something clear, easy to read and understandable by as much people as possible... it's you explaining your problem to get an answer.
Nowadays formulas like this would drop out of a *linear* regression of log(weight) against log(height), which (for biological and statistical reasons) is a more natural way to analyze these quantities.
(-1): this is not an answer and it is not even funny. The OP provided an example for the general task of automatically inferring an order from a set of examples and did not ask for solving the example problem only.
"The way I conceptually think about the theorem is that samples of size n will approach normality as n increases to infinity" is plain wrong. It's a theorem about a standardized mean.
@Nick I should have said the variance tends to be inflated more than the covariance thus the larger addend in the denomoinator leads to a smaller ratio (the acf).If you can help with a proof of this assertion that might help.
Clustering techniques will always make clusters and the strength of evidence (e.g., "significative") depends on the amount of data points and not (only) on the importance of the separation. Is this an indirect analysis? (you did a PCA to retain most of data variation in less dimensions and then clustered the data points with the first two axes). Then you have to be aware that: (1) goal of PCA is different than clustering (does not maximize the degree of separation); (2) you are clustering with "half of the evidence" [very roughly speaking] you had at the beginning in your multivariate data set
Smoothness has a well-defined meaning in the theory of stochastic processes. ("A variogram is a statistically-based, quantitative, description of a surface's roughness": http://www.goldensoftware.com/variogramTutorial.pdf, p. 16.) Smoothness is related to the *extrapolation* of the variogram to zero distance. (The SD of successive differences and the lag-one autocorrelation are quick and dirty versions of this). The essential information is contained in the coefficients of the Taylor series at 0. E.g., a non-zero constant is rough indeed; a high-order zero at 0 indicates a very smooth series.
Reminding that I am a learner. The mention of linear regression here seems to mean that the two sets of values that I am about to plot are related which is true because I am plotting CPU usage and a througput number. So if the no: of bytes increases the CPU usage may increase. Both are postitive values. Apology for alluding to capacity planning and not mentioning this properly . Both R code samples( ocram and user603 ) have negatives and the values are not continuously increasing or decreasing.
My target is to reduce the number of local optima.Yes if the smoothing is done in a wrong way then the solution can be bad. I had a look at "Local Optima Smoothing for Global Optimization" by [Addis] [Locatelli] [Schoen] and a combination smoothing "Search-space smoothing for combinatorial optimization problems" [Schneider] [Dankesreiter] [Fettes] [Morgenstern] [Schmid] [Singer],I am not sure if they will help. 

SA will help with skipping local minima, but instead of going the SA way directly i want to try out if smoothing really will work in my case, or may be some other method.
Well, that's a good question. What I didn't understand is how the information of the "real-world data" could change the way you set the "nu" parameter. 
As I see it, you need to expect that your training will generalize and, in that sense, the parameter nu is related to how many outliers are going to be permitted (or, it's a upper bound in the fraction of training points outside the estimated region). Nevertheless, a small "nu" indicates a bigger region to be estimated. 
(http://www.springerlink.com/content/978-3-540-34628-9/#section=496010&page=3&locus=9)
What you say may work in this sense.
I thought about such an approach, but this would mean that we assume a-priory that 5% of the existing observations are "atypical", "faulty" etc. Which might not be the case
You can get a nice visual of the hierarchy at Ben Fry's [zipdecode](http://benfry.com/zipdecode/).
Small errors of that scale (the absence of the absolute value) happen not all that infrequently in journals - outside stats I've seen far more obvious (and more egregious) ones. As for the notation, it's actually fairly common; [e.g.1](http://www.business.illinois.edu/Working_Papers/papers/01-0116.pdf), [e.g.2](http://arxiv.org/abs/1202.5093), [e.g.3](http://www.academia.edu/2812096/Percentage_points_of_rootb1_skewness_statistic_for_small_normal_samples) ... I could point to dozens
Are you looking for a mathematical characterization of such classes or are you inquiring about which of the generally known parametric families of distributions may have this property?
@whuber: there seems to be various ways to refer this situation (bivariate, cross, multivariate, etc.) and, with apologies, I tried to hit them all. Hopefully the edit will clear up the confusion.  I believe you have referred to it as 'cross-K' in a slightly related posting.
Could you expand on how this concept provides an answer to the question?
I think the cstheory guys are likely to kick it on to cs.stackexchange.com. But why move it? Are statisticians not concerned with the nature of randomness? Do we not use random number generators?
One consequence of the multiplicative error structure is that variance will be proportional to mean-squared, while the additive error would have variance proportional to a constant. Even ignoring other effects, this clearly weights the data differently.
I am not sure. I don't see how to separate it out, in general.
+1 comprehensive.
Thank you so much for your help and your insight.
Do you simply want to calculate SSE or also different types of measures for the Cluster Cohesion?  A quick idea might be to obtain a sample from you data and calculate the SSE (or other measure) based on that sample - would such an approach be ok? A better description of _what_ you want to achieve would help - do you want an estimate before actually identifying the clusters? or do you want a performance speed up?
See my answer in this thread:  http://stats.stackexchange.com/questions/54693/power-for-experimental-design/54717#54717

Basically in that framework, you'd be making p = 8/108. I don't think your colleagues are correct.  Basic algebra allows you to solve that equation for the minimum detectable effect that you are powered for at your sample size.
@robin Consult http://en.wikipedia.org/wiki/Cauchy_distribution#Properties
$z$ would no longer be normalized.  In fact, $y$ would not be normalized, either, in the sense of actually spanning the interval $[0,1]$.
Why `residual_variable <- resid(noise.lm)` is not suitable for you? If observation numbers really bug you, just remove them, `names(residual_variable)<- NULL`. Or convert to data.frame: `data.frame(residual=resid(noise.lm))`.
I am not sure what you want to do would be an accurate measure of the bandwidth. The bandwidth represents how much COULD be transmitted down the connection and not how much actually gets transmitted. If you want to get a measure of average tranfer speed then yes, averaging would be the way to go. Regardless, I would prefer to supply mean, median and a std deviation together, then you summarize your data better
+1 for the edit: these are valuable comments and it's especially useful to see the formula for $F$ appear.
(1) You have specified an exponential variogram, not a spherical one. (2) A variogram--as your formula for $y$ explicitly shows--is a function of *distance.* Therefore it makes no sense to ask about calculating its values at specific points. It is defined only for *pairs of points* via their mutual distances. Perhaps now you can return to your references and re-interpret the kriging equations to answer your last two questions (which are actually identically the same: finding the OK prediction amounts to finding the weights.)
@Glen_b Thank you for your comments. Because there is so little space, I have only been trying to provoke some thoughts. There is a long history to the Normal distribution, for instance, and for a long time the convention was $\\exp(-x^2)$, but it changed. I suspect the convention for $F$ was likely established by Fisher from his work on ANOVA and his seminal 1925 textbook, but that's just a guess.
"*I would like to compare these two sets of data samples and determine whether the null hypothesis holds.*" -- but you haven't stated one, nor alternatives you want power against. What is it you want to test for?
@DmitryLaptev Please refer to: H. von Storch and F. W. Zwiers (1999), Statistical Analysis in Climate Research, Cambridge, England: Cambridge University Press.
Ah, so it's exponential. Thanks!
Thanks Egon. SOM approach might be very interesting indeed. Will have a look at it.
An *offset* is a *variable*; that is, you have different values of the offset for different observations. An *intercept* is a *constant*; it is just a number. The log of a number is just another number, taking the log doesn't add anything meaningful; you could just as well subtract 2 from the intercept. It may help you to read my answer [here](http://stats.stackexchange.com/questions/71832//71840#71840). Although it's not specific to your question, it does talk a little bit about Poisson regression & may help w/ general understanding.
Have a look at this thread on R-Help: https://stat.ethz.ch/pipermail/r-help/2008-December/thread.html#182806 (particularly the message from Achim Zeileis who provides code to do what I think you're trying to do). It doesn't look like standard errors have been implemented into the `predict()` function for `zeroinfl()` at this time.
There is most likely an assumption that the events $A_i = \\text{"I have an edge with neighbor}~i"$ are $n-1$ mutually independent events, making the degree a _binomial_ random variable with parameters $(n-1,p)$. The result then follows from the properties of binomial random variables. If you want an _explicit_ derivation, use the result that $$\\text{var}(X) = E[X(X-1)] + E[X] - (E[X])^2$$ since $E[X(X-1)]$ is easier to compute for a binomial random variable than the $E[X^2]$ needed for applying the standard result $$\\text{var}(X) = E[X^2] - (E[X])^2.$$
@Anony-Mousse-is there a "standard" preprocessing procedure to follow before applying clustering algorithms? I mean, something different from a purification of the data and standardization (if this makes sense). If yes, have you any reference? Thanks!
Hi Glen, I am fitting my datas with mixed linear model using SAS's Proc Mixed (with assumptions that data are normally distributed). My experiment design.We would like to see if the stand time for each pig differs in the different treatments (3 treatment groups) with 20 pigs in each treatment. Q-q plot has a curved pattern
Coefficients of variation (CV) and confidence intervals (CI)
Some context would be nice. What are we looking at? Count data? What's the model? If the priors are based on the observed numbers, what makes them prior?
You can certainly frame a set of hypotheses each tested by contrasts (non-orthogonal) and test each of them, though many people would take account of the multiple testing in some way so as to reduce the overall type I error rate.
You can code each of the categorical variables 0-1, then you can simply multiply each combination of variables to get interactions.
The *Mathematica* help page has a nice illustration with explanations: http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationNewtonsMethodRoot.html .
@Colin: The author of the blog would need to put `<link rel="alternate" type="application/rss+xml" title="RSS" href="http://cscs.umich.edu/~crshalizi/weblog/index.rss" />` in his the `<head>` section of the HTML document for Firefox to pick it up automatically :)
I edited the post with an reproducible example.
+1 "*how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place*" pretty much sums it up. Don't go looking for a some magic number that you can somehow use to justify a given result without actually *interpreting the result*.
For the purpose of comparing shapes why not look at the distribution of proportions for each.  That puts them on the same scale.  the problem was differences of total counts so dividing by total counts seems to eliminate the problem.  Other normalizations don't seem to me to directly eliminate the problem and different normalizations can lead to different shapes.
$f(x)=x^2$ and $g(x)=-x^2$, $x\\in \\Re$. Both have an optimum at $x=0$, for the first its a minimum and for the second a maximum.
I was under the impression that PCA can eliminate latent correlations as well as allow you to drop the dimensions that contribute least to the overall variance.
It works both ways. When a list of tuples is passed it fits a sklearn.preprocessing.LabelBinarizer to it. You know a few of the algorithms work in the multiclass multilabel case. Notably RandomForest.
@Peter Flom- I'll check your idea about the scatterplot. In the meantime I did the olr is SPSS and it didn't reject the assumption. Moreover I found on the net a pdf by Agresti where he faces a similar problem and his answer is the following `The improvement in fit is statistically significant, but perhaps not substantively significant; effect of dose is moderately negative for each cumulative probability` which I think is the case in my problem since in both models the Ki67 coef. are similar. Also could you explain to me what do you mean by saying rescaling of the the parameter estimates?
I hadn't considered that (10k term vectors is actually a bit much). Does this hold even for sparse data? You can expect most features to be zero for most instances, and only selected features to be correlated at all...
There appears to be considerable confusion in the question and these comments about what a "variance" is. In Glen_b's examples, the *blue* data have greater variances than the red data around the two apparent peaks (near x=10 and x=17), because the blue data swing more between *low* and *high* values (which are plotted on the vertical axis, not the horizontal, which apparently represents *time*).
I think you're over-complicating this. If you just want to know the approximate distribution of the data, why not just plot the histogram and do a curve fit instead?
You may also want to read this: [why-levene-test-of-equality-of-variances-rather-than-f-ratio/](http://stats.stackexchange.com/questions/24022/).
I'm not so sure about this approach.  Perhaps it depends on how you do the multiplicity adjustment.  If it's something like the Bonferroni, then it seems to me it won't work well.  Consider a case where one of the probabilities is extremely small compared to the other two.  The decision then comes down to a single comparison, because the comparison with the low-count outcome tells us almost nothing.  The multiplicity adjustment ought to have (almost) no effect in such a situation.
Also, if you've found my answers helpful, you can mark the question as accepted (click the check mark next to my answer).
Does that mean that the values of a1, a2, ... an and of b1, b2, ... bm are all binomial?
http://upload.wikimedia.org/wikipedia/en/math/d/1/f/d1f59bcec903553cb6aa1d4d85866561.png - both for covariance and for variances of series.
My intent was to compute mean for first series only once and use it every time when it is needed in computation of correlation. However I need not save anything except Pearson correlation coefficient.
No, each testee has to guess/predict only 10 coin flips, and there are only 1024 possible patterns of H and T that he can call. It is extremely unlikely that he will call 10 Heads or 10 Tails or HTHTHTHTHT or THTHTHTHTH since these are "too obviously nonrandom" (this is a psychological,not a statistical, fact)
@gung I read the question so that the L1 loss does not refer to the coefficients (which are penalized by L1 in LASSO as you) but to the residuals, so that we basically try to minimize the (weighed) sum of the absolute values of the residuals and the absolute values of the coefficients. But the question seems to be ambigious.