tttjjj/elt_bench huggingface dataset:
There is also a discussion of the role of $(X'X)^{-1}$ in regression here: http://blog.stata.com/2011/03/03/understanding-matrices-intuitively-part-1/
If the sample is independent binomials with common success probability, you can just combine the observations - i.e. 52 successes in 200 trials. That might at least simplify things a little, reducing the chances of dropping a sign or something.
To be more specific. I have 8 players and only 18 games. So, there a lot of pair of players that did not play with each other and there a lot of pairs that played only one with each other. As a consequence, I cannot estimate the probability of win for a given pair of players. I also see, for example that there is a player that won 6 times in 6 games. But may be it is just a coincidence.
@Michelle  Certain surveys (e.g., NSLY) are longitudinal in design and may be better able to approximate causality. I'm not suggesting these techniques allow one to ascertain true causal inference. More advanced analytical techniques (e.g., matching, g-estimation) can be used to approximate an ideal randomized control trial and potentially reduce bias by functional form of the models specified. Even RCT analysis requires untestable assumptions (see the work of Hernán, Robins, etc. at Harvard) and common techniques (e.g., intention-to-treat) may be inappropriate
Data on real estate (location, characteristics of the house, ...). Target is the selling price
If you see the Related threads on the lower right of the page the use of logarithms has been covered a few times before. In particular see, [In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?](http://stats.stackexchange.com/q/298/1036).
I am not sure if autocorrelation is always on the residuals; I think responses can also be autocorrelated.
I used to work in insurance as well, and I never had major issues with a gamma/log GLM. In fact, it's a lot more stable than the default inverse link for the gamma, in my experience. One practical thing to note is that the gamma GLM gives unbiased estimates on the response (transformed scale), while logging the data and fitting a linear regression doesn't -- you need to add a bias correction.
Thanks Adam, I've been investigating all sorts of possibilities for complex solutions, and missed a common-sense solution!
No. I want to detect a change in mean, but the paper does not say whether the assumption of equal variances is required.
@Tim: No, "plausible" means "believable" or "reasonable". A plausible explanation "makes sense", but it's not guaranteed to be true. Once you dig down, it *could* turn out to not be true, but it's not obviously untrue.
**TRUE** "independent" variables simply do not exist - really think about that.  We simply assume and throw the label onto them that they are "independent" to be able to use methods that have been created in statistics in a practical manner.
@EMS Good point. I added a cautionary note and your link to the beginning of the text. Feel free to edit my text if you feel there is more to add.
@Myaseen208, do you mind expanding upon the link? Links go bad, and one would not even know what you are referencing if that happened.
What is a "measurement"?  Is it (a) a determination that the array has detected a signal; (b) a determination *for each pixel in the array* as to whether or not it has detected a signal; (c) an estimate of total number of photons reaching the array; (d) an estimate of photons reaching *each pixel* (*i.e.*, an image); or (e) something else?
So ML is not defined if I can not re-parametrise the log likelihood in terms of the parameter of interest?
@Didier Good questions. (1) $2\\beta n$ in my examples equals about 0, 6.7, and 13.3.  The distributions have modes at 4, 8, and 15, respectively.  Especially in the case $\\beta=0$ the approximation is pretty far off (more than 1 SD wrong).  (2) The normal approximation @Tomek proposes differs from yours: one approximates each of the $X_i$ with a Gaussian and from that does the integration to estimate the distribution of the range.  The result is decidedly non-normal yet, when $(1/2-\\beta)n \\ge 5$ or so, should work quite well with a continuity correction.
My own favourite strategy is to set to zero the negative eigenvalues and then proportionally correct the positive ones so that their sum = the initial sum of all eigenvalues = the trace of the doubly centered matrix of distances. I've tried 3 or 4 different ways and found that this one approximates the original distances by the final euclidean distances usually better than the other methods.
I believe the point of the question is that this brute-force approach for truly rare events may be impossible to carry out because the events might never be seen in such simulations.
Adding to Affine's answer (+1 to Affine for that): You want the first test. The second test would be for something like seeing if a die roll was coming up 1/6 for each number, or if a coin was coming up half heads and half tails.
You need to calibrate your model, this is a good starting point reference: http://people.dsv.su.se/~henke/papers/bostrom08b.pdf
@Macro It sure is.  I'll edit the post to stipulate that.
@Brandon I've tried few other ideas and it doesn't seem to help. This suggests that this set is just hard enough to make this happen (possibly default is just driven by some unpredictable random factors).
Thanks. What about the problem that Type II assumes that the interaction term is non-existing?

Is it possible to calculate a Type IV ANOVA in R? I couldn't find any information on that.
You should first ask yourself aren't these two measures apple and orange, to combine them. What is correlation between them as compared to the average correlation within them (i.e. between items constituting them)?
I'm guessing that you don't want to predict transformed $y$, right? Effectively, your transformation would have to shrink the distance between $y$'s on your original scale. You create prediction intervals on the transformed scale that have a similar width across transformed $y$ values, but when you undo the transformation, the prediction intervals get stretched onto the original $y$ scale.
Can you provide more details about the cross-classification structure: what are the variables?
Very good points. If anything the results of forecasting competitions would perhaps overestimate the performance of complex methods in the hands of less-experienced practioners. And it would be silly to argue that a method's performing poorly on average in a forecasting competition should rule out its use in a "specific situation under investigation". (I didn't understand what you were getting at with your first sentence though.)
+1 for pointing out the common misconception that the square root of an unbiased estimator of a parameter unbiasedly estimates the square root of the parameter. Btw, Michael, I believe there was a word missing from your second sentence - I've edited it. Please check that I didn't change the content in a substantive way.
There currently is a discussion of sample spaces going on at http://stats.stackexchange.com/questions/64167; much of it is relevant here. Note that you indeed *can* restrict the sample space to exclude outcomes that cannot occur. But that is separate from the definition of a probability distribution (or density) function of a random variable, which *always* is defined on the entire Real line. Sample spaces are almost never sets of Real numbers (except perhaps in math texts): they are sets of *things in which we are interested,* like people, experimental outcomes, or processes.
@Zhubarb, that is true, but in the comments to the question wakawaka says he only has two classes.
Hi Peter. thanks for your answer. I tried to follow and understand the explanation in the SAS documentation but I don't really. I still cannot tell the meaning of the 95% ellipse and as well not sure about the difference of a prediction and confidence ellipse. Isn't the ellipse in a PCA plot not always a confidence ellipse??
Presumably $X(t)$ is the process and the $1000$ exemplifies an unmentioned parameter, the *length* of the realization. But what else do you know about the process? Are $X(t)$ independent of $X(s)$ for all $s\\ne t$?
Welcome to our site!
i have noticed that a similar question is discussed, link below, the answer to which seems to agree with me/howick, however what in particular i want to know is whether the paper linked to in my question is disputing this (and perhaps, if so, why) or if i've just totally got the wrong end of the stick

http://stats.stackexchange.com/questions/16562/how-does-the-size-of-a-dataset-affect-confounding-in-both-randomised-trials-and?rq=1
There are some strong arguments for alternatives, especially censored distributions: this is what happens when lots of people achieve the top score (or near enough to it, up to measurement error). The scores can be viewed as censored versions of an instrument that in principle allows even higher scores, but all scores over 100% are just recorded as 100%. (I don't know if the College Board still does this, but at least once upon a time they internally graded SATs on a scale up to 900, but reported all grades in the 800-900 interval as 800: that's the same idea.)
![enter image description here][1]


  [1]: http://i.stack.imgur.com/a4tL0.png
Thanks for your answer and language check, Nick. This will be applied to real data where the decision impacts on system qualities will be domain expert guestimates in the form of most likely and (as you noted correctly) plausible limits. If I should use a Gaussian as you suggested, how do I compute the parameters here? Clearly, the distribution will be skewed in most cases. Thanks!
I get 9450 also. You need to show the number of possible selections you found for each case, and the general rule you used to get them.
I attached the graph of my data (daily values of DO and T for 365 days of the year) in this link http://tinypic.com/r/2elyqrm/6
Yes that was precisely the point my book was trying to make. The difference between the convergence of the pmf and the respective convergence of the CDF. Thank you.
You have repeated measures, which means you should use a mixed effects model. The pattern in your plot strongly suggests that you need to account for the repeated measures as there are 10 points on each of these "lines".
@jbowman Maybe my terminology is sloppy, but where did I say the prior was on the data? If you can be explicit which parts of the question are unclear, I can try to correct them. "$\\mathbf{p}_{\\mathbf{S}}$ for the marginal prior of p on \\mathbf{S}" is perhaps not clear. I meant the prior on the parameters that live in the slots corresponding to $\\mathbf{S}$. If it is useful/relevant I can give more details about the model, but I didn't want to introduce unnecessary complexity. Thanks.
Not exactly. A jump from 0.001 to 0.3 would be associated by an average increase of $\\log(y)$ by $\\hat \\beta_1 \\cdot (0.3-0.001) =0.4485$. There is no need to talk of relative changes on the log-scale.
Thank you, it makes sense. This was an analogy that I used to describe what is happening in my messaging middleware. There are a number of data consumers attached to the middleware, each being partitioned. The middleware forwards a message to each consumer, in turn, until one accepts the message. There is no way to predict which consumer will accept a particular message.
@Procrastinator There are many goodness of fit tests based on the empirical cdf.  Kolmogorov Smirnov is one.  These test can be used to compare the empirical cdf to any specific distribution (and with adjustments when unknown parameters are estimated before testing.  You are absolutely right that rejecting a distribution for normality for example does not tell how the distributions differ.  But MrBones if he wants to do this formally he can test for significant skewness and then do a KS or chi square test for skew normal.  Adjustments can be made for multiple testing.  Why be smug with the OP?
@Elvis, yes, you can assume that. I think.
I haven't done much with Satterwaite in linear models with unequal variances.  I think if you look at the formula for degrees of freedom it will probably start making sense.
Thanks, it's nice to be here! Yes, I found a 32GB RAM machine and I will apply some feature selection to solve the problem.
Searching for *svm parameter selection* on this site should help you get started.
I wasn't suggesting it was, I was just curious as to why such a difference might have arisen, what sort of level of error following the wrong advice might give and whether there was a decent explanation of the difference I could give to my students.
(+1) You may be interested in the following thread: http://stats.stackexchange.com/questions/3252/how-to-cope-with-exploratory-data-analysis-and-data-dredging-in-small-sample-stud. It seems we share a lot of links in common :-)
"Randomness" is a big too vague. You may want to look at the discrepancy of the series, goodness-of-fit tests and Hopkins statistic, for a starter. However, things that appear random to us, may be quite well ordered, see e.g. http://en.wikipedia.org/wiki/Low-discrepancy_sequence for "sub-random" series, that are more evenly distributed than uniform random is...
So, according to a long lost colleague, the best apropoach seems to be Chiranjeeb Buragohain and Subhash Suri. Quantiles on Streams.
http://www.cs.ucsb.edu/~suri/psdir/ency.pdf

I also like Ian's approach, as these medians of smaller data sets will converge to a normal distribution, and so I can form conf intervals for the medians.
Rather only weak classifiers give a space for improvement. In general strength is an abstract quality and we cannot really measure it. The only certain test is just to make an experiment and check whether ensembing significantly increases performance. If so, classifier was weak. If no, well, we still know nothing.
This question needs more information.  What do you mean by "20% of the time"?  is it 20% of the batches given by 1 supplier are bad, or that 20% of suppliers give batches which are all bad?  And does 1 supplier give only 1 batch - or can they give multiple batches?  And what is the random sample of?  is it of 15 batches from 1 shipment, or a random selection of 15 batches from multiple shipments?
So, having a good understanding of MANOVA might be helpful for us to understand LDA at feature extraction, right?
Thanks for your response. I am using a cross validation technique (repeated measures sub sampling) to estimate predictive accuracy rather than OOB estimates. I have to run 100 different models, so I wanted to make sure I was predicting using the best combination of variables.
Since none of the datasets when analyzed individually would lead one to use random effects, I'm not seeing why random effects are helpful here.  But it may be possible to use generalized least squares to allow each subject to have her own correlation structure.  Unpaired observations would have correlation zero.  Worth thinking about.  Thanks.
@PeterFlom  Thanks Peter, I'll take a look at runs test. Hope other people will join the discussion as well.
Would you happen to be the Peter Flom I knew years ago from playing go? I am on the Board and Card Games site, and occasionally answer Go questions.
Possibly related questions, though not necessarily answers to all your questions: [\\[1\\]](http://stats.stackexchange.com/questions/52131/how-to-update-forecasts-in-an-arima-model) [\\[2\\]](http://stats.stackexchange.com/questions/34139/updating-arima-models-at-frequent-intervals) [\\[3\\]](http://stats.stackexchange.com/questions/9512/how-to-update-arima-forecast-in-r)
@chl I have a few scales that measure different constructs. I wish to show that the subscales that measure construct 1 are different from subscales that measure construct 2 and the subscales that measure each construct measure that construct alone. Does that help?
In many real-life situations, count data is often overdispersed relative to a Poisson distribution, leading to a poor fit. This could be the case here, but I don't know since I haven't seen the data. You could try a negative binomial distribution. http://en.wikipedia.org/wiki/Negative_binomial_distribution
Don't oveelook Section 4.1as well, where the second theorem is for a limited case of nonconvex functions, saying it only converges (with infinite samples) to some point with gradient 0. It may not be global minimum or can even be a maximum. SGD is more interesting for more practical reasons such as distributed learning, not surely that it will "avoid" the local minimum.
@sjoh2574: I don't think you did; Glen_b was probably just trying to clarify that logistic regression was only a suggestion, not a statement of statistical law :)
This is a good suggestion for building models, in general optimized parameters will yield more accurate predictions.  However, in this case the bias exists regardless of tuning, as far as I can tell.  There is only one predictor so mtry must be 1. Any sample size less than n only serves to magnify the bias, and nodesize has little effect.
So is it clear to Frank that we are using the fact that for any constant c Var(cX) =c$^2$Var(x)? Since the sample estimate of the proportion is X/n we have Var(X/n)=Var(X)/n$^2$ =npq/n$^2$ =pq/n and SEx is the square root of that. I think it is clearer for everyone if we spell out all the steps.
Almon, Shirley (1965), "[The distributed lag between capital appropriations and net expenditures](http://www.jstor.org/stable/10.2307/1911894)," *Econometrica* **33**: 178-196.
Is the age variable supposed to be here?
I doubt it. The problem is that stuff is all post-hoc. Consider the famous Feynman quote, "[You know, the most amazing thing happened to me tonight...](https://www.goodreads.com/quotes/649893-you-know-the-most-amazing-thing-happened-to-me-tonight)".
It's not correct; solutions to the problem are given at http://stats.stackexchange.com/questions/65866/good-methods-for-density-plots-of-non-negative-variables-in-r.
If the population standard deviation is *known*, then you can use a $z$-test. As far as I understand your setup, you assume to know the population standard deviation to be $500$. If you *estimate* the population standard deviation by the sample standard deviation, a $t$-test is normally used. In large samples, the difference between the $z$-test and $t$-test are negligible. The critical $t$ value on the linked page is the $0.95$ quantile of the $t$-distribution with $34$ degrees of freedom. The $0.95$ quantile of the standard normal is $1.644$, not $1.96$ (because you have a one-sided test).
@PeterFlom I believe Roulh is referring to the proposal distribution used for importance sampling.
I haven't read the reviews. But often it is an issue of the level of the book.  This is a book at an intermediate level.  Many amazon reviewers that complain could be in need of something more elementary.  I perosnally have the third and sixth editions of the book and think it is excellent at the level of your interest.
@David, I don't recall downvoting or saying this was a bad answer. I'm only pointing out that it's odd that, by design, the wrong model was fit to the data when it would be just as easy to simulate data from the actual model you're fitting. In response to *"does it really matter ...?"*, I guess I'd just say that the function $$\\hat{f}(x) = \\hat{P}(Y = 1|X=x)$$ you've plotted is a biased estimate of the true data generating conditional probability, because the wrong model was fit to the data. But, that may be beyond the scope of the question. Cheers.
You should be very careful in interpreting p-values from these t-tests with this looping approach. If you have 20 columns, and there is no real difference between any of them, at least one comparison is likely to appear significantly different at p < 0.05. You should at least multiply the p-values by the number of tests, to produce the Bonferronoi correction. Or, you could multiply them by their inverse rank for the Holm correction.
Well, the coefficient of determination ($R^{2}$) is often interpreted in the manner that you propose. And for simple OLS (meeting its assumptions) that $R^{2}=r^{2}$, as in Pearson's $r$ *squared*. So I wonder if you are proposing a similar interpretation with Spearman's rank correlation coefficient? Hence, I wonder if you mean to square it.
@Abeer And that's exactly what I have shown for $X/s-Y/t$ and $X/s+Y/t$.
The constants in physical science are not known: practically by the definition of "physical" and "science," they are *measured.* In some cases they have been measured to extremely high precision (13 or more significant figures), but that does not alter their ontological status.  Moreover, it has been established that historically the confidence limits on estimates of physical parameters tend to be (way) too short, which rather emphasizes the fact of their uncertainty!
Can you describe your problem a bit deeper?
Your textbook may be making a distinction that is not universal: for instance, [Karl Sigman's notes on limiting distributions](http://www.columbia.edu/~ks20/stochastic-I/stochastic-I-MCII.pdf) defines "limiting" and "stationary" distributions to be synonymous (definition 2.3 at the bottom of p. 5). Therefore you *must* consult the definitions in your textbook in order to determine the difference.
@varuman - I tend to prefer voting systems when you have relatively large constituencies that are unlikely, unwilling, or unable to engage in dialog and deliberation as part of the selection process. On the other hand, small groups that already have open lines of communication would benefit strongly by attempting to select the CEO by a deliberative and consensus-oriented approach. If the group does end up needing to vote numerically, I would be agnostic about which voting system to use until I know more details about the problem space.
You've done PCA of correlations. What you display as `#Loadings` are eigenvectors, and the values say that the PCA rotation was 45 degrees. Loadings are eigenvectors normalized to column SS=eigenvalues, which in your case give values .872 in 1st column, .490 in 2nd column.
I have added heavily to my answer.. hopefully I've found the issue we're getting stuck on.
Hmmm, I do see that, section 6.6 "SEMILOG MODELS: LOG-LIN AND LIN-LOG MODELS". It's been a long time since I looked at that, I suppose I'd forgotten. Nonetheless, I would avoid using those terms, they really aren't standard in statistics. In particular, people will confuse the log-lin model w/ the log linear model, which are not the same thing at all. I would say, 'a regression model w/ a log transformed predictor' or 'a regression model w/ a log transformed response variable'.
+ forgive me that I am not a native English speaker :)
So far I found out that, glmmPQL uses multiplicative dispersion.
@Can'tTell You can find some help about formatting here: http://stats.stackexchange.com/editing-help
It is pedagogically unsound to divide a cake in unequal parts: all children are created equal and thus they should get equal parts of the cake.  Just kidding. You're right, of course (though countability has nothing to do with it: if you have a noncountable number of children, you can still give half the cake to the first one).
Very helpful, thank you, really appreciated
@Tom, I was mistaken about the package. Evidently, it's the [qualityTools](http://cran.r-project.org/web/packages/qualityTools/index.html) package. Moreover, the example seems to be taken from [here](http://www.inside-r.org/packages/cran/qualityTools/docs/qqPlot).
Just as I thought! Having trending data would make it easy to infer other catalysts such as seasonality and other shocks into determining what threshold is significant enough. Thanks for the feedback.
By error I basically mean 1/fitness, so I'm talking about the value of the best individual in a generation.

I've recorded the fitness value of the best individual for every generation.

So I have 1000*20*2 numbers, each corresponding to the "fitness" of the best individual in a particular generation of a particular run.
so basically XOR