tttjjj/elt_bench huggingface dataset:
http://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia
So if you get predicted probabilities from RF you can use `val.prob` for independent sample validation or as part of a  cross-validation loop.
Any comments or suggestions? I would really appreciate your kind reply on this.
From what I gather, I would run the `fitdist()` on the sample, then run an AIC on the output of the `fitdistr()`. So I'm not sure why I'd need to bootstrap the KS test if I am no longer using it?
The good ol' Berry-Esseen Theorem - Now on [Wikipedia](http://en.wikipedia.org/wiki/Berry%E2%80%93Esseen_theorem)
See http://stats.stackexchange.com/questions/1576/what-are-the-differences-between-factor-analysis-and-principal-component-analysis
You and your client(s) will be unhappy with the answer: assuming you can supply the information needed to solve this problem, then minimizing the control group size is likely to lead to a prohibitively costly test group size. It's a fact that the variability within the control group is just as important for determining statistical significance as variability within the test group. Thus this "business requirement" is counterproductive: you ought to work to replace it with a statistically effective requirement that actually works for the business objectives rather than against them.
@ttnphns Thank you.
Clustering first gained notoriety with the so-called "Moulton critique" discussed here: http://www.sciencedirect.com/science/article/pii/0304407686900217. A more recent example of the danger of not-clustering was presented  in: http://www.nber.org/papers/w8841. You can read about RD here: http://en.wikipedia.org/wiki/Regression_discontinuity_design
Does an observation with the max duration as the time mean that the subject didn't finish the task in the allotted time?
I'm assuming that you're using the R implementation. Unfortunately, the classwt parameter doesn't work (and I'm not sure that it allows different weights for different instances in the same class). The only way I got around the problem was to write my own RF implementation.
Hi Jeremy, thanks for your comment and for the article. It's still a bit weird to me though. Does this mean that I should always do separate tests for all groups after the anova even if it indicated that there was no sign. difference or is this something that should have been pointed out to me by post hoc tests or something? I thought that the anova replaced the separate tests. Now it seems rather useless. Also I wouldn't know how to report this; there was no sign. interaction, but group A showed a sign. decrease, whereas there was no sign. change in scores in group B? Thanks again :)
@mpiktas, I generated random correlated data, then I altered some of the values until received the wanted result (_b_ for XY = 0). As for the "theoretical prerequisite", it is deduced from the well-known formula for _b_ in the form it exists for OLS regression.
There are several - you can use change scores, you can add starting length as a covariate, if you have lots of data about each shoot you can use propensity scores, you could try matching in various ways.
A polynomial of degree $10$ (or of any positive degree) will never be a valid PDF over the range $[0,\\infty)$. Of more import is *why* you are attempting to fit a distribution to data.  What is your purpose?  And what do you mean by "basically asymptotic sample size"?
Not 100% related to the question, but my former PhD advisor told me that one of the favorities ironies of his PhD avisor, Prof. Debrabata Basu, was to point out that a point mass at an arbitrary parameter space point is a conjugate prior.
@ Nick : Exactly!
To try to keep my head straight, I tend to think about histogram density plots in terms of Riemann sums. But, I don't know if that works for others.
What you say is right. `u` is eigenvectors's value and is related to raw PC values. `u'` is called a loading and is related to the normalized (equal variances) PC values. You may want read my answer about it: http://stats.stackexchange.com/a/35653/3277.
Did the beginning of the last paragraph mean "CFA [on the same data as EFA] could be justified, [though], if ..." (which I think is reasonable)
Your approach works with an appropriate modification: A Weibull ($W$) for $X$ is an Exponential ($E$) distribution on $U=X^\\beta$.  Therefore $E_W[X^r]$ = $E_E[U^{r/\\beta}]$ = $\\Gamma(r/\\beta+1)$.
A feature selection will be carried out based on subject matter expertise, but I am still looking to first run an SVR on all features - approximately 60. I believe the experts will bring the number of features down to approximately 30, and I will then run another SVR.
@Charlie and Cyrus: thanks to both for your answer. Thanks also for your suggestion Cyrus - setting the variance of $x_{it}$ to be the same for each $i$ does the trick and yields OLS and FE estimates that are exactly the same.
If values are equally spread across a range, they are *not* normally distributed. That sounds like a uniform distribution.
Are you aware of the difference between descriptive and inductive statistics?
But why do you need such thing in the first place?
Better object-oriented behaviour---there are *lots* of modelling function returning differently organized objects. By providing `coef()`, you abstract that inner layer away.
This is a good comment, and a nice idea, however I believe it would need some modification to work. Assume wlog that $\\mathbb E X = 0$. Construct your $w_j$ as follows. Let $w_1 = 1$, $w_2 = 0$. Now, define $w_j$ inductively as follows: Set $w_j = 0$ until $\\sum_{i=1}^j w_i /j \\leq 1/4$. Then append ones until $\\sum_{i=1}^j w_i /j \\geq 1/2$. Append zeros again, then ones. Repeat ad infinitum. Now, $0$ and $1$ both occur an infinite number of times, but the variance of the rescaled mean oscillates between $1/2$ and $1/4$ (roughly). So, your stated sequence cannot converge in distribution.
The tails are "heavy" in the sense that they do not decay fast enough in either direction to cause the integral to converge. This concept has nothing to do with normal distributions (or any reference distribution).
If your question is solely about a software (and not about statistics), then this site is not the right place to ask. Have you taken a look at Stack Overflow?
Thank you. I looked into single linkage hierarhical clustering and it is somehow similar to NN method.
Thank you for pointing it out. Another question is that the plots look quite different. Does it indicate anything?
@JDav, that is one of the core principles of this website. I know a little about time-series & related topics, but try to stay away from them, & let more expert CV contributors address those issues. I frankly know next to nothing about financial modeling, & admit that openly at the beginning of this answer. I do, however, know a little about other statistical topics; I stand behind what I've written here, eg, that if the data are distributed w/ fat tails (say, as t w/ 3df) then no amount of data will make the sample converge to normallity, but that w/ enough the sampling dist of the mean will.
@Felix Dombek, the answer is "yes". In each of the three schemes, if you sorted each of the three vectors at the end, into ascending or descending order, then the resulting three vectors would all have the same joint distribution. These would be the order statistics of a multinomial random vector. But, the ordering at the end is important. Otherwise, each of the vectors have different distributions (though I think first and last would still be the same).
I took the liberty of trying to fix the grammar. I hope I understood the question correctly.
I'm still learning about this, but from what I read this technique was used by Turing in cracking the Enigma code. Turing's method was classified due to the war, but was also discovered by Wald (1947, sequential analysis). Recent research by Michael Shadlen suggests that neurons or clusters of neurons use this method. I couldn't find that page number in E.T. Jaynes's Probability Theory (different edition). Could you share which chapter/section discusses sequential inference or sequential analysis?
The idea's right although the equations make no sense. (The second equation is true when $\\theta = 0.471566$, but where did that come from?)  2/12 is the Bayes estimate using a Beta(1,1) prior.  Similarly, 1/11 is the Bayes estimate using a Beta(0,1) prior.  In effect, that's just a fancy way of saying you can get 1/11 as the answer if you assume you previously had sampled one person and they voted for Party B.  [Captain Statistics](http://stats.stackexchange.com/questions/423/what-is-your-favorite-data-analysis-cartoon/5006#5006) to the rescue!
"Unpredictable" makes intuitive sense, but doesn't it need some refinement?  If I am ignorant of the machinery of the heavens, then the phases of Venus will be unpredictable to me.  Does that make the workings of the solar system "random"?  (You could make a case either way, and in so doing, you would clarify what you really mean by "unpredictable.")
@rbatt. Yes, the Likelihood maximisation may fail, even for $\\mu$ constant. A good precaution is to scale the data. You can easily produce non-convergence with extreme values fits by making the data larger, say multiplying them by 1000. 

The shape parameter $\\xi$ has considerable impact and has no gaussian equivalent. Many estimators of $\\xi$ exist for the stationary case, and maybe these can be used as initial values, first ignoring covariates?
$P(+|H)+P(−|¬H)$ seems like a ridiculous quantity to consider. I would interpret "95% correct" as either $P(+|H)=.95$,$P(-|\\lnot H) = .95$ (as @Glen_b does in his answer), or possibly $P(+|H) P(H) + P(-|\\lnot H) P(\\lnot H) = .95$ (though not on this kind of question).
Pinheiro/Bates is the classic reference but doesn't have the details on how to do this; I had to do it once and basically ended up copying the code from one of the existing structures and changing it to meet my needs.  It was trickier than I expected to be, though, and long enough now that I don't quite remember what I did.  I too hope someone knows of a better reference.
@kwak Concerning your deleted response: I see vaguely that there is some kind of limited permuting going on until somehow a correlation coefficient below 0.5 (e.g.) is attained, but I cannot quite connect that with the conditions of the original question.  Are you saying you just keep applying "small" random permutations until you obtain a correlation within a targeted range of correlations?  If that's so, I think it's a really creative idea but some analysis is needed to show it can actually work.  Regardless, thanks for sharing it (however briefly!).
@chl, both, i have data and need to determine the proper fit and how well it fits
Can you cite an scientific literature showing that these strategies will predict better than chance whether the market will continue or reverse? I'm not an expert in this area, but I have been under the impression that no such approaches will work & I'd rather not have answers posted on CV that are incorrect. (I apologize for my bluntness here; I'd be happy to delete this comment if I'm wrong, & FTR, I didn't downvote.)
Discrete wavelets -- or at least the vast majority of ones that are implemented and commonly used -- are typically designed to have useful frequency-based properties under the compact support constraint.  The vanishing moment condition of Daubechies, for instance, is more or less equivalent to flatness in the pass-band.  The frequency localization properties of wavelets are usually what lead the coefficients to be sparse representations and allow estimation of the noise variance under the "signal + additive zero-mean noise" assumption.
What is it that you want to find out?

You may find [this post from my blog](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) helpful.
(+1) This is close to the point I was trying to make in the comment to ConjugatePrior's answer, though you've stated it more concisely and clearly. In some special cases, we can calculate the sampling distribution of the test statistic exactly *under the empirical distribution obtained from the sample*. But, usually, we can't and so we're forced into simulation. :)
@Glen_b, it's pretty much what I was going to put. You could elaborate a little, but I don't see much in the question you didn't cover (maybe the curvilinear part, eg).
[This site](http://www.andrews.edu/~calkins/math/edrm611/edrm13.htm) has a convenient list of possible correlation measures, as does [this document](http://www.napce.org/documents/research-design-yount/22_correlation_4th.pdf).
(1) That sounds like a bad idea, if it's more than a provisional identification to be reviewed by someone with domain knowledge. (2) Any such decision rule could perform very badly on average depending on the kind of data-sets to which it's commonly applied - you could sample some & look for patterns.
Thank you for your suggestion. Please see the updated question.
Hypothesis testing and calculating distances is not quite the same. So I wonder if you could be more precise?
What do you mean under `nonlinear regression ... with the help of genetic programming`? Could you provide more details? Why don't you simply use polynimial fitting?
I am afraid there is some confusion there about the nature of the accept-reject method. For one thing, you cannot use an exponential to approximate Gamma distributions with $\\alpha<1$. For another, the idea of using a sum of $n$ exponential variates will produce a Gamma$(n,\\lambda)$ variate. I suggest you look at our Example 2.19 (page 52) in our [Monte Carlo Statistical Methods book](http://www.amazon.com/gp/product/1441919392/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1441919392&linkCode=as2&tag=chrprobboo-20) with George Casella.
@user862, The sum of residuals equal to zero is a standard feature of regression with intercept. Differentiate sum of squares with respect to intercept to see why.
Thanks a lot! Before reading your comment, I was quite confused about why self-adjointness is crucial here. Now I have some clue, thanks!
@B_Miner Usually you will start by looking at the autocorrelation function of the residuals. If the autocorrelations are significant and large for large orders (i.e. the ACF does not decay exponentially to zero) then you may consider applying a unit root test on 
the residuals. If the analysis of the residuals suggests that there is unit root, that would mean that you should probably take first differences twice on the original data (i.e. take differences again in the differenced series).
Or, just remove $\\{1,3\\}$. It doesn't match the hint as closely, but has the virtue of being an even simpler example. (+1)
Thanks for the recommendation. I will give that a try.
The second vignette should be what you need (course notes): http://cran.r-project.org/web/packages/MCMCglmm/vignettes/CourseNotes.pdf
On the question\\* "which pairs of numbers appear most often together", I'd start with cross-tabulations. $\\quad\\,\\,\\,$ $\\, $\\* the 'question' is actually a host of somewhat different questions, depending on things like whether the pair (Num1=3,Num2=5) is considered the same as (Num1=5,Num2=3) (does order matter?), or whether it's considered the same as (Num1=3,Num5=5) (do the column labels matter?)
Given this evidence that on the average a *random* strategy performs better than yours, is there any need to consider your strategy further?
That is an improper scoring rule that uses only 1 bit of information from the predictions.  Improper scoring rules are optimized by bogus models.
I am a little confused about what you are considering as your variables. From what I understand, your matrix $M$ consists of 1517   samples from a 1000-dimensional space. When you do PCA (or ICA) on this matrix, you are able to recover the underlying modes of variation pretty well: for instance, in column 3 in your figures, row 1,2,3 correspond to the bases C, A, B respectively. This makes sense. 
However, in your code, when you perform LEM, you call the function on $M^T$ (`mixedSignal'`), which is not consistent with the above.
+1 for suggesting language modelling (I had never even heard of it!). I suspected there was a markov property to the sequences though ... all in all, great links for reading up on this subject. I'll wait and see if there are any more suggestions/answers.
If the method you are using does not yield probabilities I suggest finding another method.
There are simple closed-form to update an SVD, and more involved formulas for QR.  The computational savings can be significant when solving thousands of models. They can be implemented in R, but it requires a bit of work. It a nice project.
Check out [Kruskal Wallis](http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance) test (GraphPad info [here](http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_checklist_kw.htm)). It's non-parametric version of analysis of variance (ANOVA). The outcome would be the inflammation grade, and the factor would be the strain.
@user862: Your statement is wrong the variance is **not** the same in the second example, when shifted in time. You are misunderstanding the phrase "shifted in time". In your second example: $Y_t = a + \\sqr{t}\\epsilon_t$, for the values near $t=1$, you variance is proportional to 1 ($a+\\epsilon$), but near $t=5$, your variance is proportional to $\\sqrt{5}$ ($a+\\sqrt{5}\\epsilon$). So the variance is the same **with respect to the square root of time**. But time is changing, so your variance is also changing.
+1, that's a really nice demonstration; clear & thorough
@tomka I would define lurking variable as an unanticipated confounding variable.
+1, I enjoyed reading this explanation too. Very clear and structured!
I think you mean AC and BD in the first case ? (not CB)
**very** roughly is right. What if $P( {\\rm category \\ 1} ) = 1 - 10^{-200}?$ ;)
@whuber Can you say a bit more? The motivation for MaximumMath's procedure is to create a filtered body of data. My reasoning in the update is based on attending to the properties of the resulting filtered data. Although, I do see that I ignored the clustering issue.
I'm using a hash function to determine the integral points, which is why they have uniform distribution already. Continuity is important, however, between the points. I don't want completely random noise.
Just have to understand how to use chi square and Fisher, since i haven't understand them yes. This is why maybe I don't know which charts use, maybe Adams's idea could be useful!
@Lindsay: Unfortunately, it is not the rows you want to be centering in your case. If you want things standardized (centered and rescaled) you can just do `princomp(X,cor=T)`.
@gung I think it depends on the application. E.g. if the keyword extraction is the first step to create a document clustering, it is suitable for stats.SE.
In case you want more detail, this is covered in Agresti's *An Introduction to Categorical Data Analysis*.
Enlightening. Thanks.
don't you mean $Y−(\\beta_1 X_1+ \\beta_2 X_2+ \\beta_p X_p)$, and should there be an intercept ($\\beta_0$) in there as well? Also fix the stray "epsilon" by adding a \\. I'd have put the parentheses and \\ in myself but that's not enough characters of change to successfully edit.
I think your point about "The anova F test wrongly interprets group A to have a significantly larger mean" makes sense in my setting especially after my comment to @GregSnow's answer, where I analyse why Kruskal Wallis seems to be the most appropriate choice here.
@Macro , thanks, that's fine, and yes, I understood that they are not exactly equivalent. I prefer the 2nd approach (if I can understand it!) because it avoids the problem of the fixed effects changing even though they don't change much (by the overlapping confidence intervals measure), and it seems more parsimonious. Even if it is not "better" I would still like to understand it.
Well, you could get biased inference by doing that - for example, if you're forming prediction intervals, the coverage probabilities would probably be wrong for individuals in any of the 7 insignificant levels.
Cheers Peter, updated the question to make this clear. Thinking sum of squared differences is probably the best metric here
@Glen_b Corrected the question!
You're right, Stas.  But every few years I do find occasion to use a random number table (and each time I think it's the last time I will ever do so...).
I took the predicted probabilities for the predicted values, and then the probability if winning at a certain bid amount, and did a line graph to compare whether the model values (predic probs) were near the actual values. In retrospect, this seems like a horrible idea, but I'm not sure how else to assess the predictive power of a model on 'real' and 'new' data.
@rolando2: I suppose it's all relative, heh. I still feel like I've only scraped the surface of the subject...
Any ideas or tips how do I perform this kind of correlation analysis?
check out the *forecast* package - it's great.
I am interested in the individual fixed effects. However, the Hausman test suggests I should use the within-group estimator, which eliminates the individual fixed effects. The solution I would like to adopt is this two steps method. The two-step method is illustrated in http://ftp.iza.org/dp7583.pdf page9-10
This is a bit open-ended... Broadly speaking "random effects" appear due to the use of a subset of a population, eg. your participant's IDs; "fixed effects" are (otherwise unknown) deterministic components, eg. if someone is in your control group or not. Stating the obviously probably: Go with `R`'s `lme4` package of starters, it is not a panacea but it approximates one. :) Check the "sleep deprivation data" (it comes in the `sleepstudy` dataset of `lme4`), I think it will help you build some intuitions with the worked examples it has.
have you tried looking at iohmm (input-output hmm) ? might be more suitable for this case.
I don't know SPSS, but if it gives you parameter estimates from the model, then the OR are simply those, exponentiated.  But there is surely some way to get ORs directly and I hope an SPSS person will come along to give that
I doubt you can really get an answer to this... could be 20 as well as 5000, there's no way of telling unless the software gave other clues, that is (some logs maybe?).
Also, what benefit would SUR give the authors (vs. running separate OLS regressions) in this case since: they never really test cross restrictions, there are really no omitted terms (those variable not included in certain regressions is because they are covered under other variables, for ex the "Underweight" independent variable is the same as "Child malnutrition" dependent variable, which is why it's omitted in regressions 3 and 6)? I understand the efficiency gains of SUR, but it seems like they wouldn't apply in this case since the same variables are used in each of the 3 estimations (?)
@tibL Yes, I meant proportional to the likelihood $\\times$ prior. I already change that part in my question.
Another way to say that: the slope coefficient is interpreted as an *elasticity*... Just throwing that out there...
Could you point me to any literature that has more detail on the process? thanks!
Can't thank you enough!
@AdamO Agreed - in my answer in the question linked above, I mention that it can be used to smooth over areas of no information, as long as that lack of information is due to chance and binning. But yes - there's a reason regression is awesome.
@whuber Hum, thanks :/
@caracal Good observations.  The interval definitely can be wider than the $\\rho_i$: just consider the case where both correlations are zero.  The correlation with the sum can easily be nonzero--it can range all the way from -1 to 1.  E.g., x = (1,2,3,4,5); y1 = (3,-10,2,10,1); y2 = (-8,9,-2,-9,4); y1+y2 = (-5,-1,0,1,5) has $\\rho_1=\\rho_2=0$ but $\\rho=1$.