tttjjj/elt_bench huggingface dataset:
Certainly frequentist methods are much more subjective than advertised, but I would argue that Bayesian methods are more subjective still. And please don't convince yourself that invariance of your prior under a group or the specification of a MaxEnt prior are somehow "objective" - both types prior potentially express beliefs that *I* consider informative, and at any rate neither strategy is applicable in complete generality (e.g. I don't think there are invariant or MaxEnt priors on the space of CDFs, and any prior on this space assigns probability 1 to a topologically meager set).
Yes its trivial. I was able to come up with a simple proof using sigma algebra.
Which software are you using? Googling around let me find out that SYSTAT and SPSS in version 11.5 seem to have built in the Steel–Dwass–Critchlow–Fligner-test, an generalization of the test in your reference. There is an SUGI paper by Paul Juneau that mentiones a SAS macro called %DSCF that computes the test and can obviously only obtained by contacting the author.
A sample of your data would help. (Or at least a similar example.)
I have updated the language to try and be more concrete.  Imagine we're throwing pebbles at an open top floor window, and we only care how the pebbles which go through the window bounce on the floor inside.  We've thrown millions of pebbles, and about 2000 have gone through the window.  When a pebble goes through the window we draw a track of its progress across the floor.  Given a pebble goes through the window, we want to estimate the likelihood it goes through the region of concern.
2) We are also interested in the magnitude of the effect. The treatment is considered effective only if the new level of parameter P is the same as in the third (positive control) group of subjects (non-treated). - This would be answered by comparing means of the second group (treated) with the third (unrelated, non-treated), by un-paired t-test, since these groups are unpaired.

The problem now is that we will do two comparisons with the second group - i.e. multiple testing with its inherent inflation of the type I error. That is why I saw a need for ANOVA-based approach.
There are tons of papers and even books on this topic.
I already upvoted this, but I'm glad to see that it is going to get some more love. It may also win the CV award as our longest answer ;-).
Cross posting of http://www.stata.com/statalist/archive/2013-05/msg00114.html Neither forum forbids cross-posting, but lack of answers so far on Statalist suggests to me that this posting is too broad with too many questions. Anyone minded to answer this should check the Statalist link first to see if anyone replied.
I don't think the statement "the faster it mixes, the faster the dependence decays in successive iterations" is correct. Successive iterations will always be dependent using the Metropolis-Hastings algorithm, for example. Mixing has to do with how fast your samples converge to target distribution, not how dependent successive iterations are.
I provide a general simple model and apply it, with all details worked out, in the post at http://stats.stackexchange.com/a/18609/919. This and many other posts on standard errors (almost a thousand to date) can be found by searching our site for ["standard error"](http://stats.stackexchange.com/search?q=%22standard+error%22)
Have you checked out the visreg package?
To confirm, it is okay to use K-L divergence with categorical distributions, correct? For example, if I had another golden standard like `55% from New Jersey, 5% from California, 40% from New York`?
I would go with Simon Wood's [Generalised Additive Models: an introduction with R](http://www.maths.bath.ac.uk/~sw283/igam/index.html). Even though **mgcv** has moved on a lot from what was used in this book (See `citation(package = "mgcv")` for some of the more recent proper papers on the methods), this remains one of the best books I know for learning about GAMs (and GLMs) with a practical side in R and a theoretical side.
@MichaelChernick, I don't necessarily disagree w/ you. I just wanted to put out another opinion & I often like give a big-picture, semi-philosophical, what-is-this-all-about perspective. A lot of practitioners can get bogged down in details that seem arcane to them & are left w/o a grounded understanding.
@nullpointer correct
Sometimes there's no shortcut to reading the literature.  For example, many back issues of Public Opinion Quarterly deal with this topic. Probably so do many other sources, including the American Statistical Association's Survey Methods Section Newsletter.  But it's too broad a question for this forum, imo.
Relevant question: http://stats.stackexchange.com/q/66792/7071
Hi @Joost, welcome to the site. If you have only two groups (black and white), you could do a simple two-sample t-test to compare the mean vitamin D levels between black and white skin color. By the way: what software do you use? Stata? SPSS? R?
@Michael, I believe that when $e\\to 0$ slowly and $x$ is in the interior of the essential support of $f$, then $p(x)/(2e)$ will converge to $f(x)$, not $\\infty$. This is a two-epsilon argument: when $N$ is sufficiently large, $p(x)$ will be close to $\\int_{x-e}^{x+e}f(x)dx = (F(x+e)-F(x-e))/(2e)$, which converges to $f(x)$.
(+1) Interesting paper, but confused me rather. If you have to assume a prior distribution on the unknown parameter to show that an approximate procedure for calculating a confidence interval has higher coverage on average than the exact procedure, why not go the whole hog & calculate a Bayesian credible interval?
The question is more interesting when the test is conducted at a level of $0.5\\%$ rather than $5\\%$ or $0.5=50\\%$. That is because (2) is very strongly significant whereas (1) is significant between the $1.7\\%$ and $3.5\\%$ levels, depending on how it is tested ($\\chi^2,$ Fisher, or a permutation test). As far as its efficacy in assessing the underlying *concepts* goes, the question would be far better replacing $13$ by $18$ and $130$ by $180$: then it can be answered rigorously with almost no calculation.
@mbq, Thanks for the edits. They are much appreciated. As you can probably see, I am new to statistics.
Wellcome, Stéphane. There are some osbcure points in your answer, I think by $Y \\in dy$ you mean $Y \\in [y, y+ dy]$. Moreover $g(y)$ have to be null when $y < 0$. Finally, there is a simplification between $1-p$ and $1-F(d)$, which should be explicited...
Just wondering, why not then put a uniform prior on the range (0, infinity) rather than on (-infinity, infinity)?
If you do want confidence intervals you have to define the estimators for p, q and i.
perhaps I am missing something but why can't you determine, in R, which groups have frequencies of less than 5% and then relabel them all as 'other' and re-calculate the frequencies?
Can you please explain to me why these are the limits of the integration?
Thanks for coming back to share the answer and don't hesitate to accept it (with the little check mark next to the vote number) to mark the question as “answered”.
What's being gained by running more trials is an increase in the number of *true* positives or, equivalently, a decrease in the number of false *negatives*. That the number of false positives does not change is precisely the guarantee of the test (provided its assumptions are met).
I've merged your two accounts, Toby. You'll need to [register](http://stats.stackexchange.com/faq#login) the one used in the OP.
So far there are at least 3 strands to what you've described/proposed:  1) which of the survey questions are most strongly correlated with the person's purchasing behavior  2) how best to graph the relationships between binary and scale variables 3) how to describe these relationships in terms of probability or odds.  This site is best suited to dealing with 1, maybe 2 at a time. But at any rate, try looking into point-biserial correlation, paneled histograms, and logistic regression, respectively.
+1, maybe as a caveat: not rejecting the null does not necessarily mean that we do not reject exogeneity but simply the test may not detect the difference between OLS and IV if IV is too imprecise. This can happen for instance when the first stage does not explain much in the variation of the endogenous variables, i.e. the instrument isn't very strong. Then the standard errors of IV will be larger.
(+1) or [Kernel Density estimation](http://en.wikipedia.org/wiki/Kernel_density_estimation) as a more generic approach
Upon taking the logarithms of the data, [the many methods applicable to Normal mixtures](http://stats.stackexchange.com/search?q=maximum+likelihood+normal+mixture) become available to you.
To get you started, try this: https://www.ats.ucla.edu/stat/mult_pkg/faq/general/complete_separation_logit_models.htm
Since this discussion does not seem to be converging, let me add to @cardinal's hint.  One of the two expressions is $P\\{X+Y \\leq x\\}$ and the other is $P\\{X \\leq x, Y \\leq x\\}$ where $X$ and $Y$ are independent identically distributed positive random variables with common distribution function $F(\\cdot)$.  One of the two sets $\\{X+Y \\leq x\\}, \\{X \\leq x, Y \\leq x\\}$ is a subset of the other. So no integral need be evaluated.
@Mariam, I'm not sure exactly what you mean but it seems, in that case, it would be ok to pool across subsets of the population but not across time.
Those are very broad research questions. You'll have to narrow them down to specific questions you can address with your analyses. Which specific questions will determine which specific tests you need to use; this in turn will determine what content for your questionnaires would be best. That is to say, you need to address more basic questions before this one can be answered. What specific questions can you address with the statistical analyses that you're capable of running? This implies that you need to take into account what resources you have for data collection; some questions are pricey.
In method (or instrument) comparison studies (e.g. the comparison of oximeter A and oximeter B), the DV (oxygen level) ought to be 0 when all the IVs (oxygen levels) are 0. However, the intercept should not be ignored if I want to calibrate (or exchange) oximeter A with oximter B.
(+1) Fair enough.  I see from the reference you've given that yes, perhaps it is actually not too hard anymore for even a casual user to step into R and do things right.  Thanks for challenging.
Without knowing precisely what the response variable is, it is actually difficult to imagine trying all possible models! The use of a term like "efficacy" in your 2nd paragraph should be justified, in the context of the question.
+1 but, *caveat emptor*: this book is only valuable if you are a Stata user.  I don't use Stata (in fact I never have).  On the other hand, I like Long, so I checked this out from the library.  I'm sure there's a lot of good info in there for everyone, but it is so thoroughly intertwined with the use of Stata that it's impossible to extract the domain general information.
It seems that WAR is your _dependent_ variable. You provide evidence for my assertion, disputed elsewhere on this site, that the two terms are often confused. My advice is not to recode negative values to zeros (maltreats the data) but to use a GLM with log link. Please assume zero interest in, or knowledge of, baseball minutiae on my side.
Same as above, I have corrected now, apologies
Yes but also take account of the need to adjust the p-values for mutliplicity.
Tell them about clinical trials. Clinical trials rely entirely on the results of statistics.
*There are no observation numbers!* Any reasonable solution must be invariant under permutations of the observations, because they are exchangeable.  Regression is meaningless.
@GuessGucci: "The main reason to think that the two classes would contribute differently to the price of lunch is that Class 2 is older than Class 1 on average" then wouldn't separating this into total number of students (= class 1 + class 2) and difference between class 1 and class 2 be a sensible transformation? Which of course gets you on the way to re-invent principal component regression...
Thank you, that is a good addition to this question.
Welcome to this site! Could you give some references on the aforementioned work, and maybe more details so that readers could get a better idea of how graph isomorphism or LCS might solve the problem at hand?
...The problem arises only when a scientist cannot distinguish between a belief and a fact.  There is nothing unscientific in the belief that Bayesian or frequentist statistics are superior, as there is no objective test that can decide the answer (AFAIK), so the choice is largely subjective and/or a matter of "horses for courses".
The way I see it, evaluating your methods over all possible train/test set partitions of you data would have been the most comprehensive evaluation. Since this is impossible, you are estimating this with random runs. Let's say you could evaluate all train/test partitions - you would still be left with the question of how to decide which method is better. So this is more a matter of how you define what "good" is. Does it mean high average score? Or does it mean that in many runs one method gets a higher score than the others (personally I think this would be a better version)?
Are $a$,$b$ and $c$ independent or at least uncorrelated? if not, what are the covariances between them?
Maybe you need to _not_ skip the few tedious steps and actually derive the likelihood ratio from scratch instead of jumping to maximum-likelihood _estimators._  This is a problem about _hypothesis testing_ not about maximum-likelihood estimation of an unknowm parameter $\\theta_i$.
I think your model is definitely in trouble as your second MA polynomial has a coefficient of .97708 clearly cancelling the second of your AR coefficients .99991 . Both coefficients refer to lag 7. I will use AUTOBOX a very useful time series analysis package that I have helped develop to re-state your model in terms of the pi weights i.e. a pure autoregressive form. Poor model selection (as in this case) is due to the inopportune usage of the AIC/BIC criterion ... which is all that SAS offers as part of its inferior  "automatic process".
@PatrickCoulombe Thanks for the heads up. However, this is not "home work", but me trying to understand this derivation in order to formulate a model.
The fact that tags exist alone suggests that questions have asked previously in this field. See e.g. http://stats.stackexchange.com/questions/34753/are-data-transformations-on-non-normal-data-necessary-for-an-exploratory-factor
The answer obviously has to depend on the text itself.  For instance, after shredding a sequence of 100K 'a's, a single copy of the shredded text will suffice.  The answer also depends on the actual distribution of the piece lengths, not just their mean $X$.  It is also clear that some strong simplifying assumptions have to be made, such as that certain locations should not be heavily favored as break points in the shredding process.  (E.g., if there exists one location that is certain to break, you can never be sure of the reconstruction.)  Do all these assumptions leave a realistic question?
"Trivial" mathematically, yes: but whether it is in the least *practicable* is another matter entirely!  But I take it--please feel free to correct me if this is wrong--that you are not concerned in this thread about practicability; this is more of an open-ended query about whether there may be any conceptually fruitful way to view sampling and optimization as two aspects of the same thing.
@Dan,Because the pvclust mentioned they used the same method as hclust
You don't mention a link function, which matters for interpretation. From the look of it, I assume it was a log-link. Is that the case? Is `make` a straight categorical variable - a factor - from which we can take these coefficients to be those going with 0-1 dummies?
Why would Fisher's distribution be appropriate?  It does not seem to correspond to your description of the ball-drawing process.  Perhaps you could edit the question to clarify which process is relevant.
Did you find an answer for your last questions? " However, I want to at least say that the % answering yes is large enough to determine validates that at least some proportion of the sample is seeing a change." I also want to see if yes% is large enough to validate my hypothesis ( I also have yes/no responses and they are independent). I'd be thankful if you inform me about the statistical test you chose at the end. Best, Fafa
I already flagged it to a moderator. I first opened it here, and then I realized it was better suited for math exchange.
I think you are wrong, a certain pattern is predicted by the link function: http://stats.stackexchange.com/q/25068/442
Thanks! I will have a look at it during the week.
+1 for what I anticipate will soon be a very good question. The first problem is the *independence* test doesn't use the claimed statistic. The statistic given at the start is unidimensional (a sum over $n$ categories), while a test of independence requires more than one variable. Please edit to make the name of the test and the statistic correspond.
The second link explained everything I wanted to know. Thanks! :)
In R, you can cast categorical predictors as factors. Factor coding has the advantage of taking care of the problem of breaking multiple levels of categories into binary features automatically. R's regression routines will parse and interpret these correctly as categories when fitting the model.
[This page](http://publib.boulder.ibm.com/infocenter/spssstat/v20r0m0/topic/com.ibm.spss.statistics.help/alg_glm-uni-multi_notation.htm) may be the one you want for explanation of the notation.
@Amyunimus: it gives greater weight to values with smaller variance/higher n, which are assumed to more closely estimate the "true" value.
That's a small sample size and few questions. With only three questions, you may wish to keep the analysis very simple and only report the pre- and post-differences (with confidence intervals) for the three questions. Whether you report the mean or median will depend on the distribution. I wouldn't recommend getting too fancy.
Assuming higher quality and lower price are better, you can in this particular case reject B and C since they both have a lower quality and a higher price than A.  There are other combinations of values where you need a more sophisticated approach,
That's not even remotely normal!  The enormous JB statistic is merely telling you what is plainly obvious in the histogram.
I agree with everything except on the notion to keep things simple.  To me simplicity or complexity should be a function of the cost of improper decision that you eloquently explained. Simplicity can have negligible costs in one area (e.g. serving the wrong ad to a customer) and a wildly different cost in another (administering the wrong treatment to a patient).
+1, I think what he means is that standard regression assumes the residuals are *independent & identically distributed* as a Gaussian w/ constant variance. If this is true, then by definition the logistic regression won't be able to find a relationship b/t $P(e_i\\geq T)$ & your covariates. This is what I was trying to get at, although CarrKnight has given a clearer & more comprehensive version.
Thank Nick for answering. I already saw the example, but it doesn't specify which kind of variables are in the main regression and how I can choose them. Surely the example is clear, but I really don't know how I can interpret that. It's for that I posted here.
I think that's the right reading of my problem. Both (a) it seems like the answer depends on the weights, and (b) simulation seems like a last resort if there's no closed form. And your general philosophy makes sense. Still, I was hoping for a generic calculation (analogous to the t-based P-value based on a null hypothesis of zero), presumably in terms of the weights.
Made this CW as I don't think there's a single best response.
Also, note that the equality in the inequality holds if & only if $\\mathbf{a}=c\\mathbf{S}^{-1}\\left(\\bar{\\mathbf{x}}_1 - \\bar{\\mathbf{x}}_2\\right)$. This relates to the scale invariance @whuber mentions.
+1, I should be sorry - to steal your argument is not very polite. It's great that you spelled out the difference between independence and being uncorrelated in this context.
Nice answer. One addition: sometimes, you can write $E(X)$, but it makes really no sense to write $\\mu_X$ when the expected value does not exist. Take, for example, the Cauchy distribution. Of course you can write $E(X)$ to indicate taking the integral, but that integral is undefined, so writing $\\mu_X$ as an answer really is nonsense.
right after I asked this question, I did some digging and this is the solution I decided for myself. Came back and here it is. Check please!
Hi, welcome to the site. This appears to be a standard textbook style question. Please note the [faq discussion](http://stats.stackexchange.com/faq#questions) relating to homework questions, and the tag-wiki info for the [self-study](http://stats.stackexchange.com/tags/self-study/info) tag (the two are synonyms).

So ... What have you tried? Do you know what a pivotal quantity is?
(+1) I might suggest a small tweak to your last sentence. The test statistic is distribution-free even in finite samples (though it won't be the same as the asymptotic distribution). So, this distribution-free property is what makes the test statistic nonparametric. Note that there are *lots* of examples where the *asymptotic* distribution does not depend on the underlying continuous distribution (just think about the CLT), so, unless I'm mistaken, I don't believe that's the core feature here. :)
I wrote a tutorial review of the use of Fisher's z for correlations accessible at http://www.stata-journal.com/sjpdf.html?articlenum=pr0041 I'd recommend more use of confidence intervals and calculate 0.724, 0.781 as 95% limits. I'd recommend even more looking at the data and working out a regression.
Keeping the data as they arrive minimises within-group differences at 0. More generally, what exactly is your criterion (please write down an equation)? Why do you want to throw away information?
You said:  "*Both prior and the likelihood are normally distributed. If I understood correctly, such cases are called "conjugate priors"*" -- in this case, yes, though more generally conjugacy is understood as the posterior and the prior having the same form, which may or may not require the likelihood to be of the same form (in the parameter(s), naturally, not the random variable for the data), depending on the situation. It's often the case that the likelihood is in the same form as well, however.
And frequently the cube root transformation works well, and allows zeros and negatives.  I've found cube root to particularly work well when, for example, the measurement is a volume or a count of particles per unit volume.  Cube root would convert it to a linear dimension.  A more flexible approach is to fit a restricted cubic spline (natural spline) on the cube root or square root, allowing for a little departure from the assumed form.
My answer was more general than answering for random forest. Sorry about that. I have no experience with random forest, but I found this stackexchange's answer that might interest you [Random Forest Assumptions](http://stats.stackexchange.com/questions/59124/random-forest-assumptions)
Bravo! Worth pointing out that the second paper states that "Unfortunately, this approximation to the variance... cannot be used directly to construct approximate confidence intervals around beta, since we have to take into account the bias of the estimate. Jackknife and bootstrapping might be possible methods to obtain more insight into the variability of beta". Kind of defeats the whole point for me.
The usual rule of thumb is 10 points for each independent variable.
Awesome! Thanks Matt that did the trick.
Michael, could you explain precisely *why* this is not a model selection problem? It looks like John has clearly formulated it as one.