tttjjj/elt_bench huggingface dataset:
For the special case where the degrees of freedom is 1 (the Cauchy distribution) the answer to the original question is 1. The sum (or difference) of two independent Cauchy distributed random variables is Cauchy with scale parameter $2$, but then again, the Cauchy distribution does not even have a mean value.
$= 1-P(n*k(\\frac{\\sum(\\bar{x}_{i.}-\\bar{x}_{..})^2/(k-1)}{\\sum\\sum(x_{ij}-\\bar{x}_{i.})^2/(n-1)})<=F_{k-1,n-k,1-\\alpha}) =1-P(n*k(\\frac{\\sum(\\bar{x}_{i.}-\\bar{x}_{..})^2/(k-1)}{\\sum\\sum(x_{ij}-\\bar{x}_{i.})^2/(n-1)}-f^2+f^2)<=F_{k-1,n-k,1-\\alpha}) =1-P(n*k(\\frac{\\sum(\\bar{x}_{i.}-\\bar{x}_{..})^2/(k-1)}{\\sum\\sum(x_{ij}-\\bar{x}_{i.})^2/(n-1)}-\\frac{\\sum(\\frac{1}{k}*(µ_i-µ)^2)}{\\sigma^2}+\\frac{\\sum(\\frac{1}{k}*(µ_i-µ)^2)}{\\sigma^2})<=F_{k-1,n-k,1-\\alpha})$ like in [link](http://www.statmethods.net/stats/power.html) for n=n1=...=nk.
You can use logistic regression even if you have binomial data.  You just also need the sample sizes.
Welcome to our site! We field a wide variety of questions at all levels. Did you notice this one asks for an answer at "the level of a first graduate course"? That should be interpreted as a request for rigor, clarity, and correctness. In light of that, you might want to reconsider assertions that (although suggestive) are not really true, such as "95 of the confidence intervals [out of 100 experiments]...will contain $\\mu$."
The clustering should not be on region-year pairs since, for example, the error for Bavaria in 2014 is presumably correlated with the error for Bavaria in 2013.
The main problem is that the size of an extreme variation in the $z$ values may be--and usually is--quite a bit smaller than typical spacing between points in $(x,y)$.  Your method would tend to thin the data without necessarily finding any outlying elevations at all.
I believe your translation is exactly what I was looking for!  Better, even, because I assumed that higher $p$ would describe an increasingly poor error bound for the estimate.  I didn't expect it could "feed back" and be used to correct the estimate.
I myself have found different answers with stepAIC and lars and would imagine that my issue is to be treated with group LASSO - it is not about the VIF of the whole matrix, but a number of clusters of correlated variables.
As the data is actually streamwise velocity component from a tidal estuary, I guess the peak in ACF might show the scale of strongest horizontal turbulence. But have no idea about the correlation function.
For the first part, if $X$ is binomial with parameters $n=100$ and $p = 1/6$, then $X$ is approximately normally distributed with mean $np$ and variance $np(1-p)$. You may want to use a "continuity correction": find the probability that the normal random variable is between 14.5 and 20.5. I think I was incorrect when I said that the second part is a simple application of the same method.
It's only a partial duplicate because the question and answer in the other post doesn't respond to the skewness question, which can be answered. I don't think this one should close.
Are you, in effect, trying to integrate out the categorical variable to get an estimate of the unconditional probability of the latent Bernoulli variable?
Thanks for your answer. Do you happen to know any books on the issue of knowing what method to use given some problem? I'm a math student with a focus on probability theory and feel that I'm lacking more 'practical' skills such as knowing when to apply some method, given some real world problem. Would you happen to know any literature on that problem?
Are you sure of the high correlation?  It is not necessarily the case that X.pre and X.delta will be correlated at all.  Also, is it necessary for X.pre and X.delta to appear within the same analysis?
@Glen_b: Thanks for the suggestion. I've modified my question in the original post. See if it's clearer now.
What is your $\\alpha$? Ie, what is the long-run type I error rate you are willing to live with? Relatedly, why did you specify `conf.level=0.99` in `prop.test()`?
what for ? Clustering ? Classification ? Related: http://stats.stackexchange.com/questions/26049/feature-selection-methods-for-document-classtification
No problem. If you indent with four spaces, it turns it into "computer text". At that point, I just spaced it until it looked right.
@Aksakal That idea seems to applicable to other systems - e.g. bi-pedal locomotion. If you took only the necessary elements of two systems of bi-pedal locomotion, it seems impossible that one of the two systems could have only one *necessary* difference...
Thank you for reading through the paper too. I feel more confident that this should work. :)
Thanks this helped a lot!
Amazing. Now I just feel spoiled for choice about which bars to use, and the problem of feeling spoiled for choice is nothing but.. a delightful *non-problem*. Thanks a lot! ^ ^
In general, it helps if you provide a full reference or, preferably, a link when you mention a paper.
It may lead somewhere to play with the algebra here.  Notice that the two logs in the definition of $f$ can be combined (sum of logs is the log of a product).  Subsequently, plugging this simplified version of $f$ into 13.1 might lead to an interpretable expression.  The Law of Total Probability comes to mind.
Good answer.  Why is it "very difficult"  to get the transformed distribution?  Can't you just apply the cdf of your distribution to the inverse transform?
+1 This is both interesting and answers the question as it stands. I assumed the OP wanted a monotonic transformation, but there's nothing in the question that actually says so. A refreshing and somewhat unexpected take on it.
Thanks, D L Dahly. Since recently finishing graduate school, I've only now started to understand the "differences" held among biostatisticians and epidemiologists. The purposeful selection method was taught by a biostatistician who swears by Hosmer-Lemeshow. I almost forgot the usefulness of DAGs, but I will revisit them.
Fantastic - a clear, detailed answer, covering all the bases - but also a very beautiful answer, wonderful tables and diagrams. A pleasure to read.
Already answered as part of a wider discussion: see http://stats.stackexchange.com/questions/22718/what-is-the-difference-between-linear-regression-on-y-with-x-and-x-with-y
Not without a lot more details.
@mpiktas thanks, you can concider it has been generated with a suitable AR modeling (one for each wind farm), it won't change the problem very much. Sorry, there are too much confidenciality issues with these data, can't provide you anything, even anonymized...
Excel's `LINEST` function will give you the formula directly. For one independent variable, the method is described and illustrated at http://stats.stackexchange.com/questions/8040/use-a-trendline-formula-to-get-values-for-any-given-x-with-excel. Since `LINEST` also does multiple regression (that is, it applies to multiple independent variables), that answer works just fine in your case provided you compute a third column containing the squares of `x` and place it immediately to the right of the `x` column in your spreadsheet.
What language or package is this please? The mention of proc (?PROC) leads me to guess SAS, but please spell it out. See http://meta.stats.stackexchange.com/questions/1479/how-to-ask-a-good-question-on-crossvalidated for the advice "Say what programming language you're using". (It applies to answers as well as questions.)
Was the other IV included in the ANOVA?
But that still doesn't answer my question on what's the intuitive difference in the objective functions of SVM v/s LR, which are as follows: (a) SVM: Try to maximize the margin between the closest support vectors (b) LR: Maximize the posterior class probability
I think they use what people are betting on. The odds are constantly adjusted to ensure that whatever the outcome, the bookie always wins.
Thanks for your feedback. This is significant because you can compare current prices against historic prices along different axes. For example, you can trace the lowest and average weekly prices over time and see when a new low price is significantly lower compared to that data set, and that set could be all time (or the past 30 days), or alternately you could just compare it to the lowest price for the previous day. But then if the price decreases slowly (eg. Every day a little less) then you will never see a big "significant" price decrease.
You're welcome.  I would use the model built on the whole dataset.
I want to be able to communicate what the expected false positive rate is at a given cutoff (say, 0.95).
Readers should be aware of a previous version of this question on [dsp.SE](http://dsp.stackexchange.com/q/10796/235).
In that case I would say significance is still meaningful but unimportant. If you're trying to make predictions and the predictions are bad, then your model can't be good. It's possible that there is still a positive association between the predictor and the response; it's just too weak (relative to the noise in the data) to be of predictive value. Another possibility is that the underlying relationships in your in-sample and out-of-sample data are different. Also, did you plot the out-of-sample residuals? If they're symmetric you might still be okay.
This question also addresses the issue of Type III SS in R: http://stats.stackexchange.com/q/4544/3601
Does it have a creative commons license or something similar, like the question asked?
Thanks, that's helpful. I'd combine this with @onestop's response for a full answer -- looking into the GHK algorithm provides the math behind what you're explaining. That is, while multidimensional integrals are hard, it doesn't necessarily seem to me that they'd cause extreme issue: we tend to simulate these things anyway, so adding another dimension is just generating one more set of random draws. Looking at how GHK smooths this process, though, helps demonstrate why it takes some time (I think).
If the complexity is indeed $O(n^3)$, then you should expect the full dataset to take $10^6 \\times 20 \\, min \\approx$ 40 years. So it looks like your method is doing not so badly after all.
I would have named the Chicks differently, just to be sure that "Chick 1" from one nest have nothing to do with "Chick 1" from other nest. You can just name then combining nest & chick so your factor for "chick" reads "257_1" and "257_2" for the first two rows. Second, I think you have another level hidden, cause it seems that you may get more than one mesurement fron the same chick?
Welcome to the site, @ronny. Since you're new here you may want to read our [FAQ](http://stats.stackexchange.com/faq). Among other things, this site supports $\\LaTeX$ via mathjax. I took the liberty of adding mj to make your post easier to read; make sure it still says what you want. EG, I took "p^" to mean "p-hat" ($\\hat p$), but I note you say that it is the "true" rate, whereas p-hat is often used to indicate the p estimated from your data, so we just want to be sure your answer says what you want it to say.
@Douglas Zare testing both machines costs the same.  The business case is that the cost of fully developing the prototype would only pay off if I can expect a 10% improvement.
@stefan-ock Hypothesis testing is only about approximating chance that your result is just a lucky coincidence; here the result is absolutely clear. What you can do is to write, for instance, that the difference is so small that it is not an important factor for comparison; yet judging how many bytes is equally important as 1 ms of computation is purely use-dependent and a bit subjective in its nature.
Since your attributes are **binary**, I would be careful with the term correlation. It's more of a co-occurence thing. Try rewriting the correlation formula into a count-based formula.
Is there a practical way to approach the problem I described above? One way that is common is to take some distance measure to the mean, but this data may be multi-modal and thus this distance may not mean much
Not so, as cross-posting is specifically discouraged: see http://stats.stackexchange.com/help/on-topic More to your point, I once counted four different meanings for Gini index, so just mentioning the term without a definition helps little.
A maximum $R^2$ is attained when *all* variables are included.  This is clearly the case because including a new variable cannot decrease $R^2$.  Indeed, in what sense do you mean "local" and "global"? Variable selection is a discrete problem--choose one out of $2^k$ subsets of $k$ variables--so what would a local neighborhood of a subset be?
possible duplicate of [Locating freely available data samples](http://stats.stackexchange.com/questions/7/locating-freely-available-data-samples)
$n$ has to be moderately large before a Normal approximation becomes reasonable. Simulated kurtosis statistics are reliably skewed (positively) when $n=500$; they start to look Normal for $n\\gt 1000$ or so.
Thanks! This is almost the exact same approach I took at the beginning...I added an extra observation to see if it would run, and it did. Prior to this analysis, I had no idea that perfect separation existed.
The question remains how far the way that `envfit` calculates R^2 matches the pattern of difference between the lakes. There are lots of misses as well as hits from any simple discrimination. I haven't tried finding out about `envfit`.
Anythning is useful to start with. I know how to run PCA and its basic interpretation but finding genes which are associated with first principal component is getting diffucilt in terms of code in R.
For the sake of completeness, the example provided in the HoS27 is about meta-analysis.
A simple analysis like the paired t.test, with growth as the difference between the last 2 and the first 2 seems like the best path. This is sort of a preliminary analysis to satisfy an assumption of a larger analysis. I don't think reviewers will be particularly concerned about how I tested for growth given that it is not the focus of the research. Thanks!
@jb I believe this formula has a simple, beautiful geometric description.  The LHS is the squared length of a vector orthogonal to the space spanned by $X$.  Multiplying by the denominator on the RHS therefore gives the squared volume of the hyper-paralleliped spanned by $X$ and $y$. But that's precisely what the numerator of the RHS is, QED. (This generalizes the familiar theorem of Euclidean geometry: the area of a parallelogram is its height times its base. Turned around, it says the squared height equals the squared area divided by the squared base.)
I have edited my original question to explain briefly my study and my results
Because that's what everyone else does :) Seriously. It looks like someone did some Excel macros a long time ago, and everyone else has just copied them. I had assumed that it would take too much computing power to do it directly, but that doesn't seem to be the case. Maybe it was 20 years ago.
The mean of $Z = \\frac{X}{Y}$ is **not** $\\frac{\\mu_X}{\\mu_Y}$. Whatever gave you that idea? Even for one random variable, $E[g(X)] \\neq g(E[X])$ _in general_. This is a fundamental notion that you would do well to learn well. Engrave it on your heart in letters of gold....
@MichaelChernick: For a given sufficient statistic, carrier measure, and support, you can integrate the support to find the log-normalizer.  If the log-normalizer is finite, then I think the family exists.  He has done this and he is asking if this family has a name.
Have you tried Expectation Maximization for parameter estimation? http://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm
Technically any function of the data can be an estimator. Whether or not it is a good estimator is another matter.
Great Explanation. Thank you :) +1
Come back after you have unbiasedly validated a probability (risk) model.  See my handout at http://biostat.mc.vanderbilt.edu/CourseBios330, for example case study in logistic regression.
An even better discussion here: http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output
Given the answers here, we might flag the other Q @Macro links to as a duplicate of this one so people coming across that one see the Answers here, especially that of whuber.
Have you got the data on the 21 questions themselves?
This is related to my comment and [answer](http://stats.stackexchange.com/a/21492/7224) on [another question](http://stats.stackexchange.com/q/21335/7224) dealing with joint distributions a few days ago.
Wasn't this based on an older variation?
yes, the chances of all white on 4 straight is .0625 and the chances of all red on 4 straight is also .0625.  so the chances of either all red or all white on 4 straight is .125.  I misunderstood your question.
Any common latent factors here, or do you really only have one Likert rating per construct? Also, how large is your sample dataset?
(This should be a comment to the reply but there are so many comments there it would be lost in the noise!) Note that $\\phi$ is *not* assumed to have a second derivative.  The conclusion follows from the assumptions that $\\phi$ is *nondecreasing* and *strictly* convex. Proof: If $\\phi'$ were not strictly increasing on a nontrivial interval $[a,b]$, then $\\phi'$ would be constant on some nontrivial subinterval $[a_0,b_0]$, implying $\\phi$ would not be strictly convex on $[a_0,b_0]$, contradicting the assumptions, QED.
+1 for the great answer that wraps everything up. How to apply the mentioned numerical procedures is nicely and applicably described in Tabachnik and Fidell's Using Multivariate Statistics (for SPSS and SAS): http://www.amazon.com/Using-Multivariate-Statistics-Barbara-Tabachnick/dp/0205459382/ref=sr_1_1?ie=UTF8&s=books&qid=1284969084&sr=8-1 (But see the Erratas on the accompanied web page)
"Influential observations in Bayesian inference"? In frequentist analyses *influence* is the term for how much the inclusion of a data point affects a fitted model.
I think mcmc doesnot have Gibbs sampling, do it? It does random-walk Metropolis for an arbitrary continuous distribution on R. I could find about gibbs sampling in arm, may be I missed something. The BUGS (Bayesian inference Using Gibbs Sampling) seems to have gibbs sampling facility. thanks for the suggestion.
Regarding the relationship between Eigenvectors & a regression line, it may help you to read this thread: [Making sense of principal component analysis, eigenvectors & eigenvalues](http://stats.stackexchange.com/questions/2691/), & possibly my answer here: [What is the difference between linear regression on Y with X and X with Y?](http://stats.stackexchange.com/questions/22718//22721#22721).
What an amazingly thorough and responsive answer by @whuber
Uh... t-tests for Poisson distributions? Generally you'd be looking at something like a chi-square. And why regression rather than Poisson GLM?
Thank you underminer, I would like to look at your code but I don't know R. Anyway, thank you.
Then you'll need to explain your problem better.
This is actually the solution I went with. Part of the problem with an institutional repository is that what institution I'm at is in flux, and the data isn't really important enough for one of the big data warehouses.
Thanks, @Lolly. I'm not sure you need to switch the accepted answer, though. You can also indicate than answers have been helpful for you by upvoting them (ie, clicking on the right-side-up normal distribution to their left).
Have you tried ordered probits/logits?
The term you might be looking for is the *convex hull* of the points (or potentially the alpha hull). You should be able to find an R function to calculate these and then be able to add them as layers to the plot.
The ICC is not, in itself, a parametric statistic. You can conceive of the ICC regardless of the distribution of the data. What exactly do you mean by "nonparametric ICC"?
if i get right your question you are interested in a model were `futime` has a special "weight" as a predictor? Why don't you try validating this assumption?
It looks like [the Wikipedia article on conditional probability](http://en.wikipedia.org/wiki/Conditional_probability#Conditioning_on_an_event) is clear about this.
Even if `Data2` only have integers that are multiple of 10, i.e., `0,10,20,...100`, thus allowing only 11 possible values, the distribution are still dissimilar.
Absolutely agree with you, @whuber - I just wanted to answer to the more general question "How do I carry out a test against the null $\\beta_0+\\beta_1=0$  in a linear regression?"
I see your point @whuber. So my question now is: is there a way to estimate $N$ such that $P(\\left|r_N\\right|<\\epsilon)>p$ ?
Is the null that the population proportion of hits is the same for the two groups? Then you have a two-sample proportions test (sometimes two-sample binomial proportions test); it might also be done as a chi-square.
I think the function is meboot.pdata.frame.
Thanks for your response Irish. I have read a number of your responses re: intervention detection and am becoming increasingly convinced. One thing I am a little unclear about is whether you are suggesting a multiple state approach (i.e. approach 1) or an aggregate national approach (i.e. approach 2), or simply to try both. Could you clarify that please?
A famous example is the beer-diaper association.
Glad to help! As you may already know, CRAN has an astonishing number of R packages, which might give you some working code and pointers to references. Someone else has been asking about kriging, so you might want to take a look at those questions/answers too.
It absolutely will work, Walt: the ranks correspond appropriately to the data. However, this is one of infinitely many possible solutions: for the question to have *an* answer, you need to supply additional criteria as requested by @Glen_b and myself in comments to the question.
_Equality_ will hold if $X$ and $Y$ are _independent_ random variables.
Actually, they are the same for SPSS and R. Simple example:
yi <- c(1,3,2,2,3,5,4,6); xi <- c(1,2,2,3,3,4,5,5); wi <- c(.2,.3,.2,.1,.4,.3,.2,.3); summary(lm(yi ~ xi, weights=wi))

gives the exact same results as using the same data in SPSS with:

REGRESSION
/MISSING LISTWISE
/REGWGT=wi
/STATISTICS COEFF OUTS R ANOVA
/CRITERIA=PIN(.05) POUT(.10)
/NOORIGIN
/DEPENDENT yi
/METHOD=ENTER xi  .
@psychemedia - definitely worth the time investment in my opinion. Hadley has put together some nice work on both packages [here](http://had.co.nz/reshape/) and [here](http://plyr.had.co.nz/)
Try larger burn-in values. And please do not post additional code which makes so much harder to see where the problem is and which is not related to that particular problem.
Check Chris Burger's  
"A Tutorial on Support Vector Machines for Pattern" (http://research.microsoft.com/pubs/67119/svmtutorial.pdf). It provides you a nice intuitive, geometrical view of SVMs
1. What data do you have - is it marked by time, are groups supposed to be identical (read randomly selected)? 2. You don't really need ARIMA here, but you do need to understand basic statistics and sampling errors. And there is no easy answer - read about statistical errors, confidence intervals etc., this is the only useful answer I see.
I added a graph and a little more explanation above.  The thing I worry about with the different splits is that that seems very arbitrary to me to split it different ways and see what it looks like - is that a common analytic approach for this kind of data?  The skew is always there.  When aggregating at the highest level (overall participation rates by condition) it is reduced, but my hypotheses are specifically about growth, so that's not very informative.  GEE looks applicable, so I will pursue that - another colleague suggested HLM using a Poisson distribution, so looking there too.