tttjjj/elt_bench huggingface dataset:
Interesting. I can replace my parabolic model by an elliptic one, having its center on the left edge, and by normalizing the coordinates, turn it to a half circle. Then my problem could be rephrased as testing hypothesis $d=0$ vs $d>0$.
This is the default plot, you can also access the information to plot each individually and therefore arrange them however you want (or only plot a subset).  You can do additional manipulations, or generate random observations from the distribution as well.
Duplicate of [this question](http://math.stackexchange.com/q/911492/15941) on math.SE.
Anyone please can you help me?
@ Xi'an
Thank you for this guidance!!!!
But is there anything to be done, so that this estimate could be close to true value and not far away in the middle of somewhere? What I mean, how to suppres that bad behaviour of Monte Carlo?
One thing I do in situations like this is use Excel. Then you can lay out the calculations explicitly, each step in its own column. This makes it easy to see what's going on.
Precisely what would you be testing for "randomness"?  At a minimum it seems your assertion about the "tendencies" of falsified data, in order to have some authority, needs to be circumscribed by a description of what range of applications you have in mind.  For instance, responses to polls and surveys and responses to sets of related questions would normally exhibit "clusters" and degrees of similarity.
Sorry I should have clarified, I do see it listed in SAS output but I suppose I'm asking at a more basic statistical level how does one calculate the t-value?  Which other values given in the normal SAS output come together to make that number?  It is more for my review of the concept.
You should check out the equations for linear regression, e.g. http://www.wired.com/wiredscience/2011/01/linear-regression-by-hand/ and implement them in java (really simple).
If you have multiple 2x2 tables, yes you need to correct for multiple testing. For each 2x2 table you are getting a p-value. You would expose yourself to an inflated Type I error rate if you accept any of those tests at $\\alpha=0.05$ significance level. There are lots of ways to correct for multiple testing... R has a great function called `p.adjust` or you can do something simple like Bonferroni correction (Call something significant if its p-value is less than $\\alpha/m$ where $m$ is number of tables).
The logit of $\\pi$ (the log of the odds) is linear in the parameters, but people don't refer to *logistic* regression as linear as far as I know. Can you cite who has said this?
@MarkHeckmann There really aren't 'prerequisites' as such. The chi-square distribution is always an approximation to the true null-distribution of the chi-square, and it's a matter of how rough an approximation you can tolerate (e.g. how much inaccuracy in p-values you can tolerate in which parts of the distribution). The most common rule of thumb is too strict, as numerous papers discuss. But if the expected numbers were just too small, I'd use simulation from the null distribution of the chi-square test, or choose any appropriate statistic and use the permutation approach described in (a).
Thank you, Dilip. Hanna
Sorry for theat. I had better written 'reinvited'. It means that every 2 years they go for a check-up. Thanks for asking
Both resources are helpful, thanks. Will accept the answer once I implement. Any other resources you come across would be most welcome.
Do you choose the words you suspect of being over-represented based on looking at the data, or do you have a hypothesis developed beforehand you want to check?  Which way would make quite a difference to the approach.
Calculating the K-S *statistic* should be straightforward; getting a p-value from that is more complex though, as the distribution of the test statistic (even the asymptotic distribution) isn't a standard one. If you're doing this for learning purposes, would you be satisfied with calculating the statistic?
It is slightly difficult to understand the question, but consider $X$ uniformly distributed on $[-1,1]$ and $f(x)=x^3$.  You have $(E[X])^3 = 0 = E[X^3].$
How is what you're wanting to do different from a goodness-of-fit test that your data come from the uniform measure on the $S^{p-1}$ sphere?
No, this is not a homework problem, though the simplicity of it makes it appear so! I am after an explanation that I can present to my non-statistical target audience.
Welcome to the site. Do you mind expanding upon this. What exactly does "count data is too small" mean (sample size small or intensity of events is too small?) A reference would be appreciated as well.
How are you assessing the bias in this simulation? I've replicated your code and not found any indication that they give significantly different results. (10000 reps)
See [this previous answer](http://stats.stackexchange.com/questions/9833/constructing-95-confidence-interval-based-on-profile-likelihood).
It seems you've made an edit without being logged in, thus it is showed as an anonymous one. Next time, please check it out before sending -- this way it will be applied right away.
Its just a simple factorial design. If you look up any resources on conducting statistical analysis in Python or R you would be able to figure out what functions in what libraries are needed. I see in the comments for the 20bits article 'prop.test()' in R may be what your looking for.
You beat me by 24 seconds!
traditional clustering or association rule learning approach would not work here. Noted that a consumer would have bought only a single product. So, the association rule would not probable. However, The consumer provide the aspects ratings and overall ratings. So, i think we can get the probably similarity between pair of users given these ratings info.
@PyCthon... would I need to consider the null if it is correlational...I assumed that would apply when you want to say that one variable causes another.. I am not sure
+1, even though I cannot fathom what the OP is looking for either.
You could use a $\\chi^2$ test to test for equality of proportions amongst the three groups.
$K$ is $100$ for all intents and purposes. $N$ is sitting around $10000$. This process is being looped over anywhere from $5000$ to $20000$ times.
@whuber Right, eventually the calculations will be done in excel resulting in the probability distribution of discrete values of X, and when I change the age/race the X distribution would need to update quickly, so Monte Carlo would be out unless I wanted to pre-calculate X for each combination of age and race (~200 combinations) and take weighted averages of them on the fly.
It depends on how do you understand "new distribution". Before estimation, you had an abstract distribution with unknown parameters, one incarnation of which was "picked up" by the MLE. If that means "new" to you, fine by me.
I prefer the tools that treat arrows as special objects: when I move the variables around, I hope the arrows will follow. It seems Inkscape cannot do this.
Thank you very much. Both Ancova() and Ancova(,type=3) worked for me ;)
@Andy W: That is correct. For a given threshold and (# of friends), I am trying to find a good point where I can stop and inspect the resulting data.
some examples of spurious correlations at this web site http://www.tylervigen.com/
By "efficient" I mean that it does not use your data efficiently. The beauty of K-fold CV is that 100% of your data gets used for training and 100% of your data gets used for testing, which makes efficient use of your data. The key is, of course, that all of your data does not get used for testing and training at the same time, which would yield way-optimistic test results (overfitting). By making a static division, Hold Out says that, say, 1/3 of your data will never be used for training, and 2/3 of your data will never be used for testing, wasting a lot of information in your data.
This appears to be a duplicate question, see my answer to a previous question here http://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation/27751#27751 .  Can you explain how your question differs from the earlier one?
+1 Impressive set of references.
An interesting aspect of this approach is that in finding $\\lambda^*$ based on the data, it becomes a function of the data themselves--a statistic--rather than a constant, whence $\\text{Err}_\\lambda$ no longer has the same form as your formula, implying $\\lambda^*$ is no longer necessarily optimal!
Thank you for the input. The scores for each task have potential scores around 50 or 100, that are not really reachable due to the time allotted. There are theoretically 50 levels on the test--only 19 in the data. Regardless think that is too many for a ordinal logistic model. I will give a zero-inflated count model a go on R and post back on how things work out. Thanks again.
@caseyr547 Sorry, that was the linked to the 'published' version; I have updated the link to the raw spreadsheet, or you could download an [excel version here](https://docs.google.com/spreadsheet/pub?key=0Ai_PDCcY5g2JdGNabGs0R0IyVzhrUFIxOVRoTXMzUUE&output=xls)
You're asking about performance and accuracy differences when lambda=0, where the two should theoretically be identical. I think you should add that into your question.
@IrishStat [link](https://docs.google.com/spreadsheet/ccc?key=0AjuswtkYF_sUdERaRUtJRExsQVltVEZuaTMxQnJvanc) Hope Google Docs format is ok
How many observations do you have? Also can you post one specific question at the time?
Thanks @Zach and @ Gung for your comments. There is disagreement among economist about role of political freedom and economic freedom in economic growth. Some scholars have view that economic freedom leads to political freedom. To measure the joint effect of both variable on economic growth i can introduce interaction term of economic and political freedom but i am unable to answer the question weather for less develop country is it give political freedom to its people first or economic freedom on the basis of  econometric technique .
I added a figure and clarified the explanation. I probably wouldn't be here if I could see the math more clearly! Help welcome.
@Adam Please consider editing your original question instead of posting your updates as answers. Also, your last question about Vista issues with Chrome/IE is better suited for our [Meta](http://meta.stats.stackexchange.com/) site, but check whether you don't find anything related in older threads--here or on StackOverflow [Meta](http://meta.stackoverflow.com/)--before asking.
@whuber that I don't know and I don't know of any research on this. Basically it's something I've used as a heuristic that might fit what OP is after, and I don't really claim it being a preferred approach.
I bet that it is a variance-covariance matrix scaled by the variance of the betas.  Which is bizarre.  I suppose that the purpose of reporting this is to find evidence of multicollinearity.  Anyway, google "variance-covariance matrix" in the context of regression, or "parameter covariance matrix".  Also google "multicollinearity".  Understanding those things will get you to your answer.
+1.  Nice work!  It would be wonderful to see the answer expressed in terms of intrinsic geometric properties of the rectangle: it ought to depend on things like its area, perimeter, and configuration of the four angles.  (The literature--which I have seen referenced but have not had access to--appears to focus on domains with smooth boundaries.)
Welcome to the site, @kurt. I tried to edit your Q for greater clarity. Please make sure it's still asking what you want to know. In addition, I couldn't make sense of your 2nd paragraph; did you mean "If X is not *linearly independent*, what would the rank of X'X be"?
Thank you!! That was helpful.
To be specific, I think the term $(\\beta-1)(\\beta-2)x^2$ is wrong. And I think that changes the region where the curve is concave. (Of course, I could have made a mistake, but taking derivatives in R (via `D`) and making it a function also suggests there's a problem with your answer, since yours changes sign for cases when doing it via `D` doesn't.) -- I think that $1<\\alpha,\\beta\\leq 2$ does it
I would like to know for every group the significance in enrichment rates
Thanks for your answer. I'll look for some documentation about the approaches you suggest me.
I have not expirience from Eviews programm.I want the interpretation particularly the t-statistic and Prob for my example
(Log_austria)
Could you tell us what exactly you are optimizing?
(+1) Both responses provide a handy summary of possible discrepancies between all three statistics.
Also, are you showing us the training or test error?
[Boosting](http://en.wikipedia.org/wiki/Boosting_(machine_learning)) is what you're looking for.
@EDi Lookup `adonis` in the on-line help for vegan. (And [Jari Oksanen](http://cc.oulu.fi/~jarioksa/softhelp/vegan.html) for more info on the package itself.) BTW, this is a pretty elegant answer, you can vote it up like I did a long ago :-)
I never finished this because I couldn't find a good source for traffic volume for a given road.
Nice answer. I think that you've hit a very good reason why party is better for an advanced audience, *and* why it's a good idea to educate the audience in order to help them accept the use of party.
@MichaelChernick I believe that SnowsPenultimateNormalityTest (implemented in the TeachingDemos package for R (http://cran.r-project.org/web/packages/TeachingDemos/TeachingDemos.pdf)) could fit the description of globally most powerful test (for $n \\ge 1$).  But I also believe that the documentation for the test/function is more useful than the test/function itself.
@TheodoreLytras sorry, I was mistaken. Your previous assertion is correct. From the very paper I linked, " In addition, if multiple clusters are perfectly nested, GEE clustering on the top level cluster accounts for the multilevel correlation structure through the sandwich variance estimator".
Rescaling has the effect of making the surface of the gradient of your error less elliptical, allowing you to tune the step-size for an epoch more sensibly. Cross-validate with and without normalization. If you don't have too much data (<500,000 observations), compare the results with bfgs.
Thanks for the answer Jeromy, I think you sum up the issue well.  If I were to reorganize the data with biomass type as an independent variable would that inflate my degrees of freedom? I have 70 experimental patches, can each plant group be though of as an independent group within those patches (n = 280)? I imagine I would need to nest plant group within patch?
I'd start with the package author's website, which has links to papers about the operation of Amelia.
[Gary King at Harvard](http://gking.harvard.edu/amelia/) Also, the [vignette](http://cran.r-project.org/web/packages/Amelia/vignettes/amelia.pdf) (pdf) has a fairly comprehensive introduction to the package and algorithm.
Have you ever used WEKA or RapidMiner? For instance, EM is within data mining and it applies a model. Apart of that, check out the definition given by mariana soffer and compare it with your answer. It is a couple of years ago that I read Bishop and Russell/Norvig, but as far as I remember the def. by mariana soffer is more suitable. btw data mining is ("only") the major step in prior to knowledge discovery. data mining is only grabbing for data -and subsequent for information- when using an algorithm with adequate parameters. data mining cannot explain patterns.
Thanks @Qbik, this is what I also guess. Thanks for your help.
You are confusing me with using $R^2$ (proportion of data variability explained by the model, not the size of a cause) and estimated marginal effect represented by the slope coefficient (even in a pairwise regression without intercept term) in this strange (non-sensible?) fashion. If you are interested in impacts you may consider beta-coefficients or elasticity coefficients that really do have acceptable interpretation.
Do you think (conditional on the trend) the outcomes would be independent, or could there be time-dependence as well?
Glad to help :) 1. AFAIK, yes, this logic applies equally to all generalized linear models. 2. Yes, standardizing the variables before entering them into regression will remove differences in scale, permitting direct comparison of the resultant standardized coefficients. [N.B.](https://en.wikipedia.org/wiki/Nota_bene) Different methods of comparing coefficients apply when they are estimated in one common model vs. two (or more) separate models.
I'd guess that it'd be even less.
See also http://stats.stackexchange.com/questions/63477/calculate-precision-and-recall/
About the only "natural zero" for dates that would have any meaning in this context would be either your birth date or possibly the date of the first modern Olympic games. I'm pretty sure a modern medal winner was not born in 0 A.D., and I know the modern games started well after then, so these dates are not expressed appropriately for a ratio scale.
Following on @MånsT 's comments - if your main goal is to pass the course, then it would help if you told us the syllabus and maybe the text you are using. 

For broader understanding - quite a lot of intro. stats. is "common sense thinking" supplemented by a few key ideas and formulas and such.
@PeterFlom: The global F-test has more power to detect linear relationships with the response than individual t-tests. So it might happen that every t-test has p value right above the sign. level but the global test has a p value just below it.
@drknexus. It's funny when the terminology of different fields collide.
@Glen_b That makes sense.
thanks. I was thinking that these would behave differently with values of $X_c$ around 0, but now I see that they are equally sensitive.
I am sorry, I don’t know anything about signal filtering and stuff. You should open a new question, describing in details your data, your problem and the rationale of your solution.
If, as you say, you have the population rather than a sample, then it is arguable whether you can or should do any significance tests at all (this has been discussed here quite a bit).
+1 I would just add though that if computational complexity is an issue, it isn't too difficult to construct a sparse kernel logistic regression model by greedily choosing the basis vectors to minimise the regularised loss on the training set, or other approaches.  See the papers on the "Informative Vector Machine" for example.
1- By the way, it looks that the calculated error margin has no relation with "Confidence Level". For example if you add the option ConfidenceInterval->0.9 to the LinearModelFit there would be no change in the root mean square error.
@Glen_b Can you elaborate on why non-stationarity is an issue with Box-Cox? The power transformation is designed to eliminate heteroskedasticity.
@whuber: Thank you but it didn't say more than a few words.
The risk you take is that if one or more of those interactions actually exist, then the estimates you obtain when leaving them out are biased. Residual plots are always a good idea to make sure the model fits the data.
Given the *sharp* discrepancy between this and all the previous answers, including two simulations and a theoretical (exact) one, I suspect you are interpreting the question in a different way. In the absence of any explanation on your part, we just have to take it as being wrong. (I suspect you may be counting one less, in which case your 4.8 should be compared to 5.83258...; but even then, your two significant digits of precision provide no additional insight into this problem.)
@ocram answer goes direct to the point (+1). However, you're talking about two different methods to estimate HRs, but then you mention only the Cox model. I'm curious to know what other method you use to estimate an HR. Because maybe what you're estimating using the method other than Cox model is not really an HR.
True, but without knowing the structure of his original data set and what software he is using there is really no way to answer this question I suppose.
The -.383 value id based upon the doubly differenced X series. It is then applied to the suitably differenced Y series. The reason for using the one coefficient is to NOT distort the relationship but to identify the relationship between the original (OBSERVED) Y and the originally (OBSERVED) X .
@nz You're right.  This method is great for estimating the third moment when the skewness is small.  However, because it tends to overestimate the variance, it will typically *underestimate* the absolute value of the skewness.  The correction depends on the size of the histogram bin width relative to the standard deviation (as well as on the shape of the distribution): it becomes substantial when the bin widths exceed 1 SD (which can happen for coarse histograms of highly skewed datasets).
The autobox link seems to be broken. Is there a mirror?
I can understand your point, this technique is not something I have invented - it's a quite commonly used approach for measuring variance and uncertainty in estimates. My problem is that all of the examples I can find deal with one source of estimates, whereas I want to be able to work with multiple sources of estimates and combine them.
Accuracy isn't the issue so much as speed.  Either you're generating exponentially-distributed variables or you're not; if you are, speed is what's left to consider.  Can you think of a method that a) isn't a rejection method, and b) requires fewer than 1/0.69 uniform random variates to be generated, on average, for each output exponential variate?
I think I found the answer to the quesiton in the comment here: http://stats.stackexchange.com/questions/20125/highlighting-significant-results-from-non-parametric-multiple-comparisons-on-box. I've got to display all statistically similar boxplots by letters! Thank you @Glen
@seanv507 The benefit of the RF is that the multiple trees, fit to (random subsets of) perturbed versions of the data, average out the discontinuities. The result is a better approximation to the underlying smooth line than what a single tree would produce.
2, thats good advice, my rustiness with SPSS sure is showing.
+1 I spent so long constructing my (now deleted) answer, I didn't even see this one had posted in the meantime. I would have just voted this one up instead.
A large number of the questions posted on this very site provide problems ranging from easy to impossible.  The replies provide a richer set of answers than any problem set or textbook could possibly offer, because they often represent multiple different attempts at solutions.
If Vi2 is 10 times Vi then the correlation between them will be 1 and your model will not work. So, either you entered them incorrectly into your model or you described them incorrectly here.
Welcome to our site, Matt. This does not seem like the best advice. The data are not obviously paired, because there is no basis to associate each hour before the intervention with the hours after. Furthermore, one does not need to assess normality of the data. We must contemplate the possible approximate normality of the sampling distribution of the sample mean. There is no evidence against that, so a t-test would be perfectly appropriate and have greater power than a nonparametric test.
2. This site is not just about answering individuals' questions in public but about providing a resource for others to use later. Only the examples you give can be discussed; referring to other examples we can't see is of no use or interest to anybody.
This sound quite straightforward to implement but would be specific to a given survey platform (and there are dozens of them). This seems a bit off-topic however.
What do x1 y1 represent in the real world? Is it a standard scatter plot or a time series or a histogram, or a density plot or something else?
How would you start? What are you having difficulty with? Taking (partial) derivatives? Recalling Ito's formula? Something else?
@Marco Yes.  It's a good idea also to plot the pdf for the density against which you are integrating (which you had already done for this question).  You might find it helpful to read the section on "Graphical Moments" at http://www.quantdec.com/envstats/notes/class_06/properties.htm .