tttjjj/elt_bench huggingface dataset:
@Jack. The normal approximation applies to the chi-squared as the number of degrees of freedom gets large, whatever the origin of the chi-squared statistic. If your theoretical probabilities are correct - so the data follow that model - then $\\sum \\frac{(O-E)^2}{E}$ will be approximately chi-squared, where O and E are observed and expected counts respectively. So yes; if the model fits, the test statistic is chi-squared, which can be approximated by a normal.
Did I just get called 'technical'? If so, I'd take it as a compliment. I tend to feel myself to be a pretty basic, applied sort of fellow. I do hope the OP edits to include more information; I find the triangular distribution a fun object to play with.
@henry - $F(1)=1-1^{-\\alpha}=1-1=0$, so support can't be below 1 (as CDF is a non-decreasing function)
(Apologies for taking the log when I should have been exponentiating. But it doesn't change the fact that there is a pathological data problem.)
A paired or repeated-measure t-test may be a good choice since "copy" and "online" give the measurements on the same magazine.
Gelman's thoughts & the comments to his post are well worth reading. The scoring system is almost certainly garbage: consider that the 95% CI for this teacher is $[0\\%,~52\\%]$.
well, in principle, with negative autocorrelations, the effective sample size could be larger than the actual sample size. But I doubt that is likely to happen in practice! But your hard statement that it is impossible is untrue!
A power function can have powers $>1$, so your comment (if I understand your wprding) that $y$ necessarily increases more slowly with $x$ is quite incorrect.
Alternatively, one could interpret the initial results of no relation as the "statistical fantasy," because the subsequent "meaningful" relationships found upon controlling for demographics suggest the original model was a bad fit to the data.  What results do your goodness-of-fit tests and other model diagnostic tests show?
Welcome to cross validated! I think we need more information about *why* you want to select 10 bands. The answers will differ whether that is for regularization or for instrumental reasons.
Ok, I guess I need to be more clear about the "By accident". What I meant is, assuming the null-hypothesis is true. What are the chances that my Hypothesis test will nevertheless reject it (p-value less than p*).
Trying this recipe worked no better, at least at 10000 samples.
Selecting the "best" subset of IVs is known as [variable selection](http://en.wikipedia.org/wiki/Feature_selection).  A good set of IVs should be uncorrelated to each other but correlate well with the DVs.
@Whuber, you are right, I should have pointed to something more specific. He has some recent talk presentations about small effects and multiple comparisons, but I think the following link is enough: http://www.stat.columbia.edu/~gelman/research/published/power4r.pdf
I went to a talk recently where someone presented on the methods used for netflix and how to do it using R: http://prezi.com/8fbsaa7mushs/using-r-for-data-mining-competitions/
I provided an answer but I guess this more of an R programming question than a statistical question, so maybe it should be migrated somewhere else.
You should look into the non-central chi squared distribution.  Proving the limit is normal is going to be more complicated than a simple application of the CLT I fear though.
Just to add to what owe said I think the real essence of random variable would only come with such courses. The concepts like expected value, correlation have deep implications in statistics. As some body also said it makes you more mature to deal with statistical processes.
Are you sure you want to limit yourself so severely?  If this problem has any practical application, then by the time you know the count at time x you also know all previous counts.  Why not use them to help with the prediction?
I used the formula given in the link and got an very narrow CI: 24.77 (24.12-25.42). Intuitively, I find it to narrow. On the other side the curvture is quite strong (b2=2), which may explain the narrow CI. Or did I something wrong in my calculations?
Maybe this is relevant (I'm not sure, this is all above my pay grade), https://stat.ethz.ch/pipermail/r-help/2006-July/109308.html
"The" "best-fit" transformation is the inverse cumulative normal integral of the percentiles (readily computed with almost anything more sophisticated than an adding machine), but I doubt this is what you are referring to.  For this problem to be well-posed, you need to stipulate (a) what family of transformations you have in mind and (b) how to measure the goodness of fit.  You are correct that this problem has been addressed, but as you might guess, it has many different solutions depending on what is assumed in (a) and (b).
@user16204 Equality of variance is not the only assumption of relevance when doing regression, and explicit hypothesis tests of regression assumptions are not necessarily the best way to see whether the assumptions are reasonable.
Thank you so much! This did it. Same approach for rel risk out of GENMOD with poisson and bin
You are right, because I did not think that what he wanted is increased number of articles, not the number of users.
(3) and of course processing each new data should be done incrementally (we do not redo all clustering since the beginning on all previously seen data), but this doesn't mean that the old clusters are no more so important.
$\\binom{n}{x}=\\binom{n}{n-x}$, so $\\binom{k-1}{r-1}=\\binom{k-1}{(k-1)-(r-1)}=\\binom{k-1}{k-r}$.
Possible duplicate of [Statistical inference when the sample "is" the population](http://stats.stackexchange.com/questions/2628/statistical-inference-when-the-sample-is-the-population). Also [this question](http://stats.stackexchange.com/questions/68886/).
Ok thanks, I'll stick to this solution then.
It could mean a lot of things depending on the context. There *are* answers on the Web; the very first Google hit for [ad-hoc classifiers](http://users.soe.ucsc.edu/~pang/visweek/2012/vast/papers/hoeferlin.pdf) provides one definition. If that's not what you're looking for, then please edit your question to provide some context and reference the paper you are reading.
Something like this might solve the problem, at least approximately, for *one* circle. (This can easily be done using focal counts with a GIS.) But it won't solve the multiple-circle problem.
I think there is some confusion here. The standard error of the mean is not normally distributed regardless of the parent distribution. For one thing, it has to be non-negative :) You may have the role of the _test statistics_ (e.g. the $t$ and $z$-statistics) and the standard errors mixed up.
"Suppose you can't do the numerical integral." I understand that logical uncertainty (which this is an example of) has resisted analysis despite considerable efforts.
Aren't they paired because I'm studying the same stores before and after? For example, the "Atlanta" market has 15 stores. I'm measuring those 15 stores from pre-launch and comparing the average of the same 15 stores post-launch.
"He is in a group of 2 guys" does not imply "his chances are 1/2"
@George- No worries--  We've all got parts of this whole enterprise that give is the howling fantods.. happy to help!
(**3**) If, indeed, these are high-pass filtered, then I would guess that whatever is happening around index 125 is actually something like a sudden step down, followed by a step back up of the same size. A high-pass filter would distort such a feature in exactly the way shown in the plot. (**4**) Instead of aggregating the data by averaging in blocks, you can use a low-pass filter, like a moving average (there are likely better options, though application-specific). (**5**) I would not call what you've done ensemble averaging. That would be averaging each of the points *across* the 10 series.
If you have a specific question about the paper I'm nonetheless happy to look at it :)
When you refer to `gender * hypertension` as "an interaction term", you appear to be confusing it with `gender : hypertension` (which is the interaction). In R formulas `gender * hypertension` is the same as `gender + hypertension + gender : hypertension`.
Thank you so much. I really appreciate you getting back to me.
Have you thought about using the simple [scipy.stats.curve_fit](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html) method directly on your two polynormial model ?
Added my results as requested. That's a good point about the limitations. Does this mean that KM is more helpful in this case? If so, how can I calculate survival for multiple covariates if the COX is no longer accurate?
As explained in comments at http://stats.stackexchange.com/questions/57847/formula-to-calculate-a-t-distribution, there are trade-offs between computation time, storage, and programming complexity.  What are your preferences concerning those?  What is the anticipated range of degrees of freedom?  How accurate do the calculations need to be?
@PeterFlom Do you mean that they cannot disagree? That could be the answer!
I've made a package. It'll be up on GitHub soon.
That's exactly my question to more experienced users: Are there better / other determinants than just an improved fit? Obviously I should use a serious model, but still I wonder whether there is some good strategy to follow in order to get a good first guess up to which order lags should be relevant. Data is e.g. quarterly trend deviation from some stock index.
I would add also that you can use the deviance/pearson residuals and the "model deviance" to check if the transformatiton is better. But beware of extrapolation outside the range of the transformed variable in your data set, as you may be picking up a local relationship.
In the end I used sklearn and its support for multiple observations. It seemed to work correctly. I should dig into the code and see how it actually deals with it.
Despite the negative conclusion of this reply, Kassym, what you want can be done: see http://stats.stackexchange.com/questions/7959 for some pointers.  The principal ideas are (1) accept the possibility of a little error in the results (that's the force of Michael Chernick's remark) but (2) create a data structure that enables reasonably accurate updating of the quartiles (or other percentiles) as new data come in.
You should use time series methods if you want CIs to go with your predictions.  Your model is cast in the "filtering" framework rather than the ARIMA framework.  Look up x12 for some pointers here.
Thank you, you wrote this in much cleaner way, than I did :-)
Wilson's macros are definitely the way to go if you want to use SPSS. You may also consider picking up a copy of "Practical Meta-Analysis" by Lipsey & Wilson.
thanks, rolando2!much appreciated.
Thanks +1! Near these thoughts I have in mind but it's hard to break misunderstanding during self-education...
Please **don't** cross post! http://stackoverflow.com/q/14524818/429846
Thanks to everyone for useful suggestions. I ran the GEE, because I am not interested in the differences between individuals, but rather on the effect of the intervention, pre and post. For comparison, I also ran a generalized linear model, as per @Erik, and got essentially the same results.
No. Basically I'm a humble high school maths teacher and I had this idea to get such data from around the world (one station per pupil). A geography teacher suggested the date as a guide without having seen any of the data. Well, none of my data, maybe he has seen other data.
That is an accurate description of uni-variate screening. In addition, usually a more generous p-value of 0.2 is used for cut-off for model inclusion than in stepwise. The take home point is that people forget that uni-variate screening has the same limitations as stepwise selection methods. (whether these limitations can be ameliorated by higher the p-value threshold is another issue, and one I have no opinion on). @gung has written a few answers on the limitations of stepwise (and more generally data driven model building) if you search this site.
Forgot to add that if you make the suggested modification you should choose small values of v(.01 .001 ...) otherwise you wont have many jumps as in the wikipedia pictures.
Are you asking about a situation where you have a treated and a control group sample and you are looking to understand if they were statistically similar enough pre-marketing to then do a hypothesis test regarding difference "due" to the marketing intervention?
@Tal: for smaller N, neither of both hold. And as shown in my edit, they are exactly the same.
Look up the definitions in the Algorithms document available under Help > Algorithms
Welcome to our site, TJM!
See my comment to Dmitrij Celov's answer and clarify your question.
+1 Great answer. I had a similar debate a while back because I was intrigued by Incanter (and have done some Java coding). It was clear that R was the language to use to get statistical work done quickly while Clojure was the language to use to think more like a computer scientist. Obviously there is overlap but as you say "know who you are".
By the way, the `eRm` vignette mentions that the Warm (1989) approach will be added to a future version - see p. 11 of http://cran.r-project.org/web/packages/eRm/vignettes/eRm.pdf.
Did you check out the `Orange` library, or just `sklearn`?
And how many observations do you have?
have you tried plot on the `lm` or `glm` object? How what you are trying to achieve is different from what `plot.lm` does?
(i) Can you give some context? You start by discussing [histograms](http://en.wikipedia.org/wiki/Histogram) - which have no time component, then suddenly you're talking about *when* something was measured. It seems like context might help us understand what you're asking about. (ii) What do you mean by 'an accurate transform'? You can apply an FFT to any sequence of values. They don't have to be values over time. The FFT has many different uses. (iii) Why does your title discuss the DFT while your body mentions only the FFT? Why the inconsistency in terms?
It is worthwhile to review an older version of the same question: http://stats.stackexchange.com/questions/2547/why-is-median-age-a-better-statistic-than-mean-age/2550#2550 And, a description of Robust Statistics: http://en.wikipedia.org/wiki/Robust_statistics
Sorry for not coming back to this sooner. I am indeed interested in how the treatments influence each other. I've been looking into crossover designs and multi-level models to some extent but most repeated-measures examples I've found seem to involve measuring the effect of one treatment over time rather than of multiple treatments on the same subject, and I am not sure how to develop a model that involves treatment (A, B, C, D), order (1st, 2nd, 3rd, 4th) and sequence (e.g. A-1st, C-2nd, B-3rd, D-4th)
It's the online/offline times for each user. We don't have a regular sample, we register the exact event when they come online or go offline.
Make sure that what you are weighting on is something you want to marginalize, and realize that you pay a high variance price for doing this.  If the survey over-sampled males and you want to develop a non-sex-specific estimate then weighting is for you.  If on the other hand you want conditional estimates, i.e., you want to adjust for sex as an ordinary covariate, then weighting may be counterproductive.
@onestop, presumably one or more people will grade the writing samples.  Suzanne, the more detail you provide the more detailed the feedback can be.
This question was [asked on SO](http://stackoverflow.com/questions/18475589/priors-and-loss-in-r) and has an answer there. As clearly stated in the help on both sites *please don't cross-post*. You should pick *one* site for your question and delete the other; if you post initially to the wrong site, you *flag it for moderator attention and ask them to move it*, you don't post a second copy. When you have removed one of your two copies of the same question I will withdraw my downvote of whichever one remains.
Does it make sense to look at each of the eight measurements separately?
@cardinal  I lose track of who I am talking to and I always just call the quastioner OP.  Everyone should know what OP means.
+1, but why would you not use unstandardised regression coefficients for prediction?
Did you check Gelman's [ARM package](http://www.stat.columbia.edu/~gelman/arm/)? I seem to remember there were pretty nice functions in there. (Would have to check again, though.)
In referring to your previous question and its answer (and comments), it appears that your concern comes down to whether the independence of $X_s$ and $X_t$ implies the independence of $X_s^2$ and $X_t^2$, for then the result is *immediate* (because expectations of products of independent variables multiply).  The duplicate addresses this concern.
Would you consider 'robust' rather than 'nonparametric', or must it be nonparametric?
No, based on realExperts, i take (just one) precision and recall value of analyzedExperts. And i take precision value which is same with recall value.
Sorry about that. I've included an image in the question.
If my math is right, your formula decomposes into 26 interaction terms for a total of 31 model variables to consider. That means the total number of possible models to explore is 2,147,483,647. `step` will certainly trim down the model space significantly, but it's not surprising that it's taking a while either. I'd recommend giving lasso regression a shot if you haven't already, it's a good technique for variable selection in high dimensions. You also might consider bayesian techniques for exploring the model space.
This combined with Log likelihood seems to work
@user39531: yes, they are.
@Andy: I don't see a reason not to allow for correlations between the effects and the noise in RE, but if we agree on the rest of the answer, I rather simply edit my answer.
It is a wonderful book indeed! And I like its style.
Sure you can think about it that way.
I guess @rm999 gave an answer already, so it would be inappropriate for me to expand the question.
Hmm. Good question (+1). I wish I had a book to recommend. Personally, I tend to use simulation for any but the simplest cases.
how can I increase the acuuracy so I can see than them not as exact zeros?
Why not just tabulate them?  Mathematica 8 computed and plotted every case covered by the condition $n_1n_2 \\lt 30$ in less than 1/20 second.  The trick is to look at all possible rank sums that can be attained by the *smaller* of the two datasets; this requires only $\\min(n_1,n_2) \\binom{n_1+n_2}{n_1}$ sums.
@Tie-fighter A barplot is for discrete outcomes, an histogram is for continuous ones (that will be grouped into contiguous bins).
You will also want to read @ocram's answer here: [When to use an offset in Poisson regression?](http://stats.stackexchange.com/q/11182/7290)
Note that the above result is not correct theoretically due to the finite size of the data. The result is known as the biased estimate. What you want is an "unbiased estimate."
@ttnphns : I edited my answer to include the discussion you refered to .
Is the position of the target a continuous variable or a categorical variable? I am thinking you could regress detection performance on position, condition (as an indicator/dummy variable) and an interaction term. If the interaction term is significant then the slopes are different.
Thanks everyone for all your inputs. Excel is the only analysis tool I have available so I'll set the regression aside for the time being and consider the other points that you have raised ;)
If the determinant is negative then $K$ cannot be a covariance matrix, so no absolute value is needed or even recommended.
Is this for a class assignment?
We welcome questions like this, @Cha-am Jamal, but we treat them differently (see our [help page](http://stats.stackexchange.com/help/on-topic)). Please tell us what you understand & have tried already, & we'll provide some hints to help get you unstuck.
I'm not sure this should be categorized as a Statistics question.  What features are important to music classification requires expert knowledge of what makes different music different.  You should choose quantifiable features that allow you to distinguish one genre from the next (obviously), but you would be better off asking a musician that.
Thanks for the mention, but you have my point backwards: if the processes generating the series are different, then separate models might make sense; it's processes that generate data, not vice versa, as I use terms. Also, the implication that you need to make series stationary to make sense of them does not seems to me to reflect best current thinking on time series.
IrishStat, thank you for you attention! I have daily product price data. The other important variables I mentioned above are of both types: 'singular values' (e.g. feature of the product of) and 'multiple values' by which I mean numeric values of one measure, measured let's say quarterly. Main goal of this analysis is to predict trend of the price (will it go up or down?) and the next step is to evaluate the level of this up/down movement.
Removal of data will cause a bias unless the procedure for removal is objective and that same procedure will be applied to all future data for which predictions are obtained.
@John Indeed, the polar method is available in R, see the setRNG package.
Also I added two more factors to the data set and some questions related to the modelling of those factors
I had good luck using the vegan package in R when I was doing multivariate stats. Lots of documentation (vignettes), etc - though of course you didn't say that you're looking to use R. I imagine anything you'd generally want to do related to gut flora would be essentially community ecology, which is what the package was built for.
Thanks, Alecos. I'm embarrassed to admit that, on quadruple-checking, I was misreading the correlations and simply got the sign wrong. Mea facepalm. Reassuring to know that it should be theoretically impossible - if I read your post correctly (it's been a long time since I've studied stats formally). Though it makes me wonder about this "common misconception" re: correlation and linearity: https://en.wikipedia.org/wiki/Correlation_and_dependence#Correlation_and_linearity
I think I must be misunderstanding your stated example. What quadratic functions have more than one peak?
At present this is more of a comment than an answer, would you care to expand on it a little bit?
You should then post the new warnings. But you might also want to check [this](http://thread.gmane.org/gmane.comp.lang.r.lme4.devel/11674/focus=11893).
My pleasure Jones :)
I read this, It looks pretty intuitively advantageous but I think it is yet to be proven that it works better than simply taking a lot of random initialization points. I found this simple code in case you want to try it out:     kmpp <- function(X, k) { 
        n <- nrow(X) 
        C <- numeric(k) 
        C[1] <- sample(1:n, 1) 

        for (i in 2:k) { 
            dm <- distmat(X, X[C, ]) 
            pr <- apply(dm, 1, min); pr[C] <- 0 
            C[i] <- sample(1:n, 1, prob = pr) 
        } 

        kmeans(X, X[C, ]) 
    }
This is particularly important. I remember reading a study about whether Ramadan was bad for babies whose mothers were fasting. It looked plausible (less food, lower birth weight), but then I looked at the appendix. Thousands of hypotheses, and a few percent of them were in the "significant" range. You get weird "conclusions" like "it's bad for the kid if Ramadan is the 2nd, 4th or 6th month".
How would a *change point* regression model work with only 3 points? You would only have 2 possible lines!
You need to clarify what you mean by "p != .5", Aaron, because taken literally this statement is not testable: no amount of data could ever reject it.
An alternative is also to just use Mike Lawrence's ez package.  You should look at that.
give some example data (even fake)....we could compare the proposals.
+1 Many thanks for the comment about antithetic simulation, this is cool