tttjjj/elt_bench huggingface dataset:
To partly echo @mpiktas one approach is to work on the log-scale. In practice this often improves several aspects of the model at once. While prediction intervals transform back just fine, you have to take care with mean forecasts (if normality is reasonable on the logs, you can get an estimate for the mean of the lognormal that is usually reasonable if sample sizes are large). An alternative that can sometimes work for some simple time series models is to use a Gamma model.
@user37695 how many fish in each tank? Can they be measured at baseline and then at follow-up? Or can they be measured multiple times (as opposed to, say, being sacrificed to measure liver volume)
It seems to me that the Chi Square test will be particularly *poor* at identifying differences in tails.  If the tails are covered by many bins, then--because they are tails!--there may be few data in any of the bins, invalidating the chi-squared approximation.  If the tails are covered by few bins, then you lose almost all power to discriminate their shapes, and what you do manage to discriminate might not be terribly relevant or useful.  (One problem we're up against here is that "fatness of tail" has not been defined, so the question is really too vague to answer well.)
This question appears to be off-topic because it is about looking for an R package.
Thanks for the answer. My time series are annual 19 observations so I don t think Johansen cointegration test would help me as it specifies in the results that the critical values are calculated for 20 observations. Please tell me how to choose the number of lags in Eviews if you know based om Aic and bic and where do I specify them, probably in the lagged dependent regressors box in Johasen window.
Thanks for your answer @Pat, I agree with most of what you say here, the problem as set up was a bit sloppy (though the likelihood is simply a function of the parameters, so it's fine for it to be a probability density for sake of example). I should really do the analysis for $\\mu$ and $\\sigma$ free, but do you think a diffuse prior on $\\sigma$ would always avoid the situation given in the question for the marginal posterior of $\\mu$?
I was fearful that the already high predictive accuracy of the base model would be an issue.

My study concerns an investigation into customers who stay/leave a particular company. There are 2210 customers; 2092 stay, 118 leave. I've selected a set of independent variables & am attempting to determine if any are predictive. I feel with such a small number any significant factors will be hard to identify.

The customers are segmented so I'll examine each segment. If any have a higher % departure perhaps something significant will stand out.

Any further advice would be welcome.
Many thanks
Thanks, you've definitely put me in the right direction!
These are called [censored observations](http://en.wikipedia.org/wiki/Censoring_%28statistics%29). How to include it depends on the kind of statistical analysis you are conducting.
It's fine for people to let us know about their own work. We deeply appreciate having that kind of first-hand knowledge. It is more than a good idea, though, to acknowledge one's connection with any recommendation: it gives the recommendation more credibility and provides you the credit you deserve, too :-).  Perhaps you could take this opportunity, Randy, to amplify your answer, acknowledge your authorship, and say more about why your book would be a great choice for the purposes stated in the question. And welcome to our site!
Ah, yes. You could play a trick by fitting a different classifier for each subset of predictor types (Gaussian, Binomial, etc.) using the different naive Bayes implementations. Then multiply their posterior distributions together and re-normalize to make predictions. Again, since the predictors are assumed to be conditionally independent, this should give you the same answer... though it's a little sloppy from a software engineering perspective. :)
I get it. Thanks! To fix those variables at other values, I need to specify them in the control/factor statements? Is there another statement that might let me do that?
If your target data are in (0,1) then beta regression is an option
Have you tried listing the sample space? It might help you see why $E(Y)$ doesn't depend on $X$ unless you have observed $X$ and take account of that (i.e. are looking at $E(Y|X)$).
It depends on what you mean by 'compare'. What is the actual question you wish to answer? If you're comparing with the ends, you may wish to consider doing that as two one-sided tests.
Why use how many are longer than a threshold of 30 minutes, instead of testing whether the time is greater in one sample than in the other? (You could do so using a t-test or, if the data is non-normal, has outliers, or shows other issues, with a Wilcoxon rank-sum test)
Thanks for the enlightening answer. I understand the concept now.
If you did your "individual tests" at the typical nominal significance levels of 5% or 1%, then you should *expect* up to a few dozen false positive results out of hundreds of tests, so perhaps your initial testing was meaningless...
I didn't down-vote your answer, but it is _your_ answer that is overly complicated and not very useful as an efficient way of solving the problem. Computing $P(X_{i} = m | X_{j} = k)$ for all choices of $m \\in \\{0,1\\}$, as is implied in "Simply by enumerating the possibilities" is unnecessary extra work that is really not needed to verify pairwise independence; checking one possibility is enough.  Also, computing conditional probabilities is unnecessary extra work. It is too bad that you feel my comments were unproductive but I don't intend to withdraw them or apologize for making them.
Negative predictive value and specificity are just the flip side of precision (positive predictive value) and recall (sensitivity). I think that the main reason that precision and recall are used more often is that people tend to set up tests with a positive outcome in mind. This wikipedia [link](http://en.wikipedia.org/wiki/Sensitivity_and_specificity) article has a good chart about half-way down that shows the relationship.
I believe the term was coined in this paper:
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1177012754
I actually think that in ICA you can still use the number of 'significant' (ie, 90% of energy) eigen-vectors, as the number of independent components.
Thanks so much for the advice
Despite very poor question-design, the answer is really useful
Thanks, but I need to not only evaluate them, but also somehow remove some nodes or in another simplify the neural network.
@Sam Use either -- they give the same result (the relationship is 1-1 for fixed sample sizes and test size $\\alpha$). Typically, you would report the test statistic and p-value, but not the critical value. E.g., "There was a statistically significant difference between the two distributions according to the two-sample Kolmogorov-Smirnov test (D = .6, p < .0001)." If $\\alpha = .05$ is not an accepted convention in your field, report once that all of your statistical tests were of size $\\alpha = .05$ (e.g., in your Methods section).
How is $S_t$ 3x1? Do you only have 3 observations?
I don't see why using this strategy in a mixed model would be any different, especially if you are treating the PCA score as a fixed effect, which it seems like you are in your example?
At first this seemed too generic but the book looks fascinating.
My answer here: [why-levene-test-of-equality-of-variances-rather-than-f-ratio](http://stats.stackexchange.com/questions/24022//24024#24024), might help you. Unequal n's will reduce your power, but those n's shouldn't be an insurmountable obstacle.
What you are trying to do sounds similar to [jackknife](http://en.wikipedia.org/wiki/Resampling_(statistics)#Jackknife) methods
Hint: the case $A=B$ is consistent with your assumptions.
One issue here, orthogonal to your question, is that logistic regression is probably not ideal in your situation. Everyone in your sample will eventually die (without meaning to be morbid), it's just that some haven't yet. You should consider using Survival Analysis, probably the Cox proportional hazards model.
+1. As explained in my comment to Zare's answer, I'm struggling to understand the distinction you are making between knowing in advance you will be awakened and knowing you have been awakened. What specifically is learned upon awakening, when you were *100% sure that the awakening would occur*?
How do I use distributions as inputs to the linear model? When I get around to calculating, I won't have a single age or race, I'll have age: 25 +- 4, 80% white, 10% black, 8% hispanic, 2% other (for example)
I don't think comparing ensemble forecasting to MCMC is an apt analogy. Basically if you have access to the model (i.e. you are able to run it) then you can do some rudimentary ensemble forecasting by perturbing the initial conditions based on the measurement errors of the initial state. If you don't have access to the model, for instance if you want to compare someone else's published model predictions, then Michael Chernick's answer explains what you need to do.
If you're worried about it you could try reducing the dimensions of your documents by using a continuous space word model.
Thanks for your answer! However, my variables do not represent time series. I use cross-sectional data. Therefore, I have no serial correlation in residuals. And also cannot transform my variables into percentage changes. But you are absolutely right about heteroskedasticity (even though this has more to do with large skewness of independent variables distributions, and not collinearity between them) - I use White correction to obtain heteroskedasticity-consistent t-stats.
Well the first paragraph in quotations was just the author quoting the conclusions of a epidemiological study so there is nothing wrong with it.. The next few paragraphs are him trying to discredit the study which is where he makes some dubious statements.
Wow, thank you for your amazing response. Your second point absolutely confirms what I was thinking and you have really helped me to clarify my argument.
I have individual person data as my current data so could get all summary/inferential statistics. For the prior studies I don't have individual data but have access to most summary/inferential statistics (like means, SD, SE, t-stats) as well.
Thanks so much for a real statistician's view of these techniques!
Thank you vey much. Much appreciated, Peter Florm
The only advantage I can think of would be if you wanted to go further in identifying the likelihood that i is a harder question than j. This is territory where I'm really shaky, but there might be some way of constructing a Bayesian framework where you combine two different types of evidence (matched and unmatched comparisons). But I can't see that being very valuable... it still won't change the rankings, only your level of certainty of your hypothesis tests. And given that only a small number of people have seen both, I would imagine that marginal certainty to be of negligible value.
This looks like homework - if it is, could you please add the 'homework' tag?  You'll still get an answer, but people will approach it a bit differently to maximise learning.
But of course you fit the Gaussian mixture on the log scale, right? The problem appears to be that even on the log scale neither component is Gaussian.  Regardless, I still wonder about the first question I asked: even if an improved method can be found, what would be accomplished by *any* explicit separation of your population into two subgroups? (The answer to that, if there is one, would help to identify classification methods that work particularly well.)
Each participant would have a mean reading for each condition. Then I could run a t-test and determine the significance of the differences in readings between conditions. I know I shouldn't just keep running participants until I reach p<0.05. So I'm trying predetermine the number of participants needed.
@Spaced Unfortunately, I did not save the code (a *Mathematica* notebook). As I recall, the smoothing was done with a built-in interpolation function, after wrapping the data circularly to assure continuity at the endpoints. For instance, to assure continuity when using a cubic spline, pad the data array (sorted by $\\theta$) at the end with the initial $3$ values of $(\\theta+2\\pi, \\rho)$ and at the beginning with the last $3$ values of $(\\theta-2\\pi, \\rho)$. Apply the interpolating function only to $\\theta \\in [0,2\\pi]$.
Hmm... one of my underlying objectives in this question was to be able to do outlier detection before learning association rules. Looking at your response, I'm beginning to think that isn't such a good idea. Would you agree that it's wiser to obtain association rules first, before doing outlier detection, on this kind of survey data?
$\\langle \\cdot\\rangle$ is standard terminology in physics.  It can represent both expectation and inner product.  The OP abuses this notation with expressions like $\\langle u' \\rangle=\\sqrt{\\text{var}( u'_{t})}$; this would usually be written as a norm, $||u'||$ = $\\sqrt{\\langle u', u' \\rangle}$ = $\\sqrt{\\text{cov}(u', u')}$ = $\\sqrt{\\text{var}(u')}$.  This suggestive (and correct) terminology shows the question is really asking why the cosine of an angle does not always equal $1$.
Yes the probability (under null hypothesis) to be rejected is constant. The curve gets narrower, but the bound also gets narrower.
A quantitative analysis of this situation is provided at http://stats.stackexchange.com/questions/18215.  Although it's probably TMI, it definitely answers your question!
[This website](http://www.john-uebersax.com/stat/agree.htm) may be helpful for you.
In the weka explorer, under the classify tab. Click the button More options, and check the output source code box. Then re-run the classifier and code will be output to the Classifier output box.
@ThuyAn I believe this may have been fixed in the 30 Oct 2013 update. Are you sure you are using the most recent version (13.1 from 03 Jul 2014)?
"domain knowledge" is presumably whatever background knowledge people with data have about their problem. Managers, botanists, plumbers, lawyers should know much about management, botany, plumbing, law that is relevant to interpreting their data. In that strong sense they are experts in their field in a way that a statistician or data miner may not be. The passage you cite appears just to be saying that people in any field know a lot that they can use when looking at a visualization. (Good idea always to give the reference with a question like this.)
Thanks Emre. What you say is also my understanding, although unfortunately I *think* that the cunn GPU implementations only cover a small subset of the full torch7 package, and I *think* the shared weight modules are not included in that.
@Greg: I must seriously take issue with your dismissive remarks about the role of time series analysis. Unfortunately ignoring the "time series complication" has serious side effects that may either falsely reject or falsely accept hypothesis for models that are under-specified i.e. those that ignore the ARMA COMPONENT IN THE ARMAX MODEL. There is no such thing as a free lunch when it comes to modelling correctly.
Thanks, good example.
You have only one price, so you have no information about how these proportions vary by price.
Please tell us to what procedure that rule of thumb applies. Observe, too, the change in your approach between the beginning and end of the question: the first part concerns estimating a proportion *plus or minus* some specified error ("ME" presumably) but the last part argues for estimating a proportion up to some *relative amount* of that proportion. There can no possibility of real controversy unless you adopt a consistent objective for your sampling program.
SEM is not a miraculous statistical procedure. Causality is inferred by design, not by statistics. If you don't have time information on the events (and only observe that they are correlated), it will be hard to determine for certain that one caused the other, and not vice-versa, or that they're not both caused by a third variable.
The point of decomposition is to study the effects of the seasonality, the trend and the noise separately, which will allow for a greater understanding of the series.
"*the mean values of many questions are around 4.3, with standard deviations of 0.5 or 0.6*" --- sounds like negative skew to me.
For the singularity problem, do `plot(cox.zph(...), df=2)`.  See the case studies in the course notes for `rms` examples or install the package (which also needs the `Hmisc` package) and type these commands to bring up help files: `?cph` `?validate.cph` `?calibrate.cph`
+1 for your time. Thank You! Actually I am using python as I don't have SPSS. But thank you very much for your suggestion. Just to confirm, I just need to change the distance function to calculate what you suggested and I do not have to make any modifications to my original formula. Is that correct?
"Everyone buys socks." That's rather culture-bound.
@DimitriyV.Masterov: yes, with two periods RD might be an alternative. However, If I had multiple periods I would assume a common trend and use DD instead. Concerning x, that x is chosen by the firms is also a problem in a DD framework
@whuber I haven't come across the concept of "changepoint"... so there is no tag.. but if you think it is relevant, you can add it.
In this case there is no hope for ROC. You can at most plot them as points to compare with other ROC curves.
This solution relies on assuming more than is currently stated in the problem: namely, that long hair, AX3, *and* gender are independent.  Otherwise, you cannot justify "applying" pfx to women with long hair, etc.
Please give numeric example(s) for all your calculations! Otherwise this is not a statistical but a purely `R` question and is off-topic then.
@EpiGrad: I don't consider your "silly" answer quite so silly. There is no particular reason I see that an infinite number of categories should somehow be equated with some form of continuous random variable. Indeed, the difference seems to hinge on whether or not the variable can be represented in a meaningful, natural, and unambiguous way as a real number.
@Analyst I'm not exactly certain how to split my data data into two parts using xreg. Could you possibly give me an example or point me to a link?
Thanks for that response. If you are looking for something *like* Rice's text, but "better", my personal favorite is Mood, Graybill, and Boes, *Introduction to the theory of statistics*, 3rd ed., 1974. It has long been out of print and so can be expensive to get a hold of. It's also a bit more advanced book, even then some of it's more modern competitors like Casella & Berger. At any rate, I find the writing very clear; it moves at the right pace, generally, and has great examples and exercises. Using an elementary probability book like Ross's would probably be good as a supplement.
@Irishstat what is the sign of ma coefficient should it be interpretted as -0.1391 -0.5912 -0.5491 ?
@whuber, well the data description says very clear that it is based on quantiles. But naturally these quantiles are quantiles of empirical distribution. We then make a hypothesis that the empirical distribution is approximated well by some distribution $F$. But what would be MLE for such problem? MLE is usually defined when we have sample. Here we have some observed functionals of the sample.
I think idiotic and insane are a bit strong here. The suggestion is that the Poisson model should be estimated using the Huber/White/Sandwich estimator of the variance-covariance matrix, which does not assume $E(y_i \\vert x) = Var(y_i \\vert x)$.
Perhaps your intuition is wrong? I personally like to look at boxplots by day of week. How do those look? Alternatively, you could look at seasonal plots, plotting your variable of interest against day of week for multiple weeks, like this (but with day of week instead of month on the horizontal axis): http://otexts.com/fppfigs/a10b.png
Could you add more details on your design: How were data collected (sampling scheme), what is the total sample size? Showing us a simple contingency table might help to figure out a good test in your case.
If I have answered your question satisfactorily, could you please accept my answer? Thanks.
I assume you mean *Family-Wise Error Rate* by "FWER", but I don't follow the statement "would FWER be a more appropriate measure than Bonferroni", as Bonferroni is intended to control for family-wise error rate.
I'm interested in hearing other solutions to this myself, particularly from the more statistically literate users here.
I understand, thanks!
Just make sure that you extract as much information out of the interaction itself as you can and don't make inferential errors on followups.  When all of your levels are 2 there is almost never any need for followup tests because the interactions tell you that what you see is significant.  And NEVER do follow up tests and use something like x1:x2 was significant while y1:y2 was not therefore...The interaction already told you they were different while a followup like that does not.
This is very true.  It wasn't until after I originally wrote the code that I submitted (which I wrote quite a while ago) that I realized the connection and felt a little silly for not realizing it sooner.
@cardinal, I loved the very short story.
See [here](http://stats.stackexchange.com/questions/70619/dispersion-in-summary-glm/77594#77594) for what `glm.nb` is doing (taken from the book).
How representative samples are of the population cannot be assessed just from the samples. Comparing Alice and Bob for a given day is a two-way contingency table: see almost any introductory text with good coverage of categorical data, looking for chi-square test and Fisher exact test.
Also see http://exploringdatablog.blogspot.com.au/2011/10/zipf-and-zipf-mandelbrot-distributions.html
@Shane, it is not the blocked code highlighting (the grey background) that is the problem, it is the fact that the wrong keywords within the syntax are highlighted. For instance my loop variable in the simulated data is highlighted with a lighter grey shade, suggesting the code after the pound sign is a comment. I think it is much easier to read as a blocked code segment, but the highlighting can only be confusing. Is there anyway to keep the blocked code but not have highlighting?
I should point out that when you plan to use a logistic link in the regression, which is the usual procedure, then all proportions shown in these plots should be re-expressed as log odds. To this end, an effective choice for estimating a proportion corresponding to $k$ out of $n$ is the fraction $(k+1/2)/(n+1),$ because such fractions (which range from $(1/2)/(n+1)$ through $(n+1/2)/(n+1)$) will always have well-defined log odds.
lmtest::dwtest and car::durbinWatsonTest are particular implementations of the test that has been generalized to apply 
for lags of any order. For example, in JBES, April 1987, Vol. 5, No. 2 you may find this: "Following Vinod (1973), 
the Durbin-Watson(DW) test for autocorrelation at lag k is defined by..."
My "normal interval workbook" at http://www.quantdec.com/envstats/software contains macros "Tol2," "NCTDist," and "NCTInv" to evaluate the non-central T and illustrates how to compute normal-theory tolerance intervals with it.  See the [K Tables] sheet.
Hi Ellen and welcome to the site. What regression did you perform? A linear regression? And what do the data represent? Amount of rainfall? I think it would help tremendously if you uploaded a graph of your data.
See http://stackoverflow.com/questions/9021944/efficiency-in-time-series-regression-in-r-how-can-i-do-this-better
Thanks for pin-pointing a major flaw in the question
I was unclear in my previous reply, as Glen_b highlights. Hence, I have edited the first sentence of my answer as follows: "Set aside the theoretical issues related to extreme observations..."