tttjjj/elt_bench huggingface dataset:
Because you asked: I have added my version of the derivation as a separate answer, please feel free to copy it and use it in your answer if you think it is correct.
@StéphaneLaurent: How about for other types of regression.
@thkala Research "entropy" while you're at it.  (Entropy can be interpreted as an average "importance" where importance is measured as the negative logarithm of the frequency.)
Are the "site" effects of interest to you? Or is it more that you want to take account of possible correlation within the sites?  Also do you expect to find "really good" sites and "really poor" sites?  If you do, then you may find standard GLMM will overshrink these outlying sites.
@ Matti Pastell: I have this book. It is very good indeed. My question is about the difference between the particule filter (which from what I understand is a sequential version of MCMC), and a MCMC on a rolling window (in the latter, we re run the optimisation process on a rolling window). 
Which method should be prefered, and why?
Perhaps, you could say a bit more about the level you are interested in.  Are you comfortable with measure theoretic probability?  What do you not like about Wasserman's style?  The fact that there are theorems and proofs, or the fact that the proofs are sketchy? What exactly are you looking for?  I'm not sure what theorem style has to do with QQ-plots.  You might look at [Hogg McKean and Craig](http://www.amazon.com/Introduction-Mathematical-Statistics-Robert-Hogg/dp/0130085073) it does cover the Wilcoxon test.
Permanent random numbers are available to my users, so I'm going to declare this answered - thanks!
I am really confused: If type 3 is better, than how can you compare groups in main effects alone with type 3?
That helped. I think since the answer was very obvious - I had a problem accepting it in my head somehow. I do have the whole data, and will be looking into it. Thanks again!
The criteria in your book are for interval data; that would be useful if you had the dates at which hurricanes stroke... moreover these criteria are for **constant rate** Poisson processes, which is obviously (or I hope so) not the case of hurricanes. To check if your count data follow a Poisson distribution, a first elementary approach is the chi-square test.
Size as in disk storage...as of now I am trying to get the answers without limiting the features and accuracy.
You need to be a little cautious about assuming that particular symbols like xbar and phat will always have the same meaning, as they are just symbols. However, those two are quite common and consistent. The first is a mean which is the sum of the observations divided by the number of observations. The second is a proportion, the number of 'successes' divided by the number of 'attempts'. You should be able to determine which type of summary is relevant by considering the type of problem you are dealing with.
+1 from me.  I like that this answer is starting at the beginning and so easy to understand...
+1 and thanks for your answer. I know the first video by heart :) As for your idea in the last paragraph, they perform essentially the same approach in the paper "Force platforms as ergometers" I refer to just below the graph.
I'd recommend expanding this to include journals.  :)  In CS, conferences seem to dominate relative to journals, in statistics it tends to be the other way around.  I'd also recommend making this "community wiki", as it's somewhat open subjective.  One person may think that AOS or AOAS is both "top" and "statistical", while another may say that KDD or NIPS is the top for potential applications and theory.
Please replace the images with the actual text.
Stability is not a property of the *data*, it's a property of a *model*.
@Thomas The binomial distribution link I gave gives the probability function ($P(X=x) = \\binom{n}{x} p^x (1-p)^{n-x}$).
For the 2nd `train` model you will get a slightly different result unless you set the random number seed before running it (see `?set.seed`). The accuracy values are 0.8135 and 0.8111, which are pretty close and only due to the randomness of resampling and the model calculations.
BTW, since k-means minimizes the pooled within-cluster sum of d^2 _divided_ by the number of objects in the respective cluster, your point `coincidentally minimize Euclidean distance, because the sqrt function is monotone` is, to be precise, not correct.
Principal Component Analysis worked well, but I am finding it difficult to interpret the biplot of the results. Some of it makes a lot of sense, other aspects I'm just scratching my head about.
:Shane ; DONE !
This issue has come up on other SE sites, too. The point is that answers should be stand-alone, for many reasons. The intent is to make our threads a source of information, not just some gateway of links to possibly evanescent material. The response "an answer is at such-and-such a place" is appropriate when the answer is simple, standard, and easily reachable from a search (e.g., has a good Wikipedia article).  Otherwise, *especially* when the answer reflects a significant investigation like yours, please at least provide a summary. There's no need to attack me for making this reminder.
What are you trying to do - perform a test or estimate something with an associated confidence interval?
Thanks for taking the time! This was extremely helpful!
@DrWho I would only report the results based on the random effects model (I mean, I would always report the REM results, regardless of the actual test statistics...).
"All possible regressions" *will not* allow you to "select the best possible predictors". If this doesn't make sense / you want to know why, it may help to read my answer here: [algorithms-for-automatic-model-selection](http://stats.stackexchange.com/questions/20836//20856#20856). There are some approaches that can be used w/ situations like yours. These include penalized methods (ridge regression, LASSO, LARS, elastic net) & cross-validation (try reading some of the highest voted threads under the [tag:cross-validation] tag).
Reg Latex: See this meta thread: http://meta.stats.stackexchange.com/questions/218/tex-processing-for-stats
Technically speaking, the data for the two years isn't necessarily independent. You can have a patient who visits the hospital twice in 2012 and a third time in 2013. If you want to see if how independent the samples are, you have to look at how many *unique* people you have each year.
Clever!  You could also do `dx[c(TRUE, dx$ID[-1] != dx$ID[-length(dx$ID)], ]` for non-numeric data - I get 0.03 for character, 0.05 for factors.  PS: there's an extra `)` in your first `system.time()` function, after the second zero.
...cont. The image of the data that I want to associate is at the link showing the differenced blue line and the original 'always' nonstationary red line http://imageshack.us/photo/my-images/801/redvsdifferencd.jpg/. What I would like to be able to say (show) statistically is that the red line is associated with the drastic moves in the blue line from Sep 05. With that as my objective is your advice to follow your suggestion regarding R? Thanks.
Welcome to Stats.SE, Jimbo. I disagree with your assertion. I think we've seen what Julia is able to do, but the problem at this point is that there aren't nearly as many domain-specific packages for it as there are in R. R will continue to reign supreme in open source statistics as long as researchers see more benefit to using the numerous packages in the R universe. That's my take, at least.
@whuber: though I accept the point that the median absolute deviation of the sample median depends on the underlying population distribution near the median, and this may not be so immediately easy to estimate from the sample.
@Wendy, you probably should add that information to the question, and kyle should add his responses to his amswer
This is clearly routine bookwork. In fact it looks like you're *asking us to do your assignment*. Please add the `self-study` tag, and (as @TooTone already suggested), revise your question to fit with the the guidelines on that sort of question. In particular, you don't get to just post your assigned work and expect us to do it for you; if you meet the community expectations on this sort of question (to begin by showing what you've already done and where your problems lie), you can get guidance and helpful hints. (If you leave the question as is it will very likely be closed.)
Welcome to the site, @Tim-Brown, we appreciate your help answering questions here. One thing to note is that it's best to identify other answers by the answerer's username, "this last guy" won't be very effective. Eg, the list will be re-ordered over time as people vote. You may want to read our [FAQ](http://stats.stackexchange.com/faq).
As far as I know, logit link within ordinal regression is used for a DV with any heavy-tailed distrubution, up to uniform. However, if your "uniform" distribution is clearly multimodal, cauchit link may be better.
[What is meant by a random variable?](http://stats.stackexchange.com/q/50/919) addresses most of these points. In (1) it is not apparent what is meant by "take on". (3) is also a little unclear and could provoke some objections based on distinguishing physical from mathematical concepts. (8) might also cause some trouble for people to discuss because nothing guarantees the convergence of arbitrary sequences of random variables; when such a sequence converges, it converges to a random variable--which could be a *constant* random variable.
a link to a reference paper on "Griffing and Hayman diallel analysis" would be very helpful for non geneticists
@love-stats is right on that what you're actually asking for is covariance. But for other people who stumble on this page later, I'd caution against using `var.test()` here. It uses an F-test, which assumes the variance of the two samples come from normal distributions, which doesn't apply to this sample.
@suncoolsu: Yes, practically speaking you should get the same conclusion. I should not have said "equivalent" (I work with big data so they end up the same). I added some code in the answer to help clarify.
Hi gung, thanks for the advice. I've read the discussions you linked to and I am indeed considering programming a simulation. However I do feel that if I can't answer such a question as I posted I might not be competent enough to program a simulation that is _valid_ .

Would you know if there is another answer to my question?


And if it really boils down to simulating the data, would that be a feasible approach: 
1. generate data distribution for control group (effect size known)
2. generate data for patients (range of possible effect sizes)
3. iteratively run tests with different N?

Thanks!
You're right - it's not the answer I was hoping for. It's the one I was expecting... just not hoping for ;-(
Regression methods do not make any assumptions about the distribution of the predictor variables. It may help you to read this: [What if residuals are normally distributed, but y is not?](http://stats.stackexchange.com/q/12262/7290)
@DavidJ.Harris Quite so; +1. I've seen the slash used a few times in robustness studies. Maybe $X/Y$ - as an inverted slash - should be called the "*backslash distribution*".
All variables are considered in `rpart`, the binary split is formed by searching over all variables and all possible split locations within each variable. `rpart` also stores information about surrogate splits which can be used when there is missing data in your variables. Does this help?
Thanks for the reply. We actually have a big dataset with icd9 codes, as you have correctly guessed. In your suggestion, we will indeed end up with more than 1000 levels, but our dataset has 1 million of observations, so we have enough data. Recategorization is something we considered too, since there are so many similar codes, but, again, there might be hidden useful information with many implications regarding the outcomes we are studying, in between two seemingly almost same icd9 codes. If we proceed like this, how can we control for confounders?
+1 BTW you can create links in comments: enclose the text in square brackets, followed immediately with the *full* URL in parentheses, as in `[documentation](http://www.stata.com/manuals13/tsucm.pdf)`, which (without the backticks I put there to protect it) would render as [documentation](http://www.stata.com/manuals13/tsucm.pdf).
@cardinal, I was unsure if this question should be split between other questions on the community or not as it has one single main purpose linking them all. The question in all which is composed of 6 other questions might scare people to answer it. Please let me know if there is a better way on which I should approach this I will keep checking here and will edit if necessary again. Thank you for your suggestion points on the original question!
Can you provide more details on your calculation, and explain why this solution which is different from other solutions offered on these thread would be better?
Anything with positive values that might be expected to grow can be looked at on a log scale. Whether it helps is a different matter. (As before, scaling is not to the point here, I think.) The basic idea is that exponential growth or decline becomes linear on logarithmic scale. M.H. Williamson, Analysis of Biological Populations (1972) has a good brief discussion (assuming you are an ecologist or similar).
@probability Thank you.  Those of us who frequently find outliers and large amounts of skew in good data will be less emphatic about this and say only that the assumption of normality depends on the nature of the data and the importance of the decision(s) made with the test.
Very closely related: http://stats.stackexchange.com/questions/421/what-book-would-you-recommend-for-non-statistician-scientists
Thanks! that helped
I believe that some statistical packages will automatically convert ordinal variables (factor variables in R) to dummy variables in a linear regression, so you might not actually need to create dummy variables yourself. You might have essentially been following their advice without knowing it.
I hadn't actually tried this in R. I was simply using some "rudimentary" techniques in MS Excel. I will download R and have another go at it. Forgive my ignorance; statistics is not my area but I have a project I am currently working on that has piqued my interest in time series. Thanks for the pointer(s).
In your title you say the population mean is known. The body of your post didn't seem to talk about that. Can you explain what it is that's known?
Hello, @shakir, and welcome to Cross Validated! You probably noticed that this is an old question (from 2010) and there are seven (!) answers given to it already. It would be a good idea to make sure that your new answer adds something important to the discussion that has not been covered before. At the moment I am not sure it is the case.
@gung goal would be to say "Videos on this channel can expect to hit X by 30 day mark."  
I can figure out how to make the number conservative (math may be just as simple as slicing off 33% of the average we come up with)

Forecasting value after that (money/pricing) becomes arithmetic and a lot more simple.
And of course astronauts who have come back from the international space station could have travelled even further.
That Ripley reference is somewhat oracular.
+1 for the refs, would stratification work automatically during re-sampling? can it be adjusted? if yes what argument? I couldn't see any in `createDataPartition`. For `PLS`, it is more with pre-processing, I guess, since PLS would be called before every classification, but I am on my guards, centering and scaling could be messy, how to get scores, and apply all this in CV is far from easy. Mostly I will post another question on this issue and I contacted the maintainer about adding the `pls` method in the preProcess(). I hope this could be soon (deadlines!)
Please invest some time in asking. "Please explain me this PDF that will disappear within a few weeks" is not making the Internet a better place. Thus I must close it in a current state.
@Sara I think it dates back to Karl Pearson, which uses this empirical relationship for his "Pearson mode skewness". Aside from this, you may find interesting this online article, http://j.mp/aWymCv.
@Bayes What you are using here is the well-known Empirical Cumulative Distribution Function (ECDF). You are also mixing two types of nonparametric estimators whose use might require a theoretical justification. Moreover, the proposed estimator does not consider dependencies between the entries.
As mentioned because different fitting criteria are used, the same network structure will almost certainly be optimized for different tasks. And it is not necessary you get a 95%+ classification success in any case for your out-of-sample predictions. Finally, it might also be the case that your training sample is not a good approximation of your population. So while doing a good initial training, the network has not learned "enough" true populations characteristics. (Ah! And you can always experiment with different transfer functions; currently you are using `tansig` in both cases I guess.)
If you are referring to GP-- yes -- the library of operators can apply different transformations, but at some point, as you increase the number of inputs and depth and breadth of trees, it can become computationally intensive.
If $\\epsilon$ is fixed, e.g. to mean 0 variance 1, then the $b_2$ estimate is what `lm` gives you as 'standard error' in the summary output, ie the conditional variance.  If it's not fixed then $b_2$ is not separately identified in $b_2\\epsilon$ without more information.
+1, I agree that n=99 isn't that small.
I wonder if it isn't more sensible to ask for methods of visualisation, rather than packages, especially since most of the answers are providing little detail, and many packages provide the same methods. See, for example, http://stats.stackexchange.com/questions/41326/ways-to-reduce-high-dimensional-data-for-visualization
@whuber, good comment -- see edit to address it. In econometrics, there are iterative `generalized-moments` estimators that use the converged results of the previous stage to initialize the optimization problem for the next one; their convergence properties are often dismal, as you can imagine.
@rbatt, sorry for the confusion. The simplest model is 1D, using the day-of-year that ice-on occured in the past as the predictor.  But if you want to detect trends in ice-on date, you have the full DATE, not Day Of Year, as the thing you want to predict, because the prediction for, say, 2020 could then differ from that for 2050.
... assuming independence, at least
It's actually tough to do manually if we get statistic that follow other distributions. Calculating from z-score is easy though.
As I first posted, you don't have to take the average; you can just select a particular quantile.
Strictly speaking, one cannot apply PCA to categorical data until they have been *numerically coded.* You will get different results if you encode your binary values differently. But once the values are encoded, you have--a numeric dataset. Period. There is *nothing* in any of the accounts of PCA (that I have ever read) that asserts those numeric data must be of a certain type or have certain distributions. PCA is a pithy *summary* of the locations of those numbers when they are considered to be points in a Euclidean space. From this point of view, there's nothing different about your data.
You're right, @bnaul. However, without some kind of regularization, the `LU` decomposition will not work either. I will add a comment about this in my answer.
Thanks for the suggestion. I was really confused what to do in these scenarios.
@Kingsford My fault! I was initially thinking of another citation, not from Tufte and didn't remove my first words... I UPDATED my response. Many thanks!
Thanks for the reply Michael.
Yes, my aim is to identify fluctuations in CSR rating (from FTSE index) over years for some companies & see how thier profitability has varied over those years. Asuuming that other things remained constant, my argument would be that the changes in the profitability would be due to the varying levels of CSR activity.
I do not wish to do an analysis among companies. My analysis will only be over years of some companies.
Does this make sense? Please respond.
Many Thanks
I edited the question again to include the different ways I have attempted to answer the question.
Could you please point me to a reference for more modern algorithms? I have tried till now SOM and kmeans, but the problem is that I do not know the true number of clusters, so I resorted to kmeans, just because it is faster. (data represents time-series gene expression)
Welcome to CrossValidated, Mark!
Your two sets of graphs, the bottom one fitting better than the top... what are they?  Are you trying to say that the ones on the bottom are from simulated data?
Please post only the necessary amount of numerical information. There is no value in the endless string of numbers you included.
I assume the $X_i$ should be independent, but you didnt say so ...
Kindly note, that when I spoke about the discrete output, I was explaining him as to what I am doing and not what I want as he asked me - "what question are you trying to answer with these data". Since this is a predictive model with discrete dependent then the output will always be discrete. Only change in my project is that the odds ratio is the mandatory field and it is importat so we were trying to look out for other(better) predictive models which will give an output such that each indep variables will have their respective betas or other quantifiable values (like odds ratio in logistic)
+1; still ksvm from kernlab seems to be the preferred R SVM implementation. For instance it scales variables on its own and has a nice (=working very well) heuristic procedure for picking good $\\gamma$ for RBF.
@soufanom. I am not planning to use weka. I need to do that in matlab. Sequentialfs didn't work in matlab. Am I doing something wrong or it's just not possible using sequentialfs. I need to use some heuristic to prevent searching for all possible features, I gues
At the bottom, you give a basis for all cubic polynomials.  Although its *elements* depend on (one) of the knots, the space of functions they generate is *independent* of the knots.  Moreover, splines have more degrees of freedom than polynomials: they are allowed to change at each knot.  Depending on the type of spline, this adds *at least* one degree of freedom for each knot.  Therefore I would expect a basis of cubic splines with two knots to have a dimension of *at least* 4+2=6, exactly as stated in the quotation.
Amazing answer, as usual - very clear examples.
Based on the set-up, my guess is you are looking for an extraordinarily rare event. Given this assumption, I would not expect a high likelihood of 2+ successes in any of the 1000-trials. 2+ successes are taken into account in the 2nd step here as $\\hat{p}'$ is estimated. It would appear the major factor driving your problem is the "do-until" part, which would suggest much more a geometric distribution.
Looks set to kick Gilks, Richardson & Spiegelhalter (1996) into the long grass when that comes it in May.
This result cannot hold as stated. If it did, one could replace $Y$ by a smaller open subset $Z$ with $y_0$ on its boundary, apply the desired result to $Z$ and conclude that $c(n_0,y_0)=0$ for *every* $y_0$ in $Y$.
@Jacques The classes are split 19% positive, 81% negative.  That's why I'm using precision and F-measure, because when I used accuracy it would predict everything as negative, and say I had 81% accuracy.
+1 I would suggest perhaps taking the log of the count (+1), because this kind of data will have an extreme right tail. Unless you do some kind of transform, you'll end up with the vast bulk of sites having a score of 0 or 1, and a handful with 9 or 10.