tttjjj/elt_bench huggingface dataset:
This question appears to be off-topic because it is about programming and not about statistics.
Are you estimating parameters, or is this  for fully specified distributions?
Franck, did you note [my question](http://stats.stackexchange.com/questions/23238/cross-validation-and-ordinal-logistic-regression) on ordinal regression, on the outcome of various function from your `rms` package? How do I interpret a given value of `Dxy` (your answer here helped me, but can you tell more)? and a cross-validate Dxy? If you don’t have time to write a long answer, a bibliographic pointer would be welcome...
I will have a look during the weekend and try to learn your network with bayes-scala
Dave, then please edit your question to reflect this: currently, it reads as if you are looking for a way to transform the *data*. In the process it would help for you to indicate what you think the difference is between a data transformation and a change of model, because if there is one, it is subtle.
@suncoolsu : The transform call uses the as.data.frame convention of stringsAsFactors=TRUE so the goal of getting a character column was defeated. It was another factor column. Better would have been to use as.character around the argument passed to strsplit. Full code below.
Maybe this could be of help? http://stats.stackexchange.com/questions/5250/multiple-comparisons-on-a-mixed-effects-model
Your question appears to be almost purely an R question ('how do I use this package to achieve this particular quantity?'). As such it may belong on stackoverflow.
+1 I also use FactoMineR but I remember that when I tried it's PCA method on a really large dataset, I never got results.
My pleasure. It is cool stuff.
Assumptions: http://en.wikipedia.org/wiki/Analysis_of_covariance#Assumptions_of_ANCOVA  Also, `Anova` in the `car` package allows marginal sums of squares whereas `aov` and `anova` default to sequential.
@IrishStat modeling autocorrelated data could be done by dynamic regression models. Also, other models, like mixed models, could be used for such data. So, I don't think this feature is the substantive difference.
Joe, error bars are always too small (our models never account for all possible sources of variation or uncertainty). When somebody says that they are too large when a legitimate method has been used to calculate them, that raises a big red flag because it suggests their intent is to use statistical analysis to sanctify preconceived results rather than to reveal anything about the data.
This was very helpful. Thanks a lot!
Thank you very much. This looks very promising! What if the data is not normally distributed? How bad is it to still use $\\kappa=3-L$ ? Also, I guess if I wanted to have points with equal weight, I'd just disregard the first one, right?
@AbdelhakMahmoudi: I guess parameters learned by the model are not what you want to vary. But as you can have a probability output (I did not dig into the code, so I don't know whether the probability SVM are really equvalent to the "hard" ones), why not use it? This is a very common outcome from which the ROC is generated. R's svm function is an interface to the widely used libsvm, so you are not restricted to use R to do this.
For those who are forced to do stepwise regression for a homework problem (like I was when I asked the question originally), this is a relevant link: http://orinanobworld.blogspot.com/2011/02/stepwise-regression-in-r.html . Otherwise I agree with the responder!
Does [this](http://rss.acs.unt.edu/Rdoc/library/pscl/html/zeroinfl.html) answer your question? If not, please clarify
My reply at http://stats.stackexchange.com/a/13317 addresses the negative part of this question: namely, why $R^2$ does not of itself tell you anything about linearity, especially when the independent variables in the data sets are different.  Although you did not ask it, this question begs for a positive response, too (and perhaps you should mention this explicitly): given that your proposed methods don't do the job, what *does* work for assessing linearity?
@ttnphns Good idea. And DONE!
"many packages are built by people who know a lot about programming, but not much about economics". Package authors are practically always scientists or academics first and programmers a (very) distant (last) second. Actually, I think it would be a challenge to find a package authored by a "programmer".
@gui11aume, I think I have convinced a few people to think before doing a normality test, hopefully nobody has actually used my test for a serious study.
The number of Bootstrap samples should be ***high, but must be at least equal to the number of valid observations in your dataset.*** In another words, the recommended number of bootstrap samples is **5,000.**
It's not completely clear to me what you're trying to do, nor why 'binning' helps to 'parse' data, nor how "using the conditional Chi-squared dispersion test" achieves whatever it is you are attempting. Can you clarify?
what's the raw information you have?... i.e. how did you come to two DORs?
Did you try with a different range (e.g. >40)? Also, how many variables or features are there in the data?
My PhD is in psychometrics and @Maarten is correct.
Sorry for insisting in the point, but the $1/2$ still bothers me. Suppose I have values for $\\alpha$ and $\\lambda$ in eq. (2), and a set of coefficients $\\beta$: how would I obtain from them $\\alpha$, $\\rho$ and the coefficients on eq. (1)? Are the coefficients the same?
Sounds like *driver* could be a random effect in a mixed effects model
@bigTree: If it's not Euclidean distance, there is no way you can run PCA. However, you can use [multi-dimensional scaling](http://en.wikipedia.org/wiki/Multidimensional_scaling) which is a dimensionality reduction technique that uses precisely the matrix of pairwise distances (it can be any distance). Another note: under certain assumptions about the original data-points (which you don't have) correlations can be transformed into Euclidean distances. Assumptions are: (1) having  zero mean, (2) having fixed, e.g. unit, length. Is it by any chance true for your data?
+1 Good points.  This gets me thinking: any symmetric function of the $n$ eigenvalues would qualify as "good."  All such polynomial functions are polynomials in the $n$ elementary symmetric functions, which include the determinant and the trace.
@harris what you've described *is* a vector AR, even if you didn't think about it that way.
I didn't know there was something like Accepting Answers. From now on, I will always accept the best answer to my question which has at least 1 answer. Thank you Procrastinator for letting me know!
Welcome to the site, @youglau. I think it would help the OP even more if you could elaborate on these methods a little bit & explain how they address the problem, what their pros & cons are, perhaps add some pointers to where the OP can get more information on them, etc.
I am thinking of interaction terms. Maybe defining the correct reference level will solve my problem. I need to use all males with x4=0 as reference for example (male=1 & x4=0).
Choosing "significant inputs" is [double dipping](http://www.nature.com/neuro/journal/v12/n5/abs/nn.2303.html).  It systematically biases the model in favor of extreme predictions and destroys type I error.  Tree methods increase type II error because of their ineffective way of handling continuous predictors and additivity.  The link from @gung is excellent.  To address the greatly increased type I error you would need very complex adjustment, but this wouldn't help with the type II error or with the generally poor future predictive performance of such approaches.
I feel you need to work a bit more on the question. It is unclear what you are trying to show with your diagram above and it is also unclear for what variables you wish to control. It is also not clear what you mean by a positive relationship between a nominal variable and an interval variable.
Would I use this to set up the model alone, and then use geepack to run the generalized estimating equation? Or would glmnet also run the generalized estimating equation?
The asymptotics (and power) depends much more on the total number of positives than the overall total (strictly, the total number of the whichever outcome is rarer). Your total number of positives is 14. That's pretty small. 'Expected value 5 or more in all cells' is only a rule of thumb. 

I'm not sure what the small sample adjustment used by epitools is doing exactly (can't find any explanation in the documentation), but it's certainly something more sophisticated than Yates' correction.
I dont know. That's exactly my question. Did you see my example? Everything was done with 1000 trees (the multiclass version and the single class versions). Are you suggesting that estimating the model with 12,000 tress would give an improvement of the magnitude seen in the single class versions? I really dont think that to be so.
@Procrastinator, please visit http://meta.stackexchange.com/a/15660/166924.
Not all outliers are generated from an experiment, however. I worked with a large dataset that involved the collection of real-estate information in a region (sale price, number of bedrooms, square footage, etc), and every now and then, there would be data entry mistakes and I'd have a 400,000 bedroom house go for 4 dollars, or something nonsensical like that. I would think that part of the goal of determining an outlier is to see whether it's possible to be generated from the data, or if it was just an entry error.
Exactly how did you partition the data into those three sets?
(+1) I like this hands-on-approach. Low-level, but if anything else fails ... ;)
@Max, whuber: Thank you for the answer and explanation.
Asking for a name is reasonable because it gives you a way to look up properties and uses of the distribution. But, assuming it does not have a name, is there something in particular you would like to know about it, such as its moments, cumulants, or parameter estimators?
Is it *really* better? If you simulated a normal sample and did a parametric bootstrap assuming normal distribution, then of course you are in a perfect world. But if you simulated an exponentially distributed sample and did a parametric bootstrap assuming gamma or Weibull, would the parametrics bootstrap still be better? What if you assumed lognormal or, worse yet, normal? I personally think parametric bootstrap is your last resort when you cannot get to the empirical distribution of interest (e.g., in multilevel and small area models, DOI: 10.1111/j.1467-9868.2006.00541.x).
Out of a master pool of 18,000, I have 400 that are between $600 to $6000 in donation for last year,so it's pretty skewed. I want equal numbers in both groups.
To segment customers, not their data, a guillotine
Your motive in using PCA appears to be to guide selection of regressors. Fair enough, but that's what the lesser known "principal variables analysis" is all about. See e.g. http://www.sciencedirect.com/science/article/pii/S0167947307000564
You are overfitting. It may help you to read my answer here: [Algorithms for automatic model selection](http://stats.stackexchange.com/a/20856/7290).
Note that if you a priori expect Group 1 values to be the lowest and Group 5 the highest, then comparing Group 1 with Group 5 will have the highest power to detect a difference.
How do you guarantee that the distribution is unimodal?
@gung: if the variable moved through level 0, level 1, level 2 and then stopped at level 3, only the bin for level 3 would be incremented by 1 (not the bins for levels 0,1,2,3).
1) I can understand your remark about the compatibility between Gower and the metric MDS. I will try to use a nMDS instead. 2) for the missing variables, I was thinking of replacing the NA with the group mean in cases where less than 25% of values are missing. I will take a closer look at the variables having a lot of missing values and maybe reject some (>50% of NA).
+1 ... though a minor comment: "*as it is not testing the same hypothesis with the same assumptions*" --- however, it can test the same hypothesis with additional assumptions to the usual ones made for the KW
Thanks Alexis!  That makes sense to me.  Laziness is not really a good reason (IMHO) because my coding does the testing for me....  I suppose another good reason to do omnibus testing is that if you set alpha =0.05 and you do 20 tests, you would expect at least one to be wrong even though you thought it was right.  Omnibus testing allows you to do less testing, reducing that likelihood.
@chi - I meant the usual parameterisation of the logit-normal is in terms of the mean and SD of the related normal dist, not that the mean and SD of the logit-normal are the same as that of the related normal. Sorry if that was, or still is, unclear - i'm editing in a hurry as i should really have left for work by now!
Also, if it's showing me with a 50% accept rate or something, it's a bit misleading. 3 of the 6 questions I have asked on this site have no answers which I could accept even if I wanted to. And one was closed.
Interesting approach to `MCAR` testing! But what do you think about **alternatives**: *Little's test* for `MCAR` (http://www.jstor.org/discover/10.2307/2290157), implemented in `BaylorEdPsych` R package and *tests by Jamshidian and Jalal* (http://www.jstatsoft.org/v56/i06/paper), implemented in `MissMech` R package?
This relates to the dependence I was referring to. The binomial test assumes independence, which you have if you also have an independent validation set. But using cross-validation, the results from one fold are dependent because the training set shared observations in common.
Lustrik: +1 ;-)
@user20094, I dont know what you mean by "group information of the observations is known". Can you clarify?
... and in that vein also the lme4 package (which I find easier to use than lme or nlme) and related packages from Baayen's above referenced book, languageR.
@Alexis As further explanation of my first comment, if you consider $\\nu\\gt 2$ degrees of freedom and choose $\\sigma$ to *match variances between the Student$(\\nu)$ and Normal$(0,\\sigma)$ distribution,* then--of the two of them--the Student distribution actually is more concentrated around their common mean of $0$.
That was the problem. I didn't install MeteoLab toolbox. Thanks a lot!
And yes, I am doing this multiple times, so what else would you suggest?
@NickCox Thanks for the clarification -- I updated my answer.
Freeplane and Freemind are two free alternatives that I've used for various projects. A new fork of FreeMind is SciPlore MindMappings, which includes support for BibTeX references and importing bookmarks from a PDF.
I am a bit confused with the notation. In your first paragraph $f_{x_j}$ represent a posterior, then what does $f_{x_i}(x_1,...,x_n)$ represent?
I mean some of the sorting algorithms like :insertion sort has running time(average case) theta(n2)[n square]
I added an R function based on some code I came across a long time ago, I don't remember who wrote it but I hope it helps.
http://en.wikipedia.org/wiki/Multivariate_normal_distribution .  But probably your prof mis-spoke (or you misheard): you need to understand how to work with the Normal distribution, not the multivariate normal.  To this end I added the "homework" tag because that will give you links to worked examples and clear explanations, such as http://stats.stackexchange.com/questions/5504/normal-distribution-probability .
RRs are not the "true effect size of risk" and are incapable of being constant.  It is not more conservative, it just estimates a different quantity.  RRs are at odds with the logistic model (pun intended).
I think the whole thing is a quote.
Here is another try: http://stats.stackexchange.com/questions/10141/discrete-choice-panel-models-in-r/13417
Newton-Raphson typically finds roots to $10^{-12}$ accuracy within $3$ to $4$ steps for $p=40$ and small values of $df(\\lambda)$; almost never more than $6$ steps.  For larger values, occasionally up to $30$ steps are needed.  Since each step requires $O(p)$ calculations, the total amount of computation is inconsequential. Indeed, the number of steps does not seem to depend on $p$ if a good starting value is chosen (I pick the one you would use if all the $d_i$ equal their mean).
For a discrete variable, "cumulative distribution function" is good terminology widely acceptable. In contrast "cumulative density function" divides the population. Many would think that discrete variables have probability mass functions rather than probability density functions. Everything depends on how you define density, but a diversion into measure theory can be avoided by using cumulative _distribution_ function here.
@chl I don't know whether this is a good practice for stats.stackexchange.. ?
In the linear regression model the dependent variable $y$ is considered continuous, whereas in logistic regression it is categorical, i.e., discrete. In application, the former is used in regression settings while the latter is used for binary classification or multi-class classification (where it is called multinomial logistic regression).
Can you use all three methods? If the results of the simulation are consistent across methods then it doesn't matter. If the results are different then this may give you testable hypotheses.
You could describe what you thought about the topic this far and the implicit ideas that made you "kind of get it". We can help you make it more explicit. Presumably you are talking about a linear model?
@Henrik No, an equal frequency histogram generally is *not* flat. Histograms are commonly confused with bar charts, which display values by means of the *heights* of bars. However, by definition, a histogram displays frequencies by means of *areas*. Consider (*e.g.*) the data {0,1,2,4,8,16,32,64}, to be shown in the range [0,100] with two bins. The break for an equal-frequency histogram has to be between 4 and 8.  If we put it at 6, the height of the left bar *multiplied* by (6-0) = 6 equals 4, whence the height is 4/6. The height of the right bar equals 4/(100-6) = 4/94. Not flat at all!
Depending on what you mean by "very same", the answers are `No, No, No, No, No` or `No, No, Yes, Yes, Yes`
So, this is not a problem of model selection, you want to compare effect-size measure across models?
Unfortunately this solution is not only computationally inefficient and approximate, it will often fail altogether unless some analysis is first applied to determine an appropriate distribution for the "random vector.".  I think there is merit to the underlying *idea* of just throwing some random numbers at the problem and *randomly permuting* them (*not* "sorting" them!) until an approximate $r$ is attained (because this is quick and easy to program), but that idea is not clearly expressed in this short reply.
@M.George Welcome to the site. If you found an answer to be correct, mark it as such using the check mark by the upvote arrow. You needn't upvote a question you mark as correct, since upvotes are for "good" answers (sometimes a bit tangential) and checks are for "correct" answers (which can be good or not good). This is a service to others who may have the same question in the future and use search before asking it.
I find your 2nd sentence very confusing. Do you mean: "Rates need explicit unit-times in the denominator and have the hairy possibility of multiple events per individual in a unit person-time frame. Unlike rates, proportions are  more suitable for reporting in a ratio form" (trying to sort out to what the "which"s and "them" refer)?
Luna, I tried to fix this up so that it is readable (in its previous state it was extremely difficult to tell what was being asked). I tried to transcribe your estimating equation verbatim but I'm not sure I did it correctly since what is there now is difficult to make sense of - what argument is being minimized over? The $w$s? Do the fixed effects, ${\\boldsymbol \\beta}$, come into play anywhere?
Please explain why underflow is any problem at all.  To a very high precision, it is indeed the case that $\\exp(-800) + \\exp(-3000) = \\exp(-300)$, as your analysis shows.  Do you really need $955+$ decimal digits of precision in your numbers?
If you had more p-values than you listed, then, e.g. for Bonferroni, you would divide by n, not length(p). You have to assume these are greater than those you do list because otherwise, some of them might be significant (in Bonferroni at least). But for details you'd have to look at the methods papers. Basically, you're fine if you never use this option!
So what is the specific programming question here? Do you just not understand how bootstrapping works? Or it just a problem with your code. If it's your code that's the problem, is this really a *minimal* reproducible example? Why not use an existing package that has support for bootstrapping techniques. How would this question potentially be helpful to anyone else?
I made an addition to my post giving at outline of what was involved
I can forgive you for using the word paired - that's very clear to me now, thanks a lot! :)
And I forgot to promote Rob Hyndman's excellent online resource: http://otexts.com/fpp/6/
Makes sense, but do you have a reference for this?
The accuracy percentage I have put in my question are just an example.
Can you clarify what you're looking for, perhaps with a specific example? I don't really follow the first part of the analogy you're making.
BTW, Newman states in "Power laws, Pareto distributions and Zipf’s law": "
Since power-law cumulative distributions imply a powerlaw
form for p(x), “Zipf’s law” and “Pareto distribution”
are effectively synonymous with “power-law distribution”.
(Zipf’s law and the Pareto distribution differ
from one another in the way the cumulative distribution
is plotted—Zipf made his plots with x on the horizontal
axis and P(x) on the vertical one; Pareto did it the
other way around. This causes much confusion in the literature,
but the data depicted in the plots are of course
identical.)"
+1 Good reference, thanks.  The C-test is a more rigorous version of the one I sketched, so it's well worth considering.  The E-test relates a t-statistic to an appropriate distribution.  Calculating that distribution involves a double infinite sum that will take $O(n_1 n_2)$ calculations to converge: fairly easy to code, probably overkill for checking the newspaper!
I plotted the ACF values along with the confidence interval for the correlation.  It is well beyond the confidence limit at alpha=.01.  Is there a better way to test this?
What do you mean by "biased residuals" ? If you compute the average value of the residuals, you find 0.
Thanks and excellent example
A really good guide to doing exactly what you want in BUGS/JAGS can be found here: http://mbjoseph.github.io/blog/2013/05/27/typeII/ .
I have a feeling that this question is better suited for CrossValidated (the SE site dedicated to statistics).
@SdfasvgAsfgavf: I think this will help you: http://whathaveyoutried.com/
could you please provide more details about the experimental design?
But this raises an interesting pedagogical problem, at least in Psychology, because as far as I know most introductory statistics books being used in my field do not really discuss robust measures except as an aside.
Your answer is correct in spirit, but the formula looks confusing. KPCA works with Gram matrix $K(x_i, x_j)$, not with covariance matrix (for many nonlinear kernels it's actually impossible to compute covariance matrix as the target space has infinite dimensionality). See page 2 of the paper you cite.
@evd The permutation test is usually used if you literally have a list of the every item in the data.  But you can use it on a sample  as well, in the same method I described. Unlike my previous example, though, the probability won't be exact. It's accuracy will depend on the size of the sample and the size of your population.
Correct. Of course the number of subsets should be chosen large enough, s.t. the bands will show convergence.
I am not using the package programmes in R. I am using the vector of regression coefficients and design matrix. When I obtain the likelihood, I used the random effect. So, do I need to calculate the fitted values using random effect?